JSPM

  • ESM via JSPM
  • ES Module Entrypoint
  • Export Map
  • Keywords
  • License
  • Repository URL
  • TypeScript Types
  • README
  • Created
  • Published
    • 0
  • Score
    100M100P100Q47067F
  • License MIT

Tool to assist in migrating from dataform to dbt

Package Exports

  • dataform-to-dbt
  • dataform-to-dbt/index.js

This package does not declare an exports field, so the exports above have been automatically detected and optimized by JSPM instead. If any package subpath is missing, it is recommended to post an issue to the original package (dataform-to-dbt) to support the "exports" field. If that is not possible, create a JSPM override to customize the exports field for this package.

Readme

Dataform to DBT

A tool to assist in migrating from Dataform to DBT.

What works

  • Models are converted, with Dataform ${ref()} calls replaced by appropriate DBT {{ ref() }} and {{ source }} calls.
  • DBT currently doesn't support namespacing, so model names must be unique; dataform-to-dbt supports renaming tables as part of the migration, correcting references along the way.
  • Source and model definitions are output in schema specific model and source .yml.
  • Model documentation is included in the model .yml.
  • Dataform auto-generated assertions (uniqueKey, nonNull, and rowConditions) are migrated to appropriate tests in the model .yml, using built in tests where possible and dbt_utils tests where not.

What might work

  • Dataform includes are out of scope to ever work completely, as they can do most things javascript can and translating that to DBT's jinja based macros just isn't going to happen. That said, where the include is a simple string, a DBT macro will be produced and inserted in the right location so that it's still a single replacement (instead of inlining the replacement and duplicating it at the DBT version). Includes that call functions are not supported, but a stub macro will be produced and inserted in the right place and a warning produced so that they can be manually adjusted (by amending the implementation to actually work, and by amending all the invocations to pass appropriate arguments).
  • DBT doesn't support generating temporary tables in the model header as you can with dataform pre-operations (not with a ref(), anyway), so these are extracted to their own tables and the references updated.

What doesn't work

⚠️ Disclaimer

This is very rough and ready, used to convert some private projects. It has not been well tested, it may eat your homework, kill your pets, etc. Review the created models etc carefully before using, build into test datasets, write assertions and verify the output, take all possible precautions before running in production.

Requirements

Should work with Node 14.8+ (native modules, top level await), but has not been tested.

Usage

The migration will create models, macros, etc in the target directory; the project can then be run and leftover Dataform config can be removed.

Review the DBT docs and create a template dbt_project.yml and a profiles.yml; dataform-to-dbt assumes that the production profile output will be called prod and will fail if it's not.

Some Dataform assertions convert to dbt_utils tests (row conditions and multi-column unique assertions), so install dbt_utils if that applies.

# Get help
npx dataform-to-dbt --help

# Run a migration
npx dataform-to-dbt \
  --directory /path/to/dataform/project \
  --template /path/to/template/dbt_project.yml \
  --profile /path/to/dbt/profiles.yml