Dataform can be installed via Node:
npm i -g @dataform/cli
Create new project
To create a new project in the folder
dataform init new_project --warehouse bigquery
Currently supported warehouse types are
[bigquery, redshift, snowflake, postgres]
The default project structure is as follows:
Define a table
definitions/ directory contains files that define tables, assertions, and operations.
To create a new table, create a new file such as
definitions/simplemodel.sql with the following contents:
select 1 as test
To check everything has worked, run the following command at the root of your project directory to get JSON dump of the compiled project:
You should see output along the lines of:
"query": "select 1 as test
We can take this a step further and see exactly what statements and graph will be run, by executing the following:
You should see something similar to the following:
"statement": "create or replace view `dataform.simplemodel`
as select * from (select 1 as test)"
Create a profile
A profile defines the connection parameters to your warehouse.
If you are happy with the statements that will be executed, you can execute them against your data warehouse, by providing the path to your
profile.json file created in the previous step.
dataform run --profile=<path-to-profile.json>