Command line interface

Installation

Dataform can be installed using NPM:

npm i -g @dataform/cli

Create a new project

To create a new bigquery, postgres, redshift, or snowflake project in the new_project directory, run the respective command:

dataform init bigquery new_project --gcloud-project-id<your-google-cloud-project-id>
- or -
dataform init postgres new_project
- or -
dataform init redshift new_project
- or -
dataform init snowflake new_project

Project structure

Change directory into the newly-created new_project directory and take a look at your newly created project files:

cd new_project
ls

You should see the following structure:

project-dir
├── definitions
├── includes
├── package.json
└── dataform.json

Define a dataset

The definitions/ directory should be used for files that define tables, assertions, and operations.

To create a new dataset, create a new file definitions/example.sqlx:

echo "config { type: 'view' } SELECT 1 AS test" > definitions/example.sqlx

Compile your code

To check that your Dataform code compiles, run the compile command at the root of your project directory to get JSON output of the compiled project:

dataform compile

You should see output similar to the following:

Compiling...

Compiled 1 action(s).
1 dataset(s):
  dataform.example [view]

Create a credentials file

Dataform requires a credentials file in order to connect to your warehouse. Run the init-creds command and Dataform will guide you through credentials file creation:

dataform init-creds bigquery
- or -
dataform init-creds postgres
- or -
dataform init-creds redshift
- or -
dataform init-creds snowflake

A .df-credentials.json file will be written to disk containing your provided details.

Check out our data warehouse setup guide if you need help with the init-creds wizard.

If using a source control system, we strongly recommend that you do not commit the .df-credentials.json file to your repository in order to protect these access credentials.

Run your code

In order to run your code, Dataform needs to access your data warehouse in order to determine its current state and tailor the resulting SQL accordingly. If you'd like to see the final SQL that Dataform will run on your warehouse without actually running it, you can perform a dry run:

dataform run --dry-run

You should see something similar to the following:

Compiling...

Compiled successfully.

Dry run (--dry-run) mode is turned on; not running the following actions against your warehouse:

1 dataset(s):
  dataform.example [table]

Removing the --dry-run option will result in the SQL being run in your warehouse:

dataform run

The run command's output will now include the run's execution status, including any errors encountered during the run:

Compiling...

Compiled successfully.

Running...

Dataset created:  dataform.example [view]

Next steps

You have now seen how easy it is to use Dataform to publish simple datasets. Next, jump to our guide section.