Tutorials for getting start with Dataform web.

Tutorials

Learn how to create a new dataform web project and connect to your warehouse.

Creating a new project

This tutorial demonstrates the first few steps in configuring a new project.

Google BigQuery is used as the data warehouse in this example. 

## Log in or sign up

After opening [the app](https://app.dataform.co/).

![Sign up](/static/images/how_to_guides/dataform_101/1_sign_up.png)

If not signing in with GitHub, Google or Microsoft then you will need to verify your email address.

## Create new project

This will create a new, empty project.

![New project](/static/images/how_to_guides/dataform_101/2_new_project.png)

## Give the project an inspirational name

![Project name](/static/images/how_to_guides/dataform_101/3_project_name.png)

## Choose to "create a table" from the file templates

Ignore the instruction to the configure the data warehouse; we'll do that next.

![File templates](/static/images/how_to_guides/dataform_101/4_file_templates.png)

## Connect to your warehouse

Big green button towards towards the central right.

![Create table](/static/images/how_to_guides/dataform_101/5_create_table.png)

## Select Google BigQuery

Input the project ID. For alternatives warehouses, see [here](../guides/set-up-warehouse).

![Add warehouse](/static/images/how_to_guides/dataform_101/6_add_warehouse.png)

## Select and test the server

Input, credentials, and test that they are valid. See [here](../guides/set-up-warehouse) for how to create a JSON credentials key.

![Test server](/static/images/how_to_guides/dataform_101/7_test_server.png)

## Preview the results

This shows the exact affect it will have on the warehouse.

![Preview results](/static/images/how_to_guides/dataform_101/8_preview_results.png)

## Publish the table

![Publish table](/static/images/how_to_guides/dataform_101/9_publish_table.png)

## Next Steps

You've taken the first steps towards improving your data pipeline!

Some of the next features you could use next include:

- Automating repeated runs using [scheduling](../guides/scheduling).

- Checking validity of data using [assertions](../../guides/assertions) or [unit tests](../../guides/tests).

- [Documenting datasets](../../guides/documentation).


Start building out your dataform pipelines using SQLX files and managing dependencies.

Managing dependencies

This tutorial demonstrates how dependency trees are made within a project.

## Create a new file

Create a new file by clicking on the + button next to the `definitions` folder and create a new SQLX file named `onboarding_table`.
For now, ignore the other options; these are more advanced features which will be covered in later documentation.

![New file](/static/images/how_to_guides/publish_tables/new_file.png)

## Write a query

In the text editor, write a simple SQLX statement:

```js
config { 
  type: "view" 
}

select 
  1 as one,
  2 as two,
  3 as three
```

You will see the right sidebar update with `onboarding_table` which is the name of the dataset, a tag for the type - a `view` - as well as a validation message and two action buttons.

The `Preview results` button executes the query in your warehouse and returns the output at the bottom of the page. This can be useful during query development to check that the query returns expected output.

<img
src="/static/images/how_to_guides/publish_tables/compilation.png"
style="width: 100%"
/>

## Run the query

Clicking the `Run this action` button will open a dialog box with a few options for running your actions. Click `Run now` to continue.

Upon successful completion of the run, a new view named `dataform.onboarding_table` will be created in your warehouse.

<div className="bp3-callout bp3-icon-info-sign bp3-intent-primary" markdown="1">
  By default, Dataform creates datasets and views under a schema named <code>dataform</code>. You
  can change this in the <code>dataform.json</code> in your
  <a href="../../guides/configuration">project configuration</a>.
</div>

![Run node](/static/images/how_to_guides/publish_tables/run_node.png)

## Create a table instead of a view

To create a table instead of a view, simply change your script's `config` block:

```js
config { 
  type: "table" 
}
```

You will see the tag on the right sidebar update from `view` to `table`.

<div className="bp3-callout bp3-icon-info-sign bp3-intent-none" markdown="1">
  Dataform provides many other options for dataset creation. Learn about
  <a href="../../guides/datasets">publishing datasets</a> and
  <a href="../../guides/incremental-datasets">incremental datasets</a>.
</div>

<img
src="/static/images/how_to_guides/publish_tables/table_vs_view.png"
style="width: 100%"
/>

## Referencing other datasets

Dataform provides methods that enable you to easily reference another dataset in your project using the `ref()` function:

```js
select 
  * 
from 
  ${ref('my_table')}
```

This provides two advantages:

- You don’t have provide the full SQL dataset name.
- Any dataset that is referenced by a query will be automatically added to that query's dependencies. Dependency queries are always executed before dependent queries to ensure correctness.

## Create a new file

Create a new SQLX file named `onboarding_table2` and add the following content:

```js
config { 
  type: "view" 
}

select
  * 
from
  ${ref('onboarding_table')}
```

<img
src="/static/images/how_to_guides/publish_tables/ref_query.png"
style="width: 100%"
/>

You will see the right sidebar update with the name of the view, `onboarding_table2`. Clicking on `Compiled query` will display the fully qualified query to be run in your data warehouse.
The `${ref()}` function will be replaced with the actual name of the dataset in your data warehouse. Note that the dependency is also listed just above the compiled query.

_*Your compiled script will differ depending on your cloud data warehouse. This example uses Redshift.*_

<div className="bp3-callout bp3-icon-info-sign bp3-intent-warning" markdown="1">
  Referencing a dataset that hasn't yet been created in your data warehouse will cause validation
  errors. These validation errors are warning you that the dataset you are referencing doesn't exist
  (yet). However, since Dataform will run your project's scripts in dependency order, you can safely
  ignore these errors.
</div>

## View dependency tree

The overview page (linked at the top left of Dataform) gives you an overview of your project. The overview includes a visualization of the dependency tree of your project, containing the two datasets created by your scripts.

<img
src="/static/images/how_to_guides/publish_tables/dependencies.png"
style="width: 100%"
/>

In bigger projects containing dozens or hundreds of datasets with complicated interdependencies, the overview page can help you keep track of your project's overall structure.


tutorials

Home

Learn the basics of Dataform, how it works, and where it fits in your data stack.

Introduction

An introduction to ELT and where Dataform fits in.

ELT and the modern data stack

modern-data-stack

Learn how Dataform and SQLX can help your team manage data in your warehouse.

SQLX and Dataform in 5 minutes

dataform-in-5-minutes

Learn how Dataform compiles your project and runs it in your warehouse.

How Dataform works

how-dataform-works

introduction

Learn about which data warehouses Dataform can work with and how to configure them.

Supported warehouses

Authentication, configuration options, and content for Snowflake.

Snowflake

snowflake

Authentication, configuration options, and content for BigQuery.

Google BigQuery

bigquery

Authentication, configuration options, and content for Redshift.

Amazon Redshift

redshift

Authentication and configuration options for Azure SQL Data Warehouse.

Azure SQL Data Warehouse

sqldatawarehouse

Authentication and configuration options for Postgres.

Postgres

postgres

warehouses

Learn how Dataform works with examples projects and scripts.

Example projects and scripts

A list of example projects for Snowflake, BigQuery, and Redshift.

Example projects

A simple project turning Stackoverflow public raw data into reporting tables in a BigQuery data warehouse.

Stackoverflow data on BigQuery

stackoverflow-bigquery

A fictional ecommerce store project on a Snowflake data warehouse.

Ecommerce store on Snowflake

ecommerce-snowflake

A fictional ecommerce store project on a Redshift data warehouse.

Ecommerce store on Redshift

ecommerce-redshift

projects

A list of examples of scripts to use in your Dataform projects.

Example scripts

example-scripts

examples

This tutorial is for people who are new to Dataform and want to be taught how to set up a new project. We will show you how to create your own data model, how to test and document it and how to run schedules on it.

Getting started tutorial

Learn how to create a new BigQuery project and generate warehouse credentials.

Getting set up

set-up

Learn how to connect to a warehouse and create and publish your first dataset.

Building your data model

building-your-data-model

Learn how to use the ref function in Dataform and how to view your project in the Dependency tree.

managing-dependencies

Learn how to set up a schedule and alerts in Dataform

Setting up a schedule

scheduling

Learn how to set up data quality tests using assertions and how to document your datasets

Data quality tests and documenting datasets

testing-and-documentation

Learn how to commit changes you've made in your Dataform project

Committing your changes

committing-changes

getting-started-tutorial

Guides to build your Dataform project.

Build your Dataform project

Learn how to configure, publish and document data tables in your warehouse.

Publish data tables and views

Learn how to publish tables and views in your warehouse.

Publish datasets

publish

Learn how to configure tables that update incrementally.

Incremental datasets

incremental

Learn how to add data documentation for your tables and views.

Document datasets

documentation

datasets

Learn about the structure and features of SQLX files.

SQLX

sqlx

Learn how to test data quality with assertions.

Test data quality with assertions

assertions

Learn how to declare external datasets with declarations.

Declare external datasets with declarations

declarations

Learn how to define custom SQL operations in Dataform.

Write custom SQL operations

operations

Learn how to configure your Dataform project.

Configure your project

configuration

Learn how you can use JavaScript to re-use code across your scripts and define several actions.

Power your code with JavaScript

Learn how to re-use code across your project with includes.

Re-use code with includes

includes

Learn how to define several actions in a single Javascript file.

Use JavaScript files

js-api

javascript

Learn how to organise your project with tags.

Organise your project with tags

Learn how to run unit tests on your queries.

Run unit tests on your queries

tests

Configure continuous integration/deployment workflows for your Dataform project.

Configure CI/CD

ci-cd

incremental-datasets

guides

Learn how to set up and run your projects in Dataform Web's cloud environment.

Dataform web guides

Learn how to use version control in Dataform Web.

Version control

version-control

Learn how to use Git access tokens for connecting to git providers.

Git access tokens

git-access-tokens

Learn how to schedule runs and view logs

Schedule runs

Learn about environments in Dataform web and how to configure them

Environments

environments

Learn about configuring slack notifications with Dataform web

Slack Notifications

slack-notifications

Learn how to use the REST API to control Dataform Web

Use the REST API

Web API Reference

reference

using-the-api

api-reference

set-up-warehouse

dataform-web

Best practices to scale your Dataform project and your analytics

Best practices using Dataform

Principles to start your project on the right foot

First steps with your Dataform project

start-your-dataform-project

Follow best practices to build a project that will scale and maximise productivity

Tutorials

What's next

Creating a new project

Managing dependencies

Sitemap