Now you have your BigQuery project and warehouse credentials, you’re going to set up your Dataform project using Dataform Web. In this step you'll create a project, connect your warehouse and build out your first dataset.
To create a new project in Dataform:
Create a Dataform account here.
Follow the sign up flow:
Dataform Tutorial
.New Project
.As part of the project creation flow you'll be asked to connect to a warehouse. You’re going to use the credentials you generated in the earlier part of this tutorial to connect:
On the Configure Warehouse modal click Connect
.
Select Google BigQuery
from the drop down menu.
Enter your Project ID.
Browse for the service account key JSON file you created in the Setting Up
part of this tutorial and upload it.
Check your connection is working:
Test Connection
to check that the connection is working.Save Connection
.Now that you've created your project and connected a warehouse, you're ready to start defining your data model.
Make sure you are in a development branch:
yourname_dev
.Develop
and select New Branch
.Create a new dataset:
New Dataset
button in the left hand side bar.order_stats
and click Create Table
.Define the dataset:
1SELECT 2 orders.date AS order_date, 3 orders.id AS id, 4 orders.customer_id AS customer_id, 5 orders.status AS order_status, 6 charges.status AS payment_status, 7 charges.payment_method AS payment_method, 8 SUM(orders.item_count) AS item_count, 9 SUM(charges.amount) AS amount 10 11FROM 12 dataform-demos.dataform_tutorial.shopify_orders AS orders 13 LEFT JOIN dataform-demos.dataform_tutorial.stripe_payments AS charges 14 ON orders.payment_id = charges.id 15 16WHERE 17 orders.id <= 999 18 AND orders.item_count > 0 19 AND orders.status <> 'internal' 20 AND charges.payment_method IN ('debit_card', 'subscription', 'coupon') 21 22GROUP BY 1, 2, 3, 4, 5, 6
order_stats.sqlx
below the config block.Preview Results
to check that the data looks correctCreate the table in your warehouse:
Publish Table
to create the table in your warehouseCheck Run Logs:
Run Logs
by clicking on the hamburger menu in the top left hand corner.Run Logs
you can see all the past runs in the project. You can see their status, as well as how and when they were triggered.Details
button.You now have a new table called order_stats
which has been created in your warehouse and you're ready to add to your data model!
For more detailed info on publishing datasets in Dataform, see our [docs].(https://docs.dataform.co/guides/datasets/publish)