Declare external datasets with declarations

Learn how to declare external datasets with declarations.

Introduction

Your Dataform code will likely depend on at least one external (i.e. not produced by Dataform) dataset. Declarations enable you to treat these external datasets as first-class Dataform objects.

Example

Suppose we have an external dataset input.data .

To declare this dataset you can use SQLX:

1config {
2  type: "declaration",
3  schema: "input",
4  name: "data"
5}

Alternatively you can use the JavaScript API and add multiple declarations in a single file:

1declare({
2  schema: "input",
3  name: "foo",
4});
5
6declare({
7  schema: "input",
8  name: "bar",
9});

Dependencies on declared datasets

Once declared, a dataset can be referenced or resolved in the same way as any other dataset in Dataform.

Assuming the following declaration of the input.data dataset:

1config {
2  type: "declaration",
3  schema: "input",
4  name: "data"
5}

Other actions may now use the ref() or resolve() functions as usual, e.g. ref("data") or resolve({schema: "input", name: "data"}) .

What's next

Publish data tables and views

Learn how to configure, publish and document data tables in your warehouse.

SQLX

Learn about the structure and features of SQLX files.

Test data quality with assertions

Learn how to test data quality with assertions.

Write custom SQL operations

Learn how to define custom SQL operations in Dataform.

Configure your project

Learn how to configure your Dataform project.

Power your code with JavaScript

Learn how you can use JavaScript to re-use code across your scripts and define several actions.

Organise your project with tags

Learn how to organise your project with tags.

Run unit tests on your queries

Learn how to run unit tests on your queries.

Configure CI/CD

Configure continuous integration/deployment workflows for your Dataform project.

Sitemap