API Reference

The database where the corresponding view for this assertion should be created.

A description for this assertion.

If set to true, this action will not be executed. However, the action may still be depended upon. Useful for temporarily turning off broken actions.

Declares whether or not this action is hermetic. An action is hermetic if all of its dependencies are explicitly declared.

If this action depends on data from a source which has not been declared as a dependency, then hermetic should be explicitly set to false. Otherwise, if this action only depends on data from explicitly-declared dependencies, then it should be set to true.

The schema where the corresponding view for this assertion should be created.

A list of user-defined tags with which the action should be labeled.

Returns the name of this dataset.

References another action, adding it as a dependency to this action, returning valid SQL to be used in a from expression.

This function can be called with a Resolvable object, for example:

${ref({ name: "name", schema: "schema", database: "database" })}

This function can also be called using individual arguments for the "database", "schema", and "name" values. When only two values are provided, the default database will be used and the values will be interpreted as "schema" and "name". When only one value is provided, the default data base schema will be used, with the provided value interpreted as "name".

${ref("database", "schema", "name")}
${ref("schema", "name")}
${ref("name")}

Similar to ref except that it does not add a dependency, but just resolves the provided reference so that it can be used in SQL, for example in a from expression.

See the ref function for example usage.

Equivelant to resolve(name()).

Returns a valid SQL string that can be used to reference the dataset produced by this action.

BigQuery policy tags that should be applied to this column.

These should be the fully qualified identifier of the tag, including the project name, location, and taxonomy, which can be copied from the policy tags page in GCP.

For example: "projects/1/locations/eu/taxonomies/2/policyTags/3"

Currently BigQuery supports only a single tag per column.

A description of columns within the struct, object or record.

A description of the struct, object or record.

A description of columns within the dataset.

The database in which the output of this action should be created.

A description of the dataset.

The schema in which the output of this action should be created.

A description of columns within the dataset.

The database in which the output of this action should be created.

A description of the dataset.

If set to true, this action will not be executed. However, the action may still be depended upon. Useful for temporarily turning off broken actions.

Declares that this operations action creates a dataset which should be referenceable using the ref function.

If set to true, this action should create a dataset with its configured name, using the self() context function.

For example:

create or replace table ${self()} as select ...

Declares whether or not this action is hermetic. An action is hermetic if all of its dependencies are explicitly declared.

If this action depends on data from a source which has not been declared as a dependency, then hermetic should be explicitly set to false. Otherwise, if this action only depends on data from explicitly-declared dependencies, then it should be set to true.

The schema in which the output of this action should be created.

A list of user-defined tags with which the action should be labeled.

Key-value pairs for options table, view, materialized view.

Some options (e.g. partitionExpirationDays) have dedicated type/validity checked fields; prefer using those. String values need double-quotes, e.g. additionalOptions: {numeric_option: "5", string_option: '"string-value"'} If the option name contains special characters, e.g. hyphens, then quote its name, e.g. additionalOptions: { "option-name": "value" }.

The keys by which to cluster partitions by.

For more information, read the BigQuery clustered tables docs.

Key-value pairs for BigQuery labels.

If the label name contains special characters, e.g. hyphens, then quote its name, e.g. labels: { "label-name": "value" }.

The key with which to partition the table. Typically the name of a timestamp or date column.

For more information, read the BigQuery partitioned tables docs.

This setting specifies how long BigQuery keeps the data in each partition. The setting applies to all partitions in the table, but is calculated independently for each partition based on the partition time.

For more information, see our docs.

When you create a partitioned table, you can require that all queries on the table must include a predicate filter ( a WHERE clause) that filters on the partitioning column. This setting can improve performance and reduce costs, because BigQuery can use the filter to prune partitions that don't match the predicate.

For more information, see our docs.

SQL based filter for when incremental updates are applied.

For more information, see our incremental dataset docs.

The key with which to partition the table. Typically the name of a timestamp or date column.

For more information, read the partitioning documentation for the Presto connection in use.

Sets the DISTKEY property when creating tables.

For more information, read the Redshift create table docs.

Set the DISTSTYLE property when creating tables.

For more information, read the Redshift create table docs.

A list of string values that will configure the SORTKEY property when creating tables.

For more information, read the Redshift create table docs.

Sets the style of the sort key when using sort keys.

For more information, read the Redshift sort style article.

The distribution option value.

For more information, read the Azure CTAS docs.

A list of clustering keys to cluster the table by. Only applicable to actions of type "table" or "incremental".

For more information, read the Snowflake clustering docs.

If set to true, a secure view will be created.

For more information, read the Snowflake Secure Views docs.

If set to true, a transient table will be created. Only applicable to actions of type "table".

For more information, read the Snowflake docs.

Column(s) which may never be NULL.

If set, the resulting assertion will fail if any row contains NULL values for these column(s).

General condition(s) which should hold true for all rows in the dataset.

If set, the resulting assertion will fail if any row violates any of these condition(s).

Column(s) which constitute the dataset's unique key index.

If set, the resulting assertion will fail if there is more than one row in the dataset with the same values for all of these column(s).

Combinations of column(s), each of which should constitute a unique key index for the dataset.

If set, the resulting assertion(s) will fail if there is more than one row in the dataset with the same values for all of the column(s) in the unique key(s).

Assertions to be run on the dataset.

If configured, relevant assertions will automatically be created and run as a dependency of this dataset.

BigQuery-specific warehouse options.

A description of columns within the dataset.

The database in which the output of this action should be created.

A description of the dataset.

If set to true, this action will not be executed. However, the action may still be depended upon. Useful for temporarily turning off broken actions.

Declares whether or not this action is hermetic. An action is hermetic if all of its dependencies are explicitly declared.

If this action depends on data from a source which has not been declared as a dependency, then hermetic should be explicitly set to false. Otherwise, if this action only depends on data from explicitly-declared dependencies, then it should be set to true.

Only valid when the table type is view. Only valid when using Snowflake or BigQuery.

If set to true, will make the view materialized.

For more information, read the BigQuery materialized view docs or the Snowflake materialized view docs.

Presto-specific options.

Only allowed when the table type is incremental.

If set to true, running this action will ignore the full-refresh option. This is useful for tables which are built from transient data, to ensure that historical data is never lost.

Redshift-specific warehouse options.

The schema in which the output of this action should be created.

Snowflake-specific options.

Azure SQL Data Warehouse-specific options.

A list of user-defined tags with which the action should be labeled.

The type of the dataset. For more information on how this setting works, check out some of the guides on publishing different types of datasets with Dataform.

Unique keys for merge criteria for incremental tables.

If configured, records with matching unique key(s) will be updated, rather than new rows being inserted.

Indicates whether the config indicates the file is dealing with an incremental table.

Returns the name of this dataset.

References another action, adding it as a dependency to this action, returning valid SQL to be used in a from expression.

This function can be called with a Resolvable object, for example:

${ref({ name: "name", schema: "schema", database: "database" })}

This function can also be called using individual arguments for the "database", "schema", and "name" values. When only two values are provided, the default database will be used and the values will be interpreted as "schema" and "name". When only one value is provided, the default data base schema will be used, with the provided value interpreted as "name".

${ref("database", "schema", "name")}
${ref("schema", "name")}
${ref("name")}

Similar to ref except that it does not add a dependency, but just resolves the provided reference so that it can be used in SQL, for example in a from expression.

See the ref function for example usage.

Equivelant to resolve(name()).

Returns a valid SQL string that can be used to reference the dataset produced by this action.

Shorthand for an if condition. Equivalent to cond ? trueCase : falseCase. falseCase is optional, and defaults to an empty string.

The dataset that this unit test tests.