Common data models for creating type-2 slowly changing dimensions tables from mutable data sources in Dataform.
If you would like us to add support for another warehouse, please get in touch via email or Slack
Add the package to your package.json
file in your Dataform project. You can find the most up to package version on the releases page.
Create a new JS file in your definitions/
folder create an SCD table with the following example:
1const scd = require("dataform-scd"); 2 3scd("source_data_scd", { 4 // A unique identifier for rows in the table. 5 uniqueKey: "user_id", 6 // A field that stores a timestamp or date of when the row was last changed. 7 timestamp: "updated_at", 8 // The source table to build slowly changing dimensions from. 9 source: { 10 schema: "dataform_scd_example", 11 name: "source_data", 12 }, 13 // Any configuration parameters to apply to the incremental table that will be created. 14 incrementalConfig: { 15 bigquery: { 16 partitionBy: "updated_at", 17 }, 18 }, 19});
For more advanced customization of outputs, see the example.js.
Slowly changing dimensions can only by updated as quickly as these models are run. These models should typically be scheduled to run every day or every hour, depending on the granularity of changes you want to capture.
This package will create two relations in the warehouse, for a given name
these will be:
{name}
- a view with scd_valid_from
and scd_valid_to
fields{name}_updates
- an incremental table that stores the change history of the source table