BETA package for transforming Fivetran Log data, which comes from a free internal connector. An ERD of the source data is here. The package currently only supports a single destination.
This package helps you understand:
The package's main goals are to:
| model | description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | fivetran_log_connector_status | Each record represents a connector loading data into a destination, enriched with data about the connector's status and the status of its data flow. | | fivetran_log_mar_table_history | Each record represents a table's active volume for a month, complete with data about its connector and destination. | | fivetran_log_credit_mar_history | Each record represents a destination's consumption by showing its MAR, total credits used, and credits per millions MAR. | | fivetran_log_connector_daily_api_calls | Each record represents a daily measurement of the API calls made by a connector, starting from the date on which the connector was set up.
Add the package to your package.json
file in your Dataform project. You can find the most up to package version on the releases page.
Create a new JS file in your definitions/
folder and create the Fivetran Log tables as in the following example.
By default, the package will run using the fivetran_log
schema. If this is not where your Fivetran Log data is, you can override it when calling the package:
1const fivetranLog = require("fivetran-log"); 2 3fivetranLog({ 4 // The name of your fivetran log schema. 5 fivetranLogSchema: "fivetran_log", 6 // Default configuration applied to all produced datasets. 7 defaultConfig: { 8 schema: "fivetran_log_package", 9 tags: ["fiveran_log_package"], 10 type: "view" 11 }, 12});