Fivetran Logs

dataform-fivetran-log

BETA package for transforming Fivetran Log data, which comes from a free internal connector. An ERD of the source data is here. The package currently only supports a single destination.

This package helps you understand:

  • How you are spending money in Fivetran according to our consumption-based pricing model. We display consumption data at the table, connector, destination, and account levels.
  • How your data is flowing in Fivetran:
    • Connector health and sync statuses
    • Daily API calls

The package's main goals are to:

  • Create a history of measured monthly active rows (MAR), credit consumption, and the relationship between the two
  • Enhance the connector table with sync metrics and relevant alert messages
  • Enhance the transformation table with run metrics
  • Create a history of daily API calls for each connector
  • Union log data across destinations

Models

modeldescription
fivetran_log_connector_statusEach record represents a connector loading data into a destination, enriched with data about the connector's status and the status of its data flow.
fivetran_log_mar_table_historyEach record represents a table's active volume for a month, complete with data about its connector and destination.
fivetran_log_credit_mar_historyEach record represents a destination's consumption by showing its MAR, total credits used, and credits per millions MAR.
fivetran_log_connector_daily_api_callsEach record represents a daily measurement of the API calls made by a connector, starting from the date on which the connector was set up.

Installation

Add the package to your package.json file in your Dataform project. You can find the most up to package version on the releases page.

Configure the package

Create a new JS file in your definitions/ folder and create the Fivetran Log tables as in the following example.

By default, the package will run using the fivetran_log schema. If this is not where your Fivetran Log data is, you can override it when calling the package:

1
2
3
4
5
6
7
8
9
10
11
12
const fivetranLog = require("fivetran-log");

fivetranLog({
  // The name of your fivetran log schema.
  fivetranLogSchema: "fivetran_log",
  // Default configuration applied to all produced datasets.
  defaultConfig: {
    schema: "fivetran_log_package",
    tags: ["fiveran_log_package"],
    type: "view"
  },
});

Not supported:

  • transformations
  • multiple destination databases

Supported warehouses:

  • BigQuery

What's next

Create new packages

Learn how to create new packages on Dataform.

BigQuery Audit Logs

Segment

Slowly changing dimensions

Sitemap