Dataform is a platform to manage data in BigQuery, Snowflake, Redshift, and other data warehouses. It helps data teams build data pipelines that turn raw data into new tables and views that can be used for analytics.
Dataform does the T in ELT (Extract, Load, Transform) processes. It doesn’t extract or load data in your warehouse but it’s very powerful to transform data already loaded in your warehouse.
By using Dataform and its best practices, data teams are more productive and build new data tables that are well defined, tested and documented for use by the entire company.
In its simplest form, Dataform helps you run SQL commands in your data warehouse to create new tables and views. Dataform ships many features made to improve the way you manage data and make your team more productive.
They are two main ways to work with Dataform. You can use Dataform web application with an Integrated Development Environment (IDE). You can also use Dataform locally using the Command Line Interface (CLI).
The core of Dataform is open source (Dataform compiler and runner) and can be used with the CLI.
Dataform is built for data professionals who interact with a cloud data warehouse. That includes anyone who knows how to write SQL queries, including data analysts, data engineers and data scientists.
Using Dataform requires an understanding of SQL. If you're unfamiliar with SQL, check out the Khan Academy Introduction to SQL course or Codeacademy.
Knowledge of Javascript can be useful to use Dataform most advanced features. Those are totally optional but can make developing faster and easier. If you are unfamiliar with Javascript, check the re-introduction to Javascript on MDN.
Dataform helps data teams adopt best practices and software engineering workflows to manage tables in their data warehouse.
By using Dataform and its best practices, data teams are able to manage data significantly faster and deliver data that is trusted and understood by the entire organization.
You can join our Slack group and discuss with our team and hundreds of other data professionals using Dataform.
If you are encountering any issue on the Dataform web app, please contact our team using the intercom messenger icon on the bottom right of the page.