dbt™ Models
dbt™ (Data Building Tool) is a transformation tool that enables you to view and consume the data loaded to your Destination tables into a form conducive to performing analytics and reporting by BI tools.
Hevo integrates with dbt Core™, the open-source CLI offering from dbt. In dbt, you create models containing SQL SELECT statements to query the data from the tables in your Destination. dbt compiles and runs these models against the tables specified by the query. It creates the target tables and views in your Destination, using Jinja, its template engine, and loads the results to them.
With Hevo’s dbt functionality, you can:
-
Configure dbt projects in Hevo for the following Destination types:
-
BigQuery
-
Snowflake
-
Amazon Redshift
-
Databricks
-
PostgreSQL
Read Configuring a dbt Project for the steps to do this.
-
-
Connect to your existing dbt projects created in any Git provider such as Bitbucket, GitHub, or GitLab.
Note: You should have at least
read-onlyaccess to the Git repository. -
Run the dbt models on your Destination data.
-
Configure a dbt project to run models on a specific branch in your Git repository. For example, to test your created models.
-
Schedule your dbt project to:
-
Run all of the models in your project together,
-
Run each model individually, or
-
Trigger your dbt project whenever Events are loaded to a specific Destination table.
-
-
View the model execution history and the complete activity log for the dbt project.
-
Integrate your dbt projects with Hevo Workflows.
Note: Starting Release 2.17, the Workflows feature is available only for existing users. Users signing up from Release 2.17 onwards cannot create Workflows and integrate dbt projects with them. For more information, please contact Hevo Support or your account executive.
All the dbt projects that you and your team create are hosted in a Git repository, thus allowing you to control and record the changes made to any dbt project.
Read Setting up dbt Projects in Git for the steps to create a sample dbt project.
A dbt project typically includes these resources:
-
Models: Models are the
.sqlfiles containing the SQL SELECT statements to transform your data into a form ready for analytics or as input to another model. One SQL file can contain only one model. -
Seeds: Seeds are CSV files containing data that augments the existing data in your data warehouse tables. For example, a CSV file that maps a country’s code to its name.
You should not use seeds to load data containing sensitive personal information or large amounts of data.
-
Snapshots: Snapshots are means by which you can capture data change in your tables. For example, you can create a snapshot of the
orderstable to capture the date and time when the status of the order changes. -
Tests: Tests are SQL queries that you can write to test the models in your project.
-
Macros: Macros are pieces of Jinja code that can be reused multiple times in your models. Macros are defined in
.sqlfiles and are similar to functions in other programming languages. -
Sources: Sources describe the data warehouse tables that contain the data to be transformed. You can list the schema name and table names in the
sources.ymlfile and reference them in your models. -
Docs: Docs contain the documentation files for your dbt project, which you can render as a website.
-
Analyses: Analyses contain SQL files that you want dbt to compile but not execute. For example, suppose you want to test an ad-hoc query and see the results of that query in your BI tool. For this, you can create an analysis file and compile the query in dbt. You can then run the compiled SQL file in your BI tool to see the results.
The compiled files enable you to select from models independent of your project environment.
The following is the directory structure of a sample dbt project called jaffle_shop, within the Git repository.
- Articles in this section