Starting Release 2.17, Hevo has stopped supporting Firebolt as a Destination type. As a result, you will not be able to configure new Firebolt Destinations. Any existing Pipelines will continue to be supported. You can also create new Pipelines with your existing Firebolt Destination.
Firebolt is a cloud-based data storage and analytics service built over AWS that you can set up as a Destination in Hevo.
Databases in Firebolt are attached with computing resources called engines that provide isolated, assured compute and storage capacity for different workloads and queries. These engines help you run perform SQL queries on your data.
You can attach any number of engines to a database. Read Engines for more information. The cost of each engine varies according to your selected configurations.
The ingested data from your Hevo Pipeline is first staged in your Amazon S3 Bucket and then transferred to your Firebolt database.
Prerequisites
-
An active Firebolt account.
-
The username and password of your Firebolt account.
-
A General Purpose engine attached to your database. A general-purpose engine offers read and write ability and better performance compared to other engine types.
-
An Amazon S3 bucket in the same region as your Firebolt database.
and
-
A Firebolt database to which you want to load data.
-
You are assigned the Team Collaborator, or any administrator role except the Billing Administrator role in Hevo to create the Destination.
Perform the following steps to configure Firebolt as a Destination:
Create a Firebolt Database (Optional)
Follow these steps to configure a Firebolt database and engine if you do not already have one:
-
Log in to the Firebolt dashboard.
-
Click +NEW DATABASE.
-
Enter the Database Name.
-
Select the Database Region.
Note: Your database should be in the same region as your S3 bucket.
-
Select a General Purpose engine from the list of available engines. You can edit and configure these engines according to your requirements. Also, you can add any number of engines to your database. Read Tuning Engine Performance.
Note: Hevo automatically uses your chosen engine to load data at the default replication frequency of 30 minutes. As a cost-saving measure, the engine has an auto-stop trigger of 20 minutes. This means, for each data replication run, the engine has to be restarted. To avoid the resulting delay, you can increase the Auto-stop duration. Read Auto-stop duration.
-
Click Create Database.
Configure Firebolt as a Destination
-
Click DESTINATIONS in the Navigation Bar.
-
Click + CREATE DESTINATION in the Destinations List View.
-
In the Add Destination page, select Firebolt as the Destination type.
-
In the Configure your Firebolt Destination page, specify the following:
-
Destination Name: A unique name for your Destination.
-
Firebolt Username: The username of your Firebolt account.
-
Firebolt Password: The password of your Firebolt account.
-
Database Name: The database to which you want to load data.
-
Engine Name: As soon as you select a database, the default Read/Write engine linked to the database gets selected. If required, select a different engine from the drop-down.
Note: The engine must be linked to the selected database.
-
Access Key ID: An AWS access key ID with permissions to read from and write to your S3 bucket.
-
Secret Access Key: The secret access key for the access key ID you provided above.
-
Bucket: The name of the S3 bucket that you want Hevo to use as the staging environment.
-
Bucket Region: Select the AWS region where the bucket is located.
Note: The region of the S3 bucket must be the same as that of your chosen Firebolt engine.
-
-
Click TEST CONNECTION.
-
Click SAVE DESTINATION.
Additional Information
Read the detailed Hevo documentation for the following related topics:
Destination Considerations
-
Tables once created in the Firebolt database cannot be altered. This means that you cannot add or remove columns, as Firebolt does not currently support this. You must drop the table from the Schema Mapper page or the Workbench page and create a new one.
-
If performance optimization configurations such as indexes, table types, partition keys, and non-nullable columns are not set according to your query patterns, the queries will be inefficient, and you will have to drop and create the tables again with correct configurations.
-
Partition definition occurs during table creation. Check the queries you plan to run on your data, in particular, the WHERE clauses. Make sure that either the table’s primary index or primary key includes the created partitions to enjoy maximum data pruning.
-
The partition key cannot be composed of nullable columns.
-
Currently, Firebolt only supports Append-Only queries. Due to this, the Pipeline does not deduplicate the Events.
See Also
Revision History
Refer to the following table for the list of key updates made to this page:
| Date | Release | Description of Change |
|---|---|---|
| Sep-08-2021 | 1.71 | New document. |