Buildkite is a continuous integration (CI) and continuous delivery (CD) platform for software development teams to automate their build, test, and deployment processes. It provides a centralized platform for defining and managing pipelines, which are sequences of steps that run in response to code changes. It offers a scalable, secure, and flexible environment that enables teams to focus on writing code rather than managing the underlying infrastructure.
You can replicate the data from your Buildkite account to a Destination database or data warehouse using Hevo Pipelines. Refer to section, Data Model for the list of supported objects.
Hevo uses the Buildkite REST API to replicate the data present in your Buildkite account to the desired Destination database or data warehouse for scalable analysis.
Prerequisites
Obtaining the API Credentials
Hevo uses an access token and organization slug to authenticate itself on your Buildkite account. These credentials do not expire and can be reused for all your Pipelines.
Note: You must log in as an Admin user to perform these steps.
Obtaining the access token
Perform the following steps to obtain your access token:
-
Log in to your Buildkite account.
-
Click on your user name in the top right corner, and select Personal Settings.
-
In the left navigation pane, in the Personal Settings section, select API Access Tokens.
-
On the API Access Tokens page, click New API Access Token.
-
On the New API Access Token page, specify the following:
-
Description: A unique name for your access token.
-
Organization Access: The organization from where you want to ingest data.
-
REST API Scopes: The API permissions needed by Hevo to access your data. Hevo only needs read access to your API resources to ingest data. Select the following scopes:
-
Click Create New API Access Token.
-
Copy the access token, and save it securely like any other password. Use this token while configuring your Hevo Pipeline.
Obtaining the organization slug
An organization slug is a unique identifier for an organization in your Buildkite account. It can be found in the URL, appearing after the main Buildkite domain, and forms part of the URL structure. For example, in the URL https://buildkite.com/my-organization/my-project, the organization slug is my-organization.
Perform the following steps to obtain the organization slug:
-
Log in to your Buildkite account.
-
Click on your username in the top right corner, and select Personal Settings.
-
In the left navigation pane, select Organizations.
-
On the Organizations page, select the organization from where you want to ingest data.
-
On the Organization Settings page, copy the Slug, and save it securely like any other password. Use this slug while configuring your Hevo Pipeline.
Configuring Buildkite as a Source
Perform the following steps to configure Buildkite as the Source in your Pipeline:
-
Click PIPELINES in the Navigation Bar.
-
Click + CREATE PIPELINE in the Pipelines List View.
-
On the Select Source Type page, select Buildkite.
-
On the Configure your Buildkite Source page, specify the following:
-
Pipeline Name: A unique name for the Pipeline, not exceeding 255 characters.
-
Access Token: The token that you obtained from your Buildkite account.
-
Organization slug: The unique identifier that you obtained from your Buildkite account to identify the organization from where you want to ingest data.
-
Click TEST & CONTINUE.
-
Proceed to configuring the data ingestion and setting up the Destination.
Data Replication
For Teams Created |
Default Ingestion Frequency |
Minimum Ingestion Frequency |
Maximum Ingestion Frequency |
Custom Frequency Range (in Hrs) |
Before Release 2.21 |
1 Hr |
1 Hr |
24 Hrs |
1-24 |
After Release 2.21 |
6 Hrs |
30 Mins |
24 Hrs |
1-24 |
Note: The custom frequency must be set in hours as an integer value. For example, 1, 2, or 3 but not 1.5 or 1.75.
-
Historical Data: In the first run of the Pipeline, Hevo ingests all the existing data for the selected objects from your Buildkite account and loads it to the Destination. Historical load parallelization is applicable for this Source.
-
Incremental Data: Once the historical load is complete, all new and updated records for the Builds and Jobs objects are ingested as per the ingestion frequency. The remaining objects are ingested in Full Load mode.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination database:
Data Model
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline:
Object |
Mode |
Description |
Pipelines |
Full Load |
Contains the list of the series of steps that outlines the process of performing a specific task, such as building and deploying software. |
Agents |
Full Load |
Contains the list of all the physical and virtual machines that execute the jobs defined in the pipelines within your organization. |
Builds |
Incremental |
Contains details about each execution of a pipeline step or job by agents. |
Jobs |
Incremental |
Contains details about the sequential set of tasks and steps that a user defines in a pipeline. |
Read the detailed Hevo documentation for the following related topics:
Source Considerations
-
Pagination: An API call for each Buildkite object fetches one page with up to 100 records.
-
Rate Limit: Buildkite imposes a limit of 100 API calls per minute per access token. If the limit is exceeded, Hevo defers the ingestion till the limits reset.
Limitations
-
Hevo currently does not support deletes for Buildkite. Therefore, any data deleted in the Source may continue to exist in the Destination.
-
Hevo does not load an Event into the Destination table if its size exceeds 128 MB, which may lead to discrepancies between your Source and Destination data. To avoid such a scenario, ensure that each row in your Source objects contains less than 100 MB of data.
Revision History
Refer to the following table for the list of key updates made to this page:
Date |
Release |
Description of Change |
Jan-07-2025 |
NA |
Updated the Limitations section to add information on Event size. |
Oct-22-2024 |
NA |
Updated section, Obtaining the API Credentials as per the latest Buildkite UI. |
Mar-05-2024 |
2.21 |
Updated the ingestion frequency table in the Data Replication section. |
Feb-07-2023 |
2.07 |
New document. |