GitLab is an all-in-one platform that provides comprehensive DevOps functionality, including repository management, code review, issue tracking, continuous integration, and deployment tools. It is open-source and can be run either on-premises or in the cloud, offering flexibility and customization options to organizations of different sizes and needs.
You can replicate the data from your GitLab account to a Destination database or data warehouse using Hevo Pipelines. Refer to section, Data Model for the list of supported objects.
Hevo uses the GitLab access token to replicate the data present in your GitLab account to the desired Destination database or data warehouse for scalable analysis.
Prerequisites
Creating the Access Token
You need an access token to authenticate Hevo on your GitLab account.
Note: You must log in as an Admin user to perform these steps.
Perform the following steps to create the access token:
-
Log in to your GitLab account.
-
Click on your profile picture in the top left corner, and select Preferences.
-
In the left navigation pane, under the User Settings, click Access Tokens.
-
On the Personal access tokens page, click Add new token.
-
In the Add a personal access token section, specify the following:
-
Token Name: A name for your access token.
-
Expiration date: The date on which the token expires.
Note: You must specify a future date as the expiration date to a later date to prevent reauthorization of your Hevo Pipeline with GitLab as the Source.
-
Select scopes: The API permissions needed by Hevo to access your data. To grant the required permissions and create the access token, select the following scopes for the API token:
-
api
-
read_api
-
read_user
-
read_repository
-
read_registry
-
Click Create personal access token.
-
Click Copy to copy the access token and save it securely like any other password. Use this token while configuring your Hevo Pipeline.
Configuring GitLab as a Source
-
Perform the following steps to configure GitLab as the Source in your Pipeline:
-
Click PIPELINES in the Navigation Bar.
-
Click + CREATE PIPELINE in the Pipelines List View.
-
On the Select Source Type page, select GitLab.
-
On the Configure your GitLab Source page, specify the following:
-
Pipeline Name: A unique name for the Pipeline, not exceeding 255 characters.
-
Access token: The access token that you generated in your GitLab account.
-
Click TEST & CONTINUE.
-
Proceed to configuring the data ingestion and setting up the Destination.
Data Replication
For Teams Created |
Default Ingestion Frequency |
Minimum Ingestion Frequency |
Maximum Ingestion Frequency |
Custom Frequency Range (in Hrs) |
Before Release 2.21 |
1 Hr |
1 Hr |
24 Hrs |
1-24 |
After Release 2.21 |
6 Hrs |
30 Mins |
24 Hrs |
1-24 |
Note: The custom frequency must be set in hours as an integer value. For example, 1, 2, or 3 but not 1.5 or 1.75.
-
Historical Data: In the first run of the Pipeline, Hevo ingests all the existing data for the selected objects from your GitLab account and loads it to the Destination.
-
Incremental Data: Once the historical load is complete, data is ingested as per the ingestion frequency in Full Load or Incremental mode, as mentioned in the table below.
Schema and Primary Keys
Hevo uses the following schema to upload the records in the Destination database:
Data Model
The following is the list of tables (objects) that are created at the Destination when you run the Pipeline:
Object |
Mode |
Description |
Branches |
Full |
Contains details of the separate lines of development within a project’s Git repository. It is used to work on new features, bug fixes, or other changes independently of the main branch (usually named “master” or “main”) and then merge them back when they are ready to be released. |
Commits |
Incremental |
Contains details of the latest changes in the source code to track the history of a project. |
Deployments |
Incremental |
Contains details of all the changes to the IT service that are tested and deployed in the live environment. |
Groups |
Full |
Contains the list of groups and users which allows for the streamlined management of access and permissions for all projects within the group, and serve as a shared space for collaboration and communication. |
Group Issue Boards |
Full |
Contains the list of all the issues assigned to different groups. |
Issues |
Incremental |
Contains details of all the bugs reported in a project. |
Jobs |
Full |
Contains details of a specific task or set of tasks that are executed as part of the continuous integration and continuous deployment (CI/CD) process. Jobs can include building code, running tests, deploying code, or any other task required to complete the software development cycle. |
Merge Requests |
Incremental |
Contains details of the changes to a Git repository, including the code changes, author, date, and commit message. |
Pipelines |
Incremental |
Contains details of the series of jobs that run automatically as part of a continuous integration and continuous deployment (CI/CD) process. They are made up of Jobs and Stages that automate and streamline the software development workflow, from source code to deployment. |
Projects |
Incremental |
Contains details of all the files, code, and issues to organize and manage the software development workflow, including source code, documentation, testing, and deployment. |
Releases |
Full |
Contains details of a specific version of your project that has been deployed and made available to users. |
Snippets |
Full |
Contains details of a small pieces of code or text that can be stored and shared within a GitLab instance. |
Users |
Full |
Contains details about the users, who have access to the projects in your GitLab account. |
Read the detailed Hevo documentation for the following related topics:
Source Considerations
-
Pagination: An API call for each GitLab object fetches one page with up to 100 records.
-
Rate Limit: GitLab imposes a limit of 2000 API calls per minute. If the limit is exceeded, Hevo defers the ingestion till the limits reset.
Limitations
-
Hevo currently does not support deletes. Therefore, any data deleted in the Source may continue to exist in the Destination.
-
Hevo does not provide you with the option to select the historical sync duration for loading historical data. All the historical data in your GitLab account is loaded.
-
Hevo does not load an Event into the Destination table if its size exceeds 128 MB, which may lead to discrepancies between your Source and Destination data. To avoid such a scenario, ensure that each row in your Source objects contains less than 100 MB of data.
Revision History
Refer to the following table for the list of key updates made to this page:
Date |
Release |
Description of Change |
Jan-07-2025 |
NA |
Updated the Limitations section to add information on Event size. |
Oct-29-2024 |
NA |
Updated section, Creating the Access Token as per the latest GitLab UI. |
Mar-05-2024 |
2.21 |
Updated the ingestion frequency table in the Data Replication section. |
Feb-07-2023 |
2.07 |
New document. |