Configure Azure Blob Storage as a Source in your Pipeline

Perform the following steps to configure your Azure Blob Storage Source:

1. Click Pipelines in the Navigation Bar.

2. Click + Create Pipeline in the Pipelines List View.

3. On the Select Source Type page, select Azure Blob Storage.

4. On the Select Destination Type page, select the type of Destination you want to use.

5. On the Select Pipeline Type page, click Edge, and then click Continue.

   

   This page appears only if the selected Destination type is supported in Edge and your Team has an existing Azure Blob Storage Pipeline with the same Destination type. Otherwise, you can proceed to create an Edge Pipeline.

6. On the Configure Source page, specify the following:

   

   • Source Name: A unique name for your Source, not exceeding 255 characters. For example, Azure Blob Source.

   • In the Connect to your Azure Blob Storage section:

     • Authentication Method: The method you want Hevo to use to connect to your storage account. Select one of the following:

       • Microsoft Entra ID App (Client Secret): Connects using the client secret that you generated for your application.

         • Container Name: The name of the container that you obtained in the Obtain the Storage Account and Container Names section. For example, hevo-doc-test. Specify only the name, without a URL or folder path.

         • Storage Account Name: The name of the storage account that you obtained in the Obtain the Storage Account and Container Names section. For example, hevodocblobtest.

         • Directory (Tenant) ID: The directory (tenant) ID that you obtained in the Register an Application in Microsoft Entra ID section.

         • Application (Client) ID: The application (client) ID that you obtained in the Register an Application in Microsoft Entra ID section.

         • Client Secret: The client secret value that you obtained in the Generate a client secret section.

       • Microsoft Entra ID App (Client Certificate): Connects using the certificate that you uploaded to your application and its private key.

         

         • Container Name: The name of the container that you obtained in the Obtain the Storage Account and Container Names section. For example, hevo-doc-test.

         • Storage Account Name: The name of the storage account that you obtained in the Obtain the Storage Account and Container Names section. For example, hevodocblobtest.

         • Directory (Tenant) ID: The directory (tenant) ID that you obtained in the Register an Application in Microsoft Entra ID section.

         • Application (Client) ID: The application (client) ID that you obtained in the Register an Application in Microsoft Entra ID section.

         • Client Certificate: The certificate file that you obtained in the Generate a client certificate section, in the .pem, .crt, or .cer format. You can also upload a .pem file that contains both the certificate and its private key.

         • Client Private Key (Optional): The private key file that you obtained in the Generate a client certificate section, such as private_key.pem. Leave this field blank if your certificate file also contains the private key.

         • Private Key Passphrase (Optional): The passphrase for the private key. Leave this field blank if you did not specify a passphrase while creating the private key.

       • Shared Access Signature (SAS): Connects using the shared access signature that you generated for your storage account.

         

         • Container Name: The name of the container that you obtained in the Obtain the Storage Account and Container Names section. For example, hevo-doc-test.

         • SAS URL or Connection String: The connection string or Blob service SAS URL that you obtained in the Generate a Shared Access Signature section.

     • Connection Method: The way Hevo connects to your storage account. The Connect Directly option connects to the public endpoint of your storage account.

     • Use SSH: Enable this option to connect to your storage account through an SSH tunnel, instead of directly. Use it when your storage account does not allow public network access and is available only through a private endpoint in your Azure virtual network. Hevo connects to an SSH server in that network, and the server forwards Hevo’s requests to your storage account. Read Create a Linux virtual machine in the Azure portal for the steps to set up an SSH server in Azure. If you enable this option, specify the following:

       

       • SSH IP: The public IP address of the SSH server. For example, 52.12.34.56. The server must allow connections from Hevo’s IP addresses for your region. To do this, add an inbound security rule for these IP addresses on the SSH port. Read Create a security rule for the steps.

       • SSH Port: The port on which the SSH server listens for connections. Default value: 22.

       • SSH User: The name of the user on the SSH server that Hevo connects as. For example, hevouser.

       • IP address of storage container: The private IP address of the private endpoint of your storage account, as the SSH server sees it. For example, 10.0.1.4. Hevo connects to this address on port 443, so the SSH server must be able to access it on that port.

       To allow Hevo to connect to your SSH server, add the public keys displayed on the page to the ~/.ssh/authorized_keys file of the SSH user on the server. Read Create an SSH user and group for the steps.

7. In the Files to Sync section, define rules that determine the files Hevo ingests and the tables it loads them into. Hevo checks each file against the rules from top to bottom and ingests it using the first rule that matches. Therefore, add a rule for specific files above a rule that reads their entire folder. For example, to load the daily order files in the orders folder into one table and all other CSV files in that folder into another, add the Pattern as Table rule for the daily files first, and then the Directory as Tables rule for the folder. You can add up to 20 rules.

   To add a rule, perform the following steps:

   

   1. Click Add Rule.

   2. Specify the following:

      • Folder Path: The path of the folder in the container from which you want to ingest files. For example, orders/. If you leave this field blank, Hevo reads files from the top level of the container.

      • Include Nested Folders: If enabled, Hevo also reads files from all the subfolders of the specified folder. If disabled, Hevo reads only the files present directly in it. For example, if the folder path is orders/ and this option is enabled, Hevo also reads files in orders/archive/. Default value: Disabled.

      • File Format: The format of the files that you want to ingest. Select one of the following:

        • CSV: Files with the .csv extension.

        • TSV: Files with the .tsv or .tab extension.

        Hevo identifies the format of a file from its extension. Files in any other format in the folder are not ingested by this rule. Default value: CSV.

      • Object Handling Rule: The way Hevo maps the files to Destination tables. Select one of the following:

        • Directory as Tables: Hevo loads all the files from the folder into a single table. This is the default option.

        • Files as Tables: Hevo loads each file into a separate table named after the file. Hevo can create up to 50,000 tables for this rule. If the folder contains more files, the remaining files are not ingested. If you delete a file, Hevo marks the corresponding object as Deleted. The rows loaded from the file remain in the Destination. If you add the file again, Hevo marks the object as active, but does not ingest the file until you resync the object.

        • Pattern as Table: Hevo loads the files whose names match a regular expression into a single table. If you select this option, specify the File Name Pattern. This is a regular expression that must match the entire file name, excluding the folder path. The match is case-sensitive. For example, orders_\d{8}\.csv matches orders_20261001.csv.

      • Table Name (Optional): The name of the Destination table. If you leave this field blank, Hevo names the table {dir}_{type}, where {dir} is the name of the folder and {type} is the file format. For example, orders_csv.
        
        You can use the placeholders {dir}, {type}, and $1 to $9 for the groups captured by the file-name pattern. For example, with the file-name pattern sales_(\w+)_\d{8}\.csv and the table name sales_$1, Hevo loads sales_east_20261001.csv into the sales_east table. This field is not displayed if you select Files as Tables.

      • Read Files with No Extension: If enabled, Hevo ingests files that do not have an extension, such as orders_export instead of orders_export.csv, using the selected file format. If disabled, Hevo skips such files. Default value: Disabled. This option is displayed only if you select Directory as Tables.

      • Header: Select First line is the header if the first line of each file contains the column names. Otherwise, select Headerless: Name the columns column_1, column_2, and so on, and Hevo names the columns column_1, column_2, and so on. Default value: First line is the header.

      • Delimiter: The character that separates the values in a row. For example, enter | for pipe-delimited files. To specify a tab, enter \t. Default value: A comma (,) for CSV files and a tab for TSV files.

      • Quote Character: The character that encloses a value. For example, enter ' if the values in your files are enclosed in single quotes. Default value: A double quote (“).

      • Skip Lines at the Top: The number of lines to skip at the beginning of each file. Hevo reads the header, if any, from the line after these lines. For example, enter 2 if each file starts with a two-line report title above the header. Default value: 0.

      • Skip Lines at the Bottom: The number of lines to skip at the end of each file. For example, enter 1 if each file ends with a totals row. Default value: 0.

      • Escape Character: The character used to escape a quote character within a value. If you leave this field blank, a quote within a value is escaped by doubling it. For example, the value 5” monitor is written as "5"" monitor" in the file.

      • Null Sequence: The value that Hevo must load as NULL wherever it appears, such as \N. If you leave this field blank, Hevo loads empty values as NULL.

      • Line Separator: The character that marks the end of a line. If you leave this field blank, Hevo uses a newline, and handles carriage return and line feed (CRLF) line endings automatically.

      • Values Are Never Quoted: Enable this option only if the values in your files are never enclosed in quote characters. If enabled, a value that contains the delimiter is split into separate columns. For example, with a comma delimiter, the value New York, NY is loaded as two values, New York and NY. Default value: Disabled.

      • Skip Rows after the Header: The number of rows between the header and the first data row, such as a row containing units. Default value: 0.

   3. Click Save.

   Repeat the steps above to add more rules.

   Note: After you create the Pipeline, you cannot change the File Format or Include Nested Folders setting of a rule. Any rule that you add later reads only the CSV files present directly in its folder.

8. In the Additional Settings section:

   

   • Updated File Handling: The way Hevo loads the rows of a file that is updated after it is ingested. This also determines the primary key of the Destination table. Select one of the following:

     • Update Rows by Line Number: Hevo replaces the rows previously loaded from the file with its current rows, using the file path and line number of each row as the primary key. This is the default option.

     • Keep Every Version: Hevo loads the rows of each version of the file into the Destination table and keeps the rows loaded earlier.

   • Compression: The way Hevo reads compressed files. Select one of the following:

     • Auto-detect: Hevo identifies compressed files and archives from their file names, or from their content if the name does not indicate the compression type. Hevo then reads the files from them. This is the default option. Read Supported Compression Formats for the formats that Hevo supports.

     • None: Hevo reads every file as it is stored, without decompressing it.

9. Click Test & Continue to test the connection to your Azure Blob Storage Source. Hevo verifies that it can list and read the files in your container, and displays the files that each rule matches. If a rule does not match any file, Hevo displays a warning. Click Continue to set up your Destination.

Data Replication

Hevo ingests the following types of data from the files that match your rules:

• Historical Data: The first run of the Pipeline ingests all the files that match your rules and loads them into the Destination.

• Incremental Data: Once the historical load is complete, Hevo checks the container for new and updated files as per the sync frequency and ingests them. As it is not possible to identify individual changes within a file, Hevo re-ingests the entire content of each updated file. It then replaces the rows previously loaded from that file in the Destination with the file’s current rows.

Hevo detects an update when the entity tag (ETag) of a file changes. An ETag is an identifier that Azure assigns to each blob. Azure changes it when the content of the blob is updated, and also when you update its properties or metadata. As a result, Hevo re-ingests a file even if only its properties or metadata change.

Hevo identifies each file by its path. As a result, if you rename a file or move it to another folder, Hevo ingests it as a new file. The rows loaded from the file under its earlier name remain in the Destination.

Note: Azure limits the rate of requests that a storage account can handle. If the limit is exceeded, Azure rejects the requests, and a rate limit exception occurs. To understand how Hevo handles such scenarios, read Handling Rate Limit Exceptions.

Schema and Primary Keys

Hevo creates a table in the Destination for each set of files that a rule maps to a table. On the Configure Objects page, the tables are grouped by the folder path of their rule, which is displayed as the schema name. If the folder path of a rule is blank, the schema name is root.

The name of each table depends on the object handling rule:

Hevo replaces any character other than letters, digits, and underscores in a table name with an underscore (_). If a table name starts with a digit, Hevo adds an underscore at the beginning.

In addition to the columns in your files, Hevo adds the following columns to each table:

Hevo uses the __file and __line columns as the primary key of each table. For the tables created using the Files as Tables rule, Hevo uses only the __line column as the primary key.

Hevo infers the data type of each column from its values. For example, Hevo loads values such as 2026-10-01 as dates, whole numbers as integers, and other numbers as decimals. Values that do not match any other type, including numbers with leading zeros, such as 007, are loaded as strings. If a file contains a column that is not present in the Destination table, Hevo adds the column to the table.

Supported Compression Formats

Hevo identifies compressed files and archives from their file names, or from their content if the name does not indicate the format, and reads the files in them. Hevo supports the following compression formats:

Hevo also reads the files in .zip and .tar archives, including compressed tar archives, such as .tar.gz, .tgz, .taz, .tbz, .tbz2, .txz, and .tzst.

Note: Hevo supports only the framed forms of Snappy and LZ4.