SharePoint (Edge) - Hevo Data

SharePoint (Edge) Setup Guide

SharePoint is a Microsoft 365 service that organizations use to store, organize, and share files in document libraries within SharePoint sites. Each site belongs to your organization’s Microsoft 365 account, called a tenant.

Hevo uses the Microsoft Graph API to replicate data from CSV and TSV files in a SharePoint document library into the Destination of your choice. You define rules that determine the folders to read files from, the format of those files, and how the files are mapped to tables in your Destination.

Hevo supports the following access types to connect to SharePoint:

OAuth: Uses your Microsoft work account to connect. You sign in to your account and authorize Hevo’s application to read your files. The Pipeline can then read all the sites and files that your account can access.
Microsoft Entra ID App: Uses an application that you register in Microsoft Entra ID to connect, with a client secret or a client certificate. Only the sites that you allow the application to access can be read, and the connection does not depend on any individual user’s account.

Supported Features

Feature Name	Supported
Capture deletes	No
Data blocking (skip objects and fields)	Yes
Resync (objects and Pipelines)	Yes
API configurable	No
File encoding	UTF-8

Getting Started

Setup

Data Sync

Prerequisites

An active Microsoft 365 account with SharePoint Online exists, and a document library in it contains the CSV or TSV files from which data is to be ingested.
If you are connecting to SharePoint using:
- OAuth:
  - The Microsoft work account that you use to authorize Hevo can access the files from which you want to ingest data.
  - An administrator has approved Hevo’s application for your organization, or has allowed users to request this approval.
- Microsoft Entra ID App:
  - You are assigned the Global Administrator or Privileged Role Administrator role in Microsoft Entra ID to register an application and grant admin consent for its permissions.
  - If you assign the Sites.Selected permission to the application, a user with the SharePoint Administrator or Global Administrator role is available to grant the application access to your SharePoint site.
  - The application credentials are available to provide Hevo access to your SharePoint data.

Set up a Microsoft Entra ID App

To connect to SharePoint using a Microsoft Entra ID App, you need to do the following:

Register an application in Microsoft Entra ID and assign it the permission to read your SharePoint sites.
Generate the application credentials, a client secret or a client certificate, that Hevo uses to authenticate as the application.
Grant the application access to your SharePoint site, if you assign the Sites.Selected permission to it.

1. Register an Application in Microsoft Entra ID

Microsoft Entra ID is Microsoft’s cloud-based identity and access management service. Register an application in it that Hevo uses to connect to your Microsoft 365 tenant. You can use one application for all your Pipelines. To do this, perform the following steps:

Log in to the Azure portal.
In the top search bar, search for and select Microsoft Entra ID.
In the left navigation pane, under Manage, click App registrations.
On the App registrations page, click + New registration.
On the Register an application page, do the following:
1. Specify the following:
  - Name: A unique name for your application. For example, Hevo-SharePoint-App.
  - Supported account types: Select Single tenant only - <your organization>. This is the default option.
2. Click Register.
On the Overview page of your application, in the Essentials section, copy the Application (client) ID and Directory (tenant) ID, and save them securely like any other password.

Use these values while configuring your SharePoint Source in the Hevo Pipeline.

After registering the application, assign one of the following Microsoft Graph application permissions to it to allow it to read files from your SharePoint sites:

Sites.Selected: Allows the application to access only the sites that you explicitly grant it access to. If you assign this permission, you must also grant the application access to your SharePoint site.
Sites.Read.All: Allows the application to read files from all the sites in your organization.

Perform the following steps to assign the permission:

In the left navigation pane of your application page, under Manage, click API permissions.
On the API permissions page, in the Configured permissions section, click + Add a permission.
On the Request API permissions slide-in page, click Microsoft Graph.
On the Request API permissions slide-in page, do the following:
1. Click Application permissions.
2. Under Select permissions, search for the permission that you want to assign.
3. Expand Sites, and select the Sites.Selected or Sites.Read.All check box.
4. Click Add permissions.
On the API permissions page, click Grant admin consent for <your organization>, and then click Yes to confirm.

You have successfully registered an application and assigned the required permission to it. Proceed to generate the application credentials.

2. Generate the Application Credentials

To connect through your application, Hevo uses a client secret or a client certificate. Generate one of these based on your organization’s security policy.

Generate a client secret

A client secret is a confidential string that your application uses to prove its identity to Microsoft Entra ID. You generate it in your application and provide its value to Hevo. You can either use an existing client secret or create one if required.

Perform the following steps to generate a client secret for your application:

Log in to the Azure portal.
In the left navigation pane of your application page, under Manage, click Certificates & secrets.
In the Client secrets tab, click + New client secret.
On the Add a client secret slide-in page, do the following:
1. Specify the following:
  - Description: A suitable text to describe the purpose of the client secret.
  - Expires: The period after which the client secret expires. Default value: Recommended: 180 days (6 months).
2. Click Add.
In the Client secrets tab, click the Copy to clipboard icon next to the secret’s Value, and save it securely like any other password. Use this value as the client secret while configuring your Pipeline.

Note: Microsoft displays the secret value only once. If you leave the page without copying it, you must create a new client secret.

You have successfully generated a client secret. If you assigned the Sites.Selected permission to your application, proceed to grant the application access to your SharePoint site. Otherwise, proceed to obtain the library URL.

Note: If the client secret configured in the Pipeline expires or is deleted from your application, Hevo cannot authenticate with your Microsoft 365 tenant. As a result, no data is ingested. To resume data replication, create a client secret and modify the Source configuration in the Pipeline with the new value. Once the updated secret is saved, Hevo re-authenticates the Source, and data ingestion resumes.

Generate a client certificate

A client certificate consists of a public certificate that you upload to your application in Microsoft Entra ID and a private key that you provide to Hevo. You can use an existing certificate or create a self-signed one.

Perform the following steps to create a certificate and upload it to your application:

Run the following command in a terminal to create a self-signed certificate and a private key that are valid for 365 days:
```
openssl req -x509 -newkey rsa:2048 -keyout private_key.pem -out certificate.pem -days 365 -nodes -subj "/CN=Hevo SharePoint"
```
This command creates the following files:
- certificate.pem: The public certificate, which you upload to your application in Microsoft Entra ID. It identifies your application and can be shared.
- private_key.pem: The private key that belongs to the certificate, which you provide to Hevo. Hevo uses it to prove that it holds the certificate. Keep this file confidential.
  
  Note: To protect the private key with a passphrase, remove -nodes from the command. You are prompted to specify the passphrase.
Log in to the Azure portal.
In the left navigation pane of your application page, under Manage, click Certificates & secrets.
In the Certificates tab, click Upload certificate.
On the Upload certificate slide-in page, do the following:
1. Select the certificate.pem file and specify a Description for it.
2. Click Add.

Save the private_key.pem file in a secure location. Provide it, along with the certificate.pem file and the passphrase, if you specified one, while configuring your SharePoint Source in the Hevo Pipeline.

You have successfully generated a client certificate. If you assigned the Sites.Selected permission to your application, proceed to grant the application access to your SharePoint site. Otherwise, proceed to obtain the library URL.

3. Grant the Application Access to a SharePoint Site

Note: Perform the steps in this section only if you assigned the Sites.Selected permission to your application.

Grant your application read access to the SharePoint site that contains your files. You must do this once for each site from which you want to ingest data.

Perform the following steps in Microsoft Graph Explorer using an account with the SharePoint Administrator or Global Administrator role:

Log in to Microsoft Graph Explorer.
In the top-right corner of the page, click your profile icon, and then click Consent to permissions.
On the Permissions slide-in page, do the following:
1. Search for Sites.FullControl.All.
2. Click the expand icon next to Sites.
3. In the Status column, click Consent for the Sites.FullControl.All permission.
Note: If Unconsent is displayed instead of Consent, the permission is already granted. Skip to step 5.
In the pop-up window, review the permissions requested, and then click Accept. The pop-up window closes, and the status of the permission changes to Unconsent.
Close the Permissions slide-in page.
On the Graph Explorer page, perform the following steps to obtain the site ID:
1. Select GET from the drop-down, and in the URL field, specify the following URL:
```
https://graph.microsoft.com/v1.0/sites/<tenant_name>.sharepoint.com:/sites/<site_name>
```
  Note: Replace the placeholder values in the URL above with your own. For example, <tenant_name> with the name in your SharePoint URL, such as contoso, and <site_name> with the name of your SharePoint site in the URL.
2. Click Run query.
3. In the Response preview section, copy the value of the id field. The site ID is in the format <tenant_name>.sharepoint.com,<id>,<id>.
On the Graph Explorer page, perform the following steps to grant your application read access to the site:
1. Select POST from the drop-down, and in the URL field, specify the following URL:
```
https://graph.microsoft.com/v1.0/sites/<site_id>/permissions
```
2. In the Request body tab, paste the following JSON:
```
{
  "roles": ["read"],
  "grantedToIdentities": [
    {
      "application": {
        "id": "<client_id>",
        "displayName": "<application_name>"
      }
    }
  ]
}
```
  Note: Replace the placeholder values in the URL and JSON above with your own. For example, <site_id> with the site ID that you copied in step 6, <client_id> with the application (client) ID that you obtained while registering your application, and <application_name> with the name of that application.
3. Click Run query.

A response status of 201 Created indicates that your application can now read the files in the site. You have successfully granted the application access to your SharePoint site. Proceed to obtain the library URL.

Obtain the Library URL

Hevo reads files from the SharePoint document library, or a folder within it, that the library URL points to. Perform the following steps to obtain this URL:

Log in to your SharePoint site, and click Documents or the document library that contains your files.
(Optional) To read files from a specific folder instead of the entire library, open that folder.
Copy the URL from your browser’s address bar, and save it securely. For example, https://contoso.sharepoint.com/sites/Finance/Shared%20Documents/Forms/AllItems.aspx.

Note: You must specify the full URL. Specifying only a path, such as Documents/Reports, or the URL of the site instead of a library causes the connection test to fail.

Configure SharePoint as a Source in your Pipeline

Perform the following steps to configure your SharePoint Source:

Click Pipelines in the Navigation Bar.
Click + Create Pipeline in the Pipelines List View.
On the Select Source Type page, select SharePoint.
On the Select Destination Type page, select the type of Destination you want to use.
On the Configure Source page, specify the following:
- Source Name: A unique name for your Source, not exceeding 255 characters. For example, SharePoint Source.
- In the Connect to your SharePoint site section:
  - Authentication Method: The method Hevo uses to connect to SharePoint. Select one of the following:
    - OAuth: Uses your Microsoft work account to connect to SharePoint. Hevo reads data using the permissions of the account you sign in with. OAuth supports only Microsoft work or school accounts. You cannot authorize Hevo using a personal Microsoft account. Click Authenticate Your Microsoft Account and perform the following steps to allow Hevo to read your files:
      1. If you are not already signed in to your Microsoft work account, sign in when prompted. Otherwise, proceed to step 2.
        
        If the sign-in page displays a Need admin approval message, contact your administrator. Microsoft requires an administrator to approve Hevo’s application once for your organization. Your administrator can also enable the admin consent workflow, so that users can request this approval from the sign-in page.
      2. If prompted, review the permissions requested, and then click Accept to authorize Hevo to read your files. Hevo requests the following delegated Microsoft Graph permissions to read your files and identify your account:
        
        Files.Read
        
        Sites.Read.All
        
        User.Read
        
        offline_access
        
        openid
        
        profile
        
        email
        
        If you are a Global Administrator, select the Consent on behalf of your organization check box before you click Accept. Users in your organization can then authorize Hevo without administrator approval.
      You are redirected to the Configure Source page.
    - Microsoft Entra ID App: Uses the application that you registered in Microsoft Entra ID to connect to SharePoint. Hevo can read only the sites that the application is allowed to access. Select one of the following, based on the credential that you generated:
      - Microsoft Entra ID App (Client Secret): Connects using the client secret that you generated for your application.
        
        Directory (Tenant) ID: The directory (tenant) ID that you obtained while registering your application. You can also specify a verified domain of your tenant, such as contoso.onmicrosoft.com.
        
        Application (Client) ID: The application (client) ID that you obtained while registering your application.
        
        Client Secret: The client secret value that you generated for your application.
      - Microsoft Entra ID App (Client Certificate): Connects using the certificate that you uploaded to your application and its private key.
        
        Directory (Tenant) ID: The directory (tenant) ID that you obtained while registering your application. You can also specify a verified domain of your tenant, such as contoso.onmicrosoft.com.
        
        Application (Client) ID: The application (client) ID that you obtained while registering your application.
        
        Client Certificate: The certificate file that you uploaded to your application, in the .pem, .crt, or .cer format. You can also upload a .pem file that contains both the certificate and its private key.
        
        Private Key (Optional): The file containing the private key of the certificate, such as private_key.pem. Leave this field blank if your certificate file also contains the private key.
        
        Private Key Passphrase (Optional): The passphrase for the private key, if you specified one while creating it.
  - Library URL: The URL of the document library or folder that you obtained from your SharePoint site. Hevo reads files from the location that this URL points to. For example, https://contoso.sharepoint.com/sites/Finance/Shared%20Documents/Forms/AllItems.aspx.
    
    Note: You cannot change the library URL after creating the Pipeline.
In the Files to Sync section, define rules that determine the files Hevo ingests and the tables it loads them into. Hevo checks each file against the rules from top to bottom and ingests it using the first rule that matches. Therefore, add a rule for specific files above a rule that reads their entire folder. For example, to load the daily order files in the orders folder into one table and all other CSV files in that folder into another, add the Pattern as Table rule for the daily files first, and then the Directory as Tables rule for the folder. You can add up to 20 rules.

To add a rule, perform the following steps:
1. Click Add Rule.
2. Specify the following:
  - Folder Path: The path of the folder, relative to the library URL, from which you want to ingest files. For example, orders/. If you leave this field blank, Hevo reads files from the location that the library URL points to.
  - Include Nested Folders: If enabled, Hevo also reads files from all the subfolders of the specified folder. If disabled, Hevo reads only the files present directly in it. For example, if the folder path is orders/ and this option is enabled, Hevo also reads files in orders/archive/. Default value: Disabled.
  - File Format: The format of the files that you want to ingest. Select one of the following:
    - CSV: Files with the .csv extension.
    - TSV: Files with the .tsv or .tab extension.
    Hevo identifies the format of a file from its extension. Files in any other format in the folder are not ingested by this rule. Default value: CSV.
  - Object Handling Rule: The way Hevo maps the files to Destination tables. Select one of the following:
    - Directory as Tables: Hevo loads all the files from the folder into a single table. This is the default option.
    - Files as Tables: Hevo loads each file into a separate table named after the file. Hevo can create up to 50,000 tables for this rule. If the folder contains more files, the remaining files are not ingested.
    - Pattern as Table: Hevo loads the files whose names match a regular expression into a single table. If you select this option, specify the File Name Pattern. This is a regular expression that must match the entire file name, excluding the folder path. The match is case-sensitive. For example, orders_\d{8}\.csv matches orders_20261001.csv.
  - Table Name (Optional): The name of the Destination table. If you leave this field blank, Hevo names the table {dir}_{type}, where {dir} is the name of the folder and {type} is the file format. For example, orders_csv.
    
    You can use the placeholders {dir}, {type}, and $1 to $9 for the groups captured by the file-name pattern. For example, with the file-name pattern sales_(\w+)_\d{8}\.csv and the table name sales_$1, Hevo loads sales_east_20261001.csv into the sales_east table. This field is not displayed if you select Files as Tables.
  - Read Files with No Extension: If enabled, Hevo ingests files that do not have an extension, such as orders_export instead of orders_export.csv, using the selected file format. If disabled, Hevo skips such files. Default value: Disabled. This option is displayed only if you select Directory as Tables.
  - Header: Select First line is the header if the first line of each file contains the column names. Otherwise, select Headerless: Name the columns column_1, column_2, and so on, and Hevo names the columns column_1, column_2, and so on. Default value: First line is the header.
  - Delimiter: The character that separates the values in a row. For example, enter | for pipe-delimited files. To specify a tab, enter \t. Default value: A comma (,) for CSV files and a tab for TSV files.
  - Quote Character: The character that encloses a value. For example, enter ' if the values in your files are enclosed in single quotes. Default value: A double quote (“).
  - Skip Lines at the Top: The number of lines to skip at the beginning of each file. Hevo reads the header, if any, from the line after these lines. For example, enter 2 if each file starts with a two-line report title above the header. Default value: 0.
  - Skip Lines at the Bottom: The number of lines to skip at the end of each file. For example, enter 1 if each file ends with a totals row. Default value: 0.
  - Escape Character: The character used to escape a quote character within a value. If you leave this field blank, a quote within a value is escaped by doubling it. For example, the value 5” monitor is written as "5"" monitor" in the file.
  - Null Sequence: The value that Hevo must load as NULL wherever it appears, such as \N. If you leave this field blank, Hevo loads empty values as NULL.
  - Line Separator: The character that marks the end of a line. If you leave this field blank, Hevo uses a newline, and handles carriage return and line feed (CRLF) line endings automatically.
  - Values Are Never Quoted: Enable this option only if the values in your files are never enclosed in quote characters. If enabled, a value that contains the delimiter is split into separate columns. For example, with a comma delimiter, the value New York, NY is loaded as two values, New York and NY. Default value: Disabled.
  - Skip Rows after the Header: The number of rows between the header and the first data row, such as a row containing units. Default value: 0.
3. Click Save.
Repeat the steps above to add more rules.
In the Additional Settings section:
- Updated File Handling: The way Hevo loads the rows of a file that is updated after it is ingested. This also determines the primary key of the Destination table. Select one of the following:
  - Update Rows by Line Number: Hevo replaces the rows previously loaded from the file with its current rows, using the file path and line number of each row as the primary key. This is the default option.
  - Keep Every Version: Hevo loads the rows of each version of the file into the Destination table and keeps the rows loaded earlier.
- Compression: The way Hevo reads compressed files. Select one of the following:
  - Auto-detect: Hevo identifies compressed files and archives from their file names, or from their content if the name does not indicate the compression type. Hevo then reads the files from them. This is the default option. Read Supported Compression Formats for the formats that Hevo supports.
  - None: Hevo reads every file as it is stored, without decompressing it.
Click Test & Continue to test the connection to your SharePoint Source. Hevo verifies that it can access the library and that your rules match at least one file in it. Once the test is successful, you can proceed to set up your Destination.

Data Replication

Hevo ingests the following types of data from the files that match your rules:

Historical Data: The first run of the Pipeline ingests all the files that match your rules and loads them into the Destination.
Incremental Data: Once the historical load is complete, Hevo checks the library for new and updated files as per the sync frequency and ingests them. As it is not possible to identify individual changes within a file, Hevo re-ingests the entire content of each updated file. It then replaces the rows previously loaded from that file in the Destination with the file’s current rows.

Hevo detects an update when the content of a file changes. Changes made only to the properties of a file in SharePoint, such as its metadata columns or check-out status, do not cause Hevo to re-ingest it.

Hevo identifies each file by its path. As a result, if you rename a file or move it to another folder, Hevo ingests it as a new file. The rows loaded from the file under its earlier name remain in the Destination.

Note: Microsoft limits the number of requests that an application can make to SharePoint in a given period. This limit is shared with all the other applications in your tenant. If the limit is exceeded, a rate limit exception occurs. To understand how Hevo handles such scenarios, read Handling Rate Limit Exceptions.

Schema and Primary Keys

Hevo creates a table in the Destination for each set of files that a rule maps to a table. On the Configure Objects page, the tables are grouped by the folder path of their rule, which is displayed as the schema name. If the folder path of a rule is blank, the schema name is root.

The name of each table depends on the object handling rule:

Object Handling Rule	Table Name
Directory as Tables	The table name that you specify in the rule. If you do not specify one, Hevo names the table {dir}_{type}. For example, orders_csv for CSV files in the orders folder. If the folder path is blank, {dir} is root.
Files as Tables	The name of the file, without its extension. For example, region_summary for the file region_summary.tsv.
Pattern as Table	The table name that you specify in the rule. If you do not specify one, Hevo names the table {dir}_{type}.

Hevo replaces any character other than letters, digits, and underscores in a table name with an underscore (_). If a table name starts with a digit, Hevo adds an underscore at the beginning.

In addition to the columns in your files, Hevo adds the following columns to each table:

Column	Description
__file	The path of the file from which the row was ingested, relative to the library URL. For example, orders/orders_20261001.csv. This column is not added to the tables created using the Files as Tables rule.
__line	The position of the row in the file, starting from 0 for the first data row. The header row is not counted.
__modified	The date and time when the file was last modified in SharePoint.

Hevo uses the __file and __line columns as the primary key of each table. For the tables created using the Files as Tables rule, Hevo uses only the __line column as the primary key.

Hevo infers the data type of each column from its values. For example, Hevo loads values such as 2026-10-01 as dates, whole numbers as integers, and other numbers as decimals. Values that do not match any other type, including numbers with leading zeros, such as 007, are loaded as strings. If a file contains a column that is not present in the Destination table, Hevo adds the column to the table.

Supported Compression Formats

Hevo identifies compressed files and archives from their file names, or from their content if the name does not indicate the format, and reads the files in them. Hevo supports the following compression formats:

Compression Format	File Extensions
gzip	.gz, .gzip
bzip2	.bz2, .bzip2
Zstandard	.zst, .zstd
zlib (deflate)	.deflate, .zz, .zlib
Unix compress	.Z
Snappy (framed)	.sz, .snappy
LZ4 (framed)	.lz4
xz	.xz

Hevo also reads the files in .zip and .tar archives, including compressed tar archives, such as .tar.gz, .tgz, .taz, .tbz, .tbz2, .txz, and .tzst.

Note: Hevo supports only the framed forms of Snappy and LZ4.

Source Considerations

If a row in a file contains more values than the file’s header, Hevo adds columns to the Destination table for the additional values. If a row contains fewer values than the header, Hevo loads NULL in the remaining columns of the Destination table.
If the first line of a file shares no column name with its Destination table, Hevo loads that line as a data row. This allows Hevo to ingest all the rows from exports split into multiple files, such as Spark or MySQL exports, where only the first file contains the header.
Hevo ingests only the files stored in your document library and skips the following items, as they are not files:
- Shortcuts, which point to folders stored in another library or site. To ingest files from such a folder, create a Pipeline with the URL of the library that contains it.
- OneNote notebooks, which SharePoint stores as notebook packages instead of individual files.

Limitations

Hevo currently supports only SharePoint Online in the Microsoft global cloud. SharePoint Server, which is the on-premises version of SharePoint, and Microsoft 365 national clouds, such as Microsoft 365 GCC High and Microsoft 365 operated by 21Vianet, are not supported.
Hevo supports only Unicode Transformation Format, 8-bit (UTF-8) encoded files. If your files use another encoding, any characters that are not valid in UTF-8 are loaded as the replacement character (�).
Hevo ingests data only from CSV and TSV files stored in document libraries. Data from SharePoint lists and files in any other format is not ingested.
Hevo does not capture deletes. If you delete a file from SharePoint, the rows loaded from it remain in the Destination.

Last updated on Oct 16, 2026