Prerequisites

• The REST API endpoint, or the location from where data is to be ingested, is available.

• You have registered the app for Hevo in your REST API Source if you want to use OAuth 2.0 for authentication. Read your Source API documentation for the steps to do this.

  Note: It is recommended to add https://<your-hevo-region>/rest-api/oauth as the redirect URL while registering your app. Replace the placeholder value with your Hevo region.

• The credentials, such as a username and password or an API key, to authenticate Hevo’s connection with your REST API Source are available. You can obtain these from your Source administrator or generate them. Read your Source API documentation for more information.

• You have a basic understanding of JSONPath and JSONPath expressions.

• You are assigned the Team Administrator, Team Collaborator, or Pipeline Administrator role in Hevo, to create the Pipeline.

---

Configuring REST API as a Source

Perform the following steps to configure REST API as a Source in Hevo:

1. Click PIPELINES in the Navigation Bar.

2. Click + CREATE PIPELINE in the Pipelines List View.

3. In the Select Source Type page, select REST API.

4. In the Configure your REST API Source page:

   

   1. Specify a unique Pipeline Name, not exceeding 255 characters.

   2. Set up your REST API Source.

   3. Specify the data root, or the path, from where you want Hevo to replicate the data.

   4. Select the pagination method to read through the API response. Default selection: No Pagination.

5. Click FINISH SETUP and proceed to set up the Destination.

   

1. Set up the REST API Source

To allow Hevo to ingest data from your REST API Source in the Set-up Source section, you must:

1. Set up the connection to your REST API endpoint.

2. Specify the authentication method. Default selection: No Auth.

3. Provide the headers and query parameters.

4. Click TEST & CONTINUE to ensure that your connection is set up correctly. Next, proceed to set up the Data Root.

1.1 Set up the connection to your REST API Source endpoint

Perform the following steps to connect to your REST API endpoint:

1. Select the Method for making API requests:

   • GET (Default): The HTTP method for requesting data from an API endpoint.

   • POST: The HTTP method for sending data to an API endpoint.

2. Specify the complete URL of your REST API endpoint. For example, https://slack.com/api/users.list.

3. If you selected the POST method, specify the Request Body, which contains the data to be sent to your API endpoint. Else, skip to setting up the authentication method.

   The request body must be a valid JSON or Form Data in a key-value pair format. For example, the request body in the image below is in JSON format and contains the data for creating a PagerDuty incident.

   

1.2 Specify the authentication method

Select one of the following methods to authenticate Hevo and authorize it to ingest data from your REST API Source:

• No Auth (Default): This is the default selection. You can leave it unchanged if your REST API Source does not require any authentication or authenticates using credentials received in custom headers.

• Basic Auth: This method uses HTTP Basic Authentication, where authentication is done via a username and password. To set up Basic Auth, specify the following:

  

  • Basic Auth User: Your API username or API key.

  • Basic Auth Password: Your API password or API secret.

• OAuth 2.0: This method uses the Open Authorization protocol, where authentication takes place via an exchange of tokens.

  Note: Hevo supports only the Authorization Code grant type with the OAuth 2.0 authentication method. A grant type is used by an app to get the access token.

  To set up OAuth2.0, specify the following:

  

  • Redirect URL: The URL to which the Source API authorization server directs Hevo after authorizing the app you registered for Hevo. This URL must match the one you provided while registering your app. Default value: https://<your-hevo-region>/rest-api/oauth. For example, https://in.hevodata.com/rest-api/oauth.

  • Select Access Token: The access token used by Hevo to make API requests. Select an existing token for your Source from the drop-down list. For example, Asana, as seen in the image above.

    You can also create a token at this time. To do this, click + Create New Token, and in the Create New Token pop-up window, specify the following:

    

    • Token Name: A unique name for the access token.

    • Client ID: The ID assigned by your REST API Source to uniquely identify the app registered for Hevo.

    • Client Secret: The secret key associated with the client ID of your app registered for Hevo.

    • Auth URL: The URL of the Source API authorization server from where your app retrieves the authorization code.

    • Token URL: The URL of the Source API authentication server where your app exchanges the retrieved authorization code for an access token.

    • Scopes: A comma or space-delimited list of permissions that you want to grant Hevo for accessing your Source data. You can leave this field blank to assign all the scopes required by the Source API to the access token. Read your Source API documentation for information on the list format and the supported OAuth 2.0 scopes.

1.3 Specify the headers and query parameters

To provide any additional information required by your REST API Source, you can use:

• Headers: Headers are a part of the HTTP request and provide metadata about the request or the request body. These contain the additional information required by an API to complete the call and are given as key-value pairs.

  For example, specifying Authorization as the key and the token as the value for authorization or the email address of a user in the From key.

  

• Query Params: Query parameters are a part of the URL in the HTTP request and can be used to filter, sort, or paginate the API results. These are appended to the URL after a question mark ( ? ) and are usually given as key-value pairs. Hevo supports the following types of query parameters:

  • Text: A static query parameter.

    • Key: The name of the parameter.

    • Value: The value of the parameter.

    For example, consider the API endpoint https://api.thedogapi.com/v1/images/search that, by default, returns a single image. Now, if you want the API to return ten images, you can specify the key as limit and the value as 10.

    

  • Date: A dynamic Date type query parameter.

    • Key: The name of the parameter.

    • Format: The string value describing the pattern in which the date must be generated. Select a format from the drop-down list.

      Note: Hevo generates DateTime values according to Java’s SimpleDateFormat.

    • Offset Unit (Optional): The unit in which the DateTime value should be generated. Select a unit from the drop-down.

    • Offset Value (Optional): A numerical value for the selected unit. If you want to generate a date in the past, provide a negative offset value. A positive offset value generates future date times.

    The Offset Unit and Offset Value are used together to generate a DateTime value relative to the time at which the API call is made. For example, suppose you want to fetch the incidents that were reported only in the past 24 hours. For this, you need to generate a DateTime that is 24 hours ago. To do so, provide the Offset Unit as Hours and the Offset Value as -24.

    

2. Specify the data replication path

You can direct Hevo to the element in the API response from where you want to fetch data. For this, in the Data Root section, you must:

1. Specify the data root and check its validity.

2. Click CONTINUE and proceed to set up pagination.

Specify the Data Root

The Data Root is the JSONPath expression for the element in the API response from where you want to replicate data. Hevo suggests possible data roots based on the response received from the URL specified in the connection settings. You can select a suggested data root or provide one of your own. In the latter case, you must check the validity of your data root. To do this, click CHECK PARSED RESPONSE to view the parsed response.

By default, the Data Root field displays the first suggested data root and the parsed response for it in the Sample Response from API section. For example, the image below shows the first suggested data root, $.*, and the parsed response for it.

Note: The data root can contain only a JSONPath expression that returns an array or an object, and each element of the data root should return a JSON object. Read JSONPath Expression Examples for more information on writing valid expressions.

Sample Response from API: A view-only field that displays a sample of the response from the API. Hevo shows the following types of responses:

• Raw: This displays the API response in JSON format. Hevo suggests data roots from this response. You can also use this response to derive an alternate data root.

  For example, in the image below, the Hevo-suggested data roots are $.* and $.*.location.* and $.*.location.*.locations.* is the alternate data root.

  

• Parsed: This displays the API response parsed based on the data root specified above. If the API response cannot be parsed as per the selected data root, Hevo displays an error.

  For example, the image below displays the parsed response for the $.*.locations.*.locations.* data root.

  

3. Set up Pagination

You can set up pagination to enable Hevo to read through long API responses in smaller chunks. For this, in the Pagination section, select from one of the following pagination methods:

Pagination Type	Description
No Pagination (Default)	This option does not require any setup. Use this when your Source does not paginate data.
Session	Use this option when your Source paginates data using pointers or cursors.
Page Number	Use this option when your Source paginates data using page numbers.
Offset and Limit	Use this option when your Source paginates data using an offset and a limit.

For all pagination options except No Pagination, you must check the API response to ensure that it is as expected. If you are unsure of the pagination option supported by your Source API, refer to its documentation, or contact the API developer.

Note: If you set up pagination, Hevo stops calling the API when the current API response does not contain information for fetching the next set of records or returns zero results.

No Pagination

Hevo attempts to fetch the entire result set of the API in a single call. However, depending on the way pagination is implemented within the API, the complete response may be fetched as a single page or divided into multiple pages. In the latter case, you need to select a different pagination option to retrieve the entire result set.

For example, consider a REST API that fetches a list of 1000 books and their reviews in a single call. If you configure this API as a Source with no pagination, then, in each Pipeline run, the API fetches the details of all the 1000 books. Now, suppose the Source has the data of 3000 books. Since the API only gets 1000 records in a single call, with no pagination, Hevo fetches the same list of 1000 books in every call. You would need to select a different pagination option to fetch the remaining 2000 records.

Hevo imposes no limit on the payload size sent or received by a REST API Source configured with no pagination. However, it is recommended to use the pagination method supported by your Source to limit the response size, especially while testing the pagination method. Additionally, there may be restrictions imposed by the APIs, the HTTP clients, or the web servers hosting the API endpoints.

For example:

• A web server such as Apache allows a payload size of 2 GB.

• Google’s Photo APIs allow photos with a maximum file size of 200 MB.

Session-based Pagination

In session-based pagination, one of the parameters in the API response serves as a pointer to the next set of records. For example, in the image below, the parameter next_page.uri contains this pointer.

The pointer can be derived in one of the following ways:

• Specifying Query-based parameters: This is the default method whereby the next page is retrieved using query parameters. To set this up, specify the following:

  

  • Session Query Param: The name of the query parameter that Hevo appends to the URL for fetching subsequent pages from the second call onwards.

  • Session Response Field: The JSONPath expression for the field containing the pointer to the next set of records.

• Specifying URL-based parameters: In this method, the next page is retrieved from a URL. To set this up:

  1. Enable the Is the next page URL present in the response? option. Hevo extracts the URL to the next page from the specified parameter.

  2. In the Next URL Field specify the JSONPath expression to the field that contains the URL for the next set of records. You can find this in the API response.

     

• Specifying Field-based parameters: In this method, the next page is retrieved from a field in the last record of the current API response. To set this up:

  1. Enable the Extract from Last Record in Response option. Hevo extracts the pointer to the next page from a field in the last record.

  2. In the Session Query Param, specify the name of the query parameter that Hevo must append to the URL for fetching subsequent pages from the second call onwards.

  3. In the Session Response Field, specify the JSONPath expression of the field that contains the value from where Hevo must start fetching the next set of records.

  Note: When you select this option, the JSONPath expression to the Session Response Field must be relative to the object being fetched.

  For example, suppose the API response contains a field, id, used to identify each record. And, the after field in the last record holds the information for the next set of records. Then, to retrieve all the pages in the API response, you must specify the Session Query Param as after and the Session Response Field as id. With these settings, when Hevo fetches the after field, it uses the value in the id field and picks records for the next page starting from that value.

  

Page number-based Pagination

Page number-based pagination works through a page number, either provided by you or extracted from the API response. To set this up, specify the following:

• Page Number Query Param: The name of the query parameter that Hevo appends to the URL to specify the page number.

• Initial Page Number: The value of the page number in the API response from which Hevo must fetch records. It increments the page number for each call until the API stops returning results. Default value: 1.

• Extract Page Number from Response: If enabled, Hevo extracts the page number from the response of the API call as per the settings you provide.

  

  • Page Response Field: The JSONPath expression to the field that contains the next page number.

  • Increment for next page: If enabled, Hevo increments the page number value when it makes the subsequent API call.

Offset and Limit-based Pagination

Offset-based pagination uses a numerical value to determine the starting point for fetching data. This value defines the number of records to be skipped each time from the API response. To set this up, specify the following:

• Offset Query Param: The name of the query parameter that specifies the offset. Hevo appends this parameter to the URL. You can find this name in the API response.

• Offset Starting Value: The value for the offset parameter from which the API starts fetching data.

• Offset Increment Value: The number of records that the API must skip before it starts to fetch data from the second call onwards.

• Limit Query Param: The name of the query parameter that Hevo appends to the URL to limit the number of records that the API fetches per page.

• Limit Initial Value: The number of records fetched by the API on each call.

For example:

Suppose an API result set contains 100 records. If you want to generate a sample data set that contains the first five records out of every 20 records, set the parameters as follows:

• Offset Query Param: offset

• Offset Starting Value: 0

• Offset Increment Value: 15

• Limit Query Param: limit

• Limit Initial Value: 5

With this configuration, the API:

1. Starts from an offset of 0, indicating the first record.

2. It fetches the first five (0-4) records.

3. Next, it skips 15 records (5-19). The offset is now 20.

4. Again, it fetches five records (20-24) and skips the next 15 (offset is now 40), and so on.

5. It repeats this pattern on every call to fetch five records every 20 until it receives zero results.

Test the pagination method

Perform the steps in this section if you set up a pagination method other than No Pagination.

To verify the pagination settings and check the API response:

1. In the Pagination section, specify the required parameters to set up the selected pagination method. For example, consider the following settings:

   

   • Pagination type: Page Number

   • Page Number Query Param: page

   • Initial Page Number: 1

2. Click TEST PAGINATION to see the API response paginated as per the settings you specified. Click NEXT PAGE to retrieve the page(s) in sequence until you reach the last page.

   

---