Why is the historical data not getting ingested?

Hevo ingests historical data from the Source objects based on the default historical sync duration or the duration that you select during Pipeline creation. This behavior also applies to objects that you select post-Pipeline creation.

However, sometimes the Pipeline may not ingest historical data for various reasons, some of which are described below:

The API rate limit is exceeded

In the case of API-based Sources, the historical data ingestion may sometimes halt if the API rate limit for fetching data from the Source is exceeded. The rate limit applies to all the Pipelines created with that Source for an account, and the issue is mostly observed in Sources with low API rate limits. This issue automatically resolves based on the rate limits set by the Source, which could be per minute, hour, or 24 hours. You can try reducing the ingestion frequency to reduce the probability of such issues. For example, if the Pipeline ingests data every 15 minutes, the API is used every 15 minutes and reaches its limit sooner than if the ingestion were to happen every 3 hours.

No data in the included Source objects

In the case of Pipelines created with SaaS Sources, the historical data may not be ingested if the objects do not contain data for the configured historical sync duration. To ingest data for a period prior to that, use the Change Position action and shift the offset to the desired date. Doing so resets the incremental data ingestion task to that position, thus fetching data for the desired period.

In the case of Pipelines created with database Sources, if the objects that you select while configuring the Pipeline do not contain any data, no Event Types are created for them and no data is ingested. This behavior also applies to objects that you select post-Pipeline creation. However, for Source objects created or restored after the Pipeline is active, Hevo ingests all data as incremental data.

Insufficient permissions to access data

The historical data, and also the incremental data, may not be ingested if the configured user and/or the API token does not have the permission to read data from the selected objects. You need to change the permissions granted to the user and/or the API token and reconfigure the Pipeline or create a new one. Refer to your Source documentation page for information on how to grant the necessary permissions.