Why is there a delay in my Pipeline?

Hevo is built on real-time data ingestion architecture. This means that Hevo ingests data in real-time and writes to the Destination as soon as possible. Hevo’s architecture allows it to scale horizontally whenever it detects a higher volume of Events being ingested through the Pipelines. Still, there are situations where you might see a delay in your Pipelines, depending on your Pipeline configuration such as:

• Ingestion and Loading Frequencies of your Pipeline, which affect how often the data is replicated from your Source to the Destination. You should configure a high ingestion and loading frequency if you need your Source data to be updated quickly in the Destination. However, based on the Destination, increasing the load frequency may increase the cost of your load queries. Read Ingestion and Loading Frequency. Hevo’s ingestion speed is influenced by:

  • API rate limits imposed by the Source.

  • Network throughput affects how fast data can be transferred.

  • Source throughput determines how quickly data is made available for ingestion.

  • Large Source datasets or frequent updates can introduce delays, as the ingestion process may not be able to keep up with the volume or frequency of data updates.

• Applying complex Transformations on your Source data, which can cause delays in loading the data to the Destination.

• Replay of a large number of failed Events in one or more of your Pipelines. Replayed Events are fed back to the Pipelines and share the same resources that are used by the Pipeline itself. In this scenario, you can stop replaying the Events in the Pipeline from the Pipeline Overview page.

  Note: Pipelines across different accounts in Hevo do not affect each other.

• Destination performance and limitations.

  • In cases of Destinations, such as MySQL and PostgreSQL, where data is not staged before loading, the Pipeline may experience delays. This happens because Hevo is not able to write data into the Destination as fast as it is ingesting it from the Source. If you think the slowness in the Destination is temporary, you may wait until it gets resolved. Otherwise, you should upgrade the hardware configuration of the Destination to allow it to accept a higher rate of writes.

  • Some Destinations, such as MySQL, are not ideal for handling large query traffic and concurrent transactions. For data warehouses, Hevo writes data at a default loading frequency optimized to reduce synchronization costs.

  • Certain Destinations, such as Amazon Redshift, Google BigQuery, Snowflake, and S3, rely on batch-based loading rather than individual record inserts. Batching improves efficiency and reduces costs by limiting frequent table scans and deduplication processes. However, this introduces a 5-15 minute delay before data appears in the Destination.

• Some Sources impose query execution limits or timeouts to prevent excessive resource consumption. If a query takes too long to execute, the Source may automatically cancel it, preventing Hevo from retrieving the data. Frequent query cancellations may disrupt data ingestion, leading to delays.

• A Pipeline consists of multiple tasks that handle data ingestion, transformation, and loading. If any of these tasks fail, it can cause a delay. A task may fail due to any of the following reasons:

  • Schema changes in the Source: If a table structure changes, such as a column is added, renamed, or removed, and Auto Mapping is disabled, the Pipeline may fail until the schema is updated in Hevo.

  • Authentication or connectivity issues: If Hevo loses access to the Source or Destination due to expired credentials, incorrect permissions, or network disruptions, it may fail to process data.

Note: If you are unable to determine the exact cause of the delay, contact Hevo Support for further assistance.