Modifying the Pipeline Configuration

Perform the following steps to change your Edge Pipeline’s settings:

1. On your <Pipeline Name> editing page, update the settings in the desired sections.

2. Click NEXT. This action takes you to the Object Configuration tab.

3. In the Object Configuration tab, click NEXT.

4. On the Summary page, scroll to the Pipeline Configuration section and verify all the changed settings in the NEW VALUE column.

5. If applicable, click the acknowledgment check box at the bottom of the page.

6. Click SAVE & EXIT to save the changed configuration.

Note: If you click CANCEL at any point before saving, the changes made to your Pipeline and or Object configuration are not retained.

Hevo applies the updated configuration to your Pipeline in the next sync schedule.

Update the Pipeline settings

You can change the values of any of the fields in the following sections:

Note:

• The sections that cannot be altered are grayed out.

• If you click CANCEL at any point, the changes made to the Pipeline and or Object configuration are not retained.

Pipeline Setup

In this section, you can modify the:

• Pipeline Name: Renaming the Pipeline does not affect your data. The modified name should not exceed 255 characters.

Select a Load Mode

In this section, select the desired Load Mode. You can change the mode in which your Pipeline loads data into the Destination tables, from Append to Merge, and vice versa.

Note: The change in load mode applies only to Source objects that are using the load mode selected while configuring your Pipeline. If you have individually changed the load mode for any objects, changing the load mode at the Pipeline level does not affect them.

When you change your load mode from:

• Append to Merge: Hevo adds new records only if it does not find a match in the existing data; otherwise, it updates the matched records. When you switch to the Merge mode, Hevo deduplicates your data using primary keys. Hence, you need to ensure that primary keys are defined for all the selected objects in your Pipeline. If any of the selected objects do not have a primary key, you can define one for them in the Object Configuration screen.

  Once you have saved your configuration changes to ensure data integrity between your Source and Destination data, Hevo performs the following actions:

  • Delete all existing data from your Destination table.

  • Re-ingest data from your Source table and load it into the Destination table.

• Merge to Append: Hevo appends new records to the end of the dataset in your Destination table. As a result, you may see duplicates in your Destination table.

Failure Handling Policy

In this section, select the desired option from the Object Level Settings drop-down. Based on the current selection, you can change the policy to one of the following:

• Don’t load data for Objects with failed Events: In this mode, Hevo loads data from only those Source objects that were successfully processed by a job.

• Load data for Objects with less than 10K failed Events: In this mode, data from the Source objects processed by a job is not loaded to the Destination if the number of failed Events across all objects exceeds 10K.

  Note: Selecting this option may lead to a mismatch between the Source and Destination data.

Schema Evolution Policy

In this section, select the desired Schema Evolution Policy to propagate or block all Source schema changes to your Destination schema.

When you change your Schema Evolution Policy to:

• Allow all changes: Any changes made to the Source schema, columns, and tables are synced with the Destination schema.

• Block all changes: Any updates made in the Source schema, columns, and tables are not synced with the Destination. For example, if a schema is added to the Source after the policy is changed, Hevo does not automatically add the schema to your Destination.

• Allow column-level changes only: Only the changes made to the columns of the selected Source object(s) are synced with the Destination. For example, if a column is added after the policy is changed, Hevo automatically syncs this change with the Destination.

Sync Frequency

In this section, select the desired data sync mode. You can switch the Pipeline sync frequency between Scheduled Run and Sync On Demand.

Note:

• Changing the sync frequency to Sync On Demand removes all existing schedules.

• The sync frequency can only be changed at the Pipeline level, not for individual objects.

When you change the sync frequency to:

• Scheduled Run: The Pipeline syncs data on a fixed schedule. In this mode, set the Frequency Type to one of the following:

  • Every n minutes: Choose one of the values provided by Hevo. For example, 30 minutes.

  • Hourly: Specify a value between 1 and 24. For example, 2.

    For Pipelines set to sync data every 24 hours, if the Start Time field is displayed, you can set it to a value between 00:00 and 23:59.

    Note: The start time must be in UTC, converted from your local time zone. For example, 07:00 (UTC) to specify 00:00 (MST).

  The new frequency is applied immediately, and the next sync job runs at that interval in your current time zone. For example, if you change the sync frequency from one hour to two hours and your last sync job ended at 11:00, the next sync job runs at 13:00 instead of 12:00. For Pipelines that sync data every 24 hours at a fixed time, the sync job runs at that time in UTC.

• Sync On Demand: The Pipeline syncs data on demand and can be triggered using Hevo APIs, an external scheduler, such as Airflow, or the Sync Now option. For log-based Pipelines, the sync frequency must be higher than the database log retention period. This ensures that Hevo processes the transaction logs before they are purged by the Source database.

Pipeline Latency

In this section, you can enable or disable the Send alerts for Pipeline Latency option to receive notifications for latency in your Pipeline. When enabled, in the Pipeline Latency (in minutes) field, specify a value that is greater than or equal to your Pipeline’s sync frequency.