Configure the Pipeline Settings

You can specify your Pipeline settings after you have set up your Destination. To do this, perform the following steps on the Configure Pipeline page:

1. In the Pipeline Setup section, specify the following:

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Destination Prefix: A string value, not exceeding 32 characters, that you want to add at the beginning of your Destination table names. The string can have alphanumeric characters and underscores, but it cannot start with an underscore. Also, as Hevo adds a trailing underscore, you do not need to provide one at the end of the specified prefix string. For example, pp_1 but not _pp_1 or _pp_1_.

     Note: Hevo creates the tables following the conventions defined by the Destination for naming identifiers. Read your Destination documentation for the rules.

2. In the Select a Load Mode section, specify one of the following:

   • Merge: In this mode, existing data records are updated based on the matching criteria, such as primary keys, and new records are added to your Destination table if they do not match the criteria.

     Note: You must define a primary key for the Source object if it does not have one.

   • Append (Recommended): In this mode, new data records are added at the end of your Destination table without modifying or deleting any existing data. This is the default selection.

     Note: This option may result in duplicate data in your Destination tables.

   If required, you can change the load mode for any Source object from the Configure Objects page later.

3. In the Failure Handling Policy section, specify the policy for handling object-level failures:

   • Moderate: In this mode, data from all successfully processed objects in the job is ingested and loaded into your Destination tables. Select an action from the drop-down to be performed in case of failures:

     • Don’t load data for Objects with failed Events: In this case, Hevo loads data from only those objects in a job that are processed successfully.

     • Load data for Objects with less than 10K failed Events: In this case, Hevo marks the entire job as failed if the number of failed Events across all objects in it exceeds the pre-defined threshold value of 10K.

4. In the Schema Evolution Policy section, specify the option to sync the Source schema changes with the Destination schema:

   • Allow all changes: In this option, Hevo replicates any changes in the Source schema, such as the addition of new columns, schemas, tables, and or updates to existing ones, to the Destination schema. This is the default selection.

   • Block all changes: In this option, Hevo does not replicate any changes in the Source schema to the Destination schema. This means that any modifications in the Source schema, such as the addition of new columns, tables, schemas, and or updates to existing ones, are not synced with the Destination schema.

   • Allow column-level changes only: In this option, Hevo replicates any changes only in the columns of the selected Source object(s) to the Destination. This means that any modifications to the Source columns, such as adding, deleting, and renaming, are synced with the Destination schema.

     Note: Any tables or schemas created in the Source post-Pipeline creation are not synced.

5. In the Replication type section, specify the kind of data that you want Hevo to replicate:

   • Replicate existing data and ongoing changes (Historical and Incremental): In this option, Hevo first replicates all existing data from the selected Source tables and thereafter captures only data changes. Hevo recommends this method to ensure integrity between your Source and Destination data.

   • Replicate data changes only (Incremental only): In this option, Hevo replicates only the new and changed data from your Source table.

6. In the Sync Frequency section, specify an interval in hours or minutes at which the Pipeline should run jobs to process data. You can set the sync frequency in hours to an integer value in the range of 1-48 and minutes to 30 or 45. By default, the sync frequency is set to 1 hour.

   Note:

   • The specified hours must be an integer value, such as 1, 2, or 3, but not 1.5 or 1.75.

   • The default minimum sync frequency is 30 minutes. If you need a shorter interval, contact Hevo Support.

7. In the Pipeline Latency section:

   • Send alerts for Pipeline Latency: Enable this option to receive alert notifications when your Pipeline latency goes beyond the set value. If enabled, Hevo displays the following field:

     • Pipeline Latency (in minutes): The latency threshold value for your Pipeline. Hevo sends an alert when this threshold is exceeded. Latency is the difference between when the data was created or updated at Source and when it was loaded to your Destination. Specify a value equal to or greater than 5 minutes.

8. Click SAVE & CONTINUE. Once the Pipeline settings are saved, Hevo fetches all the objects from the database specified while configuring the Source.