Configure the Pipeline Settings

You can specify your Pipeline settings after you have set up your Destination. To do this, perform the following steps on the Pipeline Configuration page:

1. In the Pipeline Setup section, specify the following:

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Destination Prefix: A string value, not exceeding 32 characters, that you want to add at the beginning of your Destination table names. The string can have alphanumeric characters and underscores, but it cannot start with an underscore. Also, as Hevo adds a trailing underscore, you do not need to provide one at the end of the specified prefix string. For example, pp_1 but not _pp_1 or _pp_1_.

     Note: Hevo creates the tables following the conventions defined by the Destination for naming identifiers. Read your Destination documentation for the rules.

2. In the Select a Load Mode section, specify one of the following:

   • Merge: In this mode, existing data records are updated based on the matching criteria, such as primary keys, and new records are added to your Destination table if they do not match the criteria.

     Note: You must define a primary key for the Source object if it does not have one.

   • Append (Recommended): In this mode, new data records are added at the end of your Destination table without modifying or deleting any existing data. This is the default selection.

     Note: This option may result in duplicate data in your Destination tables.

   Later, you can change the load mode for any Source object individually on the Object Configuration page.

3. In the Failure Handling Policy section, specify the policy for handling object-level failures:

   • Moderate: In this mode, data from all successfully processed objects in the job is ingested and loaded into your Destination tables. Select an action from the drop-down to be performed in case of failures:

     • Don’t load data for Objects with failed Events: In this case, Hevo loads data from only those objects in a job that are processed successfully.

     • Load data for Objects with less than 10K failed Events: In this case, Hevo marks the entire job as failed if the number of failed Events across all objects in it exceeds the pre-defined threshold value of 10K.

4. In the Schema Evolution Policy section, specify the option to sync the Source schema changes with the Destination schema:

   • Allow all changes: In this option, Hevo replicates any changes in the Source schema, such as the addition of new columns, schemas, tables, and or updates to existing ones, to the Destination schema. This is the default selection.

   • Block all changes: In this option, Hevo does not replicate any changes in the Source schema to the Destination schema. This means that any modifications in the Source schema, such as the addition of new columns, tables, schemas, and or updates to existing ones, are not synced with the Destination schema.

   • Allow column-level changes only: In this option, Hevo replicates any changes only in the columns of the selected Source object(s) to the Destination. This means that any modifications to the Source columns, such as adding, deleting, and renaming, are synced with the Destination schema.

     Note: Any tables or schemas created in the Source post-Pipeline creation are not synced.

5. In the Replication type section, specify the kind of data that you want Hevo to replicate:

   • Replicate existing data and ongoing changes (Historical and Incremental): In this option, Hevo first replicates all existing data from the selected Source tables and thereafter captures only data changes. Hevo recommends this method to ensure integrity between your Source and Destination data.

   • Replicate data changes only (Incremental only): In this option, Hevo replicates only the new and changed data from your Source table.

6. In the Sync Frequency section, specify an interval in hours or minutes at which the Pipeline should run jobs to process data. You can set the sync frequency in hours to an integer value in the range of 1-48 and minutes to 30 or 45. By default, the sync frequency is set to 1 hour.

   Note:

   • The specified hours must be an integer value, such as 1, 2, or 3, but not 1.5 or 1.75.

   • The default minimum sync frequency is 30 minutes. If you need a shorter interval, contact Hevo Support.

7. In the Pipeline Latency section:

   • Send alerts for Pipeline Latency: Enable this option to receive alert notifications when your Pipeline latency goes beyond the set value. If enabled, Hevo displays the following field:

     • Pipeline Latency (in minutes): The latency threshold value for your Pipeline. Hevo sends an alert when this threshold is exceeded. Latency is the difference between when the data was created or updated at Source and when it was loaded to your Destination. Specify a value equal to or greater than 5 minutes.

8. Click SAVE & CONTINUE. Once the Pipeline settings are saved, Hevo fetches all the objects from the database specified while configuring the Source.

   

Select and Configure the Source Objects

Once Hevo fetches all the Source objects, you are directed to the Object Configuration page of your Pipeline. On this page, all the objects from the Source database are displayed.

Note: Hevo marks the Source object as Inaccessible if the publication key specified while configuring the PostgreSQL Sources does not include the object or if the publication is defined without the insert, update, and delete privileges.

On the Object Configuration page, perform the following steps to configure the Source objects for data ingestion:

1. In the Object Configuration section, do one of the following:

   • Click the check box next to Select All Objects to ingest data from all objects included in the publication key.

   • Select the check box next to each object whose data you want to replicate.

2. Define the primary keys for the selected Source objects if you want to load data to the Destination table in the Merge mode. This step is not required for the Append mode.

3. (Optional) Click the Edit ( ) icon under Actions to do the following:

   • Select or deselect the Source object fields.

     Note: If data is to be loaded in the Merge mode, you cannot deselect a field that is the primary key of a Source object.

   • Change the load mode for the object

4. Click SAVE & CONTINUE. You need to select at least one object for data ingestion in order to proceed to the Review page.

5. In the Summary screen, review all the configurations defined for the Pipeline.

6. Click the acknowledgment check box.

7. Click SAVE PIPELINE. This button is enabled after you have performed the step above.

You are automatically redirected to the Job History tab, which displays the jobs running in your Pipeline. You can click on a job to view its details.