Configure the Pipeline Settings

You can specify your Pipeline settings after you have set up your Destination. To do this, perform the following steps on the Pipeline Configuration page:

1. In the Pipeline Setup section, specify the following:

   • Pipeline Name: A unique name for your Pipeline, not exceeding 255 characters.

   • Destination Prefix: A string value, not exceeding 32 characters, that you want to add at the beginning of your Destination table names. The string can have alphanumeric characters and underscores, but it cannot start with an underscore. For example, pgsql_sf_2 but not _pgsql_sf_2.

     Note: Hevo creates the tables following the conventions defined by the Destination for naming identifiers. Read your Destination documentation for the rules.

2. In the Select a Load Mode section, specify one of the following:

   • Append (Recommended): In this mode, new data records are added at the end of your Destination table without modifying or deleting any existing data. This is the default selection.

     Note: This option may result in duplicate data in your Destination tables.

   • Merge: In this mode, existing data records are updated based on the matching criteria, such as primary keys, and new records are added to your Destination table if they do not match the criteria.

     Note: You must define a primary key for the Source object if it does not have one.

3. In the Failure Handling Policy section, specify the policy for handling object-level failures:

   • Strict (N/A): In this mode, data is ingested and loaded to your Destination tables only if all objects in a job are processed without failures.

   • Moderate (Default): In this mode, data from all successfully processed objects in the job is ingested and loaded into your Destination tables. Select an action from the drop-down to be performed in case of failures:

     • Don’t load data for Objects with failed Events: In this case, Hevo loads data from only those objects in a job that are processed successfully.

     • Load data for Objects with less than 10K failed Events: In this case, Hevo marks the entire job as failed if the number of failed Events across all objects in it exceeds the pre-defined threshold value of 10K.

4. In the Schema Evolution Policy section, specify the option to sync the Source schema changes with the Destination schema:

   • Allow all changes: In this option, Hevo replicates any changes in the Source schema, such as the addition of new columns, schemas, tables, and or updates to existing ones, to the Destination schema. This is the default selection.

   • Block all changes: In this option, Hevo does not replicate any changes in the Source schema to the Destination schema. This means that any modifications in the Source schema, such as the addition of new columns, tables, schemas, and or updates to existing ones, are not synced with the Destination schema.

   • Allow column-level changes only (N/A): In this option, Hevo replicates any changes in the columns of the Source object(s) to the Destination. This means that any modifications only to columns of the Source table(s), such as addition, deletion, renaming, and reordering of columns, are synced with the Destination.

     Note: Any tables or schemas created in the Source post-Pipeline creation are not synced.

5. In the Sync Frequency section, specify the frequency at which the Pipeline should run jobs to process data. You can select from one of the pre-defined values or click Custom to set a value (in Hours) other than those provided by Hevo. The default value is 1 Hour.

   Note: The custom frequency must be an integer value between 1-24. For example, 1, 2, or 3, but not 1.5 or 1.75.

6. Click SAVE & CONTINUE. Once the Pipeline settings are saved, Hevo fetches all the objects from the database specified while configuring the Source.

   

Select and Configure the Source Objects

Once Hevo fetches all the Source objects, you are directed to the Object Configuration page of your Pipeline. On this page, all the objects from the Source database are displayed.

Note: Hevo marks the Source object as Inaccessible if the publication key specified while configuring the Source does not include the object or if the publication is defined without the insert, update, and delete privileges.

On the Object Configuration page, perform the following steps to configure the Source objects for data ingestion:

1. In the Object Configuration section, do one of the following:

   • Click the check box next to Select All Objects to ingest data from all objects included in the publication key.

   • Select the check box next to each object whose data you want to replicate.

2. Define the primary keys for the selected Source objects if you want to load data to the Destination table in the Merge mode. This step is not required for the Append mode.

3. (Optional) Click the Edit icon (icon image here) under Actions to select or deselect the Source object fields.

   Note: You cannot deselect a field that is the primary key of a Source object.

4. Click SAVE & CONTINUE. This button is enabled after you have selected at least one object for data ingestion.

5. In the Summary screen, review all the configurations defined for the Pipeline.

6. Click the acknowledgment check box.

7. Click SAVE PIPELINE. This button is enabled only after you have performed the step above.

You are automatically redirected to the Job History tab, which displays the jobs running in your Pipeline. You can click on a job to view its details.