Pipelines (Edge)

A Pipeline moves data from a Source system to a Destination system. Your Source can be a database, such as PostgreSQL, and the Destination system can be a data warehouse, such as Snowflake.

You can connect only one Source and Destination in a Pipeline. However, multiple Pipelines may load to the same Destination.

Creating a Pipeline in Hevo involves configuring the following:

• Source: You can configure a new Source by providing the required connection parameters or use an existing one. Read Sources for the various Sources that Hevo integrates with.

• Destination: You can use an existing Destination or specify the necessary connection parameters to configure a new one. At this time, you can replicate data from a Source database only to a Snowflake Destination. Read Destinations.

• Pipeline: In the Pipeline configuration, you can define the way in which you want Hevo to load data to your Destination. Here, you can specify how your Destination schema should evolve and also how often the data from the Source should be synchronized with the Destination.

• Objects: Hevo allows you to select the objects from which you want to replicate data. You can also define a primary key for your object here if you have opted to load data to the Destination with the Merge data loading policy.

---

Working with Edge Pipelines

• Familiarizing with the Pipelines UILearn how to view, filter, and open your Pipelines from the Pipelines List View and Detailed View.
• Creating an Edge PipelineLearn how to create a Pipeline in Hevo to synchronize data from your Source with a Destination.
• Working with Edge PipelinesLearn how to customize, edit, and manage your Edge Pipeline's data replication process.
  • Editing Edge Pipelines
  • Orchestrating Pipelines with External Tools
  • On-Demand Syncing Data in Edge Pipelines
  • Activating Edge Pipelines
  • Modifying the Source and Destination Configuration
  • Deleting Edge Pipelines
• Pipeline Job HistoryLearn how a Job replicates data from your Source to your Destination, and how to view its details.
  • Job Types
  • Job and Object Statuses
  • Viewing Pipeline Job Details
  • Offset and Latency of Jobs and Objects
  • Canceling Jobs
  • Downloading Session Logs
• Object and Schema ManagementLearn how to select objects for replication and manage how the Destination schema evolves.
  • Familiarizing with the Object Configuration UI
  • Managing Objects in Pipelines
  • Pipeline Schema Management
• Activity LogLearn how to view a consolidated log of user actions, job statuses, and Pipeline state changes.