Share

Pipelines

A Pipeline moves your data from a Source system to a Destination database or data warehouse. It has the following components:

  • Source: A Source can be a database, a SaaS-based application (an API endpoint) or a file storage which has the data that you want to analyze. Hevo integrates with a variety of Sources. Read Sources.

  • Transformations: Transformations are useful when you want to clean, enrich, or transform your data before loading it to your Destination.

  • Schema Mapper: Schema Mapper helps you map your Source schemas to tables in your Destination.

  • Destination: Destination is a data warehouse or database where the data fetched from a Source is loaded. Read Destinations.

    Note: At present, SaaS Destinations are not supported.

You can connect only one Source and Destination in a Pipeline. However, multiple Pipelines may load to the same Destination.

Refer to the documents in this section to know how you can create and manage your Pipelines and their operations using the Hevo UI.

Creating and managing Pipelines using the Hevo API

In addition to the user-interface based option, Hevo also provides you its API to automate the Pipeline operations. Hevo API is free for all users, including any trial users.

Hevo API provides you endpoints to:

  • Create, fetch, and update Pipelines

  • Update Source configurations

  • Update Pipeline position, priority, and schedule

  • Resume, skip, include, pause, and restart Pipeline Objects

  • Fetch and Update object position

  • Use and test Transformations

  • Manage schema mapping

Read Hevo API for the steps to create your API request.


In this Section

The following interactive diagram illustrates the features and actions offered by the Pipelines and provides a summary view of the section’s content. Click on any item in the diagram to view the related information.

Create Pipelines Navigate the Pipeline User Interface Manage Pipelines Log-based Pipelines Manage Pipeline Objects References Map Source schema Transform data Log-based Pipelines View file log Schedule Pipelines Modify Pipelines Prioritize Pipelines View Pipelines progress Data flow in a Pipeline Best practices for creating database Pipelines Connectivity check for RDBMS Sources Activity log Database objects and actions SaaS objects and actions Query mode optimization Pipelines

Revision History

Refer to the following table for the list of key updates made to this page:

Date Release Description of Change
Dec-29-2023 NA Merged FAQ, Can I use Hevo to migrate data from one SaaS application to another SaaS application? to this document.
Apr-10-2023 NA Merged FAQ, Does Hevo provide APIs for automating Pipeline operations? into this document as a new section, Creating and managing Pipelines using the Hevo API.
Last updated on Dec 28, 2023

Tell us what went wrong

Skip to the section