Scheduling Data Load for a Destination

You can define the frequency at which data must be loaded to the Destination based on when you need the data. As the loading frequency affects all Pipelines using the Destination, you cannot modify it for any individual Pipeline.

Prerequisites

• You are assigned any role except the Billing Administrator in Hevo, to modify Destination configuration.

---

For a Destination, depending on the type, you can:

• Modify the default loading frequency.

• Create a custom data loading schedule.
• Manually trigger data loading for the Destination or specific objects.

You cannot change the data loading frequency for the following Destination types:

• Databases: In Destinations such as MySQL, Amazon Aurora MySQL, MS SQL Server, and PostgreSQL, data is loaded in near real-time.

• Google BigQuery data warehouse with streaming enabled, as data is loaded in near real-time.

Read through the following sections for the details.

---

Changing the Default Loading Frequency

The ingested data is loaded to a Destination at a default loading frequency, which is set when the Destination is created. You can change this frequency to one of the values provided by Hevo. Some Destinations also allow you to set a custom loading frequency. Refer to section, Creating a Custom Data Loading Schedule.

To change the default data loading frequency for a Destination:

1. Click DESTINATIONS in the Navigation Bar.

2. In the Destinations List View, click on the required Destination to open its Detailed View. You can see the default data loading frequency in the Summary Bar.

   

3. Click Change.

4. Select the desired frequency, and click SCHEDULE.

   

   The revised frequency is displayed in the Destination’s Detailed View.

   

---

Creating a Custom Data Loading Schedule

Most database Destinations do not allow you to set a custom loading frequency. For other Destinations, you can create a custom schedule to load the data.

Perform the following steps to create a custom loading schedule for your Destination:

1. Click DESTINATIONS in the Navigation Bar.

2. In the Destinations List View, click on the required Destination to open its Detailed View. You can see the currently configured data loading frequency in the Summary Bar.

   

3. Click Change.

4. In the CHANGE DESTINATION SYNC SCHEDULE pop-up window, click Custom, and select one of the following schedule types:

   • Run at a fixed interval: Select this option to set the data loading frequency to a value other than those provided by Hevo. For example, you may want to load data to your Amazon Redshift Destination every four hours instead of the default five minutes.

     Specify the number of hours, and click SCHEDULE to save the custom schedule.

     

     The changed schedule is applied immediately and is visible in the Destination’s Detailed View.

     

   • Run daily: Select this option to set a fixed, daily loading schedule. For example, you may want to load data every four hours during peak hours, and every two hours after peak hours. To configure:

     

     1. Select a time from the drop-down. To add multiple custom time(s), click + ADD MORE. You can add up to 24 custom time values to your schedule.

     2. Click SCHEDULE to save your custom schedule.

   The changed schedule is applied immediately and is visible in the Destination’s Detailed View.

   

Note: You can change the custom schedule type later, and also set the data loading frequency back to one of the values provided by Hevo.

---

Manually Triggering Data Loading

You can explicitly trigger the data loading task to load any pending Events displayed in the UI.

To do this, select Load Events Now from the actions available for the Destination. However, this loads all the pending Events for all Pipelines which use that Destination. Therefore, it is not recommended to explicitly trigger a data load.

To trigger a data load for a specific table, select the Load Events Now option from the Actions menu of the table.

Similar to the data loading schedule, you can define a schedule for ingesting data from your Source in the Pipeline. Read Scheduling a Pipeline.