Transformations

The data received from the Source system might, at times, not be suitable to be replicated to the Destination database or data warehouse as is. Using Transformations in Hevo, you can prepare the data in multiple ways before loading it to the Destination. Read Data Transformation to know more about scenarios where you may need to transform your data.

Note: For any Transformation that is applied on an Event, a maximum of 10,000 child Events can be generated.

---

Prerequisites

• You are assigned the Team Administrator, Team Collaborator, Pipeline Administrator, or Pipeline Collaborator role in Hevo, to work with transformation.

---

Writing the Transformation

You can design and apply Transformations on Events and Event Types in two ways:

• Writing Python-based Transformation script

• Using Drag-and-drop Transformation blocks provided by Hevo

To change the Transformation method, click the More () icon in the Transformation canvas.

Note: You cannot use both methods in the same Transformation. Also, Transformation of one type cannot be converted to the other automatically. If you switch from one method to the other, you need to rewrite the Transformation completely.

---

Testing the Transformation

Hevo retains the first few Events ingested from your Source as sample data, which enables you to test your Transformation code. Once you write or modify your Transformations, you can test them against these real sample Events.

To test your Transformation script:

1. In the Transformations UI, click the Select Event Type drop-down, and select the Source Event Type on which you want to test the Transformation.

   

2. Click GET SAMPLE. You receive an Event from the sample data that Hevo stores. No changes are applied till you deploy the Transformation.

3. Click TEST to test the Transformation and view the results.

   

Sometimes, if the data in the Source changes, you may not see sample Events for it immediately. Or, you may want to test an edge-case scenario but do not have a sample Event for it. In that case, you can paste a sample Event containing the changes and test your Transformation.

You can also test the Transformation on a random sample Event not specific to any Event Type.

Testing Transformations on failed Events

You can test any Transformation you have written to resolve failed Events directly within the Transformation page.

To do this:

1. Click the Select Event Type drop-down.

2. Enable the Events not Loaded option to view only the Event Types with failed Events.

3. Select the required Event Type and click GET SAMPLE.

4. Click TEST.

---

Deploying the Transformation

Once you have tested the Transformation, click DEPLOY. The Transformation is applied to all applicable incoming Events.

You can click RESET to return to the last deployed stage.

We would advise you to monitor the Failed Events for any failures due to the newly deployed code.

---

Transformation Version History

Hevo maintains a history of the last five versions of your deployed Transformation scripts and configurations. You can revert to the previous version of a script by selecting it from the displayed list.

Note: The restored Transformation script is in a Draft state. You should test this version before deploying it. The system warns you if you try to deploy it directly.

To view the history of the deployed Transformations and restore an earlier version, do the following:

1. Click the More ( ) icon and and select Version History.

   

2. From the VERSION HISTORY pane, select the version you want to restore, and then click RESTORE THIS VERSION.

   

3. Click TEST and then DEPLOY to test and deploy the restored Transformation script.

   

Note: The steps given above are for Drag and Drop Transformations. You can also restore a previous version of your Python code-based Transformation using similar steps.

---

• Articles in this section
  • Python Code-Based Transformations
    • Supported Python Modules and Functions
    • Transformation Methods in the Event Class
      • Create an Event
      • Retrieve the Event Name
      • Rename an Event
      • Retrieve the Properties of an Event
      • Modify the Properties for an Event
      • Fetch the Primary Keys of an Event
      • Modify the Primary Keys of an Event
      • Fetch the Data Type of a Field
      • Check if the Field is a String
      • Check if the Field is a Number
      • Check if the Field is Boolean
      • Check if the Field is a Date
      • Check if the Field is a Time Value
      • Check if the Field is a Timestamp
    • TimeUtils
      • Convert Date String to Required Format
      • Convert Date to Required Format
      • Convert Datetime String to Required Format
      • Convert Epoch Time to a Date
      • Convert Epoch Time to a Datetime
      • Convert Epoch to Required Format
      • Convert Epoch to a Time
      • Get Time Difference
      • Parse Date String to Date
      • Parse Date String to Datetime Format
      • Parse Date String to Time
    • Utils
      • Check if an Object is a Dictionary Object
      • Retrieve the Type for a Value
      • Replace Unicode Values
      • Round off BigDecimal Values
      • Trim Unicode Values
      • Convert JSON String to Dictionary Object
      • Convert Python Dictionary Object to JSON String
    • Examples of Python Code-based Transformations
      • Adding Time to a Timestamp Value
      • Splitting an Event into Multiple Event Types
      • Splitting JSON Objects into ArrayLists
      • Splitting Multiple Values in a Key into Separate Events
      • Splitting Nested Events into Multiple Events
  • Drag and Drop Transformations
    • Special Keywords
    • Transformation Blocks and Properties
      • Add a Field
      • Change Datetime Field Values
      • Change Field Values
      • Drop Events
      • Drop Fields
      • Find & Replace
      • Flatten JSON
      • Format Date to String
      • Format Number to String
      • Hash Fields
      • If-Else
      • Mask Fields
      • Modify Text Casing
      • Parse Date from String
      • Parse JSON from String
      • Parse Number from String
      • Rename Events
      • Rename Fields
      • Round-off Decimal Fields
      • Split Fields
    • Examples of Drag and Drop Transformations
  • Effect of Transformations on the Destination Table Structure
  • Transformation Reference
    • Datetime Field Values
    • Operators
    • Rounding Modes
    • Text Casing
    • Values and Formulas
  • Transformation FAQs
    • Why do I not see the newly ingested Source data in the sample Event?