How do I filter out specific fields before loading data to the Destination?

You can filter specific fields of an Event before the Destination table gets created for the corresponding Event Type, by using any one of the following:

• Python Code-Based Transformations

• Drag and Drop Transformations

• Schema Mapper

Note: Event Types from the Source are mapped to Tables in your Destination, while fields are mapped to the table columns.

Prerequisites

• The Pipeline is created with Auto Mapping disabled.

  

To filter columns using Python-based Transformations:

1. Create the Pipeline with Auto Mapping disabled.

2. Click Transformations to access the Python-based Transformation interface.

   

3. Write the following code in the CODE console of the Transformation:

    from io.hevo.api import Event
   
    def transform(event):
   
    properties = event.getProperties()
    #list of columns to be deleted, below is the sample for same
    del_columns=["item","qyt"]
    for i in properties.keys():
        if i in del_columns:
            del properties[i]
   
    return event
   
   

4. Click GET SAMPLE and then click TEST to test the transformation.

5. Click DEPLOY.

6. Once the Transformation is deployed, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Python Code-Based Transformations.

To filter columns using Drag and Drop Transformations:

1. Create the Pipeline with Auto Mapping disabled.

2. Click Transformations to access the Python-based Transformation interface.

3. In the Python-based Transformation interface, click ENABLE DRAG-AND-DROP INTERFACE.

   

4. Drag the Drop Fields Transformation block to the canvas.

   

5. Specify the Event Type from which you want to drop the fields. Alternatively, skip this filter to apply the transformation to all Event Types.

6. Select the Field filters to specify the fields to be dropped.

7. Click DEPLOY.

8. Once the Transformation is deployed, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Drag and Drop Transformations.

To filter columns using Schema Mapper:

1. Once the Pipeline gets created, in the Schema Mapper, select the required Event Type.

2. Click CREATE TABLE & MAP.

   

3. Deselect the columns that you want to exclude from the Destination table.

   

4. Specify the Destination Table Name and click CREATE TABLE & MAP.

5. Once the table is created, enable Auto Mapping for the Event Type from the Schema Mapper. Auto Mapping automatically maps Event Types and fields from your Source to the corresponding tables and columns in the Destination excluding the filtered columns.

Read more at Schema Mapper.

The Destination table is created with the remaining columns, and the data is replicated into this table as per the Pipeline schedule.