Metadata Column __hevo_source_modified_at

For Pipelines created with some Sources, Hevo captures the time when an Event is created, modified, or deleted, using the __hevo_source_modified_at metadata field. This is a timestamp value extracted from the Source. You can use this information to filter Events for loading or applying transformations. The Sources for which this feature is currently available include:

• Amazon DynamoDB (DynamoDB Streams and Kinesis Data Streams ingestion modes).

• MongoDB (OpLog and Change Streams ingestion modes)

• MySQL (BinLog ingestion mode)

• File-based Sources: Drive, Amazon S3, GCS and FTP/SFTP. Prior to Release 1.66, __hevo_source_modified_at was available as a field that could be mapped via the Schema Mapper.

The _hevo_source_modified_at field contains the timestamp of the update in the form of number of milliseconds since January 1, 1970, 00:00:00 GMT.

This field is not available for mapping in the Schema Mapper. However, you can access it in Transformations using the event.getSourceModifiedAt() method.

Example: Using __hevo_source_modified_at Timestamp to Transform Events

from io.hevo.api import Event
from datetime import datetime
def transform(event):

    modifiedTs = event.getSourceModifiedAt()
    properties = event.getProperties()

    # get particular time in ms format
    startDate = datetime(2021, 1, 1)
    epoch = datetime.utcfromtimestamp(0)
    startTsInMillis = (startDate - epoch).total_seconds() * 1000.0


    # Add a new field to the event based on modifiedTs
    if modifiedTs is None :
        properties['employeeInfoUpdate'] = 'Not Specified'
    elif modifiedTs >= startTsInMillis :
        properties['employeeInfoUpdate'] = 'New'
    else :
        properties['employeeInfoUpdate'] = 'Old'

    return event

The above transformation uses the event.getSourceModifiedAt() method to retrieve the time when the Event was modified in the Source. Next, it compares this value against the startTsInMillis field, which is the date 1 Jan, 2021 in Millisecond format. Based on the result of the comparison, the appropriate value is assigned to the new field employeeInfoUpdate. If modifiedTs is not present, then the new field gets the value Not Specified. Else, it is assigned the value Old or New if it is earlier or later than 1 Jan, 2021, respectively.

Result: