Audit Tables
This feature is currently available for Early Access. Please contact your Hevo account executive or Support team to enable it. Alternatively, request for early access to try out one or more such features.
Audit Tables in Hevo provide improved visibility into the Pipeline activities such as data ingestion and loading. These are tables that you can enable Hevo to create in your Destination for maintaining logs of all the jobs performed by your Pipelines for data replication. Hevo maintains the logs within a database, ingests them, and loads them to the Audit Tables as per the schedule that you define.
Hevo creates the Audit Tables at the job and object level. This means that the details are present for all jobs executed across Pipelines and each object in those Pipelines in your account. Each row in an Audit Table represents a job performed by a Pipeline.
Users who signed up after October 31, 2022, can use the same Audit tables across all the workspaces in their account. However, they must define these table settings within each workspace.
Audit Tables help you do the following:
-
Track and monitor Event usage based on the number of Events ingested, loaded, and failed.
-
Monitor the stages of data replication on the basis of:
-
Start and end timestamp of a data replication status.
-
Mode of data replication: Historical or Incremental.
-
The type of data being replicated: Refresher or Incremental.
-
Total time taken by Hevo to complete the execution for a replication status.
-
Hevo creates the Audit Tables in the Destination with the following names and schemas:
- Job_details table for Job-level logs
| Field | Field Type | Description |
|---|---|---|
| job_id | varchar | The UUID of the job performed by the Pipeline. |
| pipeline_id | long | The sequential serial number of the Pipeline. |
| source | varchar | The Source associated with the Pipeline ID. |
| destination | varchar | The Destination associated with the Pipeline ID. |
| start_timestamp | long | The start timestamp of the job. |
| end_timestamp | long | The end timestamp of the job. |
| status | varchar | The status of the Pipeline related to the job. For example, Scheduled or In Progress. |
| mode | varchar | The type of data being replicated. For example, Refresher or Incremental. |
| duration | long | The time taken for the job to be completed. |
| custom_stage | bool | The boolean value confirming whether a Transformation was applied to the Pipeline ID or not. This field is TRUE if a Transformation is applied and vice-versa. |
- Object_details table for object-level logs
| Field | Field Type | Description |
|---|---|---|
| job_id | varchar | The UUID of the job performed by a Pipeline. |
| object | varchar | The name of the Source object associated with the job ID. |
| event_name | varchar | The Source Event name associated with an object. For example, if the object, city contains address and zipcode as the associated objects, the Event names will be city.zipcode and city.address. |
| stage | varchar | The stage of data replication related to the job. For example, Ingestion or Load. |
| start_timestamp | long | The start timestamp of the job. |
| end_timestamp | long | The end timestamp of the job. |
| status | varchar | The status of data replication. For example, In Progress or Completed. Note: This field may or may not display the accurate status if Transformations are involved. |
| mode | varchar | The mode of data replication. For example, Historical or Incremental. |
| input_rows | long | The number of Events ingested for the object associated with the job. |
| output_rows | long | The number of Events loaded for the object associated with the job. Note: This may not match the input rows in case of errors or Transformations. |
| error_rows | long | The number of Events that are not replicated due to some error. |
| billable_rows | long | The number of loaded Events that are counted towards your Events usage. |
| duration | long | The time taken for the job to be completed. |
Logs are added as per the sequence in which jobs are completed in Pipelines. You can run suitable queries to view the logs as per your requirements. The following is a sample of how the logs are visible when you query the Audit Tables from the Destination workbench:
Hevo does not charge you for the logs loaded to the Destination or queries run directly from the Hevo UI. However, you may be charged by the Destination for data storage and querying. Refer to your respective Destination for details about such costs.
- Articles in this section