Pausing a Log-based Pipeline
Each log has a retention period associated with it which specifies the duration a log is maintained before it is automatically deleted. Logs are purged periodically to prevent them from taking up too much disk space.
Consequences of Pausing a Log-based Pipeline
Pausing a Pipeline stops the process of ingestion. Pausing a log-based Pipeline for more than 24 hours may lead to data loss as a result of the log being deleted. The log can get deleted due to the expiry of its retention period or limited disk storage space in the case of large log files.
Note: Pipelines in the PAUSED state do not process any data and, hence, do not affect your Events quota.
In case there is a loss of data after resuming a paused Pipeline, restart the historical load for all the objects to ingest the lost data.
To do so, perform the following steps:
-
On the Pipeline Overview page, select the Objects check box to select all the objects in the Pipeline. You can also select specific objects by selecting the check box next to their names.
-
Select the Restart option from the MORE drop-down to start the historical data ingestion.
The historical load starts immediately. The re-ingested data does not count towards your quota consumption and is not billed.
Note: You can similarly recover lost data by restarting individual jobs configured with change tracking mode in the case of an SQL Server Source.