ETL Cost Monitor
Overview
Most ETL providers today use usage-based pricing models that are both complex and opaque. Teams often lack the ability to:
- Visualize their costs at a granular level (by connector, schema, or table)
- Forecast monthly spend based on current usage
- Get alerted when there are budget overruns, unusual usage spikes, or other unexpected cost behaviors
ETL Cost Monitor solves this by giving you complete visibility and control over your ETL costs — all from within Snowflake.
What It Does
- Tracks and visualizes current month cost, forecasted cost, and total MAR
- Breaks down usage and cost by connector, schema, and table
- Highlights top gainers and most expensive tables
- Supports budget and spike alerts via Snowflake Tasks and Notification Integrations
- All logic runs inside your Snowflake account - no data is sent externally
Post-Installation Configuration
🔓 Granting Access to Usage Data
The first time you launch the app, you will be prompted to grant SELECT
access to your ETL provider’s usage table - such as fivetran_metadata.incremental_mar
📬 Alerting Setup (Optional but Recommended)
Step 1: Grant App Role Access to a User Role
GRANT APPLICATION ROLE etl_cost_monitor_app.app_public TO ROLE <your_account_role>;
Alternatively, manage this via the Access Management tab in the UI.
Step 2: Create a Notification Integration
Note: The integration must be named
etl_cost_monitor_email
.
CREATE NOTIFICATION INTEGRATION IF NOT EXISTS etl_cost_monitor_email
TYPE = EMAIL
ENABLED = TRUE
DEFAULT_RECIPIENTS = (
'user1@example.com',
'user2@example.com'
);
Step 3: Grant Usage on Integration to the App
GRANT USAGE ON INTEGRATION etl_cost_monitor_email TO APPLICATION etl_cost_monitor_app;
Step 4: Create a Recurring Task
CREATE OR REPLACE TASK etl_cost_monitor_alert_task
WAREHOUSE = <warehouse>
SCHEDULE = '24 HOURS'
AS
CALL etl_cost_monitor_app.code_schema.evaluate_and_alert();
Step 5: Resume the Task
ALTER TASK etl_cost_monitor_alert_task RESUME;
Note: The app cannot verify whether the task is running — please confirm manually.
Stored Procedures
etl_cost_monitor_app.code_schema.evaluate_and_alert()
Evaluates current and forecasted usage against thresholds and triggers email alerts if needed.
Application Role
etl_cost_monitor_app.app_public
Required Privileges
-
USAGE
on the notification integrationetl_cost_monitor_email
-
SELECT
access on the ETL usage table is requested via the app UI when launched.
Example SQL Queries
1. Top 5 Connectors by Volume This Month
SELECT connection_name, SUM(incremental_rows) AS total_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
GROUP BY connection_name
ORDER BY total_rows DESC
LIMIT 5;
2. Month-over-Month Growth by Connector
WITH current_month AS (
SELECT connection_name, SUM(incremental_rows) AS current_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
AND free_type = 'PAID'
GROUP BY connection_name
), last_month AS (
SELECT connection_name, SUM(incremental_rows) AS last_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE() - INTERVAL '1 MONTH')
AND free_type = 'PAID'
GROUP BY connection_name
)
SELECT
c.connection_name,
c.current_rows,
COALESCE(l.last_rows, 0) AS last_rows,
c.current_rows - COALESCE(l.last_rows, 0) AS growth
FROM current_month c
LEFT JOIN last_month l ON c.connection_name = l.connection_name
ORDER BY growth DESC
LIMIT 5;
3. Table-Level Usage Breakdown (Current Month)
SELECT destination_id, schema_name, table_name, SUM(incremental_rows) AS total_rows
FROM fivetran_metadata.incremental_mar
WHERE DATE_TRUNC('month', measured_date) = DATE_TRUNC('month', CURRENT_DATE())
AND free_type = 'PAID'
GROUP BY destination_id, schema_name, table_name
ORDER BY total_rows DESC
LIMIT 10;
Last updated on Jul 28, 2025