Skip to main content
Audience: Platform Teams, Data Platform, SRE Data incidents need the same rigor as application incidents. This guide helps you set up 24/7 monitoring with proper escalation, on-call routing, and incident response.

​
The Goal

On-call data alerting flow from detection to resolution

​
Architecture Overview

Alerting architecture showing event routing to different destinations

​
Setting Up PagerDuty Integration

​
Step 1: Create PagerDuty Service

In PagerDuty:
  1. Go to Services → New Service
  2. Name: Data Observability - AnomalyArmor
  3. Integration: Select Events API V2
  4. Copy the Integration Key

​
Step 2: Add PagerDuty Destination in AnomalyArmor

  1. Go to Alerts → Destinations
  2. Click Add Destination
  3. Select PagerDuty
  4. Enter the Integration Key
  5. Name: PagerDuty - Data On-Call
  6. Test and Save

​
Step 3: Configure Escalation Policy

In PagerDuty, set up escalation: Escalation policy levels

​
Alert Urgency Framework

Define how urgently different data incidents need response:

​
Critical (Page Immediately)

Criteria:
  • Production data pipeline completely down
  • Core revenue tables missing or stale >4 hours
  • Discovery failures for >24 hours
Examples:
  • Column removed from orders table
  • payments table data >4 hours stale
  • Can’t connect to production database
Destination: PagerDuty → On-Call

​
High (Respond Within 4 Hours)

Criteria:
  • Important tables stale (1-4 hours)
  • Schema changes in production
  • Non-critical discovery failures
Examples:
  • Column type changed in production
  • Analytics tables 2 hours stale
  • Staging discovery failed
Destination: Slack #data-incidents

​
Medium (Respond Within 24 Hours)

Criteria:
  • Non-production schema changes
  • Warning thresholds reached
  • New assets discovered
Examples:
  • Staging schema changed
  • Freshness approaching SLA (warning)
  • New table discovered in production
Destination: Slack #data-alerts

​
Low (Informational)

Criteria:
  • Development changes
  • Expected changes
  • Routine discoveries
Destination: Email digest (daily)

​
Alert Rule Configuration

​
Rule 1: Critical - Production Breaking Changes

​
Rule 2: Critical - Revenue Table Freshness

​
Rule 3: High - Production Schema Changes

​
Rule 4: High - Data Freshness Violations

​
Rule 5: High - Discovery Failures

​
On-Call Runbook

​
When Paged for Schema Change

On-call runbook for schema changes

​
When Paged for Freshness Violation

  1. ACKNOWLEDGE the alert
  2. CHECK ETL STATUS
    • Is the ETL job running? Failed? Stuck?
    • Check Airflow/Dagster/orchestrator
  3. CHECK SOURCE SYSTEM
    • Is the source database accessible?
    • Is source data actually updating?
  4. IDENTIFY ROOT CAUSE
    • ETL failure → Fix and restart
    • Source delay → Communicate delay
    • Connection issue → Troubleshoot connection
  5. MITIGATE
    • Restart failed jobs
    • Notify stakeholders of delay
  6. RESOLVE and document

​
Slack Integration Best Practices

​
Channel Setup

Slack Channels:
  • #data-incidents - Breaking changes (notifications on)
  • #data-alerts - All schema changes (lower priority)
  • #data-digest - Daily/weekly summaries

​
Alert Message Format

AnomalyArmor alerts include:

​
Maintenance Windows

​
Scheduled Maintenance

Before planned changes:
  1. Go to Alerts → Rules
  2. Toggle OFF relevant rules
  3. Set a reminder to re-enable (e.g., calendar event)
  4. Proceed with maintenance
  5. Verify changes detected correctly
  6. Toggle rules back ON

​
Quick Disable

For unexpected but known issues, quickly disable a rule:
  1. Go to Alerts → Rules
  2. Find the rule
  3. Toggle it OFF
  4. Remember to re-enable when the issue is resolved

​
Metrics to Track

Review these weekly in your on-call handoff.

​
Checklist

Before going live with on-call alerting:
  • PagerDuty integration configured
  • Escalation policy set up
  • Critical/High/Medium/Low rules defined
  • Slack channels created and configured
  • On-call runbook documented
  • Team trained on response procedures
  • Test alert sent and verified

​
Common Questions

​
How do I page my on-call engineer when data breaks?

Create a PagerDuty service with an Events API V2 integration, copy the integration key, and add a PagerDuty destination in Alerts → Destinations. Then route only your Critical rules (breaking schema changes, revenue-table freshness >4h) to that destination. See Setting Up PagerDuty Integration.

​
Which data incidents should actually page someone?

Page on production pipelines being completely down, core revenue tables stale for more than 4 hours, or discovery failures lasting over 24 hours. Everything else should go to Slack, not PagerDuty, to protect on-call from alert fatigue. See the Alert Urgency Framework.

​
How do I suppress alerts during planned maintenance?

Go to Alerts → Rules and toggle off the relevant rules before the maintenance window, then re-enable after. Set a calendar reminder so rules don’t stay off indefinitely. For recurring windows, use operating schedules and blackouts in the contract config instead.

​
What metrics should I track for data on-call health?

MTTD (under 1 hour, driven by discovery frequency), MTTN (under 5 minutes from alert to page), MTTR (under 4 hours), false-positive rate (under 20%), and pager load (under 5 critical alerts per week). Review these weekly in your on-call handoff. See Metrics to Track.

​
Can I send different alerts to different Slack channels?

Yes. Create separate destinations for #data-incidents (breaking changes), #data-alerts (all schema changes), and #data-digest (daily summaries), then route each alert rule by severity. That keeps high-signal alerts out of the noisy firehose and stops people from muting the wrong channel.

PagerDuty Setup

Detailed PagerDuty integration guide

Alert Best Practices

Reduce alert fatigue