Freshness Monitoring - AnomalyArmor

Freshness monitoring tracks when your data was last updated and alerts you when it becomes stale. Stale data can be just as damaging as wrong data. If your dashboard shows yesterday’s numbers, decisions made today could be wrong.

Prerequisites: Before setting up freshness monitoring, you need:

A connected data source with discovery completed
Assets with timestamp columns (created_at, updated_at, or similar)

Timeline showing expected update time, warning threshold, and violation with alert trigger — Freshness timeline showing expected vs actual data updates

Why Freshness Matters

Data has an expected update cadence. When that cadence breaks, something is wrong:

Stale data problem with and without monitoring

How Freshness Monitoring Works

AnomalyArmor tracks freshness by monitoring timestamp columns:

You specify which column indicates “when data was updated”
Discovery queries the maximum value of that column
We compare against your configured SLA
Alert fires if data is older than allowed

-- What AnomalyArmor checks
SELECT MAX(created_at) FROM orders;
-- Result: 2024-01-15 07:58:32

-- Your SLA: Updated within 1 hour
-- Current time: 2024-01-15 08:30:00
-- Data age: 32 minutes
-- Status: OK Within SLA

Setting Up Freshness Monitoring

Navigate to Asset

Go to Assets and click on the table you want to monitor.

Open Freshness Tab

Click the Freshness tab in the asset details.

Select Timestamp Column

Choose the column that best represents data recency:

Column Type	Best For
`created_at`	Append-only tables (events, logs)
`updated_at`	Tables with updates (user profiles)
`loaded_at`	ETL destination tables
`_etl_timestamp`	Warehouse staging tables

Choose a column that updates when new data arrives. For tables with updates, use updated_at instead of created_at to track the most recent changes.

Set Expected Frequency

Define how often you expect new data:

Every 15 minutes
Hourly
Every 6 hours
Daily
Weekly
Custom (specify hours)

Configure Alert Threshold

Set when to trigger alerts:Alert Configuration:

Warning threshold: 80% of expected (optional)
Violation threshold: 100%+ of expected

Example (hourly expectation):

Warning: Data is 48+ minutes old
Violation: Data is 60+ minutes old

Save Configuration

Click Save to activate freshness monitoring.

SLA Strategies

Start Conservative

Set SLAs with buffer room to avoid false positives:

Actual Cadence	Recommended SLA	Why
Every hour	2 hours	Allows for occasional delays
Every day	36 hours	Accounts for timing variations
Every 15 min	30 minutes	Double the expected window

Start lenient and tighten over time. It’s easier to make SLAs stricter than to deal with alert fatigue from SLAs that are too tight.

By Data Criticality

Criticality	SLA Approach	Example
Critical	Expected + 25%	Revenue: 1hr expected → 75min SLA
High	Expected + 50%	Orders: 1hr expected → 90min SLA
Medium	Expected × 2	Analytics: 1hr expected → 2hr SLA
Low	Expected × 3	Reports: daily expected → 3 day SLA

By Update Pattern

Streaming/Real-time
Batch/Hourly
Daily

For data expected within minutes:

Setting	Value
Expected	Every 5 minutes
Warning	10 minutes
Violation	15 minutes
Alert	Slack + PagerDuty

Setting	Value
Expected	Every hour
Warning	80 minutes
Violation	2 hours
Alert	Slack

Setting	Value
Expected	Daily by 6am
Warning	8am (2hr buffer)
Violation	12pm (6hr buffer)
Alert	Email

Freshness Alerts

Setting Up Alert Rules

Create rules to notify you of freshness violations:

Field	Value
Rule	Critical Table Freshness
Event	Freshness Violation
Assets	orders, payments, users
Destinations	Slack, PagerDuty

Alert Content

Freshness alerts include:

[!] Freshness Violation

Asset: production.public.orders
Column: created_at

Expected: Updated every 1 hour
Last update: 3 hours 15 minutes ago
Data age: 3h 15m (SLA: 1h)

Detected: January 15, 2024 at 8:00 AM UTC

[View Asset] [View Dashboard]

Handling Special Cases

Weekends and Holidays

Some data doesn’t update on weekends: Options:

Longer weekend SLA: Set different thresholds for Saturday/Sunday
Pause monitoring: Temporarily disable freshness checks
Adjust expectations: Set SLA to “72 hours” to cover full weekends

Maintenance Windows

During planned maintenance:

Disable rules: Toggle OFF freshness alert rules for affected assets
Document: Note expected staleness
Re-enable: Toggle rules back ON after maintenance

Infrequently Updated Tables

Some tables legitimately update rarely:

Table Type	Update Pattern	SLA Approach
Reference/lookup	Monthly	45-day SLA or no monitoring
Historical archive	Never	Don’t monitor freshness
Dimension tables	Daily/weekly	Match actual pattern

Viewing Freshness Status

Asset List View

In the Assets list, freshness status appears as indicators:

Indicator	Meaning
Green	Within SLA
Yellow	Warning threshold reached
Red	SLA violated
Gray	Freshness not configured

Asset Detail View

Click an asset to see:

Current freshness status
Last update timestamp
Freshness history over time
SLA configuration

The detail view shows a status card with the current freshness state, last update timestamp, data age, and a progress bar indicating how much of your SLA window has been consumed. Below that, a 7-day history shows freshness status over time. Example freshness status:

Field	Value
Status	✓ Within SLA
Timestamp Column	`created_at`
Last Update	January 15, 2024 at 2:15 PM UTC
Data Age	45 minutes
SLA Threshold	2 hours
Progress	37% of SLA (45 min / 2 hr)

Freshness History: All checks in the last 7 days were within SLA.

Troubleshooting

False positive alerts

Problem: Freshness alerts fire when data is actually fine.Solutions:

Verify timestamp column choice. Is it the right one?
Loosen SLA threshold
Check timezone handling (UTC vs. local)
Review discovery schedule vs. SLA timing

Missed violations

Problem: Data was stale but no alert fired.Solutions:

Verify freshness monitoring is enabled for the asset
Check alert rule configuration
Confirm discovery is running frequently enough
Verify timestamp column has recent values

Wrong timestamp column

Problem: Freshness showing incorrect values.Solutions:

Review column choice
For tables with updates, use updated_at not created_at
For ETL tables, use the load timestamp column
Ensure column is always populated (no NULLs)

Timezone issues

Problem: Freshness calculations seem off by hours.Solutions:

Check timestamp column timezone
AnomalyArmor normalizes to UTC
Ensure consistent timezone handling in your ETL

Best Practices

1. Start with Critical Tables

Focus monitoring on:

Revenue-impacting tables
Customer-facing dashboard sources
Compliance-required data

2. Align with Business Needs

Ask: “When would stale data cause a problem?”

Scenario	Acceptable Staleness	SLA
Real-time dashboard	Minutes	15 min
Daily executive report	Hours	Same-day
Monthly compliance	Days	1 week

3. Coordinate with ETL Schedules

Phase	Time
ETL Schedule	2:00 AM daily
Processing time	~30 minutes
Data available	~2:30 AM
SLA	4:00 AM (90 min buffer)

4. Use Warning Thresholds

Configure two levels:

Warning: “Heads up, getting stale” → Slack
Violation: “SLA breached” → PagerDuty

Common Questions

Which timestamp column should I pick?

Pick the column that updates when new data arrives. Use created_at for append-only tables like events and logs, updated_at for tables that get updated in place, and the load timestamp for ETL destination tables. Avoid columns that can be NULL.

What’s a sensible starting SLA?

Start lenient, around double the expected cadence. If data is expected hourly, set a 2-hour SLA. It’s easier to tighten thresholds than to fight alert fatigue from SLAs set too tight on day one.

How do I handle tables that don’t update on weekends?

Either extend the SLA to cover the weekend (for example, 72 hours), set different thresholds for weekend days, or pause the freshness rule during known quiet periods. Choose based on how critical the data is. For row count and metric monitors on the same table, operating-period awareness handles weekends automatically by baselining active and dormant periods separately.

Why does the freshness calculation look off by several hours?

Almost always a timezone issue. AnomalyArmor normalizes to UTC, so a timestamp column stored in local time will appear skewed. Confirm your ETL writes timestamps consistently and check the column’s declared timezone.

What’s the difference between a warning and a violation?

A warning fires before the SLA is breached (often at 80% of the threshold) as an early heads-up. A violation fires once the data is officially past SLA. Route them separately, warnings to Slack, violations to PagerDuty.

Do I need freshness monitoring on every table?

No. Focus on tables that power dashboards, revenue reporting, customer-facing products, or compliance processes. Reference tables and historical archives usually don’t need freshness checks.

What’s Next

Set Up Metrics

Track row counts, null percentages, and detect anomalies

Configure Alerts

Get notified when freshness SLAs are violated

Report Badges

Embed freshness status in dashboards

Best Practices

Reduce alert fatigue

​Why Freshness Matters

​How Freshness Monitoring Works

​Setting Up Freshness Monitoring

​SLA Strategies

​Start Conservative

​By Data Criticality

​By Update Pattern

​Freshness Alerts

​Setting Up Alert Rules

​Alert Content

​Handling Special Cases

​Weekends and Holidays

​Maintenance Windows

​Infrequently Updated Tables

​Viewing Freshness Status

​Asset List View

​Asset Detail View

​Troubleshooting

​Best Practices

​1. Start with Critical Tables

​2. Align with Business Needs

​3. Coordinate with ETL Schedules

​4. Use Warning Thresholds

​Common Questions

​Which timestamp column should I pick?

​What’s a sensible starting SLA?

​How do I handle tables that don’t update on weekends?

​Why does the freshness calculation look off by several hours?

​What’s the difference between a warning and a violation?

​Do I need freshness monitoring on every table?

​What’s Next

Set Up Metrics

Configure Alerts

Report Badges

Best Practices

Why Freshness Matters

How Freshness Monitoring Works

Setting Up Freshness Monitoring

SLA Strategies

Start Conservative

By Data Criticality

By Update Pattern

Freshness Alerts

Setting Up Alert Rules

Alert Content

Handling Special Cases

Weekends and Holidays

Maintenance Windows

Infrequently Updated Tables

Viewing Freshness Status

Asset List View

Asset Detail View

Troubleshooting

Best Practices

1. Start with Critical Tables

2. Align with Business Needs

3. Coordinate with ETL Schedules

4. Use Warning Thresholds

Common Questions

Which timestamp column should I pick?

What’s a sensible starting SLA?

How do I handle tables that don’t update on weekends?

Why does the freshness calculation look off by several hours?

What’s the difference between a warning and a violation?

Do I need freshness monitoring on every table?

What’s Next