Row Count Monitoring - AnomalyArmor

Row Count Monitoring tracks row counts in your tables over time. It detects when data volumes drop unexpectedly (data loss) or spike unusually (duplicate loads), helping you catch ETL issues before they impact downstream consumers.

Why Row Count? Row count monitoring used to be part of Data Quality Metrics. We moved it to its own feature with enhanced capabilities: ML-based pattern learning, time-windowed counting, and explicit threshold support.

Example scenario: The orders table typically receives 45,000-55,000 rows daily. On Jan 30, only 15,234 rows were loaded — a 70% drop flagged as an anomaly, indicating a potential ETL failure.

Configuration Reference

Monitoring Mode

Row Count Monitoring offers two approaches to fit different needs:

Auto-Learn Mode (Recommended)

Let AnomalyArmor learn your table’s normal row count patterns:

Aspect	How It Works
Learning period	Collects data for 7+ days to establish baseline
Pattern detection	Identifies daily, weekly, and seasonal trends
Anomaly detection	Uses statistical analysis (mean +/- stddev * sensitivity)
Best for	Tables with consistent, predictable patterns

Auto-learn example (orders table):

Day 1-7:    Learning... collecting baseline data
Day 8+:     Baseline established (avg: 48,000, stddev: 3,200)
            Alerts if row count deviates significantly

Explicit Mode

Set specific row count thresholds when you know exactly what to expect:

Setting	Description
Min rows	Alert if row count falls below this value
Max rows	Alert if row count exceeds this value
Best for	Tables with known, fixed expectations

Explicit example (daily_summary table):

Expected: Exactly 1 row per day
Min: 1, Max: 1
Alert if row count != 1

Sensitivity

For auto-learn mode, sensitivity controls how strict the anomaly detection is. It’s the multiplier applied to the standard deviation when calculating expected ranges.

Sensitivity	Behavior	Use When
1 (Tight)	Very strict, catches small deviations	Critical data, low tolerance for anomalies
2 (Balanced)	Default, catches moderate deviations	Most tables, standard monitoring
3 (Relaxed)	Less strict, allows more variation	High natural variability, noisy data
4 (Loose)	Very permissive, only catches large deviations	Highly variable patterns, initial setup

Default: 2 (balanced detection) Formula: Expected range = mean ± (stddev × sensitivity)

Example with sensitivity = 2:
Mean: 48,000 rows
StdDev: 3,000 rows
Expected range: 42,000 - 54,000 rows
(48k - 6k to 48k + 6k)

If actual count = 35,000 → ANOMALY (outside range)
If actual count = 51,000 → HEALTHY (within range)

Start with sensitivity 2-3 for new monitors, then tighten to 1-2 once patterns are stable.

Timestamp Column

Optional column used to filter rows within the time window. Without a timestamp column, all rows in the table are counted regardless of when they were created. When to use: -Append-only tables that grow over time -Event streams or log tables -Tables where you care about recent data arrival When to skip: -Tables that are fully replaced on each load -Snapshot tables with fixed row counts -Dimension tables with slow-changing data

With timestamp column (orders.created_at):
  Query: SELECT COUNT(*) FROM orders
         WHERE created_at >= NOW() - INTERVAL '24 hours'
  Result: 48,000 (recent rows only)

Without timestamp column:
  Query: SELECT COUNT(*) FROM orders
  Result: 5,000,000 (all rows ever)

Column requirements:

Must be a timestamp or datetime type
Should represent when the row was created/ingested
Should be indexed for performance

Time Window

How far back to count rows when a timestamp column is specified. Choose based on your data load frequency:

Window	Duration	Best For
1 hour	Last 60 minutes	Real-time streaming, high-frequency events
6 hours	Last 6 hours	Hourly batch jobs, frequent updates
12 hours	Last 12 hours	Twice-daily pipelines
24 hours	Last day	Daily batch ETL (most common)
168 hours	Last 7 days	Weekly aggregates, slow-changing data

Example: Daily batch job loads orders every night at 2 AM

Time window: 24 hours
Check interval: 6 hours (runs at 2 AM, 8 AM, 2 PM, 8 PM)

Check at 8 AM:
  Counts rows WHERE created_at >= 8 AM yesterday
  Includes last night's batch + today's streaming data

Without a timestamp column, the time window setting is ignored and all rows are counted.

Check Interval

How often to run the row count check and evaluate for anomalies:

Interval	Frequency	Best For
1 hour	Every hour	Real-time monitoring, critical tables
6 hours	4x per day	Standard monitoring, daily tables
12 hours	2x per day	Less critical tables, longer time windows
24 hours	Once per day	Weekly tables, slow-changing data

Cost considerations: More frequent checks = more compute resources. Choose the interval that matches your SLA requirements.

Example check intervals for different scenarios:

Scenario: Real-time event stream
  Time window: 1 hour
  Check interval: 1 hour
  Result: Hourly checks on last hour of data

Scenario: Daily batch job
  Time window: 24 hours
  Check interval: 6 hours
  Result: 4 checks per day on last 24h of data

Scenario: Weekly report table
  Time window: 168 hours
  Check interval: 24 hours
  Result: Daily checks on last week of data

Check interval should be ≤ time window for meaningful monitoring. A 24-hour check interval with a 1-hour time window would miss most anomalies.

Time-Windowed Counting

For tables that accumulate data over time, use a timestamp column to count rows within a specific window:

Window	Use Case
1 hour	Real-time event streams
6 hours	Frequent batch loads
12 hours	Twice-daily pipelines
24 hours	Daily batch ETL (most common)
168 hours	Weekly aggregates

Time-windowed counting (orders table with created_at):

Without time window:  COUNT(*) = 5,000,000 (all time)
With 24h window:      COUNT(*) WHERE created_at >= now() - 24h = 48,000

Use time-windowed counting for append-only tables. Without it, row counts only grow, making anomaly detection less useful.

Setting Up Row Count Monitoring

Navigate to the Asset

Go to Assets and select the table you want to monitor.

Open Data Quality Tab

Click the Data Quality tab on the asset detail page, then scroll to the Row Count Monitoring section.

Create Schedule

Click Create Schedule and configure:

Table: Select the table to monitor
Timestamp column: (Optional) For time-windowed counting
Time window: How far back to count rows
Check interval: How often to check (1h, 6h, 12h, 24h)

Choose Monitoring Mode

Select your monitoring approach:Auto-Learn Mode:

Toggle Auto-learn on
Set Sensitivity (1-4, lower = more sensitive)
Wait for learning period to complete

Explicit Mode:

Toggle Auto-learn off
Set Expected min rows
Set Expected max rows

Save and Monitor

Click Create. The first check runs immediately, then continues on your configured interval.

Understanding Results

Status Indicators

Status	Meaning	Action
Healthy	Row count within expected range	None needed
Anomaly	Row count outside expected range	Investigate the cause
Learning	Collecting baseline data	Wait for learning to complete
No Data	No checks have run yet	Check will run on next interval

Anomaly Types

Anomaly	Possible Causes
Row count too low	ETL failure, data loss, filter bug, source issue
Row count too high	Duplicate load, removed filter, upstream spike
Row count zero	Complete ETL failure, wrong table, permissions

Best Practices

Choose the Right Mode

Scenario	Recommended Mode
Data patterns vary naturally	Auto-learn with sensitivity 2-3
Exact expectations known	Explicit with min/max thresholds
New table, unknown patterns	Auto-learn with sensitivity 3-4
Critical data, low tolerance	Auto-learn with sensitivity 1-2

Set Appropriate Windows

Data Pattern	Recommended Window
Real-time streaming	1 hour
Hourly batch jobs	6 hours
Daily batch jobs	24 hours
Weekly aggregates	168 hours

Start Conservative, Then Tighten

Week 1: Use auto-learn with sensitivity 3 (less sensitive)
Week 2-4: Review any anomalies, adjust if too noisy
Month 2+: Tighten to sensitivity 2 once patterns are stable

Row Count vs. Metrics

Feature	Row Count	Data Quality Metrics
Purpose	Monitor row counts	Monitor column statistics
Scope	Table-level	Column-level
ML-based	Yes (auto-learn)	Yes (anomaly detection)
Time windows	Yes	No
Explicit thresholds	Yes	Via checks

Use Row Count Monitoring for: “Did the right amount of data arrive?” Use Metrics for: “Is the data quality correct?” (nulls, duplicates, ranges)

Troubleshooting

Status shows 'Learning' for too long

Causes:

Not enough data points collected yet
Check interval is very long (weekly)

Solutions:

Wait for at least 7 data points (7 days for daily checks)
Consider switching to explicit mode if you know expected values

Too many false positive anomalies

Causes:

Sensitivity is too low (too sensitive)
Natural data variation is high
Seasonality not yet learned

Solutions:

Increase sensitivity (e.g., 2 to 3)
Allow more baseline data (30+ days)
Switch to explicit mode with wider thresholds

Missing real anomalies

Causes:

Sensitivity is too high (not sensitive enough)
Baseline includes anomalous data

Solutions:

Decrease sensitivity (e.g., 3 to 2)
Switch to explicit mode with tighter thresholds

Row count always zero with time window

Causes:

Timestamp column has no recent data
Wrong timestamp column selected
Time window too narrow

Solutions:

Verify timestamp column has data in the window
Check column data type (should be timestamp/datetime)
Widen the time window

Common Questions

Auto-learn or explicit mode, which should I pick?

Use auto-learn when row counts fluctuate naturally, AnomalyArmor builds a statistical baseline and flags deviations. Use explicit when you know the exact min and max (for example, a daily summary that should always have exactly one row).

Why configure a timestamp column?

Without one, COUNT(*) returns all rows ever, so counts only grow and anomaly detection loses meaning. A timestamp column lets AnomalyArmor count only rows inside a time window like the last 24 hours, which is what you actually want to monitor for append-only tables.

How long does auto-learn mode need before it starts alerting?

At least 7 data points. For a daily check interval that’s 7 days; for hourly it’s 7 hours. Until enough baseline accumulates, the status shows Learning and no anomalies fire.

What does a sensitivity of 2 actually mean?

It’s the multiplier on the standard deviation used to define the expected range. Expected range = mean ± (stddev × sensitivity). Lower values (1) are stricter and catch smaller drifts; higher values (3-4) tolerate more variation.

What’s the difference between row count monitoring and data quality metrics?

Row count is table-level (“did the right volume arrive?”) with ML pattern learning and time windows. Metrics are column-level (“are the values correct?”), tracking things like null percentages, distinct counts, and numeric ranges.

What typically causes a row count spike?

The most common cause is a duplicate load, where the same batch ran twice or a filter was removed upstream. Sudden drops are usually ETL failures, wrong source, or a filter change that excluded valid data. The anomaly page shows the timing so you can correlate with deploys.

What’s Next

Set Up Alerts

Get notified when row count anomalies are detected

Data Quality Metrics

Monitor column-level statistics like null percentages

Freshness Monitoring

Track when data was last updated

Report Badges

Embed row count status in dashboards

Getting Started

Core Concepts

Data Sources

Detect Schema Changes

Monitor Data Health

Coverage Tiers

Get Notified

Understand Your Data

Organize & Tag

Guides

Account & Settings

Security

Help

Downloads

Documentation Index

​Configuration Reference

​Monitoring Mode

​Auto-Learn Mode (Recommended)

​Explicit Mode

​Sensitivity

​Timestamp Column

​Time Window

​Check Interval

​Time-Windowed Counting

​Setting Up Row Count Monitoring

​Understanding Results

​Status Indicators

​Anomaly Types

​Best Practices

​Choose the Right Mode

​Set Appropriate Windows

​Start Conservative, Then Tighten

​Row Count vs. Metrics

​Troubleshooting

​Common Questions

​Auto-learn or explicit mode, which should I pick?

​Why configure a timestamp column?

​How long does auto-learn mode need before it starts alerting?

​What does a sensitivity of 2 actually mean?

​What’s the difference between row count monitoring and data quality metrics?

​What typically causes a row count spike?

​What’s Next

Set Up Alerts

Data Quality Metrics

Freshness Monitoring

Report Badges

Configuration Reference

Monitoring Mode

Auto-Learn Mode (Recommended)

Explicit Mode

Sensitivity

Timestamp Column

Time Window

Check Interval

Time-Windowed Counting

Setting Up Row Count Monitoring

Understanding Results

Status Indicators

Anomaly Types

Best Practices

Choose the Right Mode

Set Appropriate Windows

Start Conservative, Then Tighten

Row Count vs. Metrics

Troubleshooting

Common Questions

Auto-learn or explicit mode, which should I pick?

Why configure a timestamp column?

How long does auto-learn mode need before it starts alerting?

What does a sensitivity of 2 actually mean?

What’s the difference between row count monitoring and data quality metrics?

What typically causes a row count spike?

What’s Next