> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anomalyarmor.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema Drift Detection

> Detect and track schema changes over time to prevent data pipeline failures

<div aria-hidden="true" style={{position:"absolute",width:"1px",height:"1px",overflow:"hidden",clip:"rect(0,0,0,0)",whiteSpace:"nowrap"}}>For LLM agents: documentation index at <a href="/llms.txt" tabIndex={-1}>/llms.txt</a>, full text at <a href="/llms-full.txt" tabIndex={-1}>/llms-full.txt</a>. Append .md to any page URL for plain markdown.</div>
Schema drift occurs when your database structure , intentionally or accidentally. AnomalyArmor detects drift automatically and alerts you before it causes downstream failures.

<Frame caption="Schema diff visualization showing column changes">
  <img src="https://mintcdn.com/anomalyarmor/CZXBGa_D1aE9spAI/images/diagrams/schema-diff-light.svg?fit=max&auto=format&n=CZXBGa_D1aE9spAI&q=85&s=69d71ee100b11f9a5179e71a773d84d5" alt="Before/after comparison showing column added, removed, and modified" className="block dark:hidden" width="800" height="340" data-path="images/diagrams/schema-diff-light.svg" />

  <img src="https://mintcdn.com/anomalyarmor/CZXBGa_D1aE9spAI/images/diagrams/schema-diff-dark.svg?fit=max&auto=format&n=CZXBGa_D1aE9spAI&q=85&s=b35cf8efb0ed828427eda9e7a68d11bf" alt="Before/after comparison showing column added, removed, and modified" className="hidden dark:block" width="800" height="340" data-path="images/diagrams/schema-diff-dark.svg" />
</Frame>

## What is Schema Drift?

Schema drift is any change to database structure:

| Column             | Before (Discovery T1) | After (Discovery T2) | Change                                            |
| ------------------ | --------------------- | -------------------- | ------------------------------------------------- |
| id                 | `integer`             | `integer`            | No change                                         |
| user\_id           | `integer`             | `integer`            | No change                                         |
| status             | `varchar`             | --                   | **Removed** (or renamed)                          |
| order\_status      | --                    | `varchar`            | **Added** (possibly renamed from `status`)        |
| total              | `decimal`             | `decimal`            | No change                                         |
| shipping\_date     | `timestamp`           | --                   | **Removed** (or renamed)                          |
| shipping\_estimate | --                    | `timestamp`          | **Added** (possibly renamed from `shipping_date`) |
| delivery\_fee      | --                    | `decimal`            | **Added**                                         |
| created\_at        | `timestamp`           | `timestamp`          | No change                                         |

## Types of Schema Drift

### Breaking Changes (High Risk)

Changes that will cause immediate failures:

| Change                       | Example                   | Impact                         |
| ---------------------------- | ------------------------- | ------------------------------ |
| **Column Removed**           | `shipping_status` deleted | Queries referencing it fail    |
| **Table Removed**            | `legacy_orders` dropped   | All queries to table fail      |
| **Column Renamed**           | `status` → `order_status` | Queries using old name fail    |
| **Incompatible Type Change** | `varchar` → `integer`     | Type mismatches, cast failures |

### Non-Breaking Changes (Low-Medium Risk)

Changes that usually don't cause immediate failures:

| Change                     | Example                        | Impact                                    |
| -------------------------- | ------------------------------ | ----------------------------------------- |
| **Column Added**           | New `delivery_fee` column      | Existing queries work; new data available |
| **Table Added**            | New `order_items` table        | No impact on existing queries             |
| **Compatible Type Change** | `varchar(50)` → `varchar(100)` | Usually transparent                       |
| **Constraint Added**       | New unique constraint          | May affect inserts, not selects           |

## How Drift is Detected

AnomalyArmor compares consecutive discovery runs:

| Column        | Run #1 (Baseline) | Run #2 (Comparison) | Result      |
| ------------- | ----------------- | ------------------- | ----------- |
| id            | Present           | Present             | No change   |
| user\_id      | Present           | Present             | No change   |
| status        | Present           | Missing             | **Removed** |
| total         | Present           | Present             | No change   |
| order\_status | Missing           | Present             | **Added**   |
| created\_at   | Present           | Present             | No change   |

**Detected drift**: Column `status` removed, column `order_status` added (possibly a rename, investigate).

### Detection Timing

| Discovery Schedule | Max Detection Delay |
| ------------------ | ------------------- |
| Every 15 minutes   | 15 minutes          |
| Hourly             | 1 hour              |
| Daily              | 24 hours            |

<Tip>
  For production databases, hourly discovery provides a good balance of timeliness and resource usage.
</Tip>

## Viewing Schema Drift

### Asset Schema History

1. Navigate to **Assets**
2. Click on a table
3. Select **Schema History** tab

You'll see:

<Frame caption="Schema change history showing column additions, removals, and modifications over time">
  <img src="https://mintcdn.com/anomalyarmor/vEjoFDGoOMk1yENS/images/screenshots/schema-history-timeline.png?fit=max&auto=format&n=vEjoFDGoOMk1yENS&q=85&s=8a3f6347d4f62678ce0d346a2832292c" alt="Schema change timeline displaying multiple changes with timestamps and details" width="1902" height="842" data-path="images/screenshots/schema-history-timeline.png" />
</Frame>

### Change Details

Click any change to see details:

* **Before state**: Previous column definition
* **After state**: New column definition (or "Removed")
* **Detection time**: When the change was discovered
* **Discovery run**: Which scan found it

## Schema Drift Alerts

### Setting Up Alerts

Create rules to notify you of drift:

| Field            | Value                                       |
| ---------------- | ------------------------------------------- |
| **Name**         | Production Breaking Changes                 |
| **Event**        | Schema Change Detected                      |
| **Data source**  | `production-postgres`                       |
| **Schema**       | `public`, `analytics`                       |
| **Change type**  | Column Removed, Table Removed, Type Changed |
| **Destinations** | Slack, PagerDuty                            |

### Alert Content

Schema drift alerts include:

```
⚠️ Schema Change Detected

Asset: production.public.orders
Changes detected:
  • Column removed: status (varchar)
  • Column added: order_status (varchar)

Detected: January 15, 2024 at 8:00 AM UTC
Discovery Run: #1234

[View Details] [View Asset]
```

## Common Drift Scenarios

### Scenario 1: Column Rename

**Symptom**: Column removed + column added with similar name

```
Detected changes:
- Column removed: user_email
- Column added: email_address

This is likely a rename.
```

**Action**:

1. Confirm it's a rename (ask the upstream team)
2. Update all downstream references
3. Verify queries work with new name

### Scenario 2: Unannounced Migration

**Symptom**: Multiple columns changed in related tables

```
Detected changes in users:
- Column type changed: phone (varchar → jsonb)
- Column added: phone_verified (boolean)

Detected changes in user_profiles:
- Column removed: phone_number
```

**Action**:

1. Identify the migration (check with upstream team)
2. Review impact on your queries
3. Update affected models

### Scenario 3: Table Dropped

**Symptom**: Table no longer exists

```
Asset removed: public.legacy_orders
Last seen: January 14, 2024

This could be:
- Intentional removal
- Renamed table
- Permission change
- Wrong database connected
```

**Action**:

1. Verify if intentional
2. Update any queries referencing it
3. Check if data was migrated elsewhere

### Scenario 4: Type Change

**Symptom**: Column data type modified

```
Column: products.price
Type change: integer → decimal(10,2)
```

**Action**:

1. Review aggregations (SUM, AVG) for precision changes
2. Check comparisons and arithmetic
3. Verify no type casting issues

## Managing Expected Drift

### Planned Schema Changes

When you know changes are coming:

1. **Communicate**: Tell your team about upcoming changes
2. **Prepare**: Update downstream code in advance (staged rollout)
3. **Execute**: Make the schema change
4. **Verify**: Confirm AnomalyArmor detects it
5. **Acknowledge**: Mark alerts as expected

### Disabling Rules During Migrations

For large migrations:

1. Go to **Alerts → Rules**
2. Toggle OFF relevant rules
3. Perform migration
4. Review detected changes
5. Toggle rules back ON

<Warning>
  Don't forget to re-enable rules after migration! Set a calendar reminder.
</Warning>

## Drift Prevention

### Schema Versioning

Coordinate schema changes with version control:

* Use migration tools (Flyway, Alembic, Atlas)
* Review migrations before applying
* Test in staging first

### Change Communication

Establish processes:

* Schema change announcements
* Dependency mapping
* Impact assessment before changes

### Backward Compatibility

When changing schemas:

* Add new columns instead of modifying existing
* Keep old columns during transition periods
* Use views to maintain compatibility

## Troubleshooting

<AccordionGroup>
  <Accordion title="Drift detected but change was intentional">
    This is expected behavior. AnomalyArmor doesn't know intent. It reports all changes. Consider:

    1. Acknowledging the alert
    2. Documenting the change in your records
    3. Disabling rules during planned migrations
  </Accordion>

  <Accordion title="Drift not detected">
    1. **Check discovery schedule**: Has discovery run since the change?
    2. **Check scope**: Is the table in a monitored schema?
    3. **Check permissions**: Can AnomalyArmor user see the table?
    4. **Run manual discovery**: Trigger discovery to check now
  </Accordion>

  <Accordion title="Too many drift alerts">
    1. **Filter by change type**: Alert only on breaking changes (removed columns/tables)
    2. **Exclude dev/test**: Don't monitor volatile environments
    3. **Use patterns**: Exclude `*_temp`, `*_backup` tables
  </Accordion>

  <Accordion title="False positive - rename detected as remove+add">
    This is expected. AnomalyArmor can't automatically determine if a remove+add pair is a rename. You should:

    1. Investigate the pair manually
    2. Document the rename in your records
    3. Update downstream references
  </Accordion>
</AccordionGroup>

## Best Practices

1. **Monitor production closely**: Run frequent discovery, alert immediately
2. **Differentiate environments**: Different rules for prod vs. dev
3. **Route by change type**: Breaking changes → PagerDuty; additions → Email
4. **Review history regularly**: Check drift patterns to identify risky areas
5. **Coordinate changes**: Communicate schema changes before making them

## Common Questions

### What counts as a breaking schema change?

Column removals, table removals, column renames, and incompatible type changes (such as `varchar` to `integer`) will break queries immediately. Additions and compatible type widenings (`varchar(50)` to `varchar(100)`) usually don't.

### How do I avoid alert storms during a planned migration?

Disable the relevant alert rules in **Alerts → Rules** before the migration, let the changes flow through, then toggle the rules back on. Set a calendar reminder, leaving rules off is a common mistake.

### Can AnomalyArmor detect renames automatically?

No. A rename shows up as a column removed plus a column added with a similar name. The UI flags the pair as a possible rename, but confirming intent still requires a human to check with the upstream team.

### Why did drift get detected for a change I made intentionally?

AnomalyArmor reports all structural changes, it has no way to know which ones you planned. Acknowledge the alert, document the change, and optionally disable the rule during known migration windows.

### What should I do first when a breaking change is detected in production?

Open the asset's schema history to see exactly what changed and when. Then check downstream dependencies, such as views, models, and dashboards, that reference the affected column or table, and coordinate fixes before more queries fail.

### Does detection work on views and materialized views, not just tables?

Yes. Discovery catalogs tables, views, and their underlying columns, so drift on views is detected the same way. See [Asset Discovery](/schema-monitoring/asset-discovery) for what's captured.

## Related Topics

<CardGroup cols={2}>
  <Card title="Schema Monitoring Overview" icon="eye" href="/schema-monitoring/overview">
    Broader schema monitoring capabilities
  </Card>

  <Card title="Alert Rules" icon="bell" href="/alerts/alert-rules">
    Configure drift alerts
  </Card>

  <Card title="Schema Changes Concept" icon="book" href="/concepts/schema-changes">
    Deep dive into change types
  </Card>

  <Card title="Asset Discovery" icon="magnifying-glass" href="/schema-monitoring/asset-discovery">
    How assets are discovered
  </Card>
</CardGroup>