Data Drift Alerting in Production: What to Monitor and When to Act

Data Drift Alerting in Production: What to Monitor and When to Act

A model that passes all your offline tests will still degrade in production. The data changes. The world changes. Features that were stable during training shift over time. At Intuition Machines, I built and maintained data drift alerting for production classification models. This post is about what actually matters to monitor, how to structure the alerting, and how to use the signals to make rollback decisions.

Why Drift Alerting Fails in Practice

Most teams start with statistical tests on input features: KS tests, PSI (Population Stability Index), chi-squared for categoricals. These have two problems:

  • They alert too early. Any live system has day-of-week patterns, seasonal patterns, and event-driven spikes that look like drift but are not. An alert that fires every Monday is noise, not signal.
  • They alert on the wrong thing. Drift in an irrelevant feature is harmless. What matters is drift in features the model depends on heavily. A feature with high SHAP importance that drifts is a real problem. A low-importance feature that drifts is not.

The right starting point is not "is the input distribution stable?" but "is model performance stable?" Performance degradation is the outcome we care about. Input drift monitoring is useful when it can predict performance degradation before it becomes visible in metrics.

Monitoring Model Performance with Delayed Ground Truth

In our system, ground truth labels arrived with a delay: the model made a prediction at time T, and the true label was available at T+N days. This is common in fraud detection, churn prediction, and clinical risk scoring.

We stored model predictions and ground truth in ClickHouse, which was already used for the production event stream. ClickHouse's columnar storage made it efficient to run aggregate metric queries over large windows.

The core monitoring query looked like this: 

SELECT
    toStartOfDay(prediction_time) AS day,
    countIf(label = 1 AND prediction = 1) AS true_positives,
    countIf(label = 1 AND prediction = 0) AS false_negatives,
    countIf(label = 0 AND prediction = 1) AS false_positives,
    countIf(label = 0 AND prediction = 0) AS true_negatives,
    true_positives / (true_positives + false_negatives) AS recall,
    true_positives / (true_positives + false_positives) AS precision
FROM predictions
WHERE prediction_time >= now() - INTERVAL 30 DAY
  AND label IS NOT NULL  -- only rows with observed ground truth
GROUP BY day
ORDER BY day

We ran this daily and plotted recall and precision over a rolling 30-day window. A sustained downward trend (3+ consecutive days outside the baseline interval) triggered a review.

Input Drift: Monitoring the Right Features

Once performance monitoring was in place, we added input drift monitoring as an early warning system. But rather than monitoring all features, we prioritized by SHAP importance.

We computed SHAP values on a weekly holdout sample and ranked features by mean absolute SHAP. The top 10 features got drift alerts. The rest were logged but not alerted on.

For each top feature, we computed PSI (Population Stability Index) comparing the current week's distribution to the baseline training distribution: 

def psi(expected, actual, buckets=10):
    breakpoints = np.linspace(0, 100, buckets + 1)
    expected_perc = np.histogram(expected, bins=np.percentile(expected, breakpoints))[0] / len(expected)
    actual_perc = np.histogram(actual, bins=np.percentile(expected, breakpoints))[0] / len(actual)
    # clip to avoid log(0)
    expected_perc = np.clip(expected_perc, 1e-6, None)
    actual_perc = np.clip(actual_perc, 1e-6, None)
    return np.sum((actual_perc - expected_perc) * np.log(actual_perc / expected_perc))

PSI below 0.1 is stable. PSI 0.1 to 0.2 warrants monitoring. PSI above 0.2 is significant drift. We used these thresholds as soft alerts: a Slack notification for PSI above 0.1 on a top-10 feature, a PagerDuty alert for PSI above 0.2.

When to Roll Back

The hardest part of production ML monitoring is the rollback decision. Rollback is disruptive. A model that is degrading might stabilize. A model that triggers alerts might be responding correctly to a real change in the underlying phenomenon.

We used a simple decision framework:

  1. Performance degradation is the trigger, not drift. Drift alone is not sufficient to roll back. If recall is stable and PSI is high, the model is handling the drift. If both are bad, the drift is causing degradation.
  2. Check the error composition before rolling back. Is the degradation uniform, or is it concentrated in a specific segment? If it is a segment issue, a targeted fix (segment-specific threshold, segment-specific model) is better than a full rollback.
  3. Compare against the previous version, not just baseline. If the previous model version is in the artifact registry with its performance metrics, you can quickly estimate whether rolling back would actually improve things.

We maintained the previous two model versions in the registry at all times. Rollback was a one-command operation in the deployment system.

What Not to Monitor

Monitoring everything is as bad as monitoring nothing. A dashboard with 50 metrics creates alert fatigue. We explicitly excluded:

  • Low-importance features (below the top 10 by SHAP)
  • Metrics with high natural variance (daily prediction volume, which varies by day of week)
  • Metrics where we had no actionable response (data quality issues upstream of the pipeline that required a separate team to fix)

Every alert should have a clear owner and a clear response procedure. If you cannot define what you would do when the alert fires, the alert should not exist.

Conclusion

Effective drift alerting in production is not about monitoring everything. It is about monitoring performance directly, using input drift as a leading indicator only for features the model actually depends on, and having a clear framework for when drift warrants action. The infrastructure is secondary: ClickHouse worked well for us, but the same logic applies in BigQuery, Redshift, or a simple Postgres table.

If you are setting up production monitoring for a classification system and want to discuss the approach, feel free to reach out.