Your ML Model Passed the Test. Here's Why It Failed in Production.
Training-serving skew, distribution shift, the feedback loop — and what to actually monitor.
Your model scored 94% accuracy in evaluation. You shipped it. Six months later, someone notices a metric drifting. You dig in. The model was already wrong for months. It just didn't tell you.
This is the most common failure pattern in production ML. Not catastrophic crashes — the API keeps returning 200s, predictions keep flowing, no alarm fires. Silent degradation. The model is confidently wrong and nothing in your system is designed to catch that.
The first way models fail: training-serving skew
Training-serving skew happens at the moment of deployment, not months later. It's the gap between how features are computed during training and how they're computed at inference time.
The canonical version: your team trains a purchase propensity model using days_since_last_order as a feature. During training, this is computed from a warehouse snapshot taken nightly at midnight. In production, the feature comes from a real-time pipeline that doesn't include orders completed in the last 15 minutes due to replication lag. For most users the difference is irrelevant. But for users who placed an order 10 minutes ago, training says days_since_last_order = 0 and serving says days_since_last_order = 90. You've trained on one thing and are predicting on another.
Skew is almost never dramatic. It's not a feature returning null where you expect a number — that raises an exception. It's two systems that both work, computing the same concept slightly differently. The model produces numbers. The numbers are just wrong.
How to catch it: compute your feature distributions separately in training and in production. If days_since_last_order had a median of 30 in training and a median of 45 in production, something is different. The check most teams skip is verifying the computation logic, not just the resulting distributions — because two different code paths can produce similar-looking distributions while disagreeing on precisely the users your model cares about most.
The second way models fail: the world changes
Distribution shift is what happens after deployment. Your model was correct when it shipped. Then the inputs started looking different.
There are two kinds, and conflating them causes bad decisions.
Covariate shift: the distribution of input features P(X) changes, but the true relationship P(Y|X) stays the same. Your fraud model was trained when 80% of transactions came from desktop browsers. Mobile transactions are now 60% of volume. The features look different — screen dimensions, session lengths, geolocation precision — but fraud is still fraud. The model may degrade because it saw few mobile transactions in training and its decision boundaries don't generalize well, even though the underlying fraud patterns are real. Fix: retrain with representative mobile data.
Concept drift: the relationship P(Y|X) changes. The features haven't changed. The patterns that used to predict the label no longer do. This is harder.
The clearest example is spam detection. In 2022, spam emails had certain patterns — all-caps subject lines, specific link structures, suspicious sender domains. Spammers are adaptive. By 2024, the inputs that used to reliably predict "spam" predict it 30% less often, because spammers optimized around your features. Your model has high confidence in its predictions. The predictions are wrong. And you won't see this in your monitoring unless you're looking at prediction-vs-label agreement using fresh human labels, which most teams don't collect continuously.
Concept drift is the failure mode that requires the most uncomfortable operational response: you need fresh labeled data, regularly, as an ongoing cost. Most teams treat labeling as a one-time expense at project launch. Concept drift is what tells you it isn't.
Why your offline metrics don't predict any of this
Your test set was drawn from the same time window and distribution as your training data. It tells you whether your model learned the training distribution correctly. It tells you almost nothing about whether the model will work on data it sees six months after deployment.
The starkest illustration: COVID-19. Every model trained on behavior before March 2020 became wrong, some of them dramatically, within weeks. E-commerce demand forecasting predicted inventory needs based on pre-pandemic buying patterns — models had never seen gym equipment and home office furniture purchased at that scale. Travel recommendation systems confidently suggested destinations. Churn models trained on a stable economy predicted retention rates that were structurally impossible as millions of users lost income.
None of these models were poorly designed. They were correctly trained on the data they had. The data was from a different world.
Your evaluation set cannot catch this because it was drawn from the same world as your training data. The only thing that catches distribution shift is periodically re-evaluating on fresh labeled data — new labels from the current time period. Most teams don't build this because it requires ongoing labeling investment. Most teams also get surprised when models degrade.
The problem that makes all of this worse: the feedback loop
When your model's predictions influence what data you collect, your training data becomes contaminated by your own decisions.
Search ranking is the most studied version. You rank result A above result B. Users are more likely to click A, partly due to its position — this is position bias, extensively measured. Your model observes many clicks on A and few on B. It learns A is more relevant. It ranks A higher. Loop closed.
If A is genuinely more relevant, this is fine. If A ranked higher due to a training artifact or early randomness, the feedback loop reinforces the mistake permanently. B will never naturally resurface unless you deliberately force it.
Fraud scoring has a version that's less obvious but equally damaging. You only get ground-truth labels on transactions you decided to investigate. Your model scores transactions low-risk → you don't review them → you never get labels on them → you retrain without knowing whether those low-risk predictions were correct. Your model learns from a dataset biased toward cases it was already confident about.
The operational response is deliberate exploration: randomly route a small fraction of decisions that override the model. Hold out some transactions for random review regardless of model score. Force your ranking system to occasionally surface lower-ranked results. This costs short-term metric performance. It is the only mechanism that keeps the feedback loop from becoming self-sealing.
What to actually monitor
Most teams monitor model accuracy. This requires labels, which requires either human review (expensive and lagged) or ground-truth events that take days or weeks to materialize.
The thing you can monitor immediately, at low cost, is distributions.
Input feature distributions. For every feature entering your model, track its distribution over time — mean, variance, null rate, the fraction of values outside the training range. A feature whose distribution has shifted significantly is a warning sign even before you have updated labels. The Population Stability Index (PSI) is the standard measure: below 0.1 is stable, 0.1–0.2 warrants investigation, above 0.2 indicates significant shift that almost always affects model performance.
Prediction score distributions. Track the distribution of your model's outputs over time. If your fraud model's average score was 0.12 in January and is 0.31 in June, something changed — either the world changed or a feature changed. The model sees more risk. You need to know whether that's real or an artifact.
Calibration. A well-calibrated model assigns probabilities that match real frequencies. If it says 80% probability, roughly 80% of those cases should actually be positive. Track this over time. A model can maintain its rank-ordering ability (AUROC stays flat) while becoming miscalibrated — it's sorting correctly but its scores no longer mean what they used to. This matters for any decision using a threshold, which is most decisions.
Business metrics as lagged ground truth. Conversion rate, churn rate, fraud rate, click-through rate. These are ultimately what the model drives, but they're the last thing to move. By the time a business metric degrades visibly, the model has often been wrong for weeks.
The monitoring stack that works: input and prediction distributions as continuous early warning, calibration tracked weekly, business metrics as final confirmation. Most teams have the last one and skip the first two.
Shadow deployment before you ship
Before you replace a live model with a new one, run them side by side.
Shadow deployment means routing production traffic through both models but only serving the live model's predictions to users. You collect what the new model would have predicted on real requests and compare. No user is affected. You see exactly where the two models agree and disagree on real data, before any decision you can't take back.
This catches training-serving skew immediately — you're running the new model on production data, not evaluation data. It surfaces systematic disagreements that may indicate a problem with either model. It gives you real latency numbers, not benchmarks.
The step most teams skip: actually investigating the high-disagreement cases. Teams run shadow mode, look at aggregate accuracy numbers, and ship if the new model looks marginally better. The cases where old says 0.9 and new says 0.1 — or vice versa — are the highest-information cases in your entire dataset. Those are the ones worth reviewing with domain experts before launch.
What a good answer looks like in an interview
The standard question is: We just shipped a new recommendation model. How do you know it's working?
"Track CTR and engagement" is not wrong, but it won't distinguish you.
The answer that lands has three parts.
First, separate offline from online. Your pre-launch evaluation tells you the model learned correctly from training data. Your post-launch measurement tells you whether it works on real users in the real world. These are different questions answered by different instruments. Treating offline metrics as a proxy for production performance is the root of most silent failures.
Second, acknowledge the latency problem. Engagement metrics tell you something is working or not, but they're lagged. If you're optimizing for long-term retention rather than session engagement, your key metric might not move for weeks after deployment. You need monitoring that catches problems earlier — input distributions, prediction distributions — as leading indicators. By the time your business metric degrades 10%, the model has often been wrong for a long time.
Third, name the feedback loop risk. A recommendation model optimizing for engagement will, over time, learn to serve the same popular content repeatedly. High engagement in the short term. Low diversity and discovery in the medium term. User dissatisfaction in the long term — and you won't see it coming in your aggregate CTR numbers until it's already happened. The right monitoring includes a diversity metric: what fraction of the content catalog is the model actually surfacing, and how is that changing week over week?
Most candidates answer this question with a metric list. The answer above answers it with a mental model of how models fail, which is what the question is actually testing.