Your A/B Test Said +3%. Here's Why the Launch Flopped.
Novelty effects, SUTVA violations, metric selection — and why your experiment measured something different from what you thought.
Your experiment ran for two weeks. Statistical significance hit 95%. The primary metric moved 3.2% in the right direction. You launched.
Six weeks later, the numbers are flat. The post-launch data contradicts the experiment, and nobody can explain why.
This happens more often than most teams admit. The experiment wasn't necessarily wrong. It just measured something different from what you thought it measured.
The first way tests fool you: novelty
When users encounter something new, a redesigned flow, a different recommendation algorithm, a new layout, they often engage with it more intensely than they will once the newness wears off. Novelty triggers attention. Attention looks like engagement. Engagement looks like lift.
The problem is that a two-week experiment window often captures the novelty spike, while the post-launch period reveals the steady state. Those aren't the same number.
A redesigned checkout flow might show a 4% conversion lift in the experiment because users are reading it more carefully, the new layout demands attention in a way the familiar one doesn't. Run the same experiment for eight weeks and the lift often compresses as users habituate and revert to their default behavior patterns.
The effect is stronger for engagement metrics than conversion metrics, and stronger for frequent users than occasional ones. A user who visits your product daily will habituate within days. A user who visits monthly might still be in the novelty window at the end of your experiment. Your aggregate metric is a blend of both, and that blend shifts as novelty wears off.
The second way tests fool you: the control group isn't really isolated
Most A/B testing frameworks assume that what happens to treatment users doesn't affect control users. Technically this is the Stable Unit Treatment Value Assumption, SUTVA. It's built into the math that computes your p-values. When it breaks, those numbers are wrong in ways that the significance calculation doesn't surface.
SUTVA breaks in predictable places.
Social products are the clearest case. If you test a new notification system on 50% of users, and those users start posting more and messaging more, that activity reaches their connections in the control group. Control users receive more content and more reasons to engage, because of what's happening on the treatment side. The control group is contaminated. The measured effect is wrong: sometimes understated, sometimes artificially amplified, depending on the mechanism.
Shared infrastructure is less obvious but equally real. If the treatment involves heavier server-side computation, a more expensive ranking model, a richer personalization layer, and that computation runs on shared infrastructure, treatment users consuming more resources can slow response times for control users. You measure a real lift for the treatment group while control users experience slightly worse performance. The gap between groups looks larger than it is.
Marketplace and supply systems have a third version. Treatment-side changes to pricing or matching affect what control users see, because both groups draw from the same underlying pool of supply.
The question worth asking before you run: is there any mechanism by which what happens to treatment users could affect control users? On a social product, the honest answer is almost always yes.
The third way tests fool you: the metric moved, the thing behind it didn't
The clearest version: you redesign a homepage to be more visually prominent. CTR goes up 8%. You measure CTR because it's immediate and easy. What you actually care about is whether users find what they're looking for and take a valuable action. A more clickable homepage can increase CTR while decreasing satisfaction, more clicks, less value found, users leaving slightly more frustrated than before. The metric moved in the direction you wanted. The behavior it was supposed to represent moved the other way.
This happens because the relationship between proxy metrics and goal metrics isn't fixed. It holds under certain conditions and breaks under others. When you change the design or the ranking, as experiments do, you can sever the relationship you were relying on without realizing it.
Engagement metrics are especially prone to this. Time on page, session length, click depth, these correlate with user satisfaction under normal conditions. Design changes that add friction or confusion can increase all of them while making the product worse. The experiment reports a win. The quarterly review asks why retention is flat.
A useful question before running: if this metric moves in the direction I want and nothing else changes, is that unambiguously good? If the honest answer is "it depends," the metric probably isn't specific enough.
The fourth way tests fool you: when you looked
A 95% significance threshold means you expect a false positive 5% of the time when there's no real effect, but only if you look at the results exactly once, at the end, with a pre-specified sample size.
Most teams check the dashboard daily. When significance is reached, they call the experiment. Sometimes they extend it when the numbers aren't moving the right way.
Every time you look and make a decision based on what you see, you use some of your false positive budget. Check a 20-day experiment daily and stop when significance is reached: your actual false positive rate is materially higher than 5%, even though your threshold said 5%. The significance calculation assumes a single look. Multiple looks require sequential testing methods, always-valid p-values, sequential probability ratio tests, that most dashboards don't implement by default.
This compounds with volume. Teams running many small experiments and calling them at first significance will see a meaningful fraction of their "wins" fail to replicate at launch. The false positives look exactly like the real results, until they don't.
Why the launch is a different experiment than the test
Even when none of the above apply, the launch can still contradict the test, because the launch is structurally different.
In the experiment, 50% of users saw the treatment. At launch, 100% do. If the feature has network effects, infrastructure effects, or interactions with other features, these behave differently at full deployment than at 50% penetration.
The common version: a new feed ranking algorithm tested on half the network affects what half the users produce and consume. At 100%, every post and every feed is shaped by the new algorithm. The system equilibrium is different. The experiment measured the algorithm in a half-deployed world. You shipped it to a fully-deployed world. Those aren't the same system.
What the interview question is actually testing
Tell me about a time your experiment results didn't match expectations after launch.
The weaker answer describes process: "We investigated and found the root cause."
The answer that distinguishes you names the mechanism before needing to find it, and explains why it's structurally predictable. Novelty effects are always present for engaged users on short experiments. SUTVA always breaks on social products. CTR is always a leaky proxy for satisfaction. Peeking always inflates false positive rates. These aren't surprises discovered after the fact. They're known failure modes that experienced practitioners check before launch, not after.
What the question is really asking: is your mental model of how experiments fail sophisticated enough that you'd catch the problem before it costs you a launch?