P-Values Don't Mean What You Think They Mean
And it's costing your team good decisions. A clear guide for data scientists and the PMs they work with.
P-Values Don't Mean What You Think They Mean And It's Costing Your Team Good Decisions
Every data scientist has been in this meeting. The experiment results are in. The PM leans forward and asks: "Is it significant?" You say "p equals 0.03." They nod. Someone writes "statistically significant ✓" in the deck and you move to launch.
The problem? Almost everyone in that room just misread what happened.
P-values are one of the most widely used and most widely misunderstood numbers in product analytics. Here's how to actually understand them and how to explain them to anyone on your team.
What a P-Value Actually Is
A p-value answers one specific question:
If there were truly no effect (no difference, no change), how likely would we be to see data this extreme just by chance?
That's it. That's the whole thing.
It is not the probability your hypothesis is correct. It is not the probability you're wrong. It is a measure of surprise under a world where nothing is happening.
The shaded region is the p-value. A small p-value means your observed result landed far into the tail surprising under a world where nothing is happening. That's all.
The Three Misinterpretations That Derail Product Decisions
1. "P = 0.03 means there's a 97% chance we're right"
This is the most common mistake, and it's understandable the framing is backwards from what people expect.
A p-value of 0.03 does not mean:
- "97% chance our feature works"
- "3% chance we're wrong"
- "The hypothesis is probably true"
It means: if the feature genuinely did nothing, there's only a 3% chance we'd see results this extreme from random variation alone.
The smoke detector analogy: A p-value is like a smoke alarm going off. If there's no fire, the alarm would only trigger 3% of the time. But the alarm going off doesn't tell you the probability there's a fire. You still need to look at how much smoke there is, whether you were cooking, and whether the building is at risk. The alarm is evidence not a verdict.
2. "P < 0.05 means ship it. P > 0.05 means kill it."
The 0.05 threshold is a convention from 1920s agricultural statistics. It was never meant to be a universal decision switch.
A p-value of 0.049 and 0.051 represent essentially identical evidence. The line between them is not the line between "real" and "noise."
What actually matters is the cost of each type of error in your specific context:
- False positive (shipping something with no real effect): What does that cost? Engineering time, user confusion, opportunity cost?
- False negative (missing something that works): How big is the opportunity you'd walk away from?
A fraud detection model and a button colour test should not use the same decision threshold. The business context always outranks the convention.
3. "Statistical significance = business significance"
With large sample sizes and most mature products have them almost anything will reach p < 0.05. A 0.1% lift in conversion rate across 10 million users is statistically significant. It is not necessarily worth three sprints of engineering work.
Statistical significance answers: Is this probably real?
It does not answer: Does this actually matter?
You always need both:
| Question | Tool |
|---|---|
| Is this real, not noise? | P-value |
| How big is the actual change? | Effect size / confidence interval |
| Is it worth acting on? | Business judgement |
How to Run the Conversation with Your PM
When experiment results come in, replace "Is it significant?" with these four questions:
- How surprised should we be if nothing was really different? (p-value)
- How big is the actual change? (effect size the number that tells you if it moves the needle)
- What's the cost of being wrong in each direction? (false positive vs false negative tradeoff)
- Does this change something meaningful for the user or the business?
A p-value is a useful signal in that conversation. It is not the conversation.
The One-Sentence Summary
A p-value tells you how surprising your data is under a world where nothing is happening it does not tell you how likely your hypothesis is to be true, and it should never be the sole basis for a product decision.
This question comes up in DS interviews at Google, Meta, and Stripe not because interviewers want you to recite the definition, but because how you explain it to a PM reveals whether you actually understand it. Practice it on PrepGraph.