Bayesian Updating: The Intuition Behind It
Three mental models that make Bayesian thinking click — the rope, the GPS, and the probability of being better.
1. The Tug-of-War Intuition
Imagine a rope. On one side, you have your Prior (your existing knowledge). On the other side, you have your Evidence (new data). The Posterior is where the knot on the rope ends up after they pull against each other.
- If your Prior is strong (e.g., "I've run 100 tests and conversion is always 5%"), the knot barely moves even if you see a weird spike in data today. You assume the spike is just noise.
- If your Evidence is strong (e.g., "I just saw 10,000 users convert at 10%"), it pulls the knot far toward the new data, even if your Prior was different.
The Result: Your new belief is always a compromise based on the quality and quantity of both sides.
2. Why the Tech Product "Daily Update" Works
Think of Bayesian updating like using a GPS.
The Frequentist approach is like looking at a paper map, calculating a 50-mile route, and refusing to look at the road again until exactly 50 miles have passed. You might have missed your exit 40 miles ago, but you won't know until the "test" is over.
The Bayesian approach is like Google Maps. It has a Prior (the planned route), but as you drive, it receives Evidence (your GPS coordinates). It updates your Estimated Time of Arrival (the Posterior) every second.
If the GPS sees you take a turn that is 99% likely to be a shortcut, it doesn't wait for you to finish the whole trip to tell you that you'll arrive early. It updates the ETA immediately.
This is why Meta could stop tests early: the "ETA" for the test results became clear long before the "trip" was scheduled to end.
3. The "Probability of Being Better"
In a standard A/B test, you usually get a p-value — which is confusing (it's the probability of seeing these results if there was actually no difference).
In the Bayesian A/B testing Meta used, the output is much more human:
"There is a 97% probability that B is better than A."
This is intuitive because:
- It's Cumulative: Every new user who clicks adds a tiny bit of weight to one side of the scale.
- It's Actionable: As a Product Manager, if the math says there is a 95% chance B is better on Day 4, why wait until Day 14? You've already learned enough to move on.
The Core Difference in One Sentence
Frequentist: "I know nothing except what this specific data tells me."
Bayesian: "I'm going to use what I already know to make sense of this new data faster."
That's why updating is the secret sauce for speed.