Your metrics might be lying to you. Simpson's Paradox is why.

A cool DS concept with a direct relevance to the PM function.

Marjuk Ahmad · 2026-04-30 · 8 min read

Simpson's Paradox illustrated with Berkeley admissions Aggregate admission rates suggest bias against women, but department-level rates show women admitted at equal or higher rates in each department. Aggregate University-wide admit rate Men 44% admitted Women 35% admitted Looks like bias against women By department Same data, segmented 62% 82% Dept A Easy admits 63% 68% Dept B Mid selectivity 6% 7% Dept F Selective, women apply more The reversal Women were admitted at higher rates in every department. The aggregate looked biased because women applied more to selective departments.

Today is the time to think about Simpson's Paradox. A cool DS concept — but with a direct relevance to the PM function.

The concept: a trend that appears in grouped data disappears or reverses when you look at the subgroups. The classic example is UC Berkeley admissions data from the 1970s. The university appeared to be biased against women in aggregate, but when they broke it down by department, women were being admitted at higher rates than men in almost every department. The paradox came from which departments women were applying to.

The PM angle is relevant here:

Once Simpson's Paradox is in your head, you start seeing how aggregate metrics get gamed — not always maliciously, but structurally. A team optimises for the headline number, the headline number moves, the underlying user experience stays flat or gets worse. "When a measure becomes a target, it ceases to be a good measure." Simpson's is what makes Goodhart's possible. The aggregate gives you somewhere to hide.

Here are a few more PM-adjacent case studies of the same phenomenon.

The Cohort Mix Problem

Every PM has lived this without naming it. You ship a feature, retention goes up, you celebrate. Then six weeks later someone in a review meeting asks why power users are churning, and you realise the aggregate retention number was masking it. Your new casual users were sticky enough to drag the average up while your most valuable cohort was quietly leaving. This is Simpson's Paradox wearing a hoodie. The mix shifted, and the mix shift hid the truth. The fix isn't more dashboards — it's a discipline of never reporting a metric without its cohort.

Survivorship in Funnels

Conversion rate improved 4% quarter-over-quarter. Cause for celebration. Except the top of the funnel got smaller and harder, so the people reaching checkout were a more committed, pre-filtered group. The funnel didn't get better — the population got narrower. Same math, different costume. The lesson generalises: any time a ratio improves, ask whether the numerator changed or the denominator did.

The Aggregate Trap in A/B Tests

You run a test, it ships positive, +2% on the north star. You roll out. Three months later revenue from your top decile of users is down. What happened? The treatment helped the bottom 80% a little and hurt the top 20% a lot, but the top 20% drives a disproportionate share of revenue. The test was positive on users and negative on dollars. Average treatment effect is one of the most misleading numbers in the PM toolkit, and most experimentation platforms make it the default view.

The Segmentation Reflex

The takeaway isn't "don't trust data." It's that the first question on any metric should be: what is this hiding? Good PMs build a reflex. Every chart gets cut by user tier, by tenure, by platform, by geo — before it gets believed. The Berkeley story works because the dean almost got sued before someone bothered to look at the departments. Most product orgs are one segmentation away from the same embarrassment.