Base Rate Fallacy Quiz: Can You Beat the Denominator?
Doctors, judges, and hiring managers make the same mistake daily: they stare at the evidence in front of them and forget to ask how common the thing was to begin with. Twelve scenarios — see if you can defend the most ignored number in statistics: the base rate.
0 / 12 answered
Q1
After interviewing candidate Zheng, the hiring manager says: “He comes across exactly like a top salesperson — confident, articulate, magnetic. I bet he'll be top-10% here.” To judge whether that prediction is reliable, which piece of data is the base rate he's ignoring?
Q2
Kahneman & Tversky's classic (1973): subjects were told a description was drawn at random from 70 engineers and 30 lawyers. The description: “Dick, 30, married, no children. High ability, high motivation, well liked by colleagues.” — deliberately zero diagnostic information. What should P(Dick is an engineer) be?
Q3
A friend describes his neighbor: “Introverted, wears plaid shirts, spends weekends on open-source projects.” Your town has about 200 programmers and 20,000 non-programmers. Which is more likely — he's a programmer, or he's one of the non-programmers who happens to fit that picture?
Q4
A disease has 1% prevalence. The test has 90% sensitivity (90% of patients test positive) and a 9% false-positive rate (9% of healthy people wrongly test positive). You test positive. Your probability of actually having the disease is closest to —
Q5
A bank's fraud model is “99% accurate.” Roughly 10 in every 100,000 transactions are actual fraud. The system just flagged a transaction. A junior analyst says: “The model's 99% accurate — this one's basically confirmed.” Where's the error?
Q6
The classic taxicab problem (Kahneman & Tversky): 85% of a city's cabs are Green, 15% Blue. A cab was involved in a nighttime hit-and-run. A witness says it was Blue, and testing shows the witness identifies colors correctly 80% of the time under those conditions. The probability the cab really was Blue is closest to —
Q7
A bestseller declares: “We studied 500 successful founders — 90% wake up at 5 a.m. Early rising is the key to success.” Statistically, which missing piece is fatal to this argument?
Q8
An airport explosive detector has a “99.9% hit rate” (it alarms on 99.9% of real explosives). It just went off. How should the security officer read the alarm?
Q9
Your spam filter is quite accurate at flagging emails containing “You've won.” Today one arrives — but the sender is the official domain of a sweepstake you actually entered. The sound reasoning process is —
Q10
In court the prosecutor says: “If the defendant were innocent, the DNA would match by chance only one in a million — so a match makes him all but guilty.” The name of this move, and its flaw:
Q11
Which of these pieces of new evidence is strong enough to let you nearly disregard the base rate?
Q12
A colleague can't crack “1% prevalence, 90% sensitivity, 9% false-positive rate — what's P(sick | positive)?” Without formulas, the rewrite most likely to make it click instantly is —
Answer all 12 questions to see your result 👆
The four numbers everyone mixes up
- Base rate
- The prior frequency in the population — the most ignored denominator (1% prevalence, 0.01% fraud).
- Sensitivity
- P(positive | sick): how many of the sick get caught. It does NOT answer “I'm positive — am I sick?”
- False-positive rate
- P(positive | healthy): how many healthy people get misflagged. At low base rates it decides who dominates the alert pool.
- Positive predictive value (PPV)
- P(sick | positive): the number you actually care about — true positives ÷ all positives, dragged down hard by the base rate.
- Representativeness
- “Looks like X” ≠ “is X”: resemblance carries no headcount information — always multiply by group size.
- Prosecutor's fallacy
- Passing off P(evidence | innocent) as P(innocent | evidence) — base rate neglect in courtroom dress.
- Likelihood ratio
- Evidence strength: P(evidence | true) ÷ P(evidence | false). Evidence near 1 doesn't deserve to update your belief.
- Natural frequencies
- Rewrite percentages as “out of 10,000 people…” — the framing that lifted accuracy from 16% to 46% (Gigerenzer & Hoffrage 1995).
What is the base rate fallacy?
| Metric | The question it answers | The classic misreading |
|---|---|---|
| Sensitivity P(pos | sick) | Of the sick, how many does the test catch? | Mistaken for “of the positives, how many are sick” — direction flipped |
| False-positive rate P(pos | healthy) | Of the healthy, how many get misflagged? | “Only 9%, negligible” — multiplied by a huge healthy population, it mans the alert pool |
| PPV P(sick | pos) | Given my positive result, how likely am I actually sick? | Assumed equal to sensitivity — the base rate drags it far lower |
| Base rate P(sick) | Ignoring all tests — how common is this to begin with? | Crowded out entirely by case details — the fallacy itself |
The base rate fallacy (base rate neglect) is the tendency to let case-specific information — a test result, an eyewitness account, a personality sketch — absorb all your attention while the prior frequency of the event gets ignored. It is not an obscure curiosity: false-positive panic after screenings, statistical wrongful convictions, “gut feel” hiring, and alert fatigue in fraud ops all run on this one error.
Its most famous students are Kahneman and Tversky — the engineer/lawyer experiment (1973) and the taxicab problem both come from their representativeness research program: people substitute “how much does this resemble X?” for “how common is X?”, and resemblance contains no headcount information whatsoever.
The most effective known antidote is not memorizing Bayes' theorem but the natural-frequency format: translate every percentage into “out of 10,000 people, N are…”. Gigerenzer and Hoffrage (1995) showed this single rewrite nearly tripled lay accuracy on Bayesian problems — and later work showed it rescues physicians too. Every explanation on this page uses it; finish the quiz and the habit comes free.
One caution: base-rate thinking does not mean “always trust the prior.” Full Bayesian posture is “start from the base rate, weight the evidence by its strength” — weak evidence (an impression, a mediocre test) barely moves the needle, while strong evidence (clear footage, near-unforgeable records) can legitimately overwhelm a low prior. Questions 9 and 11 test that direction specifically, so you don't climb out of one pit into another.
If “felt certain, computed shocked” is your kind of fun, PurrLearn's Thinking Traps course has 50 such questions: the Monty Hall switch, the gambler's fallacy, survivorship bias, the prosecutor's fallacy — each with original-experiment citations and natural-frequency explanations. Free to start, no sign-up.
FAQ
How does the base rate fallacy relate to the representativeness heuristic?
Representativeness is the cause; base rate neglect is a consequence. When judging “is he a programmer?”, people actually answer “does he resemble a programmer?” — substituting similarity for probability. Similarity carries no population information, so the prior gets squeezed out. Kahneman and Tversky's 1970s experiments established the chain.
Do doctors commit the base rate fallacy?
Famously, yes. Eddy (1982) gave physicians a breast-cancer screening problem; most overestimated the positive predictive value by nearly 10×. Gigerenzer's later studies of gynecologists were equally brutal. It's not a competence problem — conditional-probability notation is simply incompatible with human brains. Rewritten in natural frequencies, physicians' accuracy recovers substantially.
What's the fastest way to estimate P(sick | positive)?
The three-step natural-frequency method: ① imagine 10,000 people; ② split them by the base rate, then apply sensitivity to the sick group and the false-positive rate to the healthy group; ③ divide true positives by all positives. Multiplication and division only — no formula required.
How does this show up on the LSAT or GRE?
Logical reasoning questions love survivor-only samples and flipped conditionals: “90% of successful people do X, so X causes success” (missing the failure baseline), or “the match would be nearly impossible if innocent, so he's guilty” (the prosecutor's inversion). Recognize “missing baseline group” and “inverted conditional” and these questions become free points.
Does respecting base rates mean never trusting evidence?
No — that's the opposite error. Bayesian updating weights evidence by its likelihood ratio: evidence near 1 (vague impressions) shouldn't move you; evidence with an extreme ratio (clear footage, corroborated forensic chains) can overwhelm a low prior. The mantra: prior as the floor, evidence as the weights — neglect neither.