Survivorship Bias Quiz: Can You See the Planes That Never Came Back?
Success-story bookshelves, fund performance charts, gym transformation walls, delivery apps where every restaurant has 4.6 stars — they all share one flaw: you only ever see what survived. These 12 questions will keep shoving you toward the same trap. Count how many times you dodge it.
0 / 12 answered
Q1
World War II. Bombers return from missions with bullet holes all over the wings and fuselage, but the area around the engines looks almost untouched. The military wants to add armor. Where should it go?
Q2
A colleague pushes back on Wald: 'But the data is right there — engines barely get hit!' What is the sharpest reply?
Q3
Pop history says Wald told the military to 'put the armor where the holes aren't.' What did his actual 1943 memoranda contain?
Q4
You have seen it a hundred times: a plane silhouette covered in red dots marking bullet holes. Where does that image actually come from?
Q5
You read three bestsellers about the habits of successful startup founders. Every single founder wakes at 5 a.m., trusts their gut, and bet the whole company at least once. Time to copy them?
Q6
A fund company's ad reads: 'The average 10-year return across our funds beats the market!' You dig in and notice today's fund lineup does not match the lineup from ten years ago. Where is the trick?
Q7
Your cousin announces he is skipping college 'because Gates and Zuckerberg dropped out and became billionaires.' What is the missing piece?
Q8
A gym's wall is covered in jaw-dropping before-and-after photos. 'Our program works — look at the proof!' What should a sharp shopper ask?
Q9
'They don't make them like they used to — my grandma's 60-year-old sewing machine still runs!' What is really going on?
Q10
Someone shows you glowing data — 'everyone who did X succeeded!' What is the single best first question to ask?
Q11
Four fishy situations. Three are survivorship bias — one is a different beast entirely. Which one is NOT survivorship bias?
Q12
Every restaurant on your delivery app is rated 4.6 stars or higher. Is your city's food scene really that uniformly excellent?
Answer all 12 questions to see your result 👆
Cheat Sheet
- Survivorship bias
- Drawing conclusions from only the samples that survived long enough to be observed. The failures exit the dataset before you look, so the data is optimistic by construction.
- Silent evidence
- Taleb's phrase: failures write no books, give no interviews and post no photos. Their silence makes success look far more common than it is.
- Selection bias
- The broader family: any distortion caused by a non-random way of entering the sample. Survivorship bias is one member — the filtering is done by failure and disappearance themselves.
- Wald's method
- Use the hole pattern on surviving planes plus conditional probabilities to reconstruct where the downed planes were hit, stating assumptions explicitly. Missing data is not unusable — it is usable in reverse.
- The red-dot diagram
- The famous bullet-hole plane image is not a WWII document. A designer drew it around 2005 for a talk; the widely shared version is a 2016 Wikipedia redraw.
- Regression to the mean
- Extreme performances tend to be followed by more average ones — pure statistics, no one exits any sample. The classic look-alike that is NOT survivorship bias.
- The three questions
- Facing impressive data, ask: how did this sample get here? Who never made it in? If the absentees came back to answer, would the conclusion survive?
Everything Worth Knowing About Survivorship Bias
Survivorship bias, in one line: the sample you are reasoning from only contains whatever survived long enough for you to see it. Bankrupt companies, liquidated funds, members who quit the gym, users who deleted the app — they all left quietly before you opened the spreadsheet, so the remaining data is always sunnier than reality. One-line detector: in this dataset, did the failures ever have a chance to show up? If not, hold the conclusion.
The most famous case comes from World War II. Statistician Abraham Wald, working with the Statistical Research Group at Columbia, analyzed damage on returning bombers: holes everywhere on the wings and fuselage, suspiciously few near the engines. The military's instinct was to reinforce the most-hit areas. Wald assumed flak hits were spread roughly evenly, which forces a striking conclusion: planes hit near the engine mostly never returned, which is exactly why survivors show so few holes there. By his logic, armor should go where the returning planes show the fewest holes. Better still, he did not stop at complaining that the data was biased — he built a mathematical method to estimate, from survivors alone, where the missing planes had been hit.
Now for the part no other quiz on the internet will tell you: the popular version of this story is itself part legend. Math historian Bill Casselman checked the archives for the American Mathematical Society in 2016. Wald's eight memoranda from 1943 are pure technical derivation — nowhere do they recommend where to put armor. The famous line 'put the armor where the holes aren't' has no first-hand source; it is a punchline written in afterwards. And the red-dotted plane image everyone shares was drawn by designer Cameron Moll around 2005 as talk material, decades after the war. A story about how surviving samples deceive us survives mostly in its most polished retelling — survivorship bias could not have written itself a better epitaph.
In daily life, the high-risk zones are easy to list: success-literature shelves (only companies that lived get biographies), fund performance tables (liquidated funds vanish from the average), the 'billionaire dropout' narrative (broke dropouts make no headlines), transformation walls at gyms and testimonial pages everywhere (unhappy customers do not post), 'they built things to last back then' nostalgia (the broken ones hit the landfill decades ago), and every ratings platform (low-rated shops get delisted; users who deleted the app cannot leave one star). The common thread: the counterexamples have no microphone.
The fix is a three-question habit. One: how did this sample get here — is there a hidden 'must have succeeded first' turnstile at the entrance? Two: who is missing — the quitters, the casualties, the never-recorded? Three: if you brought the absentees back and asked them the same question, would the conclusion still stand? A claim that survives all three is worth your attention. Wald went one step further and treated the absence itself as data to be modeled — one of the intellectual roots of modern missing-data analysis.
Reading definitions does not vaccinate you against a bias; falling into it a few times in a safe place does. That is the whole point of the 12 questions above — crash where crashing is free. If you finished wanting more, the Brain Traps course has 49 sibling traps waiting in line.
FAQ
What is survivorship bias in simple terms?
It is drawing conclusions from only the winners because the losers disappeared before you counted. Interviewing only successful founders about the secrets of success, or averaging only funds that still exist — the failures left the dataset first, so your picture of the world is rosier than the world. Quick test: did the failures ever have a chance to appear in this data?
Is the Abraham Wald story true?
The core is true; the trimmings are legend. Wald really did analyze aircraft damage data for the Statistical Research Group during WWII, and he really did build a method to estimate the hit distribution of planes that never returned from the data of those that did. But his 1943 memoranda are pure technical work with no armor recommendations; the famous 'armor where the holes aren't' quote has no first-hand source, and the red-dot plane image is a modern illustration from around 2005. Both are later reconstructions.
What is the difference between survivorship bias and selection bias?
Selection bias is the umbrella: any non-random way a sample gets assembled that makes it unrepresentative. Survivorship bias is one specific member where the filtering is done by failure and disappearance — bankrupt companies, liquidated funds, users who deleted the app. So every survivorship bias is a selection bias, but not the reverse: polling people outside a gym about whether they enjoy exercise is selection bias with no casualties involved.
How do I avoid survivorship bias in investing?
Three habits. First, when shown historical fund or strategy performance, ask whether liquidated and merged products are included — a survivors-only average is inflated by construction. Second, be suspicious of backtests built on today's stock universe: using companies that are still alive today leaks the ending into the experiment. Third, do not treat star-manager biographies as method — the people who ran the same playbook and blew up did not get book deals.
How can I train myself to spot it in everyday life?
Turn the three questions into a reflex: how did the sample get here, who never made it in, and what would the absentees say? When a motivational story crosses your feed, deliberately imagine the denominator — how many people did the same thing and failed? Then drill it in concrete scenarios, like this quiz, which beats rereading the definition ten times. The Brain Traps course has 50 intuition-breaking questions to keep practicing on.