❤️ 15/15

Should you armor where the bullet holes are? — Survivorship bias and Wald's inversion

World War II. Bombers return to base riddled with bullet holes: wings and fuselage densely peppered, the area around the engines relatively clean. Armor is limited — add too much and the plane can't fly. So where should it go?

The intuitive answer: where the holes are — that's where the planes get hit most. According to Wallis (1980), the military indeed leaned toward reinforcing the most-hit parts of the returning planes.

Abraham Wald of Columbia University's Statistical Research Group (SRG, founded in the summer of 1942) pointed out the fatal flaw in that reasoning: the only damage data available came from planes that returned (the survivors); the planes that were shot down were entirely unobserved — using the survivors' bullet-hole distribution directly to assess aircraft vulnerability produces a systematic bias. Under the assumption that hits were roughly uniformly distributed, he inverted the inference: planes hit in vital parts were less likely to make it back, so the survivor sample shows fewer holes exactly there — the area around the engines looks 'clean' not because it wasn't hit, but because the planes hit there never came home.

Wald's actual method is far more hardcore than the legend: he wrote 8 memoranda, 'A Method of Estimating Plane Vulnerability Based on Damage of Survivors,' introducing the conditional probability p_i = 'the probability of being downed by the i-th hit, given survival of the first i−1 hits,' and building equations under explicitly stated assumptions such as 'planes are downed only by enemy fire (L0=0 — an unhit plane doesn't fall),' to infer the hit distribution of the planes that never returned from the data of those that did — one memo specifically estimates the vulnerability of each section of the aircraft. The method remained in use through the Korean and Vietnam wars; the memoranda were only reprinted publicly in 1980 by the Center for Naval Analyses (CNA, report CRC 432).

⚠️The self-referential twist: that 'survivorship bias' image you may have shared is itself survivorship bias. In the popular version, Wald fires back at the military on the spot — 'the armor goes on the engines, where there are no holes!' — accompanied by a diagram of a plane covered in red bullet-hole dots. The historical check (Casselman, AMS 2016): the 8 memoranda are pure technical derivation, containing no recommendation whatsoever about where to place armor (SRG convention was to answer statistical questions only, never to make decisions for the military); the famous quote has no primary source, and the vivid dialogue is a later 'plausible reconstruction' — the only primary sources are the memoranda themselves and two brief, unnamed mentions in Wallis's memoir. The red-dot airplane diagram is a modern illustration drawn by designer Cameron Moll around 2005 for a talk (the 2016 Wikipedia redraw is the most widely circulated version) — it is not Wald's. What 'survives' in circulation is the most dramatic version of the story, while the dry primary memoranda go unseen — sharing it means stepping on the trap while explaining the trap.
The bullet-hole pattern of returning planes is only the survivor sample: planes hit in vital parts never came back (the red-dot diagram is a post-2005 modern illustration, not a historical artifact)

Test yourself