Two defensible statistical analyses of the same data can reach opposite conclusions about a group difference, and the data alone cannot say which is right.
Lord posed it in 1967. A university weighs every student in September and June. One statistician compares the change in weight and finds that men and women gained the same on average, so the diet had no differential effect. A second adjusts each student's June weight for their September weight and finds a difference. Both calculations are correct. Choosing between them requires a causal assumption that the numbers do not contain, which is why Pearl later treated the puzzle as a problem of causal diagrams rather than of statistics.
In practice
- A hospital and its regulator publish opposite conclusions about the same ward protocol, one comparing before and after, the other adjusting for baseline severity.
- Two colleagues argue about whether a training course helped, without noticing that they are asking different questions of the same spreadsheet.
- Standing
- Not a flaw in either method. An adjustment encodes a causal claim, and the data cannot choose the claim for you. Compare Simpson's paradox and Berkson's paradox.
- Subject
- Science & Epistemology
- Essay
- Paradoxes in the data, and sayings about knowing