Under combinations of low prior odds, low power, bias and analytic flexibility, most statistically significant published claims can be false.
Ioannidis's 2005 paper is the most-downloaded in PLoS Medicine's history. The argument is arithmetic rather than cynical: he showed mathematically that given typical power, prior odds and analytic flexibility, a majority of published positives can be false.
In practice
- A field where studies typically enrol forty participants and test six outcomes will publish a great many findings that do not survive replication.
- A company running dozens of small A/B tests at 95% confidence ships several changes a year that do nothing.
- Possible explanations
- The explanation is arithmetic: with low prior odds, modest power, bias and analytic flexibility, the proportion of significant findings that are true can fall below a half, whatever the nominal error rate says.
- In its favour
- The derivation is transparent, the parameter values it assumes match those measured in several fields, and the replication crisis that followed found roughly the failure rates it implied.
- Against it
- It is a mathematical argument from assumed priors, power and bias rather than a measurement of how many published findings are false, and fields differ enough in those parameters that a single headline claim does not transfer.
- Standing
- A mathematical argument from assumed priors, power and bias rather than a direct measurement of how many findings are false.
- Subject
- Science & Epistemology
- Essay
- How knowledge grows, and how inquiry fools itself