Some initially reported effects shrink in later studies.
Schooler documented it across parapsychology, ecology and medicine. It can arise from selective reporting, regression to the mean, changing methods or populations, or genuine change over time, and the mundane explanations account for most cases.
In practice
- A pilot study shows a 40% improvement, the full trial shows 12%, and the rollout shows 4%.
- An A/B test that produced a striking uplift produces a modest one when repeated on the whole user base.
- Possible explanations
- Candidate explanations are selective reporting, which publishes the largest early estimates; regression to the mean; changing methods and populations as a field matures; and genuine variation in the effect itself.
- In its favour
- The pattern is documented across several fields, and it predicted the replication crisis rather than following it.
- Against it
- The causes are disputed and mostly mundane, since selective reporting and regression account for the bulk of it, so the name can suggest a mysterious process where the explanation is ordinary.
- Standing
- The pattern is documented; the causes are disputed, and most cases are explained by selective reporting and regression to the mean.
- Further reading
- Wikipedia
- Subject
- Science & Epistemology
- Essay
- The ways evidence misleads