Analytic choices made after seeing the data inflate false positives even with no intent to p-hack.
Gelman and Loken's 2013 point, and the sharper half of the p-hacking critique: a researcher who would have run a different analysis had the data differed inflates false positives without ever running two analyses.
In practice
- An analyst tries three date windows, two exclusion rules and a log transform before the result becomes significant, and reports one analysis honestly.
- A trial that would have been reported one way had the effect appeared in men, and another way had it appeared in women, has more paths than it declares.
- Further reading
- Wikipedia
- Subject
- Science & Epistemology
- Essay
- How knowledge grows, and how inquiry fools itself