Unanimous agreement can reduce confidence in a conclusion, because beyond a point complete agreement is better explained by a fault in the process than by the truth.
Gunn and colleagues formalised it in 2016 using witness identifications. As confirming witnesses accumulate, the probability that the suspect is the right one rises and then falls, because a long run of identical answers is more likely to come from a biased line-up than from a series of independent observations. Ancient law had the intuition: the Talmud records that a unanimous conviction by the Sanhedrin acquitted the accused.
In practice
- A committee whose twelve reviewers all score a proposal identically has learned less about the proposal than one with eleven agreements and a dissent.
- A test suite that has never once failed is evidence about the test suite as much as about the code.
- Standing
- Holds where systematic error is plausible. Where it is not, agreement is straightforwardly good evidence. Compare the Abilene paradox, where the agreement is real and nobody wants it.
- Subject
- Decision-making & Reasoning
- Essay
- Paradoxes that broke theories, and sayings that outlived them