Stated confidence is often more extreme than accuracy, particularly on difficult items.
Lichtenstein and Fischhoff's calibration studies illustrate it: people stating 90% confidence were right about 75% of the time, with the gap widening on harder questions. Calibration can reverse towards underconfidence on easy tasks, and improves greatly in expert domains with rapid feedback.
In practice
- A project team asked for a date they are 90% sure of misses it more often than one time in ten.
- A forecaster's 90% range should contain the outcome nine times in ten, and in practice contains it about seven, including among professionals.
- Further reading
- Wikipedia
- Subject
- Decision-making & Reasoning
- Essay
- The catalogue of predictable errors