The more certain designers are that a system cannot fail, the more catastrophic its failure will be.
Engineering folklore, named for the ship whose watertight compartments were open at the top. The mechanism is real: confidence in a system reduces the margins and the vigilance around it.
In practice
- A ward that installs a new alarm system withdraws the second nurse check, and the alarm is now the only check.
- A firm that has never had an outage reduces its on-call rota, and the confidence is what removed the margin.