The entries here combine rules for managing risk with sayings about what goes wrong. The rules divide over a question that runs through law, medicine and public policy: how much caution is enough. The sayings, most of them from engineers, describe the ways caution itself fails.
Two philosophies of caution
The first philosophy weighs costs against benefits. Judge Learned Hand's formula, from a 1947 negligence case, holds that a precaution is owed when the probability of harm times its size exceeds the cost of preventing it. Britain's ALARP Principle turns the same idea into regulation: reduce risk until further reduction would cost grossly more than it saves.
The second philosophy puts the burden on the action. Primum Non Nocere, first do no harm, prefers inaction where the benefit of an intervention is uncertain and its harm is not. The Precautionary Principle, written into the Rio Declaration in 1992, holds that lack of scientific certainty should not delay measures against serious or irreversible harm.
Design rules
The engineering precepts are less contested. The Fail-Safe Principle designs systems so that their failure mode is the safe state, as a railway signal drops to red when its cable breaks. Defence in Depth relies on several independent barriers, and the Law of Safety Margin keeps reserves to absorb shocks nobody predicted.
The sayings
Murphy's Law, from the American rocket-sled tests of the late 1940s, and Finagle's Law, which adds the worst possible moment, are the best-known laws in the ledger. Yhprum's Law, Murphy spelled backwards, supplies the rarer observation that things that should not work sometimes do.
The more instructive sayings describe caution defeating itself. The Titanic Effect holds that the more certain designers are that a system cannot fail, the worse the failure, because nothing was prepared for it. Lisanne Bainbridge's Irony of Automation observes that automating routine work leaves operators only the rare emergencies, for which they are now out of practice. Security Theatre, Bruce Schneier's phrase, names measures that make people feel safe without making them safer. The Turkey Problem warns that a long record of safety says little when the danger has not yet been tested.
Where it applies
The two philosophies of caution are argued out in every regulatory decision, from new medicines to new chemicals to new technologies. The design rules apply to anything built to be relied on, including software, where fail-safe defaults and layered defences are standard practice. The irony of automation is increasingly relevant as more routine work is handed to machines, from aircraft autopilots to driving aids, leaving humans to handle only the hardest cases. The turkey problem applies to any assurance that rests on a record of past safety.
Where it is argued over
The central dispute is between the two philosophies. Critics of the precautionary principle, Cass Sunstein prominent among them, argue that it paralyses, since inaction carries risks of its own and the principle cannot say which risk to avoid. Its defenders reply that cost-benefit analysis undervalues irreversible and poorly understood harms. ALARP's test of gross disproportion is decided case by case, often only after an accident. Security theatre has defenders who argue that reassurance has real value even when protection does not change.
Why it is worth knowing
Most people decide about risk without naming the philosophy they are using, and many arguments about safety are really arguments between these two. Knowing them makes the disagreement visible. The sayings add something the rules leave out: that caution has its own failure modes, that confidence breeds unpreparedness, and that the safest-looking record may be the least informative.
The entries behind this essay: Precept and Figure in Risk & Reliability. Every subject has its essays on the essays page.