Risk & Reliability

Failure rates start high from defects, fall to a low steady rate, then rise again with wear-out.

It is why burn-in testing exists: run components hard before shipping so the early-failure population dies in the factory rather than at the customer. The same shape appears in employee tenure and in new-code defect rates.

In practice

• A fleet of new vehicles has a bad first quarter, four steady years, then a maintenance bill that arrives all at once.
• A team of new hires produces its worst month first, its best years next, and its highest turnover just as everyone becomes expensive.

Attributed to

Reliability engineering

Further reading

Wikipedia

Page for this entry

Redundant components that share an environment, design, or supplier fail together, voiding the redundancy.

Two generators from the same batch, in the same flooded basement, are not redundant. Fukushima Daiichi in 2011 lost backup power this way — the redundancy was real and the independence was not.

In practice

• A data centre with two power feeds from the same substation has one power feed with extra cabling.
• An airline rosters two pilots who both ate from the same galley, which is why cockpit meals are deliberately different.

Attributed to

Reliability engineering

Page for this entry

People overreact to rare, vivid, uncontrollable dangers and underreact to frequent chronic ones.

Gigerenzer analysed the three months after 11 September 2001 and argued that avoidance of flying contributed to excess road deaths. Later reanalyses found excess fatalities but questioned the simple flight-to-driving explanation.

In practice

• A firm installs airport-style security after one incident and leaves the loading bay unlocked, where the losses actually occur.
• A parent worries about abduction and does not fit a stair gate.

Attributed to

Gerd Gigerenzer

Possible explanations

The explanation is that vivid, uncontrollable and catastrophic risks are weighted more heavily than frequent ones, so people substitute a familiar risk for a feared one without comparing rates.

In its favour

Substitution after a dramatic event is documented, including the rise in American road travel after September 2001, and the pattern fits research on risk perception.

Against it

Later reanalyses found the excess road deaths but questioned the simple flight-to-driving explanation, and the effect on behaviour fades within months, which the general claim does not predict.

Standing

Later reanalyses found excess road fatalities but questioned the simple flight-to-driving explanation.

Page for this entry

Serious injuries are claimed to sit atop a fixed ratio of minor injuries and near misses.

The original data are lost and the fixed ratios are widely disputed. It remains embedded in safety practice, which is a problem, since focusing on minor-injury counts diverts attention from low-frequency catastrophic risk.

In practice

• A site that reports 400 near misses and no serious injuries treats the record as safety, and the ratio says nothing about the rare catastrophic risk.
• A firm counting minor incidents in a chemical plant is measuring the wrong tail entirely, since the process safety risk is uncorrelated with slips and trips.

Attributed to

Herbert William Heinrich

Also called

Accident pyramid; safety triangle

Possible explanations

The explanation offered is a fixed pyramid of causes, in which the same unsafe acts produce many minor incidents and occasionally a fatal one, so counting the base predicts the top.

In its favour

Near-miss reporting, which the triangle popularised, is genuinely useful, and some hazards do produce graded outcomes from the same mechanism.

Against it

The original data are lost, the fixed ratios are widely disputed, and modern safety research finds that serious accidents often have different causes from minor ones, so driving down minor-injury counts can leave the fatal risks untouched.

Standing

The original data are lost and the fixed ratios are widely disputed; still embedded in safety practice.

Page for this entry

Complex failures rarely have one cause; naming a root cause conceals the conditions that permitted it.

Dekker's argument. Naming a root cause supplies a stopping point for the investigation and an individual to hold responsible, which is why organisations reach it before the conditions have been examined.

In practice

• A near-miss report concludes that a nurse misread a label, and the label, the lighting and the shift length are all unchanged the following week.
• An outage attributed to "human error" has closed the inquiry precisely where the interesting question — why the system permitted it — begins.

Attributed to

Sidney Dekker

Also called

Single-cause fallacy

Page for this entry

Any component whose failure halts the whole will eventually set the system's reliability.

The practical exercise is to enumerate what would halt operations and check how many have a genuine second path. Most organisations discover several they did not know were single.

In practice

• A hospital with backup generators, redundant water and one pharmacist trained on the compounding machine has one thing standing between it and closure.
• A firm with two suppliers, two data centres and one certifying laboratory has one single point of failure and does not know it.

Attributed to

Reliability engineering

Also called

SPOF

Further reading

Wikipedia

Page for this entry

Some systems gain from volatility and stress; robustness merely withstands it.

Taleb's contrast: a teacup is fragile, a rock is robust, an immune system is antifragile because exposure to pathogens strengthens it. Most institutions are built for robustness and mistake it for the third thing.

In practice

• A team that has survived three genuine outages debugs faster than one that has never had a bad night, because the runbooks were written under fire.
• A small restaurant that gets its worst review in year one and rewrites the menu outlasts the one that opened to praise.

Attributed to

Nassim Nicholas Taleb

Further reading

Wikipedia

Page for this entry

Rare, high-impact events dominate outcomes and are rationalised as predictable only afterwards.

Taleb's test is the turkey's: 1,000 days of reliable feeding are the strongest possible evidence right up to the afternoon before Thanksgiving. The 2008 crisis models assigned near-zero probability to what happened.

In practice

• A firm's risk register lists twenty scenarios and its failure comes from the twenty-first, which is afterwards described as obvious.
• A career is derailed by an event nobody's plan contained, and the retelling makes it look like a trajectory.

Attributed to

Nassim Nicholas Taleb

Further reading

Wikipedia

Page for this entry

Plans degrade in execution through countless small frictions that no plan can enumerate in advance.

Clausewitz's 1832 term for the accumulation of small unforeseeable difficulties: weather, fatigue, misheard orders. His point is that friction is not a defect in planning but a property of execution.

In practice

• A launch plan survives budget, staffing and legal review, then loses a fortnight because the only person who can sign the certificate is on paternity leave.
• A perfectly rehearsed conference talk is undone by a projector that will not accept the laptop's output.

Attributed to

Carl von Clausewitz

Page for this entry

Some organizations sustain near-error-free operation through preoccupation with failure, deference to expertise, and reluctance to simplify.

Weick and Sutcliffe's study of aircraft carriers, air traffic control and nuclear plants. The counterintuitive finding is that these organisations treat the absence of incidents as a warning sign rather than as success.

In practice

• A control room that treats a quiet month as a warning sign rather than a success finds the drift before the incident.
• A team that defers a technical decision to whoever knows the system, regardless of grade, catches problems a hierarchy would have escalated past.

Attributed to

Weick & Sutcliffe

Also called

HRO theory; mindful organizing

Page for this entry

When tasks are complementary, output depends on the weakest step, so quality clusters and small failures are costly.

Kremer's 1993 model, named for the Challenger's failed seal. Where every step must go right, high-skill workers cluster together and the value of one weak step is enormous — which explains wage gaps between countries.

In practice

• A launch with excellent engineering, excellent design and a broken payments integration converts nobody.
• An aircraft assembled from four superb components and one substandard fastener is a substandard aircraft.

Attributed to

Michael Kremer

Also called

Weakest-link production

Page for this entry

Complex systems can collapse from small initiating failures.

The 2003 North American blackout began with a sagging line touching a tree in Ohio and left 55 million people without power. Each protective trip shifted load onto the next component.

In practice

• A payment provider's certificate expires, retries pile up, the queue backs into the ordering service, and a shop that never touched payments is offline.
• A single overloaded switch reroutes traffic onto three others, which overload in turn, and the network fails outward from one component.

Attributed to

Reliability and systems engineering

Also called

Domino effect

Page for this entry

Where systems are both tightly coupled and interactively complex, serious accidents are a structural property rather than an exception.

Perrow's 1984 analysis of Three Mile Island. The two properties are what matter: tight coupling means no time to intervene, interactive complexity means the failure mode was not anticipated.

In practice

• A chemical plant where a valve fault propagates in ninety seconds through a process nobody fully models is not badly run; it is that kind of plant.
• An automated trading system whose components each behave correctly produces a cascade in four minutes that no operator could have interrupted.

Attributed to

Charles Perrow

Also called

System accidents

Page for this entry

Small departures from standard that pass without consequence become the new standard, until they do not.

Vaughan's 1996 study of the Challenger decision. O-ring erosion was observed on prior flights, tolerated, and became the expected condition — so the launch decision was made against a standard that had already moved.

In practice

• A team that has skipped the pre-deploy check forty times without incident no longer regards skipping it as skipping anything.
• A ward that has run one nurse short for two years has redefined the establishment, and the staffing paper reports full compliance.

Attributed to

Diane Vaughan

Further reading

Wikipedia

Page for this entry

A preventive measure that brings large benefit to a population offers little to each individual who takes part in it.

Rose set it out in 1981. Most cases of a common disease arise in the large group at moderate risk rather than the small group at high risk, so shifting the whole distribution prevents more cases than treating the extreme. The individual arithmetic is discouraging: of the many who take the drug, wear the helmet or accept the restriction, almost all would have been fine and one or two are saved. Population measures therefore depend on a compliance that can never be justified case by case.

In practice

• A road safety rule inconveniences a million drivers a day and prevents a handful of deaths a year, which its critics and its authors are both right about.
• A patient asks what a statin will do for him, and the honest answer is that it will probably do nothing for him and something for people like him.

Attributed to

Geoffrey Rose

Also called

Rose's theorem

Standing

A consequence of how risk is distributed rather than a disputed finding, and the standing reason that public health measures are unpopular with the individuals they protect. Compare the base rate fallacy.

Page for this entry

People adjust behaviour to hold perceived risk at a target level, offsetting safety improvements.

Strong forms are contested; partial offsetting is well documented. Wilde's version — that the target level is fixed — implies safety interventions cannot work at all, which the evidence does not support.

In practice

• A fleet fitted with collision warning sees accidents fall, and by less than the manufacturer's figures implied, because following distances shortened.
• A resort that grooms a run more thoroughly finds skiers taking it faster, and the injury rate settles between the old figure and the promised one.

Attributed to

Gerald Wilde

Also called

Risk compensation

Possible explanations

The explanation is a target level of risk: people adjust behaviour until perceived risk returns to what they are willing to bear, so a safety improvement is spent rather than banked.

In its favour

Partial offsetting is well documented, and the general point, that safety measures change behaviour as well as outcomes, is now standard in transport research.

Against it

Wilde's strong version, in which the target is fixed and offsetting complete, implies that safety interventions cannot work at all, which is contradicted by large measured reductions in deaths.

Standing

Strong forms are contested; partial offsetting is well documented.

Page for this entry

Safety is the capacity to succeed under varying conditions, not merely the absence of failure.

Hollnagel's argument that studying only failures misses how work usually succeeds — through adjustment and improvisation that no procedure specifies. It reframes safety as capacity rather than absence.

In practice

• A control room handles four unusual events in a shift and files no report on any of them, because nothing went wrong.
• A ward that meets its targets does so because staff quietly work around four broken processes, and the incident report records none of it.

Attributed to

Erik Hollnagel

Also called

Resilience engineering

Page for this entry

Accidents occur when latent weaknesses in successive layers of defence momentarily line up.

Reason's 1990 model, and the practical consequence is that the holes move: each layer's weaknesses shift, so an alignment that was impossible last month becomes possible this month.

In practice

• A patient gets the wrong dose because the prescriber was rushed, the label was ambiguous, the checking nurse was covering two wards, and the alert had been switched off in March.
• A firm that audits each control separately and never tests them together has not tested the model the controls belong to.

Attributed to

James Reason

Also called

Reason model

Further reading

Wikipedia

Page for this entry

Risk must be reduced as low as reasonably practicable, weighing residual risk against the cost of further reduction.

The gross-disproportion test comes from the reasonably practicable doctrine set out in Edwards v National Coal Board in 1949; the duties were codified in the Health and Safety at Work Act 1974, and the HSE uses the ALARP and SFAIRP terminology. In practice a duty-holder must show the cost of further risk reduction is grossly disproportionate to the benefit, not merely inconvenient.

In practice

• A rail operator installs platform-edge doors at £40m because the residual risk is real; it declines a £900m tunnel rebuild that would remove a risk one-tenth the size.
• A hospital adds a second check on high-risk drugs and declines a third, because the third would delay treatment more than it prevents error.

Attributed to

UK reasonably practicable doctrine; Edwards v National Coal Board

Also called

So far as is reasonably practicable

Possible explanations

The explanation is diminishing returns: each further reduction in risk costs more than the last, so a rule that demanded zero risk would stop the activity altogether. The gross-disproportion test puts the burden on the duty-holder to show where that point lies.

In its favour

It is workable law, tested repeatedly in British courts, and it forces a duty-holder to justify stopping rather than to justify acting.

Against it

What counts as grossly disproportionate is decided case by case and usually after an accident, so duty-holders cannot know in advance whether their judgement will stand, and the test puts a monetary value on harm that many object to in principle.

Standing

Often misattributed to the Haldane principle.

Page for this entry

Rely on several independent barriers so that no single failure can reach the harm.

The submarine's watertight compartments are the mechanical case: any single breach is survivable because no single breach reaches everything. It fails when the layers share a cause.

In practice

• An aircraft has independent hydraulic systems, a mechanical reversion mode and a checklist, so that no single failure reaches the ground.
• A finance process has a preparer, a reviewer and an external audit, and the fraud that succeeds is the one that crosses all three.

Attributed to

Military and safety engineering

Also called

Layered defence

Further reading

Wikipedia

Page for this entry

Design so that the failure mode is itself the safe state.

A dead-man's switch on a train stops the train. The design question is always which state is safe: a fire door that fails locked and a security door that fails open are the same engineering mistake in opposite directions.

In practice

• A railway signal that loses power shows red; a lift that loses power stops between floors and holds.
• A door that fails locked is right for a vault and lethal for a stairwell, and the same mechanism is both.

Attributed to

Safety engineering

Page for this entry

Organisms and institutions maintain reserves to absorb shocks.

Reliability engineering, and the biological version is well documented: organs have reserve capacity far above normal demand. Institutions that run at full utilisation have converted a margin into a fragility.

In practice

• A hospital that runs at 100% bed occupancy has no capacity for a bad week, and a bad week arrives annually.
• A household with no savings treats a broken boiler as a crisis, and the boiler was always going to break.

Attributed to

Reliability engineering

Page for this entry

A precaution is owed when the probability of harm times its magnitude exceeds the burden of taking it.

Hand's 1947 formulation in United States v. Carroll Towing: B < PL. It is an early explicit cost-benefit rule for institutional risk, and it makes negligence a question of arithmetic rather than character.

In practice

• A supermarket that could fit non-slip matting for £8,000 and faces £200,000 of foreseeable injury claims is negligent for not fitting it.
• A landlord who could install a £40 carbon monoxide alarm against a small chance of a death is doing the same arithmetic, and the answer is not close.

Attributed to

Learned Hand

Also called

Hand rule; B < PL

Standing

An early explicit cost-benefit rule for institutional risk.

Page for this entry

Where an action risks serious or irreversible harm, absence of scientific consensus does not justify delaying prevention.

From the German Vorsorgeprinzip and the 1992 Rio Declaration. Its criticism is that strong forms are self-defeating, since inaction also carries risk that the principle does not weigh.

In practice

• A council delays a cycle lane pending a full safety study, and the junction keeps the collision rate the lane was meant to cut.
• A regulator that refuses a treatment pending certainty has chosen the harm of the untreated condition, which the principle does not weigh.

Attributed to

German Vorsorgeprinzip; Rio Declaration 1992

Also called

Vorsorgeprinzip

Further reading

Wikipedia

Page for this entry

Where an intervention's benefit is uncertain and its harm is not, prefer inaction.

From the Hippocratic ethical tradition, though the Latin maxim itself is documented only in the nineteenth century. Its force is asymmetric: it applies where benefit is uncertain and harm is not, and does not license inaction where both are known.

In practice

• A surgeon declines an operation whose benefit is unproven and whose complication rate is known, and gives the patient that reason.
• It does not license inaction where both are known, which is why "first do no harm" is a poor argument against a treatment with an established benefit.

Attributed to

Hippocratic antecedent; Latin phrase documented in the 19th century

Also called

First, do no harm

Further reading

Wikipedia

Page for this entry

Proceeding on the hope that it will turn out well, treated as a stance toward risk rather than as an oversight.

The word names the decision to act without provision for failure and without pretending the provision exists. It is worth keeping as a distinct concept because it is not the same as miscalculating the odds: the odds are understood and the preparation is declined anyway.

In practice

• A driver who sets out over a mountain pass without chains has not misjudged the weather.
• A team ships without a rollback plan, knowing what a rollback plan is for.

Attributed to

Russian idiom

Also called

Reliance on chance; it'll be all right

Possible explanations

One explanation is that where provision for failure is unaffordable or unavailable, acting without it is the only option, and a word for it saves the pretence. Another is fatalism about outcomes no one controls.

In its favour

The stance it names is distinct and worth a word: proceeding without cover is not the same as failing to notice the risk.

Against it

It is a national-character commonplace, and comparative evidence does not show that risk-taking in Russia differs in the way the idiom asserts; self-descriptions of a national trait are weak evidence for it.

Standing

A national-character commonplace. There is no comparative evidence that risk-taking in Russia differs in the way the idiom asserts, and self-description is not measurement.

Page for this entry

Within about three weeks of a disaster, an official declaration that there can be no more survivors will be wrong.

Berg's third law names a recurring institutional failure: the point at which search is officially abandoned is set by the cost of continuing and by the need to manage expectations, not by the biology of survival. Because the announcement is a resource decision presented as a finding of fact, it is periodically falsified by someone pulled out alive afterwards.

In practice

• A rescue called off on the fourth day is followed on the sixth by a survivor, and no individual decision was unreasonable.
• A project declared unrecoverable by a steering group is quietly finished by two people who were not asked.

Attributed to

Mitch Berg

Also called

Berg's Third Law; the premature recovery operation

Possible explanations

The explanation is institutional: the point at which a search is called off is set by the cost of continuing and by the need to manage expectations, not by the biology of survival, and the two need not coincide.

In its favour

Survivors have repeatedly been found after official declarations, in earthquakes and mine collapses, which is the pattern the law names.

Against it

It is stated as a law from repeated cases rather than a rate, the three-week figure is not derived from data, and the survivors who are found are far more memorable than the searches that correctly concluded.

Standing

From Berg's numbered set, first codified on his blog in 2005. Stated as a law on the strength of repeated cases rather than a rate, and the three-week window is not derived from survival data. Survival past several days is rare and documented, so the claim is directionally supported and quantitatively invented.

Page for this entry

The catastrophe everyone expects any day now will not happen any day now; it will happen decades later, when everyone is worrying about something else.

John Derbyshire stated it, tongue in cheek, in a National Review Online post, 'Rumors of War', presenting it as one of the cornerstones of modern social psychology. His example was nuclear war: feared in his youth, then displaced by worries about global warming, peak oil and demographic collapse, though he suspected the first fear had been the right one.

In practice

• Warnings of a global pandemic were a staple of the early 2000s, faded from public attention, and were borne out in 2020.
• A family that spent a year bracing for a redundancy that never came is caught out by a burst pipe.

Attributed to

John Derbyshire

Possible explanations

One explanation is the issue-attention cycle: public worry rises and fades whatever the underlying risk. Another is that widely feared dangers attract the precautions that delay them.

In its favour

Captures the long record of failed forecasts of imminent catastrophe, and the drift of public attention from one danger to the next.

Against it

Some feared catastrophes arrive roughly on schedule, and the law cannot say which. Where worry leads to prevention, as with the preparations for the year-2000 computer problem, the non-event may be the worry's success rather than its error.

Standing

Compare Downs's issue-attention cycle.

Page for this entry

Anything that can go wrong will—at the worst possible moment.

Distinct from Murphy's law by the addition of timing. Murphy was about designing for the worst case; Finagle is about the worst case arriving during the demonstration.

In practice

• The projector fails during the demonstration and worked all morning.
• The one week the backup job is unmonitored is the week the disk fails, which is why the monitoring is what needs monitoring.

Attributed to

Attributed to John W. Campbell Jr.

Also called

Finagle's Law of Dynamic Negatives

Further reading

Wikipedia

Page for this entry

Automating routine work leaves operators only the rare emergencies, for which they are now out of practice.

Bainbridge's 1983 paper. Air France 447 in 2009 is the case: autopilot disengaged in conditions the crew had rarely hand-flown, and the recovery required a skill automation had made rare.

In practice

• A pilot who has hand-flown four approaches in two years is asked to hand-fly one in weather that defeated the automation.
• A controller whose alerts are handled automatically for eleven months must handle the twelfth manually, having lost the practice.

Attributed to

Lisanne Bainbridge

Also called

Automation paradox; out-of-the-loop problem

Page for this entry

Anything that can go wrong will go wrong.

Associated with Murphy, an engineer on Stapp's rocket-sled programme around 1949 to 1951, though the exact origin is disputed. The lesson usually drawn is constructive: design so that the wrong assembly is impossible, since if it can be done wrong someone will do it wrong.

In practice

• A drug cabinet holding two vials that look alike will produce a wrong dose, and the fix is different packaging rather than more care.
• A connector that can be plugged in two ways will be plugged in the wrong way, which is why the correct design is a connector that only fits one way.

Attributed to

Engineering folklore; associated with Edward A. Murphy Jr.

Also called

Sod's law

Possible explanations

One explanation is selection in memory: failures are salient and near-misses are forgotten. Another is combinatorial, since a complex system has many more ways to fail than to work, so with enough trials the possible failure occurs.

In its favour

Read as a design rule it is sound: a component that can be assembled the wrong way eventually will be, which is why connectors are keyed and interlocks exist.

Against it

As a law it cannot be tested, since it says nothing about how often or over what period, and it can encourage fatalism where the useful reading is preventive.

Standing

Exact origin disputed; the design-for-error reading is an interpretation rather than a documented original statement.

Further reading

Wikipedia

Page for this entry

Measures that produce a visible feeling of safety without materially changing risk, because the audience is the point.

Schneier popularised the term in 2003 for security measures he judged to give reassurance or visibility without commensurate risk reduction, applying it extensively to post-9/11 airport screening. Whether any particular measure is theatre requires separate evidence about its security effect.

In practice

• The candidate test is whether the measure would stop an attacker who knew about it.
• A password policy requiring quarterly changes produces predictable passwords, and the visible rigour is the whole of the benefit.

Attributed to

Bruce Schneier

Further reading

Wikipedia

Page for this entry

The more certain designers are that a system cannot fail, the more catastrophic its failure will be.

Engineering folklore, named for the ship whose watertight compartments were open at the top. The mechanism is real: confidence in a system reduces the margins and the vigilance around it.

In practice

• A ward that installs a new alarm system withdraws the second nurse check, and the alarm is now the only check.
• A firm that has never had an outage reduces its on-call rota, and the confidence is what removed the margin.

Attributed to

Engineering folklore

Page for this entry

An unbroken record of safety is weak evidence of safety when the record itself built the exposure.

Taleb's illustration: 1,000 days of reliable feeding are the strongest possible evidence right up to the afternoon before Thanksgiving. The record itself is what created the exposure.

In practice

• A bridge that has carried every load for sixty years is inspected less often each decade, and the record is the reason.
• A supplier's flawless four-year record is partly the reason your exposure to them is now large enough to be dangerous.

Attributed to

Nassim Nicholas Taleb

Also called

The Thanksgiving turkey

Page for this entry

Systems that should not work sometimes do anyway.

An inversion of Murphy's law, and a real observation about legacy systems: many run on assumptions that no longer hold and continue working because nobody has tested the edge that would break them.

In practice

• A payroll run has depended for nine years on a spreadsheet whose author left, and it has never once failed.
• A scheduled job turns out to run on a server everyone believed was decommissioned, and the discovery is made when the building loses power.

Attributed to

Murphy's law canon

Also called

Inverse Murphy's law

Standing

Inversion of Murphy's law.

Further reading

Wikipedia

Page for this entry