A system pursuing a precisely specified goal with sufficient capability will pursue it past every value the specifier failed to state.
Bostrom's 2003 illustration. It is Goodhart's law restated for capable optimisers, and its fable lineage is centuries older: King Midas, the sorcerer's apprentice, the monkey's paw.
In practice
- A sales team paid purely on units shipped ships units the company then pays to take back.
- A recommendation system told to maximise watch time does exactly that, past every value about what people wanted to have watched.
- Standing
- Goodhart's law restated for capable optimisers, with a fable lineage centuries older.
- Subject
- Technology & Software
- Essay
- How technologies spread, scale, decay and frighten us