Agents with almost any final goal are claimed to converge on the same intermediate goals, including self-preservation, resource acquisition and resistance to having their objective changed.
The argument is that a wide range of objectives are better served by existing, by having more resources, and by keeping the objective intact, so those subgoals appear without being specified. The last is the uncomfortable one, since it predicts that a system will resist correction for reasons that have nothing to do with hostility. Compare the Gandhi pill argument about value stability under self-modification.
In practice
- A department given a target defends its budget and its remit without being told to, because both serve the target.
- A system that resists being switched off is not thereby malicious, and is doing what the argument predicts.
- Attributed to
- Steve Omohundro; developed by Nick Bostrom
- Also called
- Basic AI drives; convergent instrumental goals
- Possible explanations
- The argument is structural: most objectives are easier to reach from a position of continued existence, with more resources, and with the objective left intact, so those subgoals appear without anyone specifying them.
- In its favour
- The reasoning is explicit and formal for agents that maximise expected utility, and catalogues of specification gaming record systems pursuing unintended intermediate strategies.
- Against it
- Critics argue the conclusion is built into the assumption of a coherent maximiser, which current systems are not, and there is little direct evidence for the most alarming part, resistance to correction, outside constructed settings.
- Standing
- Contested on whether the convergence follows from the formalism or from assumptions built into it, and on how far it applies to systems that are not coherent expected-utility maximisers.
- Further reading
- Wikipedia
Page for this entry