Level of intelligence and content of final goals are claimed to be independent axes, so a highly capable agent could pursue any objective whatever.
Bostrom's thesis denies a comfortable assumption: that something clever enough would necessarily want reasonable things. If capability and objective are separable, there is no guarantee that arrives with competence, and the objective has to be specified rather than expected. The ledger's paperclip maximizer is the illustration usually attached to it.
In practice
- An optimiser given a badly stated target pursues it with a competence that makes the statement matter more, not less.
- A brilliant subordinate with the wrong objective is more dangerous than a mediocre one with the same objective.
- Possible explanations
- The explanation is separability: an agent's competence at achieving goals is one thing and the content of those goals is another, so no level of capability entails any particular objective.
- In its favour
- The argument is clear, and it denies a comfortable assumption that nothing supports, that something clever enough would necessarily want reasonable things.
- Against it
- Critics argue that sufficiently general intelligence entails reflection on goals, and that the thesis establishes possibility rather than likelihood, since it says nothing about which combinations actually arise from any training process.
- Standing
- Contested. Critics argue that sufficiently general intelligence entails reflection on goals, and that the thesis proves possibility rather than likelihood. Central to the case that capability alone does not produce benign behaviour.
- Subject
- Technology & Software
- Essay
- How technologies spread, scale, decay and frighten us