As a model grows realistic enough to explain a system, it becomes as hard to understand as the system itself.
Bonini's point applies to any simulation detailed enough to be trusted: at that point nobody can say why it produced its output. Large forecasting models and machine-learning systems hit the same wall.
In practice
- A risk model with 400 parameters passes validation and cannot be explained to the board, so the board approves it on the strength of the validation.
- A supply chain simulation reproduces last year exactly and nobody can say which of its assumptions is carrying the result.
- Further reading
- Wikipedia
- Subject
- Decision-making & Reasoning
- Essay
- Paradoxes that broke theories, and sayings that outlived them