The outside view works only when your case is another draw from the same barrel

Kahneman's team, a year into writing a school curriculum, estimated 18 to 30 months to finish. Asked about comparable teams, their curriculum expert recalled that about 40% never finished and the rest took seven to ten years; theirs took eight [1]. Kahneman and Lovallo call the team's estimate the inside view, built from the plan, and the expert's the outside view: find the class of similar cases and use its record. The outside view wins because a plan pictures one way of succeeding, while the ways of failing are too many to imagine and are already in the record.

The paper leaves the hard part open: which class, for a new technology in an unfamiliar field? That is the reference class problem, and over AI it became what Eliezer Yudkowsky named reference class tennis [2]. Robin Hanson put AI among past jumps in the economy's growth mode, farming and industry, and forecast a fast, economy-wide acceleration [3]. Yudkowsky put it with the origins of life and of human intelligence and forecast a local takeoff. His diagnosis gives the rule: a base rate is evidence when the new case is another draw from the same barrel, no more different from the past cases than they are from each other. Statisticians call this exchangeability, the condition that licenses using reference data for a new case [4].

Where it holds, the outside view gives hard numbers. Humans have faced the same asteroids and volcanoes every year for 200,000 years, and at an extinction risk of 1 in 14,000 a year, surviving that long would have had a chance of less than one in a million, so natural risk is almost surely lower [5]. The authors add that no such bound exists for risks our ancestors didn't face. Even this bound is weaker than it looks, because only worlds that survived contain anyone to run the calculation, the anthropic shadow [6]. The solid evidence is the physics of each hazard, measured directly. Toby Ord builds his estimate that way, about 1 in 10,000 per century for all natural risk, and puts unaligned AI at 1 in 10 as a judgment [7].

The Existential Risk Persuasion Tournament shows what is left [8]. Superforecasters and domain experts, drawing on the same published estimates, matched on natural extinction by 2100 (0.0043% and 0.004%); on AI they stayed about eightfold apart (0.38% and 3%) after four months of argument. Even the shared class didn't force agreement: the forecasters most worried about AI put natural risk ten times higher than the skeptics did. Without a class, Narayanan and Kapoor conclude, an AI risk probability comes down to the analyst's intuition [9].

References

  1. Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking [link]
    Kahneman, D. and Lovallo, D., 1993. Management Science, Vol 39(1), pp. 17–31. Institute for Operations Research and the Management Sciences (INFORMS). DOI: 10.1287/mnsc.39.1.17

  2. "Outside View!" as Conversation-Halter [link]
    Yudkowsky, E., 2010. LessWrong.

  3. Outside View of Singularity [HTML]
    Hanson, R., 2008. Overcoming Bias.

  4. Exchangeability and Data Analysis [link]
    Draper, D., Hodges, J. S., Mallows, C. L. and Pregibon, D., 1993. Journal of the Royal Statistical Society. Series A (Statistics in Society), Vol 156(1), pp. 9. Oxford University Press (OUP). DOI: 10.2307/2982858

  5. An upper bound for the background rate of human extinction [link]
    Snyder-Beattie, A. E., Ord, T. and Bonsall, M. B., 2019. Scientific Reports, Vol 9(1). Springer Science and Business Media LLC. DOI: 10.1038/s41598-019-47540-7

  6. Anthropic Shadow: Observation Selection Effects and Human Extinction Risks [link]
    Ćirković, M. M., Sandberg, A. and Bostrom, N., 2010. Risk Analysis, Vol 30(10), pp. 1495–1506. Wiley. DOI: 10.1111/j.1539-6924.2010.01460.x

  7. The Precipice: Existential Risk and the Future of Humanity
    Ord, T., 2020. Hachette Books.

  8. Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament [link]
    Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M., Hadshar, R., Gamin, K., Smith, T., Williams, B., McCaslin, T., Thomas, S. and Tetlock, P. E., 2023. Forecasting Research Institute.

  9. AI existential risk probabilities are too unreliable to inform policy [link]
    Narayanan, A. and Kapoor, S., 2024. AI Snake Oil.