Samotsvety probabilities for potential precursors of existential risk (2024/10/22)
Seven Samotsvety forecasters give their probabilities for different possible catastrophes to lead to more than 1M direct deaths in any one year, in any of the next 10 years. This post presents results, starting with a table which presents the probabilities, continuing with a discussion of each of the items, and concluding with some thoughts on relevance for Sentinel’s work averting or mitigating incipient catastrophes.
Some forecasters found some questions in particular much more interesting, so we alternate between presenting the probability of the best model when one such exists, and the geometric mean of the odds when it’s a risk that requires more subjective judgment and we have enough judgments to aggregate. Events that could fall into different buckets resolve each buckets. For instance, an AI-invented biological weapon used during WW3 would resolve the “AI”, “biological weapons” and “WW3” buckets.
Summary plot and table
1-Year Post-mortem Analysis (2023/02/26)
Prediction: Reject ISL and overturn NCSC ruling (81%).
Outcome: Reject ISL and uphold NCSC ruling.
SCOTUS’s decision in Moore v Harper is a great case to post-mortem, because it illustrates some potential pitfalls in base rate-driven forecasting. My errors here fall into three domains: base rate specificity, inside view adjustment, and cognitive bias.
It’s easy to look at a case like this and say, “19% chance isn’t 0% chance, we just got unlucky, this case just happened to be one of the few times we’re wrong.” But this is a very detrimental mindset for forecasting. It would mean no forecast could ever be wrong, and every wrong forecast is just “bad luck”. We shouldn’t overreact to cases like this, because we do expect them to happen. But it’s also critical to find what we should’ve done differently.
Update to Samotsvety AGI timelines (2023/01/24)
Previously: Samotsvety’s AI risk forecasts.
Our colleagues at Epoch recently asked us to update our AI timelines estimate for their upcoming literature review on TAI timelines. We met on 2023-01-21 to discuss our predictions about when advanced AI systems will arrive.
Forecasts:
Definition of AGISamotsvety Nuclear Risk update October 2022 (2022/10/03)
This writeup was originally posted on the EA Forum, which has some good discussion.
After recent events in Ukraine, Samotsvety convened to update our probabilities of nuclear war. In March 2022, at the beginning of the Ukraine war, we were at ~0.01% that London would be hit with a nuclear weapon in the next month. Now, we are at ~0.02% for the next 1-3 months, and at 16% that Russia uses any type of nuclear weapon in Ukraine in the next year.
Expected values are more finicky and more person-dependent than probabilities, and readers are encouraged to enter their own estimates, for which we provide a template. We’d guess that readers would lose 2 to 300 hours by staying in London in the next 1–3 months, but this estimate is at the end of a garden of forking paths, and more pessimistic or optimistic readers might make different methodological choices. We would recommend leaving if Russia uses a tactical nuclear weapon in Ukraine.
Since March, we have also added our track record to samotsvety.org/track-record, which might be of use to readers when considering how much weight to give to our predictions.
Samotsvety’s AI risk forecasts (2022/09/09)
Crossposted to the EA Forum, LessWrong and Foxy Scout
IntroductionIn my review of What We Owe The Future (WWOTF), I wrote:
Finally, I’ve updated some based on my experience with Samotsvety forecasters when discussing AI risk… When we discussed the report on power-seeking AI, I expected tons of skepticism but in fact almost all forecasters seemed to give >=5% to disempowerment by power-seeking AI by 2070, with many giving >=10%.
Samotsvety Nuclear Risk Forecasts — March 2022 (2022/03/10)
Thanks to Misha Yagudin, Eli Lifland, Jonathan Mann, Juan Cambeiro, Gregory Lewis, @belikewater, and Daniel Filan for forecasts. Thanks to Jacob Hilton for writing up an earlier analysis from which we drew heavily. Thanks to Clay Graubard for sanity checking and to Daniel Filan for independent analysis. This document was written in collaboration with Eli and Misha, and we thank those who commented on an earlier version.
This writeup was originally posted on the EA Forum, which has some good discussion.
OverviewIn light of the war in Ukraine and fears of nuclear escalation\[1\], we turned to forecasting to assess whether individuals and organizations should leave major cities. We aggregated the forecasts of 8 excellent forecasters for the question What is the risk of death in the next month due to a nuclear explosion in London? Our aggregate answer is 24 micromorts (7 to 61) when excluding the most extreme on either side\[2\]. A micromort is defined as a 1 in a million chance of death. Chiefly, we have a low baseline risk, and we think that escalation to targeting civilian populations is even more unlikely.
Prediction Markets in The Corporate Setting (2021/12/31)
What follows is a report that Misha Yagudin, Nuño Sempere, and Eli Lifland wrote back in October 2021 for Upstart, an AI lending platform that was interesting in exploring forecasting methods in general and prediction markets in particular.
We believe that the report is of interest to EA as it relates to the institutional decision-making cause area and because it might inform EA organizations about which forecasting methods, if any, to use. In addition, the report covers a large number of connected facts about prediction markets and forecasting systems which might be of interest to people interested in the topic.
Note that since this report was written, Google has started a new internal prediction market. Note also that this report mostly concerns company-internal prediction markets, rather than external prediction markets or forecasting platforms, such as Hypermind or Metaculus. However, one might think that the concerns we raise still apply to these.
This writeup was originally posted on the EA Forum, where there was some interesting discussion in the comments.