Samotsvety Forecasting

Samotsvety Nuclear Risk update October 2022

This writeup was originally posted on the EA Forum, which has some good discussion.

After recent events in Ukraine, Samotsvety convened to update our probabilities of nuclear war. In March 2022, at the beginning of the Ukraine war, we were at ~0.01% that London would be hit with a nuclear weapon in the next month. Now, we are at ~0.02% for the next 1-3 months, and at 16% that Russia uses any type of nuclear weapon in Ukraine in the next year. 

Expected values are more finicky and more person-dependent than probabilities, and readers are encouraged to enter their own estimates, for which we provide a template. We’d guess that readers would lose 2 to 300 hours by staying in London in the next 1–3 months, but this estimate is at the end of a garden of forking paths, and more pessimistic or optimistic readers might make different methodological choices. We would recommend leaving if Russia uses a tactical nuclear weapon in Ukraine.

Since March, we have also added our track record to samotsvety.org/track-record, which might be of use to readers when considering how much weight to give to our predictions. 

Update 2022-10-04: Changed our estimates as a result of finding an aggregation error. You can see the previous version  of our post here. We also noticed that because of the relatively low number of estimates, they are fairly sensitive to each forecasts, so we are working on incorporating more forecasts.

Update 2022-10-19: These estimates seem a bit out of date now; see this comment and these forecasts from the Swift Institute.

Question decomposition

We have updated our decomposition to the following:

  1. What is the probability that Russia will use a nuclear weapon in Ukraine in the next MONTH?
  2. Conditional on Russia using a nuclear weapon in Ukraine what is the probability that nuclear conflict will scale beyond Ukraine in the next MONTH after the initial nuclear weapon use?
  3. Conditional on the nuclear conflict expanding to NATO, what is the chance that London would get hit, one MONTH after the first non-Ukraine nuclear bomb is used?

For each of those questions, we also asked forecasters for their yearly probabilities. Following up on previous feedback, we also asked forecasters for their core reasons behind their forecasts, and we’ll present those alongside their probabilities.

We also asked a range of questions about counterfactuals:

As well as a sanity check:

Summaries

Summary tables

For ≤ 1 month staggering times between each step

Event Conditional on previous step Unconditional probability
Russia uses a nuclear weapon in Ukraine in the next month — 5.3%
Nuclear conflict scales beyond Ukraine in the next month after the initial nuclear weapon use 2.5% 0.13%
London gets hit, one month after the first non-Ukraine nuclear bomb is used? 14% 0.02%

For ≤ 1 year staggering times between each step

Event Conditional on previous step Unconditional probability
Russia uses a nuclear weapon in Ukraine in the next year — 16%
Nuclear conflict scales beyond Ukraine in the next year after the initial nuclear weapon use 9.6% 1.6%
London gets hit, one year after the first non-Ukraine nuclear bomb is used? 23% 0.36%

Visualizations

This time, we are also experimenting with providing a few visualizations. Their advantage is that they may be more intuitive; the disadvantage is that they may gloss over the shape of our uncertainty, and thus mislead. Reader beware.

For the forecast with one month between each escalation step, we have:

A forecaster’s perspective

In order to understand at what level we are forecasting here, we are providing forecasters’ comments. One forecaster provided his comments in a more self-contained form—rather than question by question—so I’m presenting those comments here, lightly edited:

In general, nuclear rhetoric has been used extensively before and it seems that it was fairly successful at achieving its intended goals without having to use the weapons (e.g., Germany was hesitant to send weapons to Ukraine). I think such bluffing might be wearing off but Moscow is very good at maintaining ambiguity.

Being ambiguous about one’s willingness to use these weapons is what we have seen in the past and is what we see now. E.g., Zvi previously summarizes, when discussing a recent Putin’s speech:

> What I heard were several instances of drawing a distinction between Russia and its territorial integrity, and the territories under occupation. He said that the call-ups would be ‘sufficient for the operation.’ He declared his intention to keep the territory, if he can maintain physical control. Then, he went back to saying that Ukraine was getting weapons that could ‘threaten Russia,’ explicitly including Crimea as part of Russia but not Donbass, whereas Ukraine’s normal forces can obviously already threaten Donbass or Kherson. He framed his threats of nuclear use in response to claimed Western nuclear blackmail and what he says are Western attempts to get Ukraine to invade clearly Russian territories.

Using nukes doesn’t feel like a good choice.

Consider what will happen if the Ukrainian offensive continues. Russia is losing cities in Lugansk. I feel that Ukrainians are calling Putin’s nuclear bluff. And this gives Putin few good options to work with.

If things go nuclear:

I feel uncomfortable about my estimation process for a few reasons:

Forecaster probabilities and comments

See a later section for a comment on our aggregation method.

Russia using a nuclear weapon in Ukraine

What is the probability that Russia will use a nuclear weapon in Ukraine in the next MONTH?

What is the probability that Russia will use a nuclear weapon in Ukraine in the next YEAR?

Conditional on Russia using a nuclear weapon in Ukraine in the next year, will it be a tactical nuclear weapon?

Forecaster comments

These have been lightly edited. Reading them is probably indicative of the level at which we are thinking, which has the flavor of “we have a lot of uncertainty about this.”

This is a particularly dangerous time. Many of the gambles Putin has taken so far have gone badly and now he stands a real risk of losing power as the war drags on and he has nothing to show for it. Even still, for Putin, even without moral guardrails, the risks of using nuclear weapons of any kind should still outweigh the benefits if he is seeing things clearly. If things continue to deteriorate, the situation may change, but for now, it seems that although Putin has been weakened, he still has a very good chance of remaining in power if he can simply get to a stalemate in the territories he now controls. Although I’ve frontloaded a lot of the risk into the next month, if a nuclear weapon is going to be used, there will probably be some build-up before it is deployed with warning signs along the way. It is likely Putin will try to prepare his population, and, while declaring territories within Ukraine to be part of Russia may provide some pretense of a justification, each stage of escalation brings heightened risk. At each stage, it makes sense to escalate slowly to attempt to extract the maximum possible concessions a before taking on the increased risk of further escalation. I would expect to see nuclear tests or warning shots before seeing nuclear attacks, and for the first nuclear attack, tactical nuclear weapons would be the most logical starting point. 

I think the use of nuclear weapons tactically would be a lot easier for Putin to explain to the Russian people. Perhaps strategic use could come afterwards, if he is in a desperate situation.

I think that Putin is 100% committed to conquering Ukraine. His “special military action” has largely failed so far, so he is expanding his military efforts with a “partial” mobilization. If that fails, or perhaps in combination with increased military mobilization, it looks possible to me that he could detonate a tactical nuclear weapon in the mistaken belief that it would make NATO countries back off at least from territory that Russia currently controls. In reality, I think detonating a tactical nuclear weapon would have the opposite effect, though.

 

[My uncertainty is] primarily methodological and from skewing to uncertainty. The main errors in the Superforecaster post-mortem for predicting invasion were overreliance on certain base rates and underestimating Putin’s willingness to take major risks. I’m hesitant to make the same mistakes twice.

I also think Putin and Kremlin officials are less analyzable than most seem to think. I still don’t have a compelling explanation for why Putin wants Ukraine so bad and why he’s taken so much risk up until this point, which to me says my mental model of their decision-making isn’t good enough to do much with.

Plausible scenarios exist where Putin uses a tactical nuke, probably to scare Ukraine, divide NATO, etc. 

 

I would be higher with my first two estimates if they included an attack on a nuclear plant that could lead to a radiation disaster. This might be Putin’s preferred method because he could keep a level of ambiguity as to Russia being responsible. That said, Putin’s reason for using a tactical nuclear weapon might precisely be to let Ukraine and the world know how serious he is about not backing down. I think Putin wants to win the Ukraine War at pretty much any cost.

> […] I think Putin would almost definitely use a tactical nuke instead of a strategic one because it would make Ukraine and America/NATO more fearful of the situation without as high of a chance of a nuclear apocalypse (when compared to a strategic nuke being detonated in Ukraine).

Putin has established a land bridge to Crimea, which is a major strategic goal for Russia. In recent speeches, he has explicitly said that Russia will use everything it has on the table to protect the newly annexed region.

Using nuclear weapons would drastically upend the current geopolitical order. But I don’t have enough confidence to confidently reject that outcome.

Nuclear conflict escalating beyond Ukraine after Russia uses a nuclear weapon in Ukraine

Conditional on Russia using a nuclear weapon in Ukraine what is the probability that nuclear conflict will scale beyond Ukraine in the next MONTH after the initial nuclear weapon use? 

Conditional on Russia using a nuclear weapon in Ukraine, what is the probability that nuclear conflict will scale beyond Ukraine in the next YEAR after the initial nuclear weapon use?

Forecaster comments

I think nuclear war happening as a result of Russia using a tactical nuke in Ukraine is not extremely unlikely because the world would be in somewhat unprecedented territory, so this could make for a catastrophe as a result of miscalculations on one or both sides.

If Russia uses a nuclear weapon, the west probably would not respond with a nuclear strike, but would probably try to use other channels which I won’t speculate about publicly. Depending on the type, scale, and impact of the attack, a nuclear response is possible. If there is no Russian nuclear attack there is a minuscule chance of either a preemptive strike (based on intelligence that Russia is likely to launch a nuclear attack) or a false signal based on something that looks like an attack triggering a nuclear strike against Russia. The fact of heightened tensions makes these kinds of accidents more likely than they would otherwise be.

I don’t think Russia nuking Ukraine raises the global nuclear risk by much. I think most of the risk still comes from accidental launches due to false alarms, which I think is probably at an elevated risk currently.

I think that the MAD precludes nuclear conflict scaling up. And I think that if nuclear conflict were to expand following Russia detonating a nuclear weapon in Ukraine (or elsewhere), then that would likely happen close to immediately.

Payload and target of tactical nukes are all widely variable, if one is used I’d imagine those parameters would be chosen to minimize the risk of a nuclear response. 

NATO isn’t currently personally involved in the war, its hard to imagine them deciding to send troops or especially to send nukes in response to a hit on a military target or a demonstration blast on Snake Island or the Black Sea.

It’s possible Putin miscalculates or actually wants nuclear war, but to me the most likely outcome is negotiations (for better or for worse).

I have high confidence that nuclear weapons will not be used outside this conflict.

I don’t have high confidence that nuclear weapons will not be used in areas close to the strategic landscape (e.g., areas supporting either side in NATO, Belarus, inner Russia, etc.)

No one wants it to escalate. Escalating to NATO is suicidal, just clearly a loss for Putin and folks.

Also, I expect revolt of elites or something. As they would feel that this is totally suicidal, not worth it. I expect a lot of people to fear that nuclear war would mean guaranteed death or misery for their families etc. 

London being hit with a nuclear weapon, conditional on nuclear conflict escalating beyond Ukraine

Conditional on the nuclear conflict expanding to NATO, what is the chance that London would get hit, one MONTH after the first non-Ukraine nuclear bomb is used? 

Conditional on the nuclear conflict expanding to NATO, what is the chance that London would get hit, one YEAR after the first non-Ukraine nuclear bomb is used?

What is the unconditional probability of London being hit with a nuclear weapon in October?

Forecasters' comments
 

If nuclear conflict expands outside of Ukraine, it seems quite likely that London would get hit because I think that the UK would be the second choice of a Russian nuclear attack—the first choice being America. I also think that in the case of a nuclear war, it is a likely scenario that Russia launches a general nuclear attack on most, if not all of, NATO.

Barring accidents nd other unlikely circumstances, London will only be a target in the event of full-scale nuclear war. At each stage of escalation, prior to full-scale war, there would be attempts to take off ramps. But, it is possible, even if unlikely, that predetermined nuclear response protocols could kick in, or, in the fog of war mistakes and miscalculations could result in rapid escalation.

If there is a nuclear exchange between NATO and Russia, London will be hit very quickly.

If a nuclear conflict does expand to NATO, I would still hold out some hope that it doesn’t turn into an all-out nuclear war. Thus, my forecast for London getting hit in the event of nuclear conflict with NATO is relatively low. And, if the nuclear conflict expanded to NATO, I’d expect that if London were to get hit, then it would happen within a month. My forecast for the unconditional chance of London getting hit in October is about 10% of my forecast for any nuclear conflict in October and is barely above my forecast conditional on Russia not dropping a nuclear weapon in Ukraine.

Conflict likely wouldn’t expand to the exchange of strategic nukes after a tactical nuke exchange. Large cities are where the leaders making decisions are. Its one thing to kill soldiers and civilians but it’s another to put your own life on the line. Unlike other questions, we have a fairly strong historical track record here for mutually assured destruction during the cold war. Time has passed and tactical nukes are a key difference, but I think the core concept still applies. 

London getting targeted is also a very foreseeable scenario, I’d be surprised if NATO’s military systems aren’t ready and sophisticated enough to detect and shoot down a missile or submarine. 

There are also layers of complication from assassination, coups, and civil unrest. The risk to Putin feels much more personal than in other scenarios. 

Escalation is still possible, e.g. maybe Putin just really hates the West and that’s his true motivation, or maybe conflict simply keeps escalating once nukes are exchanged. But that type of dramatic escalation feels unlikely.

Escalation beyond Ukraine doesn’t help Russia achieve its strategic goals.

hard to see intermediate escalation

Comparison vs other sources

A few other sources which have forecasts on this are:

There is internal discord within Samotsvety about the degree to which the magnitude of the difference between our current and former probabilities is indicative of a lack of accuracy. We Samotsvety updated our endline monthly probability of London being hit with a nuclear weapon by ~2 (~0.02% vs 0.067 * 0.18 = 0.012%). The difference was higher before correcting an aggregation error, so I’ve moved discussion to a footnote[1].

In addition, a former senior U.S. government official previously gave me a 20% probability of Russia using nuclear weapons by the end of the year, and at the time I thought that this was too high, but now think that this was a reasonable belief to have, and I regret not having deferred more to him.

Estimating the value of leaving London or other major cities

Here is a template for calculating risk, given one’s probabilities (also saved here and here). 

If we input the full range of our forecasters’ probabilities together with some default values, we get the following estimate of how many lost hours one loses in expectation as a result of staying in London in the medium term—where, because of the way we prompted forecasters, the “medium term” can range from one to three months:

If we instead input the forecasters’ aggregate, rather than the range, we arrive at: 

A mixture of both estimates gives a 90% confidence interval of ~2 to 300 hours lost. Personally, I would use this second estimate, but it’s hard to say why: maybe because I think that taking the minimum and maximum out of each question does a good job of filtering the least accurate forecasts.

Compare with a previous estimate back in March:

So, the danger of staying in London has increased by ~1-10x since March. We’d guess for most people reading this post moving out of the city for 1-3 months would still cause more value in lost productivity than the updated estimates of expected lost life hours, but it might be a closer call than it was previously.

For personal purposes, we probably don’t have a better decision rule than “leave major cities if any tactical nukes are dropped in Ukraine” (as this will ~10x risk).

Miscellanea

A sanity check

We can compare the directly elicited probability of nuclear war reaching London in October with the conditional steps multiplied directly:

The conditional steps are:

  1. What is the probability that Russia will use a nuclear weapon in Ukraine in the next MONTH?  0.053025 (5.303%)
  2. Conditional on Russia using a nuclear weapon in Ukraine what is the probability that nuclear conflict will scale beyond Ukraine in the next MONTH after the initial nuclear weapon use? 0.0254 (2.5%)
  3. Conditional on the nuclear conflict expanding to NATO, what is the chance that London would get hit, one MONTH after the first non-Ukraine nuclear bomb is used? 0.1424 (14%)

And if we multiply these together, we get 0.053025 * 0.0254 * 0.1424 = .00019178930400 (0.019% ~ 0.02%), versus 0.00065 (0.066%) when elicited directly. 

I think that the conditionals multiplied directly should be higher. Because the directly elicited probability assumes a scenario where escalation happens within one month, whereas the conditionals multiplied directly would include that scenario, but also scenarios where each escalation step is more staggered.

One way to think about this difference is that a ~3x difference when eliciting unlikely, <1% events is relatively normal. Personally, I (Nuño) would give more weight to the conditionals multiplied directly.

Counterfactual baseline risk

Forecasters also predicted on these counterfactual questions. 

The first two probabilities are dwarfed by the probabilities in the Russian conflict. The third probability indicates a very low baseline risk, but is also very sensitive to the individual forecasts.

A brief note on the aggregation method

We used the geometric mean of the samples with the minimum and maximum removed to better deal with extreme outliers, as described in our previous post. Note that the minimum (resp. maximum) do matter. For example, in [0.1, 1, 10, 100, 1000], the aggregate would be (1 * 10 * 100) ^ (1/3)  = 10. But if we remove 0.1, that aggregate would become (10 * 100) ^ (½) = 31.6. 

Acknowledgements

This is a project by Samotsvety. Thanks to Jared Leibowich, Jonathan Mann, Tolga Bilge, belikewater, Greg Justice (@slapthepancake), Misha Yagudin and Nuño Sempere for providing updates. Thanks as well to Eli Lifland for comments and suggestions, and to Daniel Kokotajlo and Bhuvan Singla for their probability mass app. 

  1. ^

    Dropping into the first person, I (Nuño) felt that the degree to which we updated, or at least the degree to which I personally updated, is indicative that our/my probability wasn’t a martingale), i.e., that it didn’t accurately price the likelihood of future movements. See some discussion about this here, in the context of Nassim Taleb criticizing Nate Silver. Overall, that update to me suggests we should give probabilities closer to 50%, to better adjust for future unknowns, which we maybe aren’t pricing in.

    On the other hand, other proud Samotsvety forecasters point out that our previous forecast was only for March, even though we presented the risk in annualized units. It’s also just straight-out possible that we are in the bottom 10-20% of scenarios. So overall we are not done with our post-mortem, which would also include personal updates in April &c.