The Economist explains
Michael Ben-Gad, a professor at London’s City University who has studied the credibility of long-term promises by governments, questions whether Nato’s commitment to collective defence is absolute and asks what would happen if Russia’s border guards crossed the bridge that separates Narva from Ivangorod and took the Estonian town.
“Would the US and western Europe really go to war to defend the territorial integrity of Estonia? I think Estonia has reasons to worry. Narva is the most obvious place; it is almost completely Russian-speaking,” he says.
NATO Article 5 essentially says that an attack on one is an attack on all. This is meant to be a deterrent; presumably Russia (or whoever) does not want to go to war against the combined might of NATO. But for a deterrent to work, it must be credible. Game theory suggests the NATO threat probably isn't.
In backward induction we start by comparing the payoffs to the choice made in the final decision and working our way backward to the first decision. If we play the game forward, it seems as though Russia should not invade; NATO promises to respond with a massive attack, which means the payoff to Russia of an invasion is not attractive. Backward induction points to a different outcome.
In that case, the last decision is whether or not to respond to a Russian invasion by attacking Russia. The problem here is that the payoff to NATO'S big military powers to attacking Russia is hugely negative. A third world war fought with conventional weapons is among the best possible outcomes, with nuclear war being among the worst. The payoff to not attacking Russia, by contrast, is a small cost (to countries not called Estonia, or Latvia or Lithuania, or maybe Poland). It is difficult to imagine the key NATO governments risking thousands, or perhaps millions, of citizens' lives for the integrity of Estonian territory.
So we then move to the penultimate decision. If the payoff to invasion is higher than that to not invading we can conclude that Russia will invade. Here we run into a little trouble since, on the face of things, not invading clearly entails a higher payoff, at least in terms of Russian welfare. But the identity of the decision-taker is important here. Clearly Mr Putin is willing to accept some economic cost to Russia to obtain foreign territory, so if our western eyes reckon it's idiotic to invade we're obviously not perceiving Mr Putin's utility function correctly. The man gets something out of expanding Russia, throwing NATO for a loop, and generally reliving the bad old days. So it's possible that Mr Putin will perceive the payoff to invading Estonia as positive. In that case, it is hard to imagine that American military threats will discourage him. Odds are decent that Mr Putin will start nibbling away at the Baltics after finishing with Ukraine.
That seems bad.
Is there anyway to prevent this outcome? There are a couple of actions that might alter Mr Putin's payoff structure. One would be to make threats that are both credible and a real deterrent to Russia. If America promised to counter any further aggression by locking Russia out of the global financial system (as it did to Iran) that might work, given that the policy would mean relatively minor economic costs for NATO and enormous and immediate economic pain for Russia. Another option would be to put American soldiers in harm's way, so that Russia could not invade NATO territory without directly harming American military personnel. Given that America could not help but respond forcefully to an attack on its own people, such a move might render the NATO guarantee toothy. Not toothless. So, basically, man the NATO-Russian border with American troops.
The problem in both of these cases is that—given that we're not exactly clear on Mr Putin's utility function and political constraints—there is some not-insignificant risk that either would make war more likely, and possibly much more likely.
One will read all sorts of chest-thumping about how what is needed is resolve or leadership or the pure, distilled machismo necessary to stare down Mr Putin. That's nonsense. Neither is it the case that NATO is a fundamentally weak and flawed organisation; or, it might be, but that is not really the problem here. The problem is that when a nuclear power doesn't want peace, there is just not an awful lot that can be done to write a happy ending to the story.