The CIA was "Probably" Right

In 1951, the CIA warned that a Soviet invasion of Yugoslavia was a “serious possibility.” Sherman Kent later said he meant about 65%. His colleagues heard something lower.

Kent asked the people who approved the phrase what number they had in mind. Their answers ran from 20% to 80%. In 1964 he proposed a standard scale for phrases like “probable” and “almost certain.”¹ The CIA ignored it.

In the 1970s, researchers asked 23 NATO intelligence officers² to translate phrases like “It is highly likely that the Soviets will invade Czechoslovakia” into percentages, then published the results as a dot chart.

That chart went everywhere from CIA foundational texts and university textbooks to blog posts and LinkedIn ‘thought pieces’. You’ve likely seen some version of it before.

Some version of the chart

One casual Sunday afternoon, I was comparing the 1977 Barclay original with a 2013 redrawing from the Pherson textbook. Same data, supposedly.

Except “We Believe” looked completely different.

So I did what should be the obvious thing: I checked. I digitised three published versions — Barclay (1977), Heuer (1999), and Pherson (2013) — and recorded every dot position.³

The number of dots per statement varies from 16 to 23 across the three versions; it should always be 23. The raw data has been lost so we can’t even reproduce the original chart.⁴

So was Kent actually right?

Despite the game of chart telephone, I was curious if Kent was right anyway - that people badly disagree about what probability words mean.

I ran a new survey: 99 respondents with undergraduate or higher education were shown 25 probability phrases in random order.⁵ The original 16 from Barclay plus nine more, including “frequent,” “realistic possibility,” “rare,” and “occasional.” ⁶

Zonination had already done something similar in 2015 on /r/samplesize, with 46 respondents and 17 phrases.⁷ That makes three samples: NATO officers in the 1970s, Reddit in 2015, mine in 2026. Three nicely independent samples.

So: Kent was right. He’s been right for fifty years.

“Probable” has meant about 70% for half a century. It is also notable that despite the random order of presentation, people still rebuild the same ladder from “certain” down to “impossible.” (even if they individually are quite divergent from the trend).

Despite the population consistency; individually the results are still have a wide spread - for example “We Believe”. Put ten people in a room and ask what it means; the gap between the most confident and least confident is about 40 percentage points.

You might be tempted to label the most extreme dots as outliers and obvious junk; however I’d caution against that. The responses passed the attention filters, and most still follow the same broad hierarchy. People are not clicking at random. Some might be ‘misunderstanding’ the question - but they were able to correctly identify the percentages for a coin-flip; and the chances of being struck by lightning.

There is an interesting asymmetry as well - Humans seem worse at low-probability, high-impact events. “It is highly unlikely that the adversary will deploy this capability” is exactly the sort of sentence where you want everyone to be in agreement.

What can we do about it.

While we’ve been, as a population, surprisingly consistent with our probabilistic language since the Cold War, decisions are not made by average readers; they are made by one person writing “unlikely” and another person reading it, and at that level we can produce dramatic inconsistencies.

Probability words do preserve the ranking, but not the quantity. In casual conversation that is often good enough. In intelligence estimates, it is not. Yet, most of the world still writes as though the words alone will do.

So here is mine - and Kent’s - remedy. When the number matters, be brave, write the number - especially when it means you can be wrong.