Estimative Intelligence Definition Essay
This classic exposition of estimative intelligence, which treats both its epistemology and its importance to the policymaker, was classified Confidential and published in the Summer 1968 number of Studies of Intelligence.
There are a number of things about policymaking which the professional intelligence officer will not want to hear. For example, not all policymakers can be guaranteed to be free of policy predilections prior to the time they begin to be exposed to the product of the intelligence calling. Indeed, there will be some policymakers who could not pass a rudimentary test on the "facts of the matter" but who have the strongest views on what the policy should be and how to put it into effect. We do not need to inquire as to how these men got that way or why they stay that way, we need only realize that this kind of person is a fact of life.
Nor should we be surprised to realize that in any policy decision there are a number of issues which we who devote ourselves solely to foreign positive intelligence may almost by definition be innocent of. The bulk of them, are, of course, purely domestic ones: domestic political issues, domestic economic issues, popular attitudes, public opinion, the orientation of the congressional leadership, and so on. Even if we know in our bones of the great weight which such issues have carried in many a foreign policy decision, we do not readily and consciously acknowledge it. Our wish, is, of course, to have our knowledge and wisdom about the foreign trouble spot show itself so deep and so complete that it will perforce determine the decision. The nature of our calling requires that we pretend as hard as we are able that the wish is indeed the fact and that the policymaker will invariably defer to our findings as opposed to the cries of some domestic lobby.
But consider for a moment how people other than ourselves and our consumers view these phenomena which I have just dismissed with a mild pejorative. Look, for example, at the table of contents of any of the recent books devoted to "How Foreign Policy Is Made." Or look at the lineup of lectures and discussions in the syllabus of any of our senior service schools; look particularly at the section devoted to national security policy formulation. You will find that intelligence and what it contributes to the task, far from enjoying the overpowering importance with which we--quite understandably--like to endow it, is casually ticked off as one of a score of forces at work.
The Credibility of Intelligence
Thus a certain amount of all this worrying we do about our influence upon policy is off the mark. For in many cases, no matter what we tell the policymaker, and no matter how right we are and how convincing, he will upon occasion disregard the thrust of our findings for reasons beyond our ken. If influence cannot be our goal, what should it be? Two things. It should be relevant within the area of our competence, and above all it should be to be credible. Let things be such that if our policymaking master is to disregard our knowledge and wisdom, he will never do so because our work was inaccurate, incomplete, or patently biased. Let him disregard us only when he must pay greater heed to someone else. And let him be uncomfortable--thoroughly uncomfortable--about his decision to heed this other.
Being uncomfortable is surely his second choice. Before he becomes uncomfortable he is going to ask himself if it is strictly necessary. This is of course the equivalent of asking himself if he really thinks that the information he has received from his intelligence colleagues is relevant to his problem and if he has to believe it. When we in intelligence look at the matter in this light we might consider ourselves fortunate that our policymaking consumers find so much of our product relevant, credible, and hence useful. Is there any way of categorizing that which is most happily, gratefully, and attentively read and that which is least? Perhaps a start can be made by having a quick critical look at three classical families of intelligence utterances.
First, basic intelligence. No question but that credibility is highest in this area of intelligence. Time and time again our consumer has need of something comparable to the perfect World Almanac or the perfect reference service. We come close to giving him just that, and nine times out of ten he is warmly appreciative of the breadth and depth of our knowledge and the speed with which we can handle his requests.
Second, how about current intelligence? There is probably less enthusiasm among consumers for this than for basic. They have a tendency to compare it--and unfavorably--to the daily press or the weekly news magazines; or they gripe because they often find it a gloss upon something they have just read in a cable.
Lastly, in the formal estimate credibility is lowest. It was more than a decade ago that Roger Hilsman, after interrogating scores of policymaking consumers of intelligence, concluded thus.(2) He discovered that the people with whom he talked were extremely grateful to intelligence when it came up with the facts that they felt they had to know before they went further with their policymaking and operating tasks. They seem to have gone out of their way to praise intelligence in its fact-finding role, but to be anything but grateful for intelligence utterances in the estimate category.
Why was this so? Although Hilsman does not make the point, one may safely infer from his findings: The policymaker distinguished in his own mind between things which he thought of as factual and those which he thought of as speculative. For the first he was grateful, for the second not at all.
This puts a number of questions before the house. Why should Hilsman's respondents (implicitly, at least) have questioned the credibility of intelligence estimates? Was it because the respondents had caught intelligence out in self-serving errors? Was it because they were fearful of being misled by intelligence? Had intelligence on its part ever done anything to merit this want of confidence on the part of its customers? If not, how did it come about that the very officer who besought the help of intelligence in one area eschewed intelligence in another?
The Nature of the Estimate
Let me begin with a look at estimates and the business of making them.
Let me first be quite clear as to the general and the particular meaning of the word "estimate" in the present context. In intelligence, as in other callings, estimating is what you do when you do not know. This is the general meaning. In this broad sense, scarcely an intelligence document of any sort goes out to its consuming public that does not carry some sort of estimate. Field reports are circulated only when someone has estimated that the source is sufficiently reliable and content sufficiently credible to be worthy of attention. Current intelligence items as often as not carry one of those words of likelihood--"probable," "doubtful," "highly unlikely," etc. and so forth--that indicate that someone has pondered and decided that the report should be read with something less than perfect assurance as to its accuracy. An endless number of important sentences in even the basic intelligence category carry the same evidence of this kind of speculative evaluation, i.e, estimating.
But what I have in mind in particular when I use the word "estimates" here are the formal intelligence documents which begin to examine a subject from the point of view of what is known about it, and then move on beyond the world of knowing and well into the world of speculating. When you reflect upon a whole large subject matter--the future of Greece or the armed strength of Communist China, for example--and realize that you cannot begin to know about either with the degree of certainty you know your own name, you reach for the next best thing to "knowing." You strive for some sort of useful approximation. In pursuit of this you evoke a group of techniques and ways of thinking, and with their help you endeavor logically and rationally (you hope) to unravel the unknown or at least roughly define some area of possibility by excluding a vast amount of the impossible. You know that the resultant, while still a lot better than nothing at all, will be in essence a mix of fact and judgment. Upon occasion it turns out to be almost exactly correct, but at the time you wrote it you expressed yourself with appropriate reservation.
To the extent that your judgment and the many quite subjective things which influence it are now involved, the man who reads this estimate will by no means accept it in the attitude of relaxed belief with which he reads, for example, that "not counting West Berlin, there are ten Länder in the FRG." It is this form of intelligence document that Hilsman's respondents were cool about. What follows is an attempt to explain the chill.
Let me ask you to think of one of these estimates in terms of the geometrical form called a pyramid. Think of the perfect estimate as a complete pyramid. At its base is a coagulation of all-but-indisputable fact. With an absolute minimum of manipulation on our part, the facts have arranged themselves to form what is quite clearly the base of a pyramid. They have spread out in the horizontal dimensions to the degree that we pretty well perceive its base area, and piled up in the vertical dimension generally to indicate the slope of its sides.
Knowing the nature of the base of the pyramid, to take an illustrative case, is like saying that we now have enough solid information to know that a photo image we have been wondering about is of an aircraft--not, say, a dairy ranch; more importantly, it is a bomber aircraft, not a transport. As to the other things we want to know about it--its performance characteristics--we are not at all certain. We are, however, in a good position to speculate about them.
Raising the Pyramid
Now back to the pyramid. Let us assume that when we know the general locus in space where the sides will converge to form the apex, we will have most of what we want. Let us assume that the exact point of the apex is exactly what we want, that if we know this with certainty we will have what we are after. For the bomber, constructing the apex would be reasoned speculations about how it will perform: How far it can fly, how high, how fast, and with what bomb load. Just as classical induction revealed the base of the pyramid, so now we call upon the other classical methodologies of deduction, and with their help we reason our way up the pyramid toward the top.
The factual stuff of the base of the pyramid is likely to be largely the fruit of our own intelligence-gathering efforts and so constitute a body of material about which we are better informed than our consumers. But we enjoy no such primacy with respect to the matter above. In fact, the talent to deduce rigorously is one which we share with any other educated and intellectually disciplined human. Furthermore, the advantage we enjoy with respect to base material can be and usually is dissipated by our habit of making it available to quite an array of non-intelligence types. The point is that the studious consumer can approach our mastery at the base and match us higher up. He can be his own estimator whenever he wishes to invest the time.
Let me not even seem to pretend that all conceptual pyramids in our area of work are constructed as described. The procedure which moves from the known to the unknown with a certain amount of tentative foraying as new hypotheses are advanced, tested, and rejected is merely the most respectable way. Its very opposite is sometimes employed, though usually with a certain amount of clandestinity.
The follower of this reverse method first decides what answer he desires to get. Once he has made this decision, he knows the exact locus of the apex of his pyramid but nothing else. There it floats, a simple assertion screaming for a rationale. This, then, is worked out from the top down. The difficulty of the maneuver comes to a climax when the last stage in the perverse downward deduction must be joined up smoothly and naturally with the reality of the base. This operation requires a very considerable skill, particularly where there is a rich supply of factual base-material. Without an artfully contrived joint, the whole structure can be made to proclaim its bastardy, to the chagrin of its progenitor.
But even under the respectable method the intelligence estimator at some moment in the construction process reaches the place where he has used his last legitimate deductive crutch and must choose one of three possible courses.
The first is to let himself be propelled by the momentum of his reasoning into a final and fairly direct extrapolation. The effect of this is to put a sharpish top on the pyramid--a measure which, in turn, has the effect of telling his audience that he is pretty sure that he has discerned the outlines of what must be the truth. For the bomber it would be like saying: "Thus we conclude that the bomber in question is almost certainly a supersonic aircraft of medium range. See Table II for our estimate of its performance characteristics."
The second is not to make this final extrapolation but to leave the pyramid truncated near its apex. This has the effect of telling the reader that you have narrowed the range of possibilities down to only a few. The further down you truncate, the wider their range. Thus the most unsatisfactory kind of intelligence construction is often that which perforce has to stop where the factual stuff of the base runs out. Often it is the equivalent of issuing the most general kind of news and asking the reader to suspend judgment pending the appearance of new evidence. For example: "Thus, we are unable at this time to be more precise regarding the performance characteristics of this bomber. It is possible that it is a new supersonic medium."
The third is what I will call "the look before the leap" or the "clandestine peep ahead." It is, one may hope, less often used by the intelligence professional than by the policy officer doing his own estimating. What you do is look hard at the final extrapolation and take full stock of where you will be if you go for it. Then, having taken stock, you ask yourself if you really wish to subscribe to this conclusion.
In the case I have in mind, you recoil. It may be that by making it yours you will be depicting yourself a non-patriot, or someone soft on Communism. It may be that by implication you can be made to seem a harsh critic of a higher authority or a scoffer at one of his policies. It may be that you will be doing the budget claims of your department or agency a grave disfavor. Or most important of all you realize that your findings may be advanced to support a policy which you oppose or that they do not support with sufficient vigor a policy which you favor.
If you have taken the peep ahead and find the prospect not to your taste, you can settle for the second course and simply not complete the estimate. Or you can back down on your argument, tearing it up as you go. Then when you have found a salubrious ground for another start, you can reargue your case upwards--perhaps using a few facts which you had dismissed as irrelevant the first time through, perhaps giving more weight to this analogy and forgetting about that, etc., etc. Thus, with a small amount of tinkering you can create a somewhat different conceptual pyramid whose base is still the same, but whose apex will lie in a zone much less dangerous to your job security or much more appropriate to the requirements of your policy preconceptions.
The Policy Welcome
Irrespective of which of the three ways of handling the problem you choose and irrespective of the substantive conclusion--or lack of it--the completed estimate will be bad news to one if not more of its important readers: it may undercut a long-held position or destroy a line of painfully developed argument; it may indicate the unwisdom of a plan or the malallocation of large sums of money. Another thing you may be sure of is that he will react as any recipient of bad news reacts--the reflex is one of "I don't believe you." Need I emphasize again that estimates are far more vulnerable to the criticism which is bound to accompany incredulity than are propositions which are stated, at least, as if they were fact.
The disappointed consumer may begin with a hard look at our pyramid's factual base. He may find some loose masonry which can be jimmied apart, and then jimmy. He may find some quite substantial building stones left off to one side, stones, which, although of the same material and cut to fit some sort of geometrical form, were not incorporated into the base structure. He will speedily perceive that if these are chiseled a bit here and there they can be made to fit into this structure, with the result that they change some important aspects of its configuration. You may be sure he will soon focus on the upper zones of our pyramid.
One thing he will be most alert to is any evidence that intelligence, having taken the "peep ahead" and found the pyramid about to peak at an unwanted place, went on to take the corrective action I have indicated. If he can find evidence of this sort of disingenuous case-making, he will attack with very weighty weaponry. Before he is done he may be able to prove to himself and a number of others that the so-called intelligence contribution is a fraud--nothing more nor less than a policy brief brazenly masquerading as an intelligence estimate.
In these terms we may readily understand why a good many of Hilsman's respondents felt as they did about the value of intelligence estimates. For purposes of fuller explanation, let us suppose that an intelligence estimate on the Banana Republics had been prepared; let us suppose that our policymaking reader Mr. "A" is his department's authority on these Republics. A tour of his psyche as he reads the paper may be illuminating.
First, let us assume that the estimate accords in very high degree with his own estimate of the present and probable future situation in Banania. His psyche will begin to purr in contentment; "What a remarkably perceptive document," it will whisper. But this may be as far as the word of praise gets. When the moment comes to articulate his comment on the estimate, he is less likely to praise it than to proclaim, "This is exactly what I have been saying all along. Why in the world do we have to have someone who knows less of the matter than I say so before anyone pays attention?" In short, as far as he is concerned, the intelligence effort that went into the study was unnecessary. "A" may not always feel this way, particularly if during the policy debate he realizes that he can make points against his opponents by citing the estimate as a dispassionate outside opinion.
Alternatively, let us assume that the estimate accords not at all with the views of Mr. "B." He will be unhappy, for he will realize that if the conclusions of the estimate are believed by his peers and superiors, the policy which he has been championing will have to be modified--perhaps drastically. If he wishes to stay in the fight, then, he must be prepared to attack the intelligence estimate as misleading and erect one of his own to replace it.
Lastly, let us assume that the policy issue is one of those which is going to be settled almost entirely on the basis of some purely domestic matter: The cotton lobby, the gold flow, the budget, and so on. Our policymaking consumer does not have to attack the substance of the irrelevant estimate. He will chuckle patronizingly to himself while his psyche warms in the feeling of superiority to those poor boobs in intelligence who have thought that what they called the "Situation and Prospects in X" could have any bearing on the way US policy towards X is being shaped today. Out loud he wonders how such naivetè can persist; he has no comment on the substance of the estimate.
These views of an estimate as unnecessary, misleading, or irrelevant may coincide with those of some of the people whom Hilsman polled and explain why they were less grateful for estimates than for what they considered factual intelligence issuances.
How seriously should we in intelligence take the indictment which damns our estimating work as unnecessary, or misleading, or irrelevant? Take the misleading charge first. If it is made, and if it is true because the document was designed that way, then it must be taken very, very seriously indeed. For this accusation implied that the peep ahead had been taken and the necessary retracing of steps and reconstruction had followed so that the conclusion of the estimate suited the policy predispositions of the estimators. They have been caught out in their stupidity, and their credibility, at least for this estimate, is dead. It is dead not merely for the reader who found the conclusions abhorrent, but for all the others who found out by themselves or were told.
If the same group of estimators are caught out a second or third time, their credibility will probably be dead for good. Thereafter almost any intelligence pronouncement they or their associates make will be slightingly referred to as propaganda, and perhaps not even read. They have not only lost all hope of directly influencing policy, they have lost what is even more important because more attainable than direct influence. This is the indirect influence which they might have exercised through an honest contribution to the debate which ought to precede every substantial policy decision.
Suppose the charge of misleading is made simply as a function of a committed reader's general disbelief or annoyance, and suppose that, try as he may, he cannot show a trace of bad faith on the part of the estimators. The estimators are confronted with nothing more sinister than a human disagreement, perhaps from a reader whose nose is out of joint. This is just life.
What of the charge, unnecessary? The question here is--unnecessary to whom? To everyone involved in the policy decision? Already I have dealt with Mr. "A" to whom it was unnecessary because it accorded exactly with his views, and Mr. "B" to whom it was unnecessary and many times worse because he found it misleading. But are these the only two officers or two kinds of officers involved? Is there perhaps not a Mr. "C" or Messrs. "C" who have no more than a layman's knowledge of the subject but who must participate in the policy debate and decision? Of course there are the Messrs. "C," and important men they are. The President, upon many an occasion, is a Mr. "C," and so are members of his staff and his Security Council. They have found the estimate anything but unnecessary.
It does not follow, however, that the impact which the estimate may make upon the Mr. "C"s will in itself cause the defeat of the dissenting Mr. "B"s. What it will do is to force the Mr. "B"s to put forth a better effort. This will stimulate the Mr. "A"s themselves to better effort. At a minimum, the intelligence estimate will have made its contribution in the way it promoted a more thorough and enlightened debate and a higher level of discourse within the high policymaking echelon. At a maximum it may have denied a wrong-headed Mr. "B" an easy triumph.
Lastly, the charge of irrelevant. This rested upon the fact that the foreign policy decision was going to have to be made on the basis of a domestic consideration, something about which the estimate is wholly--and properly--mute. But it is just possible that the domestic consideration is not all that important and that the national interest is not really being served by this sort of deference to it. It may be that the estimate helped the policy people to reach this new appreciation of the national interest. Hence, even if the decision I am talking about gets made in conformity with the wish of the domestic pressure group, maybe the next such decision will not.
Truth Before Power
I suppose that if we in intelligence were one day given three wishes, they would be to know everything, to be believed when we spoke, and in such a way to exercise an influence to the good in the matter of policy. But absent the Good Fairy, we sometimes get the order of our unarticulated wishes mixed. Often we feel the desire to influence policy and perhaps just stop wishing here. This is too bad, because to wish simply for influence can, and upon occasion does, get intelligence to the place where it can have no influence whatever. By striving too hard in this direction, intelligence may come to seem just another policy voice, and an unwanted one at that.
On the other hand, if intelligence strives for omniscience and strives to be believed, giving a third place to influence, serendipity may take over. Unselfconscious intelligence work, even in the speculative and highly competitive area of estimates, may prove (in fact, has proved many times) a key determinant in policy decision.
(1) Adapted by the author from his presentation before the September 1966 Intelligence Methods Conference in London.
(2) [Editor's Note: Hilsman was then Chief of the State Department's Bureau of Intelligence and Research (INR).]
Posted: Mar 19, 2007 11:00 AM
Last Updated: Jul 07, 2008 02:09 PM
This classic piece on the need for precision in intelligence judgments was originally classified Confidential and published in the Fall 1964 number of Studies in Intelligence. Although Sherman Kent's efforts to quantify what were essentially qualitative judgments did not prevail, the essay's general theme remains important today.
The briefing officer was reporting a photoreconnaissance mission.(1) Pointing to the map, he made three statements:
- "And at this location there is a new airfield. [He could have located it to the second on a larger map.] Its longest runway is 10,000 feet."
- "It is almost certainly a military airfield."
- "The terrain is such that the Blanks could easily lengthen the runways, otherwise improve the facilities, and incorporate this field into their system of strategic staging bases. It is possible that they will." Or, more daringly, "It would be logical for them to do this and sooner or later they probably will."
The above are typical of three kinds of statements which populate the literature of all substantive intelligence. The first is as close as one can come to a statement of indisputable fact. It describes something knowable and known with a high degree of certainty. The reconnaissance aircraft's position was known with precision and its camera reproduced almost exactly what was there.
[Top of page]
The second is a judgment or estimate. It describes something which is knowable in terms of the human understanding but not precisely known by the man who is talking about it. There is strong evidence to sustain his judgment: the only aircraft on the field are military aircraft, many are parked in revetted hardstands, the support area has all the characteristics of similar known military installations, and so on. Convincing as it is, this evidence is circumstantial. It cannot justify a flat assertion that this is a military airfield. It makes the case, say, 90 percent of the way. And some sort of verbal qualifier is necessary to show that the case is a 90-percenter, not a 100. This is why the briefer said "almost certainly."
The third statement is another judgment or estimate, this one made almost without any evidence direct or indirect. It may be an estimate of something that no man alive can know, for the Blanks may not yet have made up their minds whether to lengthen the runways and build up the base. Still the logic of the situation as it appears to the briefer permits him to launch himself into the area of the literally unknowable and make this estimate. He can use possible to indicate that runway extension is neither certain nor impossible, or he can be bolder and use probably to designate more precisely a degree of likelihood, a lower one than he had attached to his estimate regarding the character of the airfield.
Generally speaking, the most important passages of the literature of substantive intelligence contain far more statements of the estimative types two and three than of the factual type one. This is the case because many of the things you most wish to know about the other man are the secrets of state he guards most jealously. To the extent his security measures work, to that extent your knowledge must be imperfect and your statements accordingly qualified by designators of your uncertainty. Simple prudence requires the qualified in any type-three statement to show a decent reticence before the unknowable.
Concern over these qualifiers is most characteristic of that part of the intelligence production business known as estimates. This is no small recondite compartment; it extends to almost every corner of all intelligence research work, from the short appraisals or comments of a reports officer to the full-dress research study of the political or economic analyst. Practically all substantive intelligence people constantly make estimates. The remarks that follow are generally addressed to all these people and their readers, but most especially are they addressed to that particular institution of the estimating business known as the National Intelligence Estimate and its audience.
The NIE, taking into account the high echelon of its initiators, producers, and consumers, should be the community's best effort to deal with the relevant evidence imaginatively and judiciously. It should set forth the community's findings in such a way as to make clear to the reader what is certain knowledge and what is reasoned judgment, and within this large realm of judgment what varying degrees of certitude lie behind each key statement. Ideally, once the community has made up its mind in this matter, it should be able to choose a word or a phrase which quite accurately describes the degree of its certainty; and ideally, exactly this message should get through to the reader.
It should not come as a surprise that the fact is far from the ideal, that considerable difficulty attends both the fitting of a phrase to the estimators' meaning and the extracting of that meaning by the consumer. Indeed, from the vantage point of almost fourteen years of experience, the difficulties seem practically insurmountable. The why and wherefore of this particular area of semantics is the subject of this essay.
Let me begin with a bit of history.(2)
[Top of page]
Early Brush with Ambiguity
In March 1951 appeared NIE 29-51, "Probability of an Invasion of Yugoslavia in 1951.(3) The following was its key judgment, made in the final paragraph of the Conclusions: "Although it is impossible to determine which course the Kremlin is likely to adopt, we believe that the extent of Satellite military and propaganda preparations indicates that an attack on Yugoslavia in 1951 should be considered a serious possibility." (Emphasis added.) Clearly this statement is either of type two, a knowable thing of which our knowledge was very imperfect, or of type three, a thing literally unknowable for the reason that the Soviets themselves had not yet reached a binding decision. Whichever it was, our duty to look hard at the situation, decide how likely or unlikely an attack might be, and having reached that decision, draft some language that would convey to the reader our exact judgment.
The process of producing NIEs then was almost identical to what it is today. This means that a draft had been prepared in the Office of National Estimates on the basis of written contributions from the IAC(4) agencies, that a score or so of Soviet, Satellite, and Yugoslav experts from the intelligence community labored over it, and that an all but final text presided over by the Board of National Estimates had gone to the Intelligence Advisory Committee. There the IAC members, with the DCI in the chair, gave it its final review, revision, and approval.
As is quite obvious from the sentence quoted above, Soviet and Satellite intentions with respect to Yugoslavia were a matter of grave concern in the high policy echelons of our government. The State Department's Policy Planning Staff was probably the most important group seized of the problem. Its chairman and members read NIE 29-51 with the sort of concentration intelligence producers can only hope their product will command.
A few days after the estimate appeared, I was in informal conversation with the Policy Planning Staff's chairman. We spoke of Yugoslavia and the estimate. Suddenly he said, "By the way, what did you people mean by the expression `serious possibility'? What kind of odds did you have in mind?" I told him that my personal estimate was on the dark side, namely, that the odds were around 65 to 35 in favor of an attack. He was somewhat jolted by this; he and his colleagues had read "serious possibility" to mean odds very considerably lower. Understandably troubled by this want of communication, I began asking my own colleagues on the Board of National Estimates what odds they had had in mind when they agreed to that wording. It was another jolt to find that each Board member had had somewhat different odds in mind and the low man was thinking of about 20 to 80, the high of 80 to 20. The rest ranged in between.
Of my colleagues on the Board at least one--maybe more--shared my concern. My most obvious co-worrier was Max Foster.(5) He and I were shaken perhaps more by the realization that Board members who had worked over the estimate had failed to communicate with its audience. This NIE was, after all, the twenty-ninth that had appeared since General Smith had established the Office of National Estimates. Had Board members been seeming to agree on five months' worth of estimative judgments with no real agreement at all? Was this the case with all others who participated--ONE staffers and IAC representatives, and even IAC members themselves? Were the NIEs dotted with "serious possibilities" and other expressions that meant very different things to both producers and readers? What were we really trying to say when we wrote a sentence such as this?
What we were trying to do was just what my Policy Planning friend had assumed, namely to quote odds on this or that being the case or taking place in the future. There is a language for odds; in fact there are two--the precise mathematical language of the actuary or the race track bookie and a less precise though useful verbal equivalent. We did not use the numbers, however, and it appeared that we were misusing the words.
[Top of page]
The No-Odds Possible
Our gross error in the Yugoslav estimate, and perhaps in its predecessors, lay in our not having fully understood this particular part of our task. As Foster and I saw it, the substantive stuff we had been dealing with had about it certain elements of dead certainty: Stalin was in charge in the USSR, for example. These, if relevant, we stated affirmatively or used impliedly as fact. There were also elements of sheer impossibility (Yugoslavia was not going to crack off along its borders and disappear physically from the face of the earth); these we did not bother to state at all. In between these matters of certainty and impossibility lay the large area of the possible. With respect to the elements herein we could perceive some that were more likely to happen than not, some less likely. These were the elements upon which we could make an estimate, choosing some word or phrase to convey our judgment that the odds were such and such for or against something coming to pass.At the race track one might say:
There are ten horses in the starting gate. It is possible that any one of them will win--even the one with three legs.
But the odds (or chances) against the three-legger are overwhelming.
Here, as in estimating Yugoslav developments, there is evidence to justify the citing of odds. But in the world that intelligence estimates try hardest to penetrate--a world of closed covenants secretly arrived at, of national business conducted behind the walls of all but impenetrable security, of skillfully planned deceptions, and so on--such evidence is by no means invariably at hand. In a multitude of the most important circumstances--situations you are duty bound to consider and report on--about all you can say is that such and such is neither certain to happen nor is its happening an impossibility. The short and proper way out is to say that its happening is possible and stop there without any expression of odds. If you reserve the use of "possible" for this special purpose--to signal something of high importance whose chances of being or happening you cannot estimate with greater precision--hopefully you will alert your reader to some necessary contingency planning. (You may not if you have dulled him by citing a lot of "possibles" of little real consequence.)
If our gross error lay in not perceiving the correctness--or at any rate the utility--of the above formulation, our particular error lay in using the word "possibility" with the modifier "serious." Foster and I felt that it was going to be difficult enough for the estimators to communicate a sense of odds even if they stuck to a fairly rigorous vocabulary; it was going to be impossible if the vocabulary were permitted to become as sloppily imprecise as in normal speech. We had to have a way of differentiating between those possible things about which we could make a statement of likelihood and the other possible things about which we could not. The first cardinal rule to emerge was thus, "The word `possible' (and its cognates(6)) must not be modified." The urge to drop into ordinary usage and write "just possible," "barely possible," "a distinct [or good] possibility," and so on must be suppressed. The whole concept of "possibility" as here developed must stand naked of verbal modifiers.(7)
[Top of page]
An Odds Table
Once Foster and I had decided upon this first cardinal rule we turned to the elements where likelihood could be estimated. We began to think in terms of a chart that would show the mathematical odds equivalent to words and phrases of probability. Our starter was a pretty complicated affair. We approached its construction from the wrong end. Namely, we began with 11 words or phrases that seemed to convey a feeling of 11 different orders of probability and then attached numerical odds to them. At once we perceived our folly. In the first place, given the inexactness of the intelligence data we were working with, the distinctions we made between one set of odds and its fellows above and below were unjustifiably sharp. And second, even if in rare cases one could arrive at such exact mathematical odds, the verbal equivalent could not possibly convey that exactness. The laudable precision would be lost on the reader.
So we tried again, this time with only five gradations, and beginning with the numerical odds. The chart that emerged can be set down in its classical simplicity thus:
|The General Area of Possibility|
|93%||give or take about 6%||Almost certain|
|75%||give or take about 12%||Probable|
|50%||give or take about 10%||Chances about even|
|30%||give or take about 10%||Probably not|
|7%||give or take about 5%||Almost certainly not|
Important note to consumers: You should be quite clear that when we say "such and such is unlikely" we mean that the chances of its NOT happening are in our judgment about three to one. Another, and to you critically important, way of saying the same thing is that the chances of its HAPPENING are about one in four. Thus if we were to write, "It is unlikely that Castro will attempt to shoot down a U-2 between now and November 1965," we mean there is in our view around a 25-percent chance that he will do just that. If the estimate were to read, "It is almost certain Castro will not . . . ," we would mean there was still an appreciable chance, say 5 percent or less, that he would attempt the shootdown.
We had some charts run up and had some discussions in the community. There were those who thought the concept and the chart a very fine thing. A retired intelligence professional thought well enough of it to put it into a book.(8) CIA officers, addressing War College audiences and the like, would sometimes flash a slide and talk about it. A few copies got pasted on the walls of estimates offices in the community. Some people were sufficiently taken that they advocated putting it on the inside back cover of every NIE as a sort of sure-fire handy glossary.
There were also those who did not think about the idea at all, and others in opposition to it. Some fairly important people who had a professional stake in this kind of thinking never took the trouble to learn what it was all about. A good many did take a little trouble and laughed. Still a third group found out all they needed to know and attacked the whole proposition from a hard semantic base point. Of these more later.
In the face of this inertia and opposition and with the early departure of my only solid ally, Max Foster, I began backing away from bold forward positions. I did continue harassing actions and in the course of making a nuisance of myself to associates and colleagues did pick up some useful converts, but I dropped all thought of getting an agreed airtight vocabulary of estimative expressions, let alone reproducing the chart in the rear of every NIE. With the passage of time it has appeared that the guerrilla strategy thrust upon me by circumstance was the only one holding any chance of success. In almost 14 years this article is my first serious and systematic attempt to get the message across, and it probably would not have been written if David Wark had not consulted me about his foray into the same semantic problem.(9)
[Top of page]
The Aesthetic Opposition
What slowed me up in the first instance was the firm and reasoned resistance of some of my colleagues. Quite figuratively I am going to call them the "poets"--as opposed to the "mathematicians"--in my circle of associates, and if the term conveys a modicum of disapprobation on my part, that is what I want it to do. Their attitude toward the problem of communication seems to be fundamentally defeatist. They appear to believe the most a writer can achieve when working in a speculative area of human affairs is communication in only the broadest general sense. If he gets the wrong message across or no message at all--well, that is life.
Perhaps I overstate the poets' defeatism. In any case at least one of them feels quite strongly that my brief for the "mathematicians" is pretty much nonsense. He has said that my likening my side to the mathematician's is a phony; that I am in fact one with the sociologists who try by artificial definitions to give language a bogus precision. He has gone on to stress the function of rhetoric and its importance. And he has been at some pains to point out how handy it would be to use expressions like "just possible," "may well," and "doubtless" as they are loosely used in conversation. Could there not be an occasional relaxation of the rule?
Suppose one wrote a sentence: "Khrushchev may well have had in the back of his mind such and such, or indeed it is distinctly possible that somebody had just primed him. . . ." Now suppose you delete the "well" and the "distinctly"; has anything been lost? There will be those who point out that "may well" and "distinctly possible" do convey a flavor which is missing without them. Of course the flavor in question is the flavor of odds, communicated without quoting them. The poets would probably argue that in a sentence of this sort the introduction of any of the terms for particular odds would make the writer look silly. Everybody knows that you could not have the evidence to sustain the use of, say, "probably" in these two instances. Hence, you can only suggest odds by the use of the "may well" and "distinctly possible" and so say something without saying it, in short, fudge it. The poets feel wounded when urged to delete the whole ambiguous sentence, arguing that this serves only to impoverish the product. They grow impatient when you advocate dropping only the "well" and the "distinctly." And as for your accusation of fudging, they generally counterattack, inviting you to write something that fudges nothing.
There is a point that the poets can make with telling effect. It is that there are probably just as many reading poets as there are writing poets, and these are going to be numb to the intended meaning of the "mathematician" writer. If you write to give no more than just the general idea or general feel you may get through with great success. Per contra, if you break your heart in an endeavor to make yourself fully and precisely understood, you may not.
I realize the truth in the above; I am not reconciled; I deplore it.
[Top of page]
The Growth of Variants
Even if there had been no poets it would have been an impractical idea to print a chart on the inside of the back page of each NIE as a sort of glossary. To have used the one on page 133 and stuck to these words exclusively would have imposed intolerable restraints upon the prose. Even if it had been desirable it would have been impossible to enforce such rigidity. But this was really never at issue: from the start a number of perfectly legitimate synonyms for the concept of possibility and a number for each of the five orders of likelihood were generally recognized.(10)
|Almost certain||virtually certain |
all but certain
odds [or chances] overwhelming
|50-50||chances about even |
chances a little better [or less]
|Probably not(14)||we believe that . . . not |
we estimate that . . . not
we doubt, doubtful
|Almost certainly not||virtually impossible |
some slight chance
If the chart were expanded to take care of these, it probably would not fit on the inside back cover of the NIE, and even if it could be made to, its complexity would probably exasperate gentle reader more than it would edify him. Still worse, he would be confused by changes that would have to be made in it from time to time, always to accommodate newcomers among the accepted expressions.
The table of synonyms above did not come into being all at once; it has grown to its present size by accretion. "We believe" came in rather early, and as I remember viaGeneral Smith himself. "We estimate" was a bit later; "we think," "we expect," and "we judge" are part way in.(15) If they make it all the way, I trust they will be used and understood in the "probably"/"we believe" bracket. "We doubt" has been accepted within the last few years as a legitimate equivalent of "probably not." There will be others--I sincerely hope not very many. Keeping them out will take some doing. In the past, whatever the rigor insisted upon at the working and drafting level, who was there to tell a General Smith or a Mr. Dulles, as he presided over the IAC or USIB, that the revision he had just written out on a piece of yellow paper is not permissible?
[Top of page]
Consistency in Usage
From my remarks about the poets, it should be clear that my sympathies lie with their mathematical opponents. But we mathematically-inclined are ourselves not in good array. You might almost say that some of us are talking in the decimal, others in the binary, and still others in the root five or seven systems.
For example, consider the letter-number device which has been standard with attach and other reporting services, A-2, C-3, F-6, etc. The numbers 1, 2, 3, 4, 5, and 6 designating the quality of a report's content stand for, respectively: (1) confirmed by other independent or reliable sources; (2) probably true; (3) possibly true; (4) doubtful; (5) probably false; and (6) cannot be judged. Note that the number 3, "possibly true," is in the middle of the scale of odds, doing the duty I have hoped it should never be asked to do.
Or consider the findings of a distinguished intelligence research project. The object was to identify certain military units with respect to the chances of their existence or nonexistence. One group of units was called "firm," another "highly probable," a third "probable," and a fourth general group "possible." Except for one important thing, this kind of ordering was wholly to my taste. The word "firm" was unfortunately not used, as one might expect, to describe a condition of 100 percent certainty. Its begetters, upon cross-examination, owned that it was meant to indicate something like 90-95 percent--roughly the equivalent of my "almost certain." This usage puts the lower categories slightly askew from the terminology of my chart--"highly probable" equating to my "probable" and "probable" to my "chances better than even." "Possible," however, was used exactly as I have felt it should be used, to designate something in the range of chances between the absolute barriers of "certainty" and "impossibility" to which no numerical odds could be assigned.
There are other heresies among the mathematicians, if they can be so proclaimed. For example, look at the way in which photointerpreters have defined their key evaluative words:
Suspect--Evidence is insufficient to permit designation of a function with any degree of certainty, but photography or other information provides some indications of what the function may be.
Possible--Evidence indicates that the designated function is reasonable and more likely than other functions considered.
Probable--Evidence for the designated function is strong and other functions appear quite doubtful.
This kind of formulation shows that someone--probably a number of people--had spent a good amount of time striving for a set of rigorous definitions. If you pause long enough to realize that the photointerpreter's first problem is identification and then take a hard look at his word "suspect," you will see that it parallels my usage for "possible." But the PIs have preempted "possible" for other duty. Their "possible" fits nicely into the slot of "probable" in my scale of values, and their "probable" into my "almost certain."
We are in disarray.
[Top of page]
To Estimate or Not
The green language of ordinary conversation abounds with estimates given lightly and with a high order of confidence: "You're a shoo-in," "Not a Chinaman's chance," "A million to one." When you hear one of these expressions or read its more decorous counterpart you may realize that the matter at issue and the related judgment required little soul-searching on the part of the estimator. In the intelligence business, too, there are many occasions when the obscurities of the unknown are easily pierced and we can launch an estimative "probably" or an "almost certainly not" with speed and conviction.
It is unfortunate that intelligence estimators are not allowed this kind of freedom in brushing off requests for estimates of the totally impenetrable. Some way or another a convention has been established by which we may not write the sentence: "It is impossible to estimate such and such." If we try this maneuver our masters will often rudely ask, "Why can't you; what are you paid for anyway?" If they do not bludgeon us thus, they employ a combination of blackmail and flattery before which even the most righteous among us are likely to fall. The play goes like this: "You say you cannot estimate the number, type, and performance characteristics of Chinese Communist long-range missiles for mid-1970. This is data which is absolutely essential for my planning. Obviously no one expects you to be wholly accurate or very confident of your findings. But you people are after all the experts, and it would be too bad if I had to go to others for this stuff who know far less about it than you. And that is exactly what I will do if you refuse my request."
At this point we do not invite our would-be consumer to seek out his own crystal ball team. We accept his charge, but with grave reservations. Sometimes we try to stay honest by introducing contingencies. "This will probably continue to be the case but only if . . . , if . . . , and if . . . ." Then without closing out the contingencies with firm estimates (which we are plainly unable to make) we merely talk about the "ifs," hoping that he will keep them in mind as time unfolds and that when sufficient returns are in he will himself make the estimate or ask us to have a second look.
At other times again, when it is the whole subject rather than one of its parts that cannot be estimated, we meet the impossible frontally. We scrupulously avoid the word "estimate" in describing the document and its findings. Rather, we proclaim these to be intelligence assumptions for planning. In our opening paragraphs we are likely to be quite specific as to where our evidence begins and ends, how we are speculating about quantities of things that the other man may produce without knowing whether he has yet made the decision to produce so many as one. We acknowledge our use of the crutch of US analogy, and so on. We promise to speak, not in discrete figures, but in ranges of figures and ranges of our uncertainty regarding them.
Some years back we were obliged by force majeur to compose some tables setting forth how the Blanks might divide up an all-but-undreamed-of stockpile of fissionable material among an as-yet-unborn family of weapons. There were of course the appropriate passages of verbal warning, and then, on the chance that the numerical tables should become physically separated from the warning, the tables were overprinted in red, "This table is based on assumptions stated in . . . . Moreover, it should not be used for any purpose whatever without inclusion, in full, of the cautionary material in . . . ." More recently we have issued a document which not only began with a fulsome caveat but was set off by a format and color of paper that were new departures.
[Top of page]
The Lurking Weasel
Unhappily, making the easy estimate is not the commonplace of our trade; making the impossible one is happily equally rare. What is the commonplace is the difficult but not impossible estimate. And how we, along with all humanity, hate the task! How fertile the human mind in devising ways of delaying if not avoiding the moment of decision! How rich the spoken language in its vocabulary of issue-ducking! "I have a sneaker that . . . ," "I'd drop dead of surprise if . . ."--expressions with sound but upon reflection almost without meaning. How much conviction, for example, do you have to have before you become possessed of a sneaker; how much of the unexpected does it take to cause your heart to fail?
Even the well-disciplined intelligence brotherhood similarly quails before the difficult but not impossible estimate and all too often resorts to an expression of avoidance drawn from a more elegant lexicon. What we consciously or subconsciously seek is an expression which conveys a definite meaning but at the same time either absolves us completely of the responsibility or makes the estimate at enough removes from ourselves as not to implicate us. The "serious [or distinct] possibility" clan of expressions is a case in point.
Look at our use of "apparently" and "seemingly" and the verbal "appears" and "seems." We, the writers, are not the unique beings to whom such and such "appears" or "seems" to be the case; with these words we have become everybody or nobody at all. So also with "suggests" and "indicates." Perhaps the "to us" is implicit, but we do not so state; and far more importantly, we practically never say why our suggestibilities were aroused or assess the weight of the reason that aroused them. So still again with "presumably," "ostensibly," and--most serious of all--"reportedly" otherwise unmodified. The latter taken literally and by itself carries no evaluative weight whatsoever, and who should know this better than we ourselves who each day handle scores of "reports" whose credibility runs up and down the scale between almost certain truth and almost certain nonsense. It is a pleasure to report--authoritatively--that you will find very few unmodified "reportedlys" in the NIEs.
We say "the Soviets probably fear that such and such action will cause thus and so." What I think we mean is "The Soviets probably estimate that if they do such and such the reaction will be disadvantageous to them." If we say "they probably hope . . ." we mean roughly the opposite. We talk of another country's willingness "to risk such and such." This is a shorthand, and probably an unconscious one, for the country's having estimated the odds against the unwanted thing's happening as well as how unacceptable the unwanted thing would be if it occurred. Its "risking the danger" removes the critical judgment a step or two from our personal responsibility.
Words and expressions like these are far too much a part of us and our habits of communication to be banned by fiat. No matter what is said of their impreciseness or of the timidity of soul that attends their use, they will continue to play an important part in written expression. If use them we must in NIEs, let us try to use them sparingly and in places where they are least likely to obscure the thrust of our key estimative passages.
Here may I return to the group to which I have especially addressed the foregoing--the brotherhood of the NIE. Let us meet these key estimates head on. Let us isolate and seize upon exactly the thing that needs estimating. Let us endeavor to make clear to the reader that the passage in question is of critical importance--the gut estimate, as we call it among ourselves. Let us talk of it in terms of odds or chances, and when we have made our best judgment let us assign it a word or phrase that is chosen from one of the five rough categories of likelihood on the chart. Let the judgment be unmistakable and let it be unmistakably ours.
If the matter is important and cannot be assigned an order of likelihood, but is plainly something which is neither certain to come about nor impossible, let us use the word "possible" or one of its stand-ins--and with no modifier.
[Top of page]
(1) This particular briefing officer was not the photointerpreter. See [p. 138 in this volume] for the special language of P/Is.
(2) Harry H. Ransom's Central Intelligence and National Security (Cambridge, MA, 1958) carries on pp. 196-197 a bob-tailed and somewhat garbled version of it.
(3) [NIE 29-51 "Probability of an Invasion of Yugoslavia in 1951" was disseminated on 20 March 1951. A follow-up study, NIE 29/1-51 "Review of the Conclusions of NIE 29 ÔProbability of an Invasion of Yugoslavia in 1951'" was issued on 9 May 1951. Both documents have been declassified and released to the National Archives.]
(4) Intelligence Advisory Committee, USIB's predecessor.
(5) Maxwell E. Foster, one of the original eight members of the Board of National Estimates, a lawyer by trade, and a gifted semanticist by avocation. Some will remember him for his elegant and precise writing; none will forget his eccentricities. He was the man who always wore his hat in the house.
(7) This usage is wholly in accord with the findings of the lexicographers, who almost invariably assign it the number one position. Further, it is readily understood and generally employed by statisticians, scientists, and the like, who sometimes define it as "non-zero probability." This is much to my taste.
At the same time there can be no question of the existence of a second usage, especially in the ordinary spoken word. The meaning here is most emphatically not the broad range of "non-zero probability," but a variable low order of probability, say anywhere below 40 or 30 or 20 percent. Thus it would fall last in a series that named descending odds: certain, probable, possible. When people use it to signify very low odds, for example below 5 percent, they may say "remotely possible" or any of its many cognates. This of course is not to my liking, but the intended meaning is clear. The serious trouble comes when another group of users lifts the word out of its position in the cellar of odds and by the addition of augmenting adjectives makes it do duty upstairs: "serious possibility," "great possibility," "highly possible."
(8) Washington Platt, Strategic Intelligence Production [New York: Frederick A. Praeger,1957]. The chart appears on the inside cover and again on page 208--not exactly as above but in full accord with my principles. The trouble comes on pp. 209-210, where General Platt departs widely, and to me regrettably, from my notion of legitimate synonyms.
(9) See the next following article. [Editors Note: David L. Wark, "The Definition of Some Estimative Expressions," Studies in Intelligence (Fall 1964).]
(10) Some of these synonymous meanings are expressed in verb forms. Thus it is syntactically possible to use them closely coupled to one of the adverbial expressions of odds, e.g., "we believe it likely that . . . " or "we estimate it is almost certain that such and such will not . . . ." If we really mean to assign an odds value to these verb forms good usage would forbid this kind of doubling-up. Mathematically, the probabilities would have to undergo a quite ridiculous multiplication. Thus "we believe" (75±percent) multiplied by "likely" (75±percent) would yield odds worse than 3 to 2 instead of 3 to 1. If we are not assigning an odds value to "we believe" and "we estimate," the purist would say we should not use them. Yet on many occasions a writer will feel uncomfortable--and justifiably so--with a bare "It is likely that. . . ." Such a bald statement is seemingly more confident than the situation would warrant. The writer will feel something akin to a compulsion towards modesty and a drive to soften the "likely" by introducing it with a "we believe" or "we estimate." Almost invariably he does not intend to change the odds associated with "likely." If one could set himself up as the arbiter, one world, I believe, rule that the "likely," or the "probably," or the "almost certainly," etc. was the operative expression of odds and that its message was unaffected by the introducing verb.
Doubling up in the "possibly" category is a different matter. We should avoid "it might (or may) be possible for the Blanks to . . . ." The verb should be present or future indicative, normally "is" and "will be."
(11) "Could" is included here because of many years' duty as a synonym for "possible." It has also served as a short way of noting a capability as in "The Soviets could develop [or "have the capability to develop"] such and such a radar though we have no evidence that they are doing so." The two usages are close, to be sure, but not identical.
(12) These synonyms must not be modified; might well, could well, just could, barely conceivable, etc. are as inadmissible as the original sin.
(13) As in, "It is almost certain that such and such will occur in the delta, perhaps in Saigon itself."
(14) This group of words poses at least one very vexing problem. Suppose you wish to make a positive estimate that there is, say, about a 30-percent chance that such and such thing is the case. Assuming that the thing in question is important, a 30-percent chance of its being the case is highly significant. If you stick with the chart and write "it is improbable [or likely, etc.] that such and such is the case" you will probably convey a much more negative attitude than you intend. There are many ways around the problem; they will, however, require a few more words.
(15) "We anticipate," used regrettably as a synonym for "we expect," is also part way in. I hope it gets out.
[Top of page]
Posted: Mar 19, 2007 11:00 AM
Last Updated: Jul 07, 2008 02:14 PM