From his perspective as a pyschology researcher, Philip Tetlock watched political advisors on the left and the right make bizarre rationalizations about their wrong predictions at the time of the rise of Gorbachev in the 1980s and the eventual collapse of the Soviet Union. (Liberals were sure that Reagan was a dangerous idiot; conservatives were sure that the USSR was permanent.) The whole exercise struck Tetlock as what used to be called an "outcome-irrelevant learning structure." No feedback, no correction.
He observes the same thing is going on with expert opinion about the Iraq War. Instead of saying, "I evidently had the wrong theory," the experts declare, "It almost went my way," or "It was the right mistake to make under the circumstances," or "I'll be proved right later," or "The evilness of the enemy is still the main event here."
Tetlock's summary: "Partisans across the opinion spectrum are vulnerable to occasional bouts of ideologically induced insanity." He determined to figure out a way to keep score on expert political forecasts, even though it is a notoriously subjective domain (compared to, say, medical advice), and "there are no control groups in history." - The Long Now Foundation
Philip E. Tetlock
Philip E. Tetlock, author of "Expert Political Judgment: How Good Is It? How Can We Know?," is Professor of Business Administration, Political Science, and Psychology at the University of California at Berkeley.
The surge didn't work, unless you're willing to include paying people to not shoot at you as part of the surge(what's going to happen when this bandaid comes off?).
You're still in Iraq, none of the major goals have been met, there is no exit strategy.
Good evening welcome to another seminar about Long Term Thinkingsponsored by the Long Now Foundation. My name is StewartBrand, Philip Tetlock is anticipated something Long Now has wantedto do by about 100 years, we have a site called longbets.org wherepeople register predictions right out in public and people vote on themand some people bet money against them. And it's all veryconspicuous and the idea is to develop an idea over 100 years or so.And who is actually good in that predicting, who is quite bad, whatkinds of predications hold up and spend spending your time payingattention to what kinds are worthless.Well Philip Tetlock jumped all that by just going ahead and doing aresearch on existing predictions by experts and finding out what shiftedout from that. And he has got I think one of the fundamental ideasabout Long Term Thinking especially as it applies to takingresponsibility about what might happen soon in the future, pleasewelcome Philip Tetlock.Thank you very much Stewart. My topic today is good politicaljudgment and it is a sign of the power of politics to fog our thinkingthat there is vastly more controversy surrounding the concept of goodpolitical judgment then there is surrounding the concepts of say goodmedical judgment, or good engineering judgment, or good financialjudgment. The vast majority of intelligent people feel that there arereasonably solid benchmarks gold standards as it were for assessinggood medical judgment. Patients either do well or they do poorly die,but did either they stand or they fall. Your portfolio either grows or itdwindles by contrast good political judgment feels to most peoplehopelessly subjective. You really don't expect Nancy Pelosi andGeorge W. Bush to agree and you certainly expect in the internationalscene to observe dramatic differences and how the Iranian leadershiplooks at good judgment or how Osama Bin Laden views it or KimJong-il or Vladimir Putin.So my first task in this talk is to persuade you that the concept of goodpolitical judgment is not hopelessly subjective. That it is possible toconstruct some reasonably objective benchmarks I call them transideological benchmarks, benchmarks that experts from wide range ofideological and theoretical orientations can at least grudgingly accept.And it's possible to learn a lot from applying these benchmarks for thepurpose of assessing the judgmental accuracy by political experts.There are two big categories of benchmarks of good judgments thatwe use there are knowledge indicators and meta-knowledgeindicators. So we are interested both in, whether experts get it right orwrong. And we are also interested in the degree to which expertsassign realistic subjective probabilities to their predications. So thedegree to which experts know something about what they know anddon't know. So do you know what you do, and do not know? And Iam going to argue that there is really only way to find out that is to getin a habit of keeping score on one zone judgmental accuracy requiresdoing something rather cognizably unnatural I think I am speaking nowas a psychologist because my disciplinary background is that of apsychologist. Its not an altogether natural mental act to mental process is that way.I'll give a quick example of what is it like to scrutinize ones mentalprocess and I am going to use some non-political examples initially butwe will slip quickly into political ones. This is a subjective probabilityscale for people have dichotomous choice. There are only two possibleanswers. So a 50% probability represents of course maximumuncertainty coin toss uncertainty, 100% probability represents completeconfidence that you are right. And, the research that I am, I started doingin the 1980's and I was then pretty much exclusively an experimentalpsychology already on confidence and confidence calibration,measuring how realistically people could judge states of knowledge. And theseinvolve constructing rather elaborate test and exercises in which people wereasked questions like this about isotopes or the next question France beinglarger in area than Spain. And we are interested both in the accuracy of youranswer and we are interested in the accuracy of your subjective probabilityassessment. And you might ask, well how would you assess the accuracy orsubjective probabilities. You know we, and, I will answer that.There are two fundamental ways to do that; one is called calibration, as theability to assign subject to probabilities to outcomes that correspond to theirobjective probabilities. Now to do that, you need to get people to take ratherlong and elaborate tests. Need to get them to make lot of predictions, one ofthe thing I was successful in doing was in getting some very thoughtful people tosit down and make a lot of different predictions. Now you're well calibrated it if,when you say there is a 70% chance of something happening, the things thatyou are saying that 70% likelihood to occur about 70% of the time. And youare well calibrated if when you say a 100% probability the thing alwayshappens. When you say 50% it only happens 50%, so there is roughcorrespondence between subjective and objective probabilities and those ofyou who have some statistical background will know that this is somewhatnoisy measurement process, so you need to make the large numbers work foryou, you need to get a lot of predictions to make this, to make the resultsreasonably reliable and in fact in the research I am going to be talking about, itsbased, most of the analysis are based on approximately 28,000 predictionsover roughly 15 year period.So calibration is one aspect of the accuracy of subjective probability judgment.Discrimination is the other key aspect I am, going to talk about today. And askyour ability to assign sharply higher probabilities to things that happen and thethings that don't happen. And your perfect calibration score is going to looklike zero, because there is no gap between subjective and objectiveprobabilities. Your perfect discrimination score is going to look like a 100%.So here is an example of a graph in which on the X-axis you've got subjectiveprobability judgments on the Y-axis you have objective frequencies. And this isan example of someone who never, this is an expert you might characterize asrather cowardly. This is an expert who never offers a subjective probabilitylower that 0.4 or never offers a subjective probability higher than 0.6. So thebest to get out of this character is the minor shades or maybe. Nonetheless thischaracter scores well on calibration because when the individual assigns aprobability of 40% things happens about 40% at the time. When the personassign to probability of 60% things happen 60% at the time. So if your datapoint is on the diagonal here you are very well calibrated. You are not verydiscriminating now. This would be an example of good calibration, very goodcalibration and much better discrimination because now the experts using amuch wider range of the probability scale from 0.1 all the way to 0.9.And the next one is what we call the God profile. This is essentially omissions.This is what happens, when you assign a probability of zero things neverhappen. When you assign a probability of 1, things always happen and sinceyou are omission and perfectly deterministic universe that is, is the way it is. Sothis is pretty permutable and I don't thing I will be surprising anybody when Itell you that no human being looks like that. Okay this is just some technicalstuff and probability scoring, which I can come back to if you are interested. Iwanted to just say a few words about how I actually came to be doing this.When you do a longitudinal study of forecasting the sorts like that I'm doing,the project starts to feel as long as your entire life. I started doing this back in1984, so I am 52 almost 53 years old now and I just got in tenure in 1984Berkley. And I found myself on a committee, the national research councilcommittee created to look at the dynamics of American-Soviet relations. And Idon't how many if you remember 1983, early 1984 the Bulletin of the AtomicScientist had pushed its clock closer to midnight and I think any other, at anyother juncture than I think the Cuban Missile Crisis. (KAL07) it has beendown and there was a lot of talk about the growing tension in the relationbetween the United States and the Soviet Union. And there were lot offoundation that we are interested in funneling money into research in this areaand this committee existed for that reason.The interesting thing about the situation was that, Jonathan Schell wrote a book"The Fate of the Earth" in which he characterized Ronald Reagan as thedemented bus driver queuing down a mountain road with sharp curves. And alot of people who were liberals thought that Ronald Reagan was indeedbringing us pretty close to the apocalypse. So they are basically, and the lot ofpeople who are conservative thought well Reagan is doing the right thing but themost we can realistically hope in this situation is to contain the Soviet Union andthe Soviet Union will retreat in the Neo-Stalinist mode and that will be that.Stewart was actually sharing an interesting story about Peter Schwartzand Robert Gates. And I actually had some similar kind of story;I can share with you later.The dominant view he was a senior Soviet analyst in the mid 1980's. Were thedominant view was that nothing very good was going to happen. There were acouple of people, couple of outliers who though that dramatic liberalizationunder Gorbachev like figure could be possible. By it was by and large certainlynot the dominant prediction. So the National Research Council is suppose tobe a nonpartisan value neutral scientific activity and they are very carefully, theydo their diligence, they bring Richards Pipes in and they bring (MarshalSolomon) and the whole political spectrum was represented in that context.And then along comes Gorbachev and it was my first encounter with what I calloutcome irrelevant learning situation. This is a situation in which no matter howdifferent the X anti expectation of the experts were, everyone is in the positionafter the fact to explain what happens. So it really didn't matter in some sense,the liberals could argue, well no Reagan didn't appropriate a nuclear war. Youknow what Reagan wasted a lot of money in unnecessary defense spending andthe Soviet Union would have evolved in this direction anyway. Allow, ofcourse very few people were predicting at X anti and the conservatives ofcourse were eager to claim credit, thought that Ronald Reagan had won theCold War. Even though they had been on record before hand saying thatGorbachev was really Neo-Stalinist in disguise and was just a (indiscernible) abreathing spell bit of Robert Gates for example has identified with that, thatpoint of view until quite late actually in the Gorbachev period. But it was anoutcome irrelevant learning situation and it convinced me that there really wouldbe some great value added if someone tried systematically to keep score onpolitical experts. And that was what I proceeded to do, I just got in tenure, soI didn't have to worry about the gestation period of the project and move forward.Now there is a very interesting similarity I think, I, not only of course youstudied the predictions about Soviet Union quite a while to go, I've alsomonitored some of the more recent predictions about the war in Iraq. Andthere is some striking similarities in the conceptual structure of the argumentsthat unfold, I don't think I had been adequately commented on. It's a little bitcomplicated but it's worth thinking about, because it illustrates I think, it gets tothe core answer as to why the concept of good political judgment is soprofoundly controversial or as my, like my former colleague (indiscernible),what is it about politics that makes people so dumb? This is a standard 2 by 2table here and the rows are defined by idealogical perspectives the top rowrepresents a view of Reagan or George W. Bush is essentially good, strong,visionary leader. The bottom row represents the view of Reagan or Bush isessentially a dogmatic stabber in somewhat op trap kind and leader.The good historical trajectory represents what happens if things break in thedirection that, in the direction of the, the policy of the United States is pursuingat the moment and the bad historical trajectory is the things break againstAmerican policy. So in the top square, in the top left you have directindications. So insofar the Cold War ended roughly as the, on American termsconservatives quick to claim that most of our policies work. Now, insofar asthe Cold War had not ended on favorable terms for the United States if in factyou round up with the kind of situation had round up in Iraq, you would haveobserve a series of belief system defenses. You would observe them saying;you probably recognize some of these arguments in the context of Iraq debatefrom Neo-Conservative recognize just half on timing, you have to stay thecourse, you have to be patient or the downward counter fact defense, you thinkthis is bad you should see what would have happened if we haven't, so that'sthe downward count of actual, the war, it's a horrible situation we are in but itwould been even worse. But we made the right mistake defend, they weren'tweapons of mass destruction in Iraq, but it was prudent to set of very lowthreshold for moving in because that was, because making the other mistake themistake of not invading when there were weapons of mass destruction wouldhave been far worse.And Robert Gates of course invoked that argument with respect to Gorbachevin commenting on why he was very slow to recognize that Gorbachev was agenuine liberalizer, he essentially said, well we are making the right mistake. Itwas prudent to overestimate rather, it's more prudent to over estimate theSoviets than to underestimate. Overestimate their capacity, overestimate theirmaliciousness. And finally you would take a bad historical trajectory as justfurther support for, an essential view of the historical adversary.So what you see is, if the Cold War had ended in an, if the Reagan policies hadbeen associated with nasty outcomes, they probably, they almost certainlywould have been invoke counter facts defenses of the forth that you see in thecontext of Iraq. Conversely you have a belief system dilemma for liberals andmany liberals were in awkward position explaining why the Cold War endeddespite the fact that they are predicting that Reagan's legacy would be a badone. So they could invoke the not worth the price defense, they can invoke theother, the upward counter factual. Things could have been just as good orbetter even if we hadn't spent all this money. And they can invoke their versionof that we made the right mistake defense.So what you have is an interesting kind of intellectual stalemate in whichbecause they are no control groups in history, because no body knows howhistories would have unfolded, if it hadn't been President Reagan, if it been atwo term Carter presidency with a two term Mondale follow up, who in theaudience really knows whether the Cold War would have ended in roughly thesame way. It's simply not something available for human empirical knowledge.And for that matter no body can really say for sure what would have happenedin the world in which we hadn't gone into Iraq. It's just that the world that weare in seems so bad, that is hard to imagine alternative world. That would beworse, but it's not impossible, certainly not impossible.So it was just experience with the outcome irrelevant learning situations in the1980's that lead me to the view that participants across the opinion spectrumare vulnerable to occasional bouts, of audiologically induced insanity and it iscrucial, it is a great public need for developing like epistemic or knowledgestandards for judging judgement and to having standards for judging judgementthat somehow transcend idealogical boundaries. From my effort to developthese standards was described in great didactic detail in the book, but the keyidea is to work out ground rules for keeping score that advocates of thecompeting points of view can agree in advance or fair and therefore they find itawkward to denounce those standards after the fact when they are in an awkward position.So we did this project over a long period of time, 18 year project, 284 expertsin international affairs and they were signing prediction, roughly about a 100each expert on average coming out to about 28,000. Virtually all of them hadsome postgraduate training, roughly 12 years of work experience and theyincluded a mixture of academics, journalist some intelligence analysts andpeople in various think tanks. There are certain key requirements fordeveloping good questions; you need to specify the outcome so clearly thatthey pass the clairvoyance test.So if Stewart were a true clairvoyant I could hand my predictions to himand Stewart could tell you whether the predictions are right or wrong without theneed to comeback to me and say, oh Tetlock what exactly did you mean by aPolish Peron or what did you mean by a back clash in South Africa or whatdid the kinds of, the types of vague verbiage you often get from experts. Onceidentified the possible features and in, sufficiently clearly, the pass ofclairvoyance test then you get the experts to place subjective probabilities oneach set and a lot of our questions have this kind of a format, so centralgovernment debt in country X is going to either hold between 35% and 40% ofGDP, is going to fall below or rise above that range.Now this is one of the more far sighted questions we asked, although thisquestion you might know was not all that far sighted because it never reallyoccurred to us that the communist party would collapse. You know the closeswe came to anticipating the future here was - its control would weaken. Butit's an illustration of what we are getting at. So we asked a lot of questions, 59different nations, states passed some questions varying on EU, NATO, WorldTrade Organization lots of different things. Economic performance, growth,inflation, unemployment, policy priorities, defense spending, leadership changes,border conflicts, entry-exit from international agreements. And we haddifferent types of forecasting arises for different variables so we had shorterranges for the faster moving variables like stock markets and we had slower,we had very long time raises were things that move slowly like border changesfor example. So some of our predictions haven't even come do yet, having todeal with border changes.Okay, among other things we discover this; we discover that contrary to whatmost political participants would have expected, namely that there a side wouldwin. We find that traditional divisions of political opinion don't matter all thatmuch. It didn't matter all that much in terms of academia we had lots adifferent theoretical contrast, and I am sound a little alien we had realist versusinstitutionalist, constructivist and we had, we had optimist verses pessimist,boomsters versus doomsters, didn't make much of a difference. We hadliberals versus conservatives it didn't make all that much of the difference. Thecontent of what people thought was not a very good predictor of how accuratethey were. But was a good predictor of how accurate they were was how theythought. At least if you are quiet interested in his aggregate accuracy, whatmatters is principally how you think and this brings us to the point of our foxesand hedgehogs. And I was actually stunned by the consistency of the result.That expert's who may classified as foxes and I can explain how we classifyexperts in to these categories in moment.But experts whom may classified as foxes beat hedgehogs and it was anadvantage we could not make a goal it was very hard to make it disappearstatistically. So we have tweaked in all sorts of different ways to see if it wasnot effective chance in some way or not effective different error avoidancepriorities all sorts of different control mechanism and the fox advantage reallydoes it does prove to be quiet robust.But who are these foxes and hedgehogs well there is a essay by Isaiah Berlincame about 45 years ago in which he draws on a fragment of Greek poetryfrom 2500 years ago by the Greek warrior poet Archilochus which is roughlytranslated as the fox knows many things but the hedgehog knows one big thing.And he defines the ideal type hedgehog has an expert or professional or athinker who relates every thing to a single central vision in terms of which allthat they say has significance. So you could be a Marxist hedgehog or youcould be a libertarian hedgehog. You could be a boomster hedgehog or youcould be a doomster malthusian hedgehog. You could be a realist hedgehog;you could be a idealist hedgehog. The important thing is that you approachhistory; you approach current events in a deductive frame of mind. You havecertain first principals and you try as hard as you can to absorb as manydifferent facts in to the frame work of those first principals, that might sound likevagility but if you think about it for a minute from a philosophy of science pointof view it also is parsimony that is what scientist are supposed to do they aresupposed to explain as much as possible with as little as possible.So we will come back to that that value attention in a movement. The idealtype hedgehogs, Berlin defined this way, he said they pursue many ends oftenunrelated and even contradictory they entertain ideas that are centrifugal ratherthan centripetal with out seeking to fit them in to or exclude them from any oneall embracing inner vision, those are the foxes. So foxes and hedgehogs, foxesare skeptical of big theories. You are not going to find very many foxes whoare true believers. And interestingly the foxes, who did the best of myforecasting exercises were at least enthusiastic about participating and theywere the most diffident about their ability to forecast. Because they really dosee history as in substantial measures is quiet unpredictable. Whereas thehedgehogs were more enthusiastic about it they tend to be more enthusiasticabout extending their favorite theories in to new domains and intend to be moreconfident in their ability to predict.Now this is some data which I discovered very quickly if you are on the perfectdiagonal the straight line here represents perfect calibration so the were thecurvy lines represent actual groups of human beings making thousands ofpredictions and these are aggregations of that. And the key thing to know hereis that one of the lines stays further from the perfect diagonal and all the otherlines so the line that strays furthest is the line in which hedgehog are making longterm predictions within the domains of their expertise. Whereas the line that'sclosest to the perfect diagonal is foxes making short term predictions within thedomains of their expertise. Now there is an argument that started to unfoldabout whether the foxes were doing better than the hedgehogs not becausethey are more perceptually accurate but rather because the foxes are, excusethey punt foxes they are chickens the foxes were unwilling to say any thingmuch more than maybe so the foxes were clinging around the subjectiveprobability point to point five. So one way to test that if that were true then thefoxes should not be as discriminating as the hedgehogs. The hedgehogs should- may lose on calibration but they should win on discrimination. So in this thelittle graph here what we do as we plot both calibration better calibration on theX-axis better discrimination on the Y-axis and as you move towards the upperright you move toward better and better performance and what you see is thatthe FST, FLT represent foxes making short term and long term predictions andyou can see that they are doing better than hedgehogs making shot term andlong term predictions HST and HLT and they are better on both calibration anddiscrimination. So I think that rules out the foxes were just chicken hypothesis.There is however one slightly disturbing result at least from the stand point ofthose who want to argue that the foxes are particularly brilliant and that is whenyou compare the fox accuracy of fox predictions to the accuracy of predictionsyou could have generated by using very simple statistical algorithms like predictno change or predict the most recent rate of change when you use that as yourbench mark of comparison that's what we are calling MC mindlesscompetition. And you can see the mindless competition is pretty close to hefoxes and so it is not exactly a great indication of human intuitive judgment.Now you might ask what do hedgehog do well and the answer is they assignhigher probabilities to big changes that do materialize so if you want to knowwho is really out front, really out front in predicting some of the bigger changesthat have occurred over the last 20 years and there were couple of hedgehogsactually out front with respect to the Soviet Union and the decline of SovietUnion, we call them Ethnic Nationalist Fundamentalists. These were peoplewho argued that the Soviet Union which essentially prisons house of nations itwas inherently unstable and it was going to fall apart. Here in mind thesepeople had been predicting the Soviet Union will fall apart back in 1960's, inthe 1970"s as well, if like the broken clock is eventually right but they wereworried about this, they are right, but they are right at a high price you arepaying a high price in false positive predictions. So when you rely on thehedgehogs, the hedgehog predictors, you are going get a lot of false positives.So the price of being a successfully predicting the disintegration of Yugoslaviaor the disintegration of the Soviet Union is that you are also predicting thedisintegration of Canada, of India, of Kazakhstan, of Indonesia, of Nigeria, ofSouth Africa it's a long list. And that of course helps to explain what thehedgehogs are doing as well, they are assigning high probabilities to lowfrequency events which makes much more interesting than the foxes, but it alsomake them substantially more incorrect.Now a friend of mine Nassim Taleb who am my, my suspect probably be hereat some point, I just finished writing a book on Black Swans in which he talksabout the radical unpredictability of the world and, if you want to know theadvantage of being the hedgehog its that there its always, there is they usually atleast a few of them who are out front in predicting outlandish outcomes. Soactual blacks swans at the hedgehogs were in the forefront of expecting, at leasta few hedgehogs, most hedgehogs were wrong but the, at least a hedgehogswho were out front. And the list here is collapse of the USSR in 1991 theYugoslav Civil War, Saddam's invasion of Kuwait Rwandan genocide financialturbulence preceding the collapse of long-term capital management in 1998.The East-Asian currency crisis and Russian debt fault, the internet bubble the90's this puncturing terrorist strikes all are known predicting 9/11 but terroristincreased incidents of terrorism.But again lots of false positives, so predictions of the great American and thenglobal depression of 1990's and Dow 36000 and the NASDAQ 5000. Weactually had NASDAQ 5000 or Dow 1500 and NASDAQ 350 predicating aNeo-Stalinist school in Moscow that brings return of the Cold War. And ofcourse predicting full bound civil wars in places such as South Africa, Nigeria,Pakistan, India, China, Korean Peninsula, Ukraine and Kazakhstan and eventhey use of nuclear weapons in the Indian Sub-continent or the KoreanPeninsula the two most popular sites for making predictions of that sort. Imean I, I think I run out of time so I should be, how much more time do I have.Okay, I want to leave lots of time for questioning. But I want to say that, I thinkits tempting for me as a psychologist who as long interested individualdifferences to treat this as a horse race between foxes and hedgehogs. I thinkthat's fundamentally the wrong way of looking at this. I think it makes moresense to think about hedgehogs and foxes existing in kind of interdependentintellectual ecosystems. So on a one hand you've got the hedgehogs, who arevery good at seizing on big ideas and pushing them as far as reasonable andthen offering good deal further. On the other hand you have the foxes almostplaying a role of intellectual scavengers they are picking bits and pieces of bighedgehog ideas and they are creating a eclectic mismatches, that on averagehave more predictive power then the original ideas had. So I think this hasinteresting implications for optimal intellectual diversity in setting up teams. Ithink it also says; I will give a quick example of how that would work in thecontexts of the Soviet Union. There were some people Jerry Hough I amthinking of a Sovietologist for example who were out front in predicting theGorbachev would be a liberalizer before a lot of other people were. But thepeople who are out front in predicting that Gorbachev would be a liberalizertended to be people who saw the soviet system has having more internallegitimacy then it did. The people who thought that Gorbachev being aliberalizer, it would be impossible to have a liberalized emerge out of the(indiscernible) Patrick's and Richard Pipes of the world who saw the SovietSystem has infallibly self reproducing Totalitarian mode, those people thoughtGorbachev liberalization be impossible because the Kremlin leadership wouldrecognized as liberalized thing will fall apart. So they wouldn't be stupidenough to do that.So who would be best position both to predict Gorbachev liberalization andsoviet disintegration? It wasn't the liberals who thought the soviet system hadmore legitimacy than that and it wasn't the conservatives who were ruling outthe possibility of liberalization in the first place, it was a small group of foxeswho had successfully manage to integrate in their mind some liberal cognitionsand some conservative cognitions, they were willing to accept the liberal insightthat the soviet leadership like most political leadership in human history was nottotally monolithic and they were a genuine differences of opinion within it. Sothey were willing to accept that, but they also tended to were the moreconservative, the soviet system didn't have legitimacy and that once to lid waslifted all hell would break loose.So that's an interesting way of looking at perhaps the situation in Iraq, youcould imagine various types of fox cognitions there. You could imagine foxcognitions that would, try this dissonant possibility for example it would well bethat the Bush administration made a mass of mistake, it's certainly is the casethat the conservatives we studied who were sympathetic to the invasion of Iraqin the first place would had been wrong on many, many predictions over thelast 2 to 3 years. But it is also the case that nobody knows what's going tohappen in the next 20 or 30 years. So foxes often have a way of taking outinteresting contrarian positions that both liberals and conservative find works in.And a Fox position might be that, might annoy conservatives by saying that theIraq was stupid and the other hand I might annoy liberals by saying its perfectlypossible and in the next 10 or 15 years the situation in Iraq would might be farbetter and it would have been if Iraq had been under the leadership of UdayHussein. So foxes have a way of annoying people across the political spectrumthey have the way of annoying people who have intrenched theoretical positions.And those are the people who in my exercises tend to do better, they don't geteverything right they get lot of things wrong. I think we live in an inherentlyprobabilistic world, nobody is really expecting omissions. But it turns out that asomewhat contrarians self critical cognitive style does translate into morerealistic subjective probabilities being assigned the possible futures. And if youthink that there is an, that good policy depends on assigning more realisticprobabilities the possible futures or if you think that making more moneydepends on assigning a realistic subject that probabilities possible futures hasprobably a result we are taking into consideration.You have enough questions.