In response to my post from a few days ago, George Brett writes:
i think you should differentiate more between prediction and explanation. the pundits are being asked to predict future events, while the Science article you discuss targets scientists who are seeking explanation for observed occurrences, which they are then able to test in at least partially controlled environments.
Good point. I agree that there is a certain difference between punditry (and social
science in general) and hard science; it’s much harder to perform experiments to disprove hypotheses in sitting at your desk in the Department of Economics than when you are standing in a sophisticated lab on a medical campus. But I don’t think that the explanation/prediction issue makes Chamberlin and Platt’s points any less valuable for social scientists – just as economists try to predict the future of the economy, biologists try to predict what will happen when you administer this drug, turn on/off this gene, etc., and presumably professionals in both fields will learn and adjust their models after observing what happens in the future/the more controlled cell model. On the other hand, Platt (and to a lesser extent, Chamberlin) felt that the access to experimentation wasn’t as problematic as the thought process leading up to it. Platt and Chamberlin wrote to remind scientists that critical thought and consideration
of multiple hypotheses would lead to choosing better experiments that could more efficiently disprove alternative hypotheses and move science forward. This is why both were so concerned with “intellectual parentage,” or favoritism toward one particular hypothesis. Social scientists don’t have the opportunities for experimentation that hard scientists do, so it becomes more and more important to consider alternatives to your main idea, and whether those alternatives could bring about a better resolution to your thought experiment. Back to George Brett:
i would compare the pundit point more to a study of economic forecasting by James Montier, “Global Equity Strategy: The Folly of Forecasting: Ignore All Economists, Strategists, and Analyst,” published in Global Equity Strategy, August 24, 2005. He makes the point that “economists are really good at telling you what has just happened!” he has some very compelling data.
Tetlock and Montier agree that hubris plays a central role in this phenomenon, although Montier calls the factor “overconfidence.” overconfidence, as Montier uses the term, refers to a “situation where people are surprised more often than they expect to be.” so a person is “well calibrated” if, when asked 10 questions and instructed to give a range within which they are 90% confident the answer will fall, their range will be correct for 9 of 10 questions.
Many experts are overconfident, but Montier tested, among others, weathermen and doctors. while weathermen seem to be fairly well calibrated (e.g. when 80% sure they are correct, they are correct 80% of the time), doctors were right only 15% of the time when they were 90% sure they were correct. investment professionals also fared poorly.
Good point, and it’s cool to see more research on the topic. I agree that the “overconfidence” factor has much to do with pride in what you have already learned/contributed and
hesitance to evaluate that information objectively at every proximal step. It’s interesting to see who’s well calibrated and who’s not: congrats weathermen, and yikes, physicians! I have seen a bit of that from personal experience: a cavalier chauvinism still permeates much of medicine (especially surgery) and is predicated on a sense of overconfidence as a defense mechanism imbued in training. This is not universal, but understandable – today’s doctors are expected to successfully manage a perhaps unrealistic amount of patients at all times, and confidence in your solutions (and avoidance of calling another doctor with more specialized knowledge in the middle of the night) is necessary to get through the day. Well-trained and more collaborative (general medicine, pediatrics) doctors probably don’t have this problem as much, I’d imagine, but plenty of doctors, especially those who are younger and had less involved or professional teachers, might fall into the overconfidence trap, which is hard to get out of. Montier’s research, at least, lets us know that a great many people in many professions may want to take a look back at Platt and Chamberlin.