Experts study

Heuer describes a study done about betting on horse races. They took eight professional handicappers (someone who sets the betting odds based on calculations of the outcome of a contest, especially a horse race) and asked them to rank 80 different pieces of data about a horse race as to what they thought was most important. Do you factor in the jockey’s record as well as the recent record of the horse? The weather? The competition? How much weight is the horse carrying? What is the length of the race? There are scores of variables. Then the handicappers were given what they felt were the five most important pieces of data and asked to project the winner of a race (actual names and races were not given, so as to not bias the projections). They were also asked to rank their confidence about their predictions. Now it gets interesting. They were then given 10, 20, or 40 pieces of what they had individually considered to be the most important information. Three of the handicappers actually showed less accuracy as the amount of information increased, two improved their accuracy, and three were unchanged. But as a group, their accuracy did not improve and in fact was slightly down. But with each increase in information, their confidence went up. In fact, by the end, their confidence had in fact doubled. If they had actually been at the track and were betting, would they have doubled their bets as they became more confident? Human nature says yes, they would. But that confidence would not have made them any better predictors. They just doubled their bets, which magnified their gains or losses. Think of it like adding leverage to your stock portfolio. “A series of experiments to examine the mental processes of medical doctors diagnosing illness found little relationship between thoroughness of data collection and accuracy of diagnosis.” Another study was done with psychologists and patient information and diagnosis. Again, increasing knowledge yielded no better results but significantly increased confidence. The inference is clear and quite important: “Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts use much less available information than they think they do.” Dr. Martin Bloom