How likely is for people to make adequate probability judgments, and how much do intelligence and formal education help them? The answers will likely surprise you.
Management, from strategy to tactics and from innovation to daily operations, depends on judging probabilities. What are the odds of a manufacturing error? What percentage of customers will like the new packaging? Is a new team member going to be a good addition? What loans are likely to be repaid? We have developed a wide variety of techniques —Six Sigma, credit ratings, experts’ educated guesses— to deal with the many uncertainties implicit in any complex situation.
But only assessing probabilities at the beginning only works in a static world. We are constantly receiving new information at a rate and depth unparalleled in human history, a characteristic of the contemporary world that becomes more important almost month to month. To avoid being outrun by events or outmaneuvered by competitors, it’s necessary to integrate new information and reassess probabilities as a continuous process.
This activity is well understood in theory, specially within the field of Bayesian statistics which studies among other things the mathematically correct procedures to include new information on an existing estimate. “Mathematically correct” has in this case a very practical meaning — mathematically incorrect answers generally end up impacting the bottom line… sometimes catastrophically, as is abundantly clear these days.
What characterizes the people that are more skilled at incorporating and making use of new information? Obvious possibilities are their educational background (after all, education does expose people to new information on an ongoing basis) and raw cognitive capabilities as measured by the IQ test and more modern and specific variations of it.
Researchers in Germany published a few weeks ago a report where they studied this question by testing a representative cross section of German society. Unlike most similar studies, they didn’t restrict themselves to college students, but included people from different educational backgrounds. Besides testing people’s effectiveness at making and modifying probability judgements in the presence of new information, they also collected data about their academic history and their cognitive capabilities using different psychometric tests.
To avoid being unduly influenced by the individuals’ cultural backgrounds, they posed a problem couched in the familiar terms of the probability of rain. The problem took into account first, the weather service’s prediction and their historical track record, and second, a direct weather observation complementing this information.
The results were very surprising: Less than 6% of the people in the study were able to make even a very approximately correct estimate of the solution. Of the 94% of people that gave wildly inaccurate responses, around a quarter basically ignored their previous information and used only the new information, while around 10% of the individuals fell into the opposite trap of not taking into account the new information.
It bears repeating, because judging probabilities and updating them using new information is a basic intellectual activity used in most professional, social, and personal contexts. It might usually be done in a qualitative way instead of using explicit percentages, but it’s nonetheless one of the cornerstones of intelligent conscious behavior.
And 94% of people not only failed to do this in a simple, controlled experiment, but they failed by a very wide margin.
More surprising still are the results the researchers got when they studied the relationship between cognitive capabilities, educational background, and the capability to perform this task. People with higher cognitive capabilities and people with more years of formal education, actually did worse and were more likely to commit analytical fallacies. “Intelligent,” educated people were no better than the average, and in fact worse, at making these judgments.
The reason for this is still unknown, but it’s clear that human performance judging probabilities using new information is positively dismal, and that formal education and selection based on cognitive capabilities do nothing to improve this performance.
With this ability being a critical one for many roles, including all managers and other knowledge workers, it’s important to both put in place analytical mechanisms to supplement or replace non-specialists in this area, and to find a way for the educational system to enhance, rather than impair, this necessary skill. It doesn’t take much statistical sophistication to understand that a 6% chance of approximately correct answers is too low a number.
IEET Blog