and Frames” (Cambridge, United Kingdom: Cambridge
University Press, 2000). See also W.M. Goldstein and
R.M. Hogarth, eds., “Research on Judgment and Decision
Making: Currents, Connections, and Controversies”
(Cambridge, United Kingdom: Cambridge University
Press, 1997); D.J. Koehler and N. Harvey, eds., “Blackwell
Handbook of Judgment and Decision Making” (Malden,
Massachusetts: Blackwell Publishing, 2004); and
D. Kahneman, “Thinking: Fast and Slow” (New York:
Farrar, Straus, and Giroux, 2011).
2. Readers can examine different probabilities of winning
in tennis at “Tennis Calculator,” 2015, www.mfbennett.
com. For analytical derivations, see F.J.G.M. Klaassen
and J.R. Magnus, “Forecasting the Winner of a Tennis
Match,” European Journal of Operational Research 148,
no. 2 (2003): 257-267.
3. E. Siegel, “Predictive Analytics: The Power to Predict
Who Will Click, Buy, Lie, or Die” (Hoboken, New Jersey:
John Wiley & Sons, 2013); and T.H. Davenport and
J.G. Harris, “Competing on Analytics: The New Science
of Winning” (Boston: Harvard Business Review Press,
4. K. Popper, “Of Clocks and Clouds,” in “Learning,
Development, and Culture: Essays in Evolutionary
Epistemology,” ed. H.C. Plotkin (Hoboken, New Jersey:
John Wiley & Sons, 1982), 109-119.
5. Notable books in this regard are J. Baron, “Thinking
and Deciding,” 3rd ed. (Cambridge, United Kingdom:
Cambridge University Press, 2000); J.E. Russo and
P.J.H. Schoemaker, “Winning Decisions: Getting It Right
the First Time” (New York: Doubleday 2001); G. Gigerenzer
and R. Selten, eds., “Bounded Rationality: The Adaptive
Toolbox” (Cambridge, Massachusetts: MIT Press, 2002);
D. Ariely, “Predictably Irrational: The Hidden Forces That
Shape Our Decisions” (New York: HarperCollins, 2008);
and M. Lewis, “The Undoing Project” (New York: W. W.
6. P.E. Tetlock and D. Gardner, “Superforecasting: The Art
and Science of Prediction” (New York: Crown, 2015).
7. P.J.H. Schoemaker and P.E. Tetlock, “Superforecasting:
How to Upgrade Your Company’s Judgment,” Harvard
Business Review 94, no. 5 (May 2016): 72-78.
8. For more details about best practices for setting up
and running prediction tournaments, see Schoemaker
and Tetlock, “Superforecasting.”
9. Prediction tournaments are scored using a rigorous,
widely accepted yardstick known as the Brier score.
For more information about the Brier score, see
G. W. Brier, “Verification of Forecasts Expressed in
Terms of Probability,” Monthly Weather Review 78,
no. 1 (January 1950): 1-3.
10. B. Fischhoff, “Debiasing,” in “Judgment Under
Uncertainty,” ed. Kahneman, Slovic, and Tversky,
422-444; and J.S. Lerner and P.E. Tetlock, “Accounting
for the Effects of Accountability,” Psychological Bulletin
125, no. 2 (March 1999): 255-275.
11. B. Fischhoff, “Debiasing;” G. Keren, “Cognitive Aids
and Debiasing Methods: Can Cognitive Pills Cure Cogni-
tive Ills?,” Advances in Psychology 68 (1990): 523-552;
and H.R Arkes, “Costs and Benefits of Judgment Errors:
Implications for Debiasing,” Psychological Bulletin 110,
no. 3 (November 1991): 486-498.
12. The term “bootstrapping” has a different meaning in
statistics, where it refers to repeated sampling from the
same data set (with replacement) to get better estimates;
see, for example, “Bootstrapping (Statistics),” Jan. 26,
13. H.A. Wallace, “What Is in the Corn Judge’s Mind?,”
Journal of American Society for Agronomy 15 (July 1923):
14. S. Rose, “Improving Credit Evaluation,” American
Banker, March 13, 1990.
15. These tasks included, among others, predicting
repayment of medical students’ loans. See R. Cooter and
J.B. Erdmann, “A Model for Predicting HEAL Repayment
Patterns and Its Implications for Medical Student Finance,”
Academic Medicine 70, no. 12 (December 1995): 1134-
1137. For more detail on how to build linear models —
both objective and subjective — see A.H. Ashton, R.H.
Ashton, and M.N. Davis, “White-Collar Robotics: Levering
Managerial Decision Making,” California Management
Review 37, no. 1 (fall 1994): 83-109. Especially useful is
their discussion of possible objections to using linear
models in applied settings, as in their example of predicting advertising space for Time magazine.
16. For a thorough analysis of the multiple reasons
for this paradox, see C.F. Camerer and E.J. Johnson,
“The Process-Performance Paradox in Expert Judgment:
How Can Experts Know So Much and Predict So Badly?,”
chap. 10 in “Research on Judgment and Decision Making,” ed. Goldstein and Hogarth.
17. Random noise can produce much inconsistency
within as well as across experts; see R.H. Ashton,
“Cue Utilization and Expert Judgments: A Comparison
of Independent Auditors With Other Judges,” Journal
of Applied Psychology 59, no. 4 (August 1974): 437-444;
J. Shanteau, D.J. Weiss, R.P. Thomas, and J.C. Pounds,
“Performance-Based Assessment of Expertise: How to
Decide if Someone Is an Expert or Not,” European Journal of Operational Research 136, no. 2 (January 2002):
253-263; R.H. Ashton, “A Review and Analysis of Research on the Test-Retest Reliability of Professional
Judgment,” Journal of Behavioral Decision Making 13,
no. 3 (July/September 2000): 277-294; S. Grimstad and
M. Jørgensen, “Inconsistency of Expert Judgment-Based
Estimates of Software Development Effort,” Journal of
Systems and Software 80, no. 11 (November 2007):
1770-1777; and A. Koriat, “Subjective Confidence in
Perceptual Judgments: A Test of the Self-Consistency
Model,” Journal of Experimental Psychology: General
140, no. 1 (February 2011): 117-139.
18. Beyond just predictions, noise reduction is a broad
strategy for improving decisions; see D. Kahneman,
A.M. Rosenfield, L. Gandhi, and T. Blaser, “Noise:
How to Overcome the High, Hidden Cost of Inconsistent
Decision Making,” Harvard Business Review 94, no. 10
(October 2016): 38-46.
19. The radiologist example was taken from P.J. Hoffman,
P. Slovic, and L.G. Rorer, “An Analysis-of-Variance Model
for Assessment of Configural Cue Utilization in Clinical
Judgment,” Psychological Bulletin 69, no. 5 (May 1968):