338-349. Note that these were highly trained professionals
making judgments central to their work. In addition, they
knew that their medical judgments were being examined
by researchers, so they probably tried as hard as they
could. Still, their carefully considered judgments were
20. The average intra-expert correlation was . 76, which
equates to a 23% chance of getting a reversal in the ranking or scores of two cases from one time to the next. In
general, a Pearson product-moment correlation of r translates into a [. 5+arcsin (r)/π] probability of a rank reversal of
two cases the second time, assuming bivariate normal
distributions; see M. Kendall, “Rank Correlation Methods”
(London: Charles Griffen & Co., 1948).
21. A provocative brief for this structured numerical
approach in medicine can be found in J.A. Swets, R.M.
Dawes, and J. Monahan, “Better Decisions Through
Science,” Scientific American, October 2000, 82-87.
22. For a general review of bootstrapping performance,
see C. Camerer, “General Conditions for the Success of
Bootstrapping Models,” Organizational Behavior and
Human Performance 27, no. 3 (1981): 411-422, which
builds on and refines the classic paper by K.R. Hammond,
C.J. Hursch, and F.J. Todd, “Analyzing the Components
of Clinical Inference,” Psychological Review 71, no. 6
(November 1964): 438-456.
23. G. Klein, “The Power of Intuition” (New York: Currency-Doubleday, 2004); and R.M. Hogarth, “Educating Intuition”
(Chicago: University of Chicago Press, 2001). See also
D. Kahneman and G. Klein, “Conditions for Intuitive
Expertise: A Failure to Disagree,” American Psychologist
64, no. 6 (September 2009): 515-526.
24. P. Goodwin, “Integrating Management Judgment
and Statistical Methods to Improve Short-Term Forecasts,”
Omega 30, no. 2 (April 2002): 127- 135; for medical
examples, see J. Reason, “Human Error: Models and
Management,” Western Journal of Medicine 172, no. 6
(June 2000): 393-396; and B.J. Dietvorst, J.P. Simmons,
and C. Massey, “Algorithm Aversion: People Erroneously
Avoid Algorithms After Seeing Them Err,” Journal of
Experimental Psychology: General 144, no. 1 (February
25. R.C. Blattberg and S.J. Hoch, “Database Models
and Managerial Intuition: 50% Model + 50% Manager,”
Management Science 36, no. 8 (August 1990): 887-899.
26. Related cognitive processes involve associative networks, scripts, schemata, frames, and mental models;
see J. Klayman and P.J.H. Schoemaker, “Thinking About
the Future: A Cognitive Perspective,” Journal of Forecasting 12, no. 2 (1993): 161-186.
27. R. Hastie, S.D. Penrod, and N. Pennington, “Inside
the Jury” (Cambridge, Massachusetts: Harvard University
28. J. Klayman and Y.-W. Ha, “Confirmation, Disconfirma-tion, and Information in Hypothesis Testing,” Psychological
Review 94, no. 2 (April 1987): 211-228; and J. Klayman
and Y.-W. Ha, “Hypothesis Testing in Rule Discovery:
Strategy, Structure, and Content,” Journal of Experimental
Psychology: Learning, Memory, and Cognition 15, no. 4
(July 1989): 596-604.
29. T. Gilovich, “Something Out of Nothing: The Misperception and Misinterpretation of Random Data,” chap. 2
in “How We Know What Isn’t So: The Fallibility of Human
Reason in Everyday Life” (New York: Free Press, 1991);
see also N.N. Taleb, “Fooled by Randomness: The Hidden
Role of Chance in Life and in the Markets” (New York:
Random House, 2004).
30. The best way to untangle the confounding effects is
through controlled experiments, and even then it may be
difficult. For a research example of how to do this, see
P.J.H. Schoemaker and J.C. Hershey, “Utility Measurement: Signal, Noise and Bias,” Organizational Behavior
and Human Decision Processes 52, no. 3 (August 1992):
31. J.D. Sterman, “Business Dynamics: Systems Thinking
and Modeling for a Complex World” (New York: McGraw-Hill, 2000).
32. For textbook introductions to some of these technologies, see J.M. Zurada, “Introduction to Artificial Neural
Systems” (St. Paul, Minnesota: West Publishing Company, 1992); and S. Haykin, “Neural Networks: A
Comprehensive Foundation,” 2nd ed. (Upper Saddle
River, New Jersey: Prentice Hall, 1998).
33. “Finding a Voice,” Economist, Technology Quarterly,
Jan. 7, 2017, pp. 3- 27; see also J. Turow, “The Daily You:
How the New Advertising Industry Is Defining Your
Identity and Your Worth” (New Haven, Connecticut:
Yale University Press, 2011).
34. R. Copeland and B. Hope, “The World’s Largest
Hedge Fund Is Building an Algorithmic Model From Its
Employees’ Brains,” The Wall Street Journal, Dec. 22,
35. “Perspectives on Research in Artificial Intelligence
and Artificial General Intelligence Relevant to DoD,”
JASON Study JSR-16-Task-003, MITRE Corporation,
McLean, Virginia, January 2017, https://fas.org/irp/
36. Prediction banks are a special case of the more
general notion of a setting up a mistake bank; see
J.M. Caddell, “The Mistake Bank: How to Succeed by
Forgiving Your Mistakes and Embracing Your Failures”
(Camp Hill, Pennsylvania: Caddell Insight Group, 2013).
37. R. Feloni, “Billionaire Investor Ray Dalio’s
Top 20 Management Principles,” Nov. 5, 2014,
38. A. Edmondson, “Psychological Safety and Learning
Behavior in Work Teams,” Administrative Science Quarterly 44, no. 2 (June 1999): 350-383.
39. R.S. Michalski, J.G. Carbonell, and T.M. Mitchell, eds.,
“Machine Learning: An Artificial Intelligence Approach”
(Berlin: Springer Verlag, 1983).
40. See, for example, H. Kunreuther, R.J. Meyer, and
E.O. Michel-Kerjan, eds. (with E. Blum),“The Future of
Risk Management,” under review with the University
of Pennsylvania Press.
Reprint 58301. For ordering information, see page 4.
Copyright © Massachusetts Institute of Technology, 2017.
All rights reserved.