Reference class forecasting

Reference class forecasting or comparison class forecasting is a method of predicting the future by looking at similar past situations and their outcomes. The theories behind reference class forecasting were developed by Daniel Kahneman and Amos Tversky. The theoretical work helped Kahneman win the Nobel Prize in Economics.

Reference class forecasting is so named as it predicts the outcome of a planned action based on actual outcomes in a reference class of similar actions to that being forecast.

Discussion of which reference class to use when forecasting a given situation is known as the reference class problem.

Overview

Kahneman and Tversky[1][2] found that human judgment is generally optimistic due to overconfidence and insufficient consideration of distributional information about outcomes.

People tend to underestimate the costs, completion times, and risks of planned actions, whereas they tend to overestimate the benefits of those same actions. Such error is caused by actors taking an "inside view", where focus is on the constituents of the specific planned action instead of on the actual outcomes of similar ventures that have already been completed.

Kahneman and Tversky concluded that disregard of distributional information, i.e. risk, is perhaps the major source of error in forecasting. On that basis they recommended that forecasters "should therefore make every effort to frame the forecasting problem so as to facilitate utilizing all the distributional information that is available".[2]:416 Using distributional information from previous ventures similar to the one being forecast is called taking an "outside view". Reference class forecasting is a method for taking an outside view on planned actions.

Reference class forecasting for a specific project involves the following three steps:

  1. Identify a reference class of past, similar projects.
  2. Establish a probability distribution for the selected reference class for the parameter that is being forecast.
  3. Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.

Reference class tennis

The reference class problem, also known as reference class tennis, is the discussion of which reference class to use when forecasting a given situation.

Suppose someone were trying to predict how long it would take to write a psychology textbook. Reference class tennis would involve debating whether we should take the average of all books (closest to an outside view), just all textbooks, or just all psychology textbooks (closest to an inside view).[3][4]

Practical use in policy and planning

Whereas Kahneman and Tversky developed the theories of reference class forecasting, Flyvbjerg and COWI (2004) developed the method for its practical use in policy and planning. The first instance of reference class forecasting in practice is described in Flyvbjerg (2006).[5] This was a forecast carried out in 2004 by the UK government of the projected capital costs for an extension of Edinburgh Trams. The promoter's forecast estimated a cost of £255 million. Taking all available distributional information into account, based on a reference class of comparable rail projects, the reference class forecast estimated a cost of £320 million. A report issued in August 2011 estimated that the final cost of the yet unfinished project would be over £1 billion, for a shorter tram line than the proposed Line 2.[6]

Since the Edinburgh forecast, reference class forecasting has been applied to numerous other projects in the UK, including the £15 (US$29) billion Crossrail project in London. After 2004, The Netherlands, Denmark, and Switzerland have also implemented various types of reference class forecasting.

Before this, in 2001 (updated in 2011), AACE International (the Association for the Advancement of Cost Engineering) included Estimate Validation as a distinct step in the recommended practice of Cost Estimating (Estimate Validation is equivalent to Reference class forecasting in that it calls for separate empirical-based evaluations to benchmark the base estimate):

The estimate should be benchmarked or validated against or compared to historical experience and/or past estimates of the enterprise and of competitive enterprises to check its appropriateness, competitiveness, and to identify improvement opportunities...Validation examines the estimate from a different perspective and using different metrics than are used in estimate preparation.[7]

In the process industries (e.g., oil and gas, chemicals, mining, energy, etc. which tend to dominate AACE's membership), benchmarking (i.e., "outside view") of project cost estimates against the historical costs of completed projects of similar types, including probabilistic information, has a long history.[8]

gollark: For what purpose?
gollark: No, that is far too little.
gollark: Only if you pay me £0.86 for my time.
gollark: Yes, I saw it.
gollark: No.

See also

References
  1. Kahneman, Daniel; Tversky, Amos (1979). "Prospect Theory: An Analysis of Decision under Risk" (PDF). Econometrica. 47 (2): 263–291. CiteSeerX 10.1.1.407.1910. doi:10.2307/1914185. JSTOR 1914185.
  2. Kahneman, Daniel; Tversky, Amos (1977). "Intuitive prediction: Biases and corrective procedures" (PDF). Cite journal requires |journal= (help) Decision Research Technical Report PTR-1042-77-6. In Kahneman, Daniel; Tversky, Amos (1982). "Intuitive prediction: Biases and corrective procedures". In Kahneman, Daniel; Slovic, Paul; Tversky, Amos (eds.). Judgment Under Uncertainty: Heuristics and Biases. pp. 414–421. doi:10.1017/CBO9780511809477.031. ISBN 9780511809477.
  3. "Week 10: Reference Class Forecasting". Conceptually. Retrieved 20 April 2017.
  4. "Outside view". LessWrong Wiki. Retrieved 20 April 2017.
  5. Flyvbjerg, Bent (2006). "From Nobel Prize to Project Management: Getting Risks Right". Project Management Journal. 37 (3): 5–15. arXiv:1302.3642. Bibcode:2013arXiv1302.3642F. doi:10.1177/875697280603700302. SSRN 2238013.
  6. "Council to borrow £231m for Edinburgh trams project". BBC News. BBC. 19 August 2011. Retrieved 19 August 2011.
  7. AACE International, "Total Cost Management Framework, Section 7.3, Cost Estimating and Budgeting", 2011. p. 147
  8. Merrow, E. and Yarossi, M., "Assessing Project Cost and Schedule Risk", 1990 AACE Transactions. pp H.6.1-7

Bibliography
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.