Applica'ons of the Golden Rule

Transcription

1 Applica'ons of the Golden Rule J. Sco7 Armstrong*, University of Pennsylvania Kesten C. Green*, University of South Australia *Ehrenberg- Bass Ins:tute at University of South Australia Interna*onal Symposium on Forecas*ng Riverside, California 24 June 2015 Slides available at ForPrin.com A&G ISF 2015 Golden- R11 1

2 Golden Rule of Forecasting: Be Conservative or Forecast unto others as you would have them forecast unto you. Be conserva*ve by adhering to cumula:ve knowledge about: 1.the situa*on, and 2.evidence- based forecas*ng methods The Golden Rule of Forecas*ng was published in June

3 Golden Rule of Forecas'ng (GR) Guidelines Procedure: By logic, we developed 28 guidelines. Validity tes*ng by analyzing prior experimental comparisons relevant to the guidelines, almost none of which were done in awareness of the Golden Rule. Direc*onal effects were consistent with compara*ve tests of accuracy. 70 papers tested effect sizes: On average, the use of a single guideline reduced forecast error by 31%. 3 3

4 Why experimental findings? We believe that experimenta*on is the basis for scien*fic advances. Not feasible to iden*fy causality from analyses of nonexperimental data in uncertain complex situa*ons. Illusions in regression analysis Direc*ons of effects from nonexperimental studies ozen differ with those from experimental studies. Armstrong & Patnaik (2009). 4 4

5 Golden Rule based on cumula've knowledge about forecas'ng methods Proved to be a large undertaking to develop the hypothesis and to accurately summarize the evidence. 1.Required three years to complete the paper. 2.Eighteen people provided peer review. 3.Eleven researchers contributed on various aspects. 4.Four editors worked on the wri*ng. 5 5

6 Support for the Golden Rule 1. We contacted all authors of key studies for whom we found addresses. 2. Of those, 84% responded. 3. All but one agreed with our summary of their work (an issue not to be taken for granted. See Fawlty Towers paper). 6 6

7 Example: Predic'ng Elec'on outcomes The Golden Rule Checklist was used to evaluate PollyVote. 1.Independent judgments were made as to whether the situaion involved: a) complexity Modest b) uncertainty Modest c) likelihood of bias Low 2.Independent judgments were made as to which guidelines were relevant, and 3.RaIngs were made as to whether the the Golden Rule was used properly or not. 4.Comparisons were made of the accuracy of alternaive methods. 7 7

8 Forecast Accuracy of the PollyVote vs. typical econometric model We rated the PollyVote against the Golden Rule checklist. 13 of the 28 guidelines were relevant to forecas*ng elec*ons. The PollyVote adheres to all 13 guidelines. 8 8

9 Forecast Accuracy of the PollyVote vs. typical econometric model (across remaining days to elec'on, ) 9 9

10 Climate Change Forecasts We rated two methods used to forecast global mean temperatures

11 The Chart Behind Global Warming: (by Anthony Waes) This is the surface temperature record, on the scale of human experience. 11

12 Warming alarmists do not forecast, they create scenarios via computer simula'ons 1. Scenarios are: a. Stories about what happened in the future b. Biased so do not provide valid forecasts (Gregory & Duran, 2001). 2. The stories are based on expert judgments. 3. According to prior research, expert judgments about what will happen in complex, uncertain situa:ons are no more accurate than forecasts from people with liele exper*se: a. Seer- sucker Theory b. Tetlock s 20- year experiment 12 12

13 No- change model is conserva've given cumula've knowledge about the situa'on Disagreement about the effects of the many variables that affect temperature. See e.g. NIPCC s Climate Change Reconsidered II: Physical Science & Climate Change: The Facts

14 Golden Rule applied to IPCC scenario Golden Rule of ForecasIng Checklist was used to evaluate IPCC business as usual global warming scenario and no- change model forecasts. Consensus raings by Armstrong and Green indicated that of the 20 relevant Golden Rule Checklist guidelines: the IPCC scenarios followed none the no- change model followed 95% Don t believe us? Rate them yourself and send us your ra:ngs and reasons! Tests of forecast accuracy over the forecasing period yielded 58 forecasts for horizons of 91 to 100 years. Average error (MAE) of no- change forecast for 50- year horizon was 0.24 C. Errors from the IPCC scenario of.03 C warming- per- year were

15 Example of a conserva've guideline Decompose to best use prior informa*on. Look for data where there are different causal causal factors (e.g., to forecast traffic deaths forecast miles driven and deaths per mile driven, then recombine as shown in this paper.) Is the weather gekng warmer in Las Vegas? Each day s high and low temperatures are averaged. Than an average is taken across all days

16 16

17 Does everyone agree that Las Vegas has go7en warmer since 1937? Any complainers? 17 17

18 18

19 19

20 Why is it warmer at night in Las Vegas, but cooler in the day? During the day, the pavements and buildings store the heat. The heat is emieed during the night. Our thanks to Anthony Waes for this example

21 Earlier evidence on accuracy of IPCC projec'ons vs. no- change forecasts Tests of forecasts over the forecas*ng period yielded 58 forecasts for horizons of 91 to 100 years. The errors of these IPCC forecasts were 12.6 *mes larger than those from the simpler no- change model

22 Conclusions for Climate Change 1. Alarming IPCC temperature projec*ons are based on procedures that are insufficiently conserva*ve to be trusted 2. Cumula*ve knowledge about the situa*on was ignored. In the belief that this *me it is different

23 Possible Applica'ons Golden Rule checklist allows commentators and decision makers to assess whether forecasters were forecas:ng as they would expect others to forecast unto them. Especially useful for areas subject to bias, such as a. corporate mergers, b. mass transporta*on systems, c. law suits (adver*sing: Lance Armstrong case) d. public policies (e.g., gun control, minimum wages)

24 Conclusions on the use of the Golden Rule Checklist 1. Rapid applica*on of evidence developed over the past century. 2. Inexpensive (raters don t need high exper*se in forecas*ng) 3. Use of the guidelines produces more accurate ex ante forecasts 4. Avoids the biggest forecast errors (e.g. Winter Storm) 5. Leads to ac*ons that can improve accuracy. 6. Allows commentators and decision makers to assess whether forecasters were forecas:ng as they would expect others to forecast unto them