Decision 411: Forecasting

Transcription

1 Decision 411: Forecasting Professor: Bob Nau Course content: How to predict the future How to learn from the past using data analysis

2 Who should be interested: Anyone on a quantitative career track (financial investments, marketing research, consulting, operations, accounting, econometrics, engineering, environmental science ) Anyone who wants more experience in computer modeling & data analysis Anyone who needs to make decisions based on forecasts provided by others

3 Forecasts are used at every organizational level Corporate Strategy Marketing Finance Accounting Production, Operations & Supply Chain Sales Many numbers. or one number?

4 2003 Nobel Prize(s) ) in Economics awarded for forecasting methods Robert F. Engle for methods of analyzing economic time series with time-varying volatility (ARCH) Clive W.J. Granger "for methods of analyzing economic time series with common trends (cointegration( cointegration)

5 Recent history (pitfalls of forecasting) (X 10000) 3 DJIA to March

6 Recent history (X 10000) 3 DJIA to March 2000 Forecasts (GRW) 2 Lower 95% Upper 95%

7 Recent history

8 Recent history (X 10000) 3 DJIA to March 2000 Forecasts (GRW) 2 Lower 95% Upper 95% DJIA since March

9 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

10 Course objectives How to use data to predict the future & aid decision-making Data acquisition and integration Statistical & graphical data analysis Regression and other forecasting models Time series concepts Management of forecasting

11 Course map Forecasting methods Statistical Non-statistical Extrapolative (one variable) Associative (many variables) Simulation (what-if) Subjective (expert consensus, field estimates) Naive Smoothing Decomposition One equation (regression) ARIMA Many equations (econometric) Nonlinear (data mining via neural nets, classification trees, etc.) We are mainly here Betting markets

12 Course outline Week 1: Data concepts & simple models: linear trend & random walk Week 2: Seasonal adjustment & exponential smoothing (HW#1( due Tues 3/27) Week 3: Regression (HW#2 due Tues 4/3) Week 4: More regression (Quiz( on Tues 4/10) Week 5: ARIMA models (HW#3( due Tues 4/17) Week 6: Additional topics (automatic, nonlinear ) Final project (due( at end of exam week Thur 5/3)

13 Readings My notes handed out in class, also on course web page faculty.fuqua.duke.edu/~rnau/decision411coursepage.html Powerpoint slides from lectures Additional materials on web page, bulletin board, & CD s Optional stats textbook by Schleifer & Bell (or any other MBA-level stats textbook)

14 Software Statgraphics XV (in lab & on your PC) Excel Library databases (Economagic( Economagic,, etc.) Google

15 Decision 411 CD s Video files that provide a tour of Statgraphics & Economagic on your own PC View with Camtasia Player (included on CD) Hit Alt-Enter to toggle the control bar

16 Bulletin board Main course b-board: b board: mba.spring2007_session4.decision411.forecasting Will be used for answers to FAQ s, additional comments on lecture topics, & discussions of statistics in the news and in the workplace check check it frequently Feel free to post your own examples of good/bad/interesting stats (extra credit for class participation!) Do not post any assignment-related questions.

17 If you have a question for me,, send it by rather than posting on a b-board b board but check main b-board b board first to see it has already been asked and answered Use a descriptive subject line beginning with Forecasting:

18 Grading basis 45% homework (3 assignments) 15% quiz 30% final project 10% class participation

19 Study group policy Work in teams of 2 (max) Try to find a partner by Friday OK to team up with someone from other section Send me e mail if still seeking a partner

20 Final project Final project may be based on a data set and modeling goal of YOUR choice Should get started by 5th week of class Alternatively, there will be several designated project options (essentially a fourth homework assignment) Can work in groups of 2 on final project as well as regular homework

21 Honor code issues You are encouraged to consult your classmates for general advice on forecasting concepts and software use Specific details of data analysis assignments should be discussed only with your study- group partner Don t post notes on b-board b board that are at all related to assignments prior to due dates send any questions to me by . e

22 Suggestions & examples welcome! If you are interested in particular forecasting problems or can suggest particular examples that might be useful for classroom discussion, please send me e mail (include data if you have it) Exception: no examples from graded assignments in other ongoing courses!

23 Today s agenda Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

24 How can we predict the future? Look for statistical patterns that were stable in the past and which can be expected to remain stable Extrapolate those patterns into the future I have seen the future and it is very much like the present, only longer.

25 Example: stable mean & variance Time Sequence Plot for X Constant mean = actual forecast 95.0% limits X

26 Example: stable trend Time Time Sequence Series Plot Plot for for Y Y Linear Random trend = walk with drift t actual forecast 95.0% limits Y

27 Example: stable seasonality ) 3.4 Time Series Plot for RetailxautoNSA 3 RetailxautoNSA /92 1/94 1/96 1/98 1/00 1/02 1/04 1/06 1/08

28 Example: stable correlations age features price sqfeet tax

29 Transformations Sometimes a stable pattern is not apparent on a graph of the raw data Transformations of the data (deflation, logging, differencing, seasonal adjustment) may help to reveal the underlying pattern

30 Example: stock prices Pattern: exponential growth curve with 1990 s bubble

31 Logged stock prices Natural log transformation linearizes the growth : slope of trend line in logged units is average percentage growth

32 Logged stock prices Logged indices since 1990

33 Difference of natural log = percent change between periods Logged & differenced stock prices 0.25 Time Series Plot for adjusted SP500monthclose adjusted SP500monthclose /80 1/84 1/88 1/92 1/96 1/00 1/04 1/08

34 Example: U.S. retail sales (excluding autos) Pattern: strong nominal growth & seasonal pattern

35 Deflated and seasonally adjusted sales x-autos 1310 Variables RetailexautoSA/CPIcityavg /92 1/96 1/00 1/04 1/08 Pattern: real growth accelerated in late 90 s, flattened after March 2000 peak, dipped in September 2001, ramped up again, but recently?

36 What if patterns are not stable? Trends, seasonality, etc., may vary in time This may limit the amount of past data that should be used for fitting the model (don t merely use all data because it is there ) More sophisticated forecasting models are capable of tracking time-varying parameters Expert opinion can also be used to anticipate changes in patterns

37 A changing pattern: Housing Starts Strong seasonal pattern, big drop in last year!

38 (A few) Forecasting Principles Use the most relevant & recent data Seek diverse & independent data sources Let model selection be guided by theory and domain knowledge,, not just fit to past data Keep It Simple Test the assumptions behind the model Validate the model on hold-out out data Report confidence intervals with forecasts

39 The best forecasting model Is the one that can be expected to make the SMALLEST ERRORS when predicting the FUTURE* Is intuitively reasonable Is no more complicated than necessary Provides insight into trends & causes Can be explained to your boss or client *not always the same thing as giving the best fit to the past!

40 Forecasting risks (sources of error) 1. Intrinsic risk (random error) 2. Parameter risk (estimation error) 3. Model risk (erroneous assumptions) Note: statistical confidence intervals are based on estimates of intrinsic risk and parameter risk, not model risk

41 Intrinsic risk Even the best model cannot be expected to make perfect predictions ( forecasting is hard, especially when it s about the future ) Intrinsic risk is measured by error statistics such as the standard error of the estimate (RMS error, adjusted for number of coefficients) Intrinsic risk can be reduced, in principle, by finding a better model based on more detailed assumptions and data

42 Parameter risk Even if you have the correct forecasting model, its parameters may not be exactly known they must be estimated from available data Parameter risk is measured by standard errors and t-statistics t of model coefficients Parameter risk can be reduced, in principle, by using more past data to estimate the model The blur of history problem: older data may be stale and not reflect current conditions Parameter risk is usually a smaller component of forecast error than intrinsic risk or model risk

43 Model risk This is often the most serious risk and its effects are not taken into account in the calculation of confidence intervals Model risk can be reduced by following good forecasting principles: Exploratory data analysis to make sure important patterns or related variables are not overlooked Statistical tests of key assumptions Out-of of-sample validation of statistical model Use of domain knowledge and expert judgment

44 Today s agenda Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

45 Where to get data Internet sources (Economagic( Economagic,, library databases, government agencies ) Your corporate database Trade associations & journals Econometric consulting firms Designed experiments and surveys

46 How to move data around Most computer programs use their own idiosyncratic binary file formats for storing data (word processors, spreadsheets, stat programs, database programs ) All programs must also read and write text files in order to communicate with people Hence, different programs can always exchange data with each other in the form of text files 1 character of text data = 1 byte of storage

47 Text files May be either fixed format or delimited In a fixed format file, data fields are delineated by character position within a line xxxxx xxxxx xxxxx xxxxx In a delimited file, data fields are separated by delimiting characters (commas, tabs, spaces) xxxxx, xxxx, xxxxx, xxxxx, Statgraphics & Excel can easily read tab- or comma-delimited files as well as XLS files

48 From Economagic to Statgraphics* Save several series to personal workspace Create Excel file or CSV (comma-delimited text) file Open the file in Excel & clean it up (delete extraneous rows, add more descriptive column headings as variable names) Save the cleaned-up file under a new name, CLOSE IT,, and open it in Statgraphics * See video for details

49 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

50 Statistical graphics Wizards & integrated plotting procedures make charting easy Complex patterns in data can be uncovered and communicated by following principles of good graphic design Charts can also be boring, confusing, or deceptive if produced thoughtlessly

51 Tufte s graphical principles* Above all else, show the data Avoid chartjunk chartjunk : dark grid lines, false perspective, unintentional optical art, self- promoting graphics Maximize the ratio of data ink to non-data ink Mobilize every graphical element, perhaps several times over, to show the data (e.g., data values printed on a bar chart) * The Visual Display of Quantitative Information by E. Tufte

52 Charts vs. tables Charts are most effective when data are numerous and/or multi-dimensional If the data are one-dimensional and not too numerous, or if numerical details are important, a table may be better than a chart A table is nearly always better than a dumb pie chart; the only worse design than a pie chart is several of them

53 Focus attention Don t embed important numbers in sentences of text set them apart in a table or chart. Treat tables & charts as paragraphs, and include them in the narrative at the appropriate points Annotate charts with appropriate comments Maximize data density: graphs can be shrunk way down so that more than one will fit on a page or slide

54 Excel & Statgraphics tips Embed small, well-labeled, labeled, well-chosen charts & tables in your reports Make points and lines thick enough to show the data Suppress gridlines where not needed Use an appropriate chart type (e.g., line plots for time series, scatterplots for cross- sectional data, bar charts or tables rather than pie charts)

55 Economagic GIF charts Often it is instructive to plot more than one variable on the same graph here different left and right axis scales were used to align the two series Economagic will superimpose bars indicating periods of recession

56 Scatterplot matrix age features price This chart provides detailed views of relationships between many variables that may be helpful in regression analysis sqfeet tax Describe/Numeric Variables/Multiple-Variable Analysis

57 Residual time series plot Residual Plot for adjusted DJIAtoMarch Random walk with drift Residual Residuals are forecast errors within the sample that was fitted by the model. Look for non-random patterns, changes in variance, outliers (this one is not bad except for a couple of outliers) /80 1/85 1/90 1/95 1/00 1/05 Residuals-vs-time or vs-row-number is an option in Forecasting, Multiple Regression, & Advanced Regression)

58 Residual probability plot (vertical) proportion Residual Plot for adjusted DJIAtoMarch Random walk with drift Residual Deviations from diagonal line reveal non-normality of error distribution (this one is not bad, except for two negative outliers) Plot/Exploratory Plots/Normal Probability Plot (also a residual plot pane option in Forecasting & Advanced Regression)

59 Residual autocorrelation plot Residual Autocorrelations for RSJEWEL Winter's exp. smoothing with alpha = , beta = , gamma = Autocorrelations lag Ideally all the autocorrelation bars should be within the red 95% significance bands. This plot shows significant autocorrelation at lag 12, indicating a poor fit to the seasonal pattern in the data.

60 Today s agenda Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

61 Consider the following time series: Time Series Plot for X X

62 How to forecast? If you have reason to believe the observations are statistically independent and identically distributed,, with no trend*,, the appropriate forecasting model is the MEAN model Just predict that future observations will equal the mean of the past values *These assumptions might be based on domain knowledge, or else they could be tested by comparing alternative models and looking at autocorrelations, etc..

63 Stats review: sampling from a population X = random variable, n = sample size μ, σ = population mean & standard deviation X n i xi = =1 n = sample mean (AVERAGE) S 1( = n i= xi X ( n 1) ) 2 = sample std. dev. (STDEV)

64 Standard error of the mean SE mean = This is the estimated standard deviation of the sampling distribution of the mean It measures the precision of our estimate of the (unknown) population mean As n gets larger, SE mean gets smaller and the sampling distribution becomes normal* S n *Central Limit Theorem

65 Std. deviation vs. std. error? The term standard deviation (usually) refers to the actual root-mean mean-squared deviation of a given population or sample around its mean The term standard error refers to the expected root-mean mean-squared deviation of an estimate or forecast around the true value under repeated sampling-- --i.e., the standard deviation of the error

66 Forecasting with the mean model Let xˆn+1 denote a forecast of x n+1 based on data observed up to period n If x n+1 is assumed to be independently drawn from the same population as the sample x 1,, x n, the forecast that minimizes mean squared error is simply the sample mean: xˆ n +1 = X

67 Forecast standard error The standard error of the forecast has two components: fcst mean SE = S + SE = S + n This term measures the intrinsic risk ( noise in the data) This term measures the paramete risk (error in estimating the signal in the data) For the mean model, the result is that the forecast standard error is slightly larger than the sample standard deviation Note that variances, rather than standard deviations, are additive

68 Confidence intervals for forecasts A point forecast should always be accompanied by a confidence interval to indicate its accuracy but what is a confidence interval?? An x% confidence interval is an interval calculated by a rule which has the property that the interval will cover the true value x% of the time under simulated conditions, assuming the model is correct. Loosely speaking,, there is an x% chance that your data will fall in your x% confidence interval but only if your model and its underlying assumptions are correct! (This is why we test assumptions.)

69 Confidence interval = point forecast ± t standard errors If the distribution of forecast errors is assumed to be normal,, a 95% confidence interval for the forecast is ˆ t SE t,n xn+ 1 ±. 05, n 1 where is the critical value of the Student s t distribution* with a tail probability of.05 and n 1 1 degrees of freedom (in Excel, t. 05,n 1 = TINV(.05,n 1)) *discovered by W.S. Gossett of Guinness Brewery fcst

70 en.wikipedia.org/wiki/william_sealey_gosset

71 t vs. normal distribution The t distribution is the distribution of ( X μ) SE mean i.e., the number of standard errors from the true mean when the standard deviation is unknown. The t distribution resembles a standard normal (z)( ) distribution but with fatter tails for small n

72 Normal vs. t: much difference? Normal t with 20 df t with 10 df t with 5 df

73 # standard errors ± computed from normal and t distributions are very close except for very low d.f.. or very high confidence Confidence level (2-sided) d.f. 90.0% 95.0% 99.0% 99.5% 99.9% Normal

74 Empirical rules of thumb For n 20 or more, the critical t value is approximately 2, so the empirical 95% CI is roughly the point forecast plus or minus two standard errors, however A prediction interval that covers 95% of the data is often too wide to be managerially useful 50% (a coin flip ) ) or 80% might be easier for a manager to understand A 50% confidence interval is roughly plus or minus two-thirds thirds of a standard error

75 Example, continued Time series X (n=20*,( d.f. =19**): 114, 126, 123, 112, 68, 116, 50, 108, 163, 79 67, 98, 131, 83, 56, 109, 81, 61, 90, 92 Statistics : X = 96.35, S = SE mean = / 20 = 6.48 SE fcst = = * True parameters : μ= 100, σ = 30

76 Confidence intervals for predictions Exact 95% CI* = ± = [34.2, 158.5] Exact 50% CI** = ± = [77.8, 114.9] *t.. 05, 19 = 2093 **t., 519= 0688.

77 Statgraphics output: mean model Time Sequence Plot for X Constant mean = actual forecast 95.0% limits X

78 Statgraphics output: mean model Time Sequence Plot for X Constant mean = actual forecast 50.0% limits X A 50% confidence interval is 1/3 the width of a 95% confidence interval.

79 What if there s really a trend? Time Sequence Plot for X Linear trend = t actual forecast 50.0% limits X Actually, t=1.61 for slope coefficient, so this model would be rejected at.05 level of significance. That s a different modeling assumption, and it leads to very different forecasts and confidence intervals.

80 Yes, it s simple, but... The mean model is the foundation for more sophisticated models we will encounter later (RW, regression, ARIMA) It has the same generic features: A coefficient to be estimated A standard error for the coefficient that reflects parameter risk A forecast standard error that reflects intrinsic risk & parameter risk and model risk too!

81 Today s agenda Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness

82 Market-based forecasting Betting markets are often an efficient way to aggregate diverse opinions (and to share risks or have fun) Probabilistic forecasts derived from contract prices are often well-calibrated Caveats: markets don t always work may exhibit herding or distortions when bettors lack independent information or have highly correlated personal stakes in events Some applications are controversial (e.g. terrorism futures )

83 March madness: forecasting basketball games via a betting market These are the price quotes on Tradesports.com at 10am on Monday, March 19, 2007.

84 As of Monday Florida s stock is rising

85 Recap of today s topics Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) March madness