Decision 411: Forecasting

Transcription

1 Decision 411: Forecasting Professor: Bob Nau Course content: How to predict the future How to learn from the past using data analysis Who should be interested: Anyone on a quantitative career track (financial investments, marketing research, consulting, operations, accounting, econometrics, engineering, environmental science, policy analysis ) Anyone who wants more experience in computer modeling & data analysis Anyone who needs to make decisions based on forecasts provided by others 1

2 Forecasts are used at every organizational level Corporate Strategy Marketing Finance Accounting Production, Operations & Supply Chain Sales Many numbers. or one number? 2003 Nobel Prize(s) ) in Economics awarded for forecasting methods Robert F. Engle for methods of analyzing economic time series with time-varying volatility (ARCH) Clive W.J. Granger "for methods of analyzing economic time series with common trends (cointegration( cointegration) 2

3 Recent history (pitfalls of forecasting) (X 10000) 3 DJIA to March Recent history (X 10000) 3 DJIA to March 2000 Forecasts (GRW) 2 Lower 95% Upper 95%

4 Recent history Recent history (X 10000) 3 DJIA to March 2000 Forecasts (GRW) 2 Lower 95% Upper 95% DJIA since March

5 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets (Tradesports( Tradesports,, etc.) Course objectives How to use data to predict the future & aid decision-making Data acquisition and integration Statistical & graphical data analysis Regression and other forecasting models Time series concepts Management of forecasting 5

6 Course map Forecasting methods Statistical Non-statistical Extrapolative (one variable) Associative (many variables) Simulation (what-if) Subjective (expert consensus, field estimates) Random walk One equation (regression) Many equations (econometric) Betting markets Smoothing Seasonal decomposition ARIMA Nonlinear (data mining via neural nets, classification trees, etc.) We are mainly here Course outline Week 1: Data concepts & simple models: linear trend & random walk Week 2: Seasonal adjustment & exponential smoothing (HW#1( due Tues 9/11) Week 3: Regression (HW#2 due Tues 9/18) Week 4: More regression (Quiz( on Tues 9/25) Week 5: ARIMA models (HW#3( due Tues 10/2) Week 6: Additional topics (automatic, nonlinear ) Final project (due( at end of exam week Wed 10/17) 6

7 Readings My notes handed out in class & on course web page: faculty.fuqua.duke.edu/~rnau/decision411coursepage.html Powerpoint slides from lectures Additional materials on web page, bulletin board, & CD s Optional stats textbook of your choice Some forecasting texts are also on reserve in the Fuqua library Software Statgraphics XV (in lab & on your PC) Excel Library databases (Economagic( Economagic,, etc.) Google 7

8 Decision 411 CD s Video files that provide a tour of Statgraphics & Economagic on your own PC View with Camtasia Player (included on CD) Hit Alt-Enter to toggle the control bar Main course b-board: b board: Bulletin board mba.fall_1_2007.decision411.forecasting Will be used for answers to FAQ s, additional comments on lecture topics, & discussions of statistics in the news and in the workplace check check it frequently Feel free to post your own examples of good/bad/interesting stats (extra credit for class participation!) Do not post any assignment-related questions. 8

9 If you have a question for me,, send it by rather than posting on a b-board b board but check main b-board b board first to see it has already been asked and answered Use a descriptive subject line beginning with Forecasting: Grading basis 45% homework (3 assignments) 15% quiz 30% final project 10% class participation 9

10 Study group policy Work in teams of 2 (max) Try to find a partner by Friday OK to team up with someone from other section Send me e if still seeking a partner Homework assignments Assignments will give detailed instructions on how to obtain the data, import it into Statgraphics,, and do much of the analysis. Your task is to discover the interesting and important patterns and determine a best model for purposes of forecasting and/or decision making from among a given set of models Have fun! 10

11 Homework guidelines Assignments should be submitted in Powerpoint form, ideally no more than 15 slides. Don t worry about fancy fonts, backgrounds, or clip art focus on the important technical issues. First one or two slides should state your most important findings, describe your final model, give your bottom-line forecasts & confidence intervals Subsequent slides should document the sequence of steps by which you reached these conclusions (data exploration, comparison of models, etc.) Cut-and and-paste key reports and graphs from Statgraphics and add your specific comments on what they tell you about the data. Final project Final project may be based on a data set and modeling goal of YOUR choice Should get started by 5th week of class Alternatively, there will be several designated project options (essentially a fourth homework assignment) Can work in groups of 2 on final project as well as regular homework 11

12 Honor code issues You are encouraged to consult your classmates for general advice on forecasting concepts and software use Specific details of data analysis assignments should be discussed only with your study-group partner Don t post notes on b-board b board that are at all related to assignments prior to due dates send send any questions to me by . e Suggestions & examples welcome! If you are interested in particular forecasting problems or can suggest particular examples that might be useful for classroom discussion, please send me e (include data if you have it) Exception: no examples from graded assignments in other ongoing courses! 12

13 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets How can we predict the future? Look for statistical patterns that were stable in the past and which can be expected to remain stable* Extrapolate those patterns into the future One important test of assumed patterns is whether unexplained variations (forecast errors) are independent and identically distributed ( i.i.d i.i.d. ). ) * I have seen the future and it is very much like the present, only longer. Kehlog Albran,, The Profit 13

14 Example: i.i.d.. variations around a horizontal line Time Time Sequence Series Plot Plot for for X X Constant mean = actual forecast 95.0% limits X Properties of objects coming off an assembly line might have this pattern. More complex patterns are often reducible to this pattern by suitable transformations. Example: i.i.d.. variations around a trend line Z Time Time Sequence Series Plot Plot for for Z Z Linear trend = t actual forecast 95.0% limits Trends are usually not perfectly linear, and variations around them are usually not i.i.d., but this assumption is often used as a 1 st -order approximation for trended data 14

15 Example: i.i.d. changes in the level of the series from one period to the next Time Time Sequence Series Plot Plot for for Y Y Random walk with drift = actual forecast 95.0% limits Y This pattern is often seen in financial markets and also in many physical processes (e.g. Brownian motion) Example: stable seasonal pattern (X ) RetailxAutoNSA FORECAST This pattern is typically seen in retail sales data and in various measures of macroeconomic activity 15

16 Example: stable correlations among variables age features price sqfeet tax Correlations provide a basis for using regression models to predict some variables from others. Transformations Sometimes a stable pattern is not apparent on a graph of the raw data Transformations of the data (deflation, logging, differencing, seasonal adjustment ) may help to reveal the underlying pattern Ideally the transformed data can be fitted by a relatively simple model 16

17 Example: stock prices Pattern: exponential growth curve with 1990 s bubble Logged stock prices Natural log transformation linearizes the growth : slope of trend line in logged units is average percentage growth. Dips are more clearly seen to coincide with recessions. 17

18 Logged stock prices Logged indices since 1990 adjusted SP500monthclose Logged & differenced stock prices Time Series Plot for adjusted SP500monthclose /80 1/84 1/88 1/92 1/96 1/00 1/04 1/08 Difference of natural log percent change between periods, which is independently and almost identically distributed. Variance, i.e., volatility, may vary over time. 18

19 Example: U.S. retail sales (excluding autos) Pattern: strong nominal growth & seasonal pattern Deflated and seasonally adjusted sales x-autos Pattern: real growth accelerated in late 90 s, flattened after March 2000 peak, dipped in September 2001, ramped up again, but recently? 19

20 What if patterns are not stable? Trends, seasonality, volatility, etc., may vary in time This may limit the amount of past data that should be used for fitting the model (don t merely use all data because it is there ) More sophisticated forecasting models are capable of tracking time-varying parameters Expert opinion can also be used to anticipate changes in patterns Prices on stock options reveal the instantaneous volatility of stock prices in the mind of the representative investor. A changing pattern: Housing Starts Strong seasonal pattern and general upward trend with cyclical variations, but big drop in When will it rebound? 20

21 (A few) Forecasting Principles Use the most relevant & recent data Seek diverse & independent data sources Let model selection be guided by theory and domain knowledge,, not just fit to past data Keep It Simple Test the assumptions behind the model Validate the model on hold-out out data Report confidence intervals with forecasts The best forecasting model Is the one that can be expected to make the SMALLEST ERRORS when predicting the FUTURE* Is intuitively reasonable Is no more complicated than necessary Provides insight into trends & causes Can be explained to your boss or client *not always the same thing as giving the best fit to the past! 21

22 Forecasting risks (sources of error) 1. Intrinsic risk (random error unavoidable, although may be reduced by more sophisticated modeling) 2. Parameter risk (estimation error may be reduced by collecting more data) 3. Model risk (erroneous assumptions often often the biggest danger!) Warning: statistical confidence intervals are based only on estimates of intrinsic risk and parameter risk, not model risk! Intrinsic risk Even the best model cannot be expected to make perfect predictions ( forecasting is hard, especially when it s about the future ) Intrinsic risk is measured by error statistics such as the standard error of the model*, mean absolute error, and mean absolute percentage error Intrinsic risk can be reduced, in principle, by finding a model that explains more of the variance by making more accurate assumptions and by using more or better predictor variables *Standard error of the model is the error standard deviation adjusted for # coefficients estimated 22

23 Parameter risk Even if you have the correct forecasting model, its parameters will not be exactly known they must be estimated from available data Parameter risk is measured by standard errors and t-statistics of model coefficients Parameter risk can be reduced, in principle, by using more past data to estimate the model The blur of history problem: older data may be stale and not reflect current conditions Parameter risk is usually a smaller component of forecast error than intrinsic risk or model risk Model risk This is often the most serious risk and its effects are not taken into account in the calculation of confidence intervals Model risk can be reduced by following good forecasting principles: Exploratory data analysis to make sure important patterns or related variables are not overlooked Statistical tests of key assumptions Out-of of-sample validation of statistical model ( hold-out out data, backtesting backtesting ) Use of domain knowledge and expert judgment to guide model selection and provide reality checks 23

24 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets Where to get data Internet sources (Economagic( Economagic,, library databases, government agencies ) Your corporate database Trade associations & journals Econometric consulting firms Designed experiments and surveys 24

25 How to move data around Most computer programs use their own idiosyncratic binary file formats for storing data (word processors, spreadsheets, stat programs, database programs ) All programs must also read and write text files in order to communicate with people Hence, different programs can always exchange data with each other in the form of text files 1 character of text data = 1 byte of storage Text files May be either fixed format or delimited In a fixed format file, data fields are delineated by character position within a line xxxxx xxxxx xxxxx xxxxx In a delimited file, data fields are separated by delimiting characters (commas, tabs, spaces) xxxxx, xxxx, xxxxx, xxxxx, Statgraphics & Excel can easily read tab- or comma- delimited files as well as XLS files 25

26 From Economagic to Statgraphics* Save several series to personal workspace Create Excel file or CSV (comma-separated separated-value) file Open the file in Excel & clean it up (delete extraneous rows, add more descriptive column headings as variable names) Save the cleaned-up file under a new name, CLOSE IT,, and open it in Statgraphics * See video for details Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets 26

27 Statistical graphics Wizards & integrated plotting procedures make charting easy Complex patterns in data can be uncovered and communicated by following principles of good graphic design Charts can also be boring, confusing, or deceptive if produced thoughtlessly Tufte s graphical principles* Above all else, show the data Avoid chartjunk chartjunk : dark grid lines, false perspective, unintentional optical art, self-promoting graphics Maximize the ratio of data ink to non-data ink Mobilize every graphical element, perhaps several times over, to show the data (e.g., data values printed on a bar chart) * The Visual Display of Quantitative Information by E. Tufte 27

28 Charts vs. tables Charts are most effective when data are numerous and/or multi-dimensional If the data are one-dimensional and not too numerous, or if numerical details are important, a table may be better than a chart A table is nearly always better than a dumb pie chart; the only worse design than a pie chart is several of them Focus attention Don t embed important numbers in sentences of text set them apart in a table or chart. Treat tables & charts as paragraphs, and include them in the narrative at the appropriate points Annotate charts with appropriate comments Maximize data density: graphs can be shrunk way down so that more than one will fit on a page or slide 28

29 Excel & Statgraphics tips Embed small, well-labeled, labeled, well-chosen charts & tables in your reports Make points and lines thick enough to show the data Suppress gridlines where not needed Use an appropriate chart type (e.g., line plots for time series, scatterplots for cross-sectional sectional data, bar charts or tables rather than pie charts) Names and units of variables should be clearly shown in titles, legends, and/or axis labels Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets 29

30 Consider the following time series: Time Series Plot for X X How to forecast? If you have reason to believe the observations are statistically independent and identically distributed, with no trend*,, the appropriate forecasting model is the MEAN model Just predict that future observations will equal the mean of the past values *These assumptions might be based on domain knowledge, or else they could be tested by comparing alternative models and looking at autocorrelations, etc.. 30

31 Stats review: population statistics X = random variable N = size of entire population (possibly infinite) n = size of a finite sample The population ( true ) mean μ is the average of the all values in the population: N = x N μ i= 1 The population variance σ 2 is the average squared deviation from the true mean: N ( xi μ i= σ = ) N The population standard deviation σ is the square root of the population variance, i.e., the root mean squared deviation from the true mean. i Stats review: sample statistics In forecasting applications, we never observe the whole population. The problem is to forecast from a small sample. Hence statistics such as means and standard deviations must be estimated with error. The sample mean is the average of the all values in the sample: n X = i = 1x n The sample variance s 2 is the average squared deviation from the sample mean, except with a factor of n 1 rather than n in the denominator: n i 2 1 ( xi X i= s = ) n 1 and the sample standard deviation is its square root, s 2 31

32 Sample statistics, continued Why the factor of n 1?? This corrects for the fact that the mean has been estimated from the same sample, which fudges it in a direction that makes the mean squared deviation around it less than it ought to be. Technically we say that a degree of freedom for error has been used up by calculating the sample mean from the same data. n X = i = 1x n The correct adjustment to get an unbiased estimate of the true variance is to divide by the number of degrees of freedom, not the number of data points i In Excel Population mean = AVERAGE (x( 1, x N ) Population variance = VARP (x( 1, x N ) Population std. dev. = STDEVP(x 1, x N ) Sample mean = AVERAGE (x( 1, x n ) Sample variance = VAR (x( 1, x n ) Sample std. dev. = STDEV (x( 1, x n ) 32

33 Why squared error? Why should we measure variability in terms of average squared deviations instead of average absolute deviations around a central value? Squared error has a lot of nice properties: The central value around which average squared deviations are minimized is the mean,, so by attempting to minimize squared deviations we are implicitly calculating means. From a decision-theoretic viewpoint, large errors usually have proportionally worse consequences than small errors, hence squared error is more representative of economic consequences of error. Variances and covariances play a key role in normal distribution theory & regression analysis. Standard error of the mean SE mean This is the estimated standard deviation of the error that we would make in using the sample mean X as an estimate of the true mean μ, if we repeated this exercise with other independent samples of size n. It measures the precision of our estimate of the (unknown) true mean from a limited sample of data. As n gets larger, SE mean gets smaller and the distribution of errors becomes normal* = s n *Central Limit Theorem 33

34 Standard deviation or standard error? The term standard deviation (usually) refers to the actual root-mean mean-squared deviation of a given population or sample around its mean The term standard error refers to the expected root-mean mean-squared deviation of an estimate or forecast around the true value under repeated sampling Thus, a standard error is the standard deviation of the error in estimating or forecasting something Forecasting with the mean model Let xˆn+1 denote a forecast of x n+1 based on data observed up to period n If x n+1 is assumed to be independently drawn from the same population as the sample x 1,, x n, then the forecast that minimizes mean squared error is simply the sample mean: xˆ n +1 = X Now, what is the standard deviation of the error can we expect to make in using X as a forecast for x n+ n+1? 34

35 Standard error of the forecast The standard error of the forecast has two components: fcst mean SE = s + SE = s + n This term measures the intrinsic risk ( noise in the data) This term measures the paramete risk (error in estimating the signal in the data) For the mean model, the result is that the forecast standard error is slightly larger than the sample standard deviation Note that variances, rather than standard deviations, are additive Confidence intervals for forecasts A point forecast should always be accompanied by a confidence interval to indicate its accuracy but what is a confidence interval?? An x% confidence interval is an interval calculated by a rule which has the property that the interval will cover the true value x% of the time under simulated conditions, assuming the model is correct. Loosely speaking,, there is an x% chance that your data will fall in your x% confidence interval but only if your model and its underlying assumptions are correct! (This is why we test assumptions.) 35

36 Confidence interval = point forecast ± t standard errors If the distribution of forecast errors is assumed to be normal,, a 95% confidence interval for the forecast is where is the critical value of the Student s t distribution* with a tail probability of.05 and n 1 degrees of freedom In Excel, t. 05,n 1 t. 05,n 1 xˆ ± n+ 1 t. 05, n 1 SE fcst = TINV(.05, n 1) *discovered by W.S. Gossett of Guinness Brewery t. 05,n 1 en.wikipedia.org/wiki/william_sealey_gosset 36

37 t vs. normal distribution The t distribution is the distribution of ( X μ) SE mean which is the number of standard errors by which the sample mean deviates from the true mean when the standard deviation of the population is unknown. The t distribution resembles a standard normal (z)( distribution but with fatter tails for small n Normal vs. t: much difference? Normal t with 20 df t with 10 df t with 5 df 37

38 # standard errors to use for calculating confidence intervals is very similar for normal and t distributions except for very low d.f.. or very high confidence d.f. Normal % Confidence level (2-sided) 95.0% % % % Empirical rules of thumb For n 20 or more, the critical t value is approximately 2, so the empirical 95% CI is roughly the point forecast plus or minus two standard errors, however A prediction interval that covers 95% of the data is often too wide to be managerially useful 50% (a coin flip ) ) or 80% might be easier for a manager to understand A 50% confidence interval is roughly plus or minus two-thirds thirds of a standard error 38

39 Example, continued Time series X (n=20*, d.f. =19**): 114, 126, 123, 112, 68, 116, 50, 108, 163, 79 67, 98, 131, 83, 56, 109, 81, 61, 90, 92 Statistics: X = 96.35, s = SE mean = / 20 = 6.48 SE fcst = = * True parameters : μ= 100, σ = 30 Confidence intervals for predictions Exact 95% CI* = ± = [34.2, 158.5] Exact 50% CI** = ± = [77.8, 114.9] *t.. 05, 19 = 2093 **t., 519=

40 Statgraphics output: mean model X Time Sequence Plot for X Constant mean = actual forecast 95.0% limits Statgraphics output: mean model X Time Sequence Plot for X Constant mean = actual forecast 50.0% limits A 50% confidence interval is 1/3 the width of a 95% confidence interval. 40

41 What if there s really a trend? X Time Sequence Plot for X Linear trend = t actual forecast 50.0% limits Actually, t=1.61 for slope coefficient, so this model would be rejected at.05 level of significance. That s a different modeling assumption, and it leads to very different forecasts and confidence intervals. Yes, it s simple, but... The mean model is the foundation for more sophisticated models we will encounter later (random walk, regression, ARIMA) It has the same generic features: A coefficient to be estimated A standard error for the coefficient that reflects parameter risk A forecast standard error that reflects intrinsic risk & parameter risk and model risk too! 41

42 Course introduction Today s agenda Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets Market-based forecasting Betting markets are often an efficient way to aggregate diverse opinions (and to share risks or have fun) Probabilistic forecasts derived from contract prices are often well-calibrated Caveats: markets don t always work may exhibit herding or distortions when bettors lack independent information or have highly correlated financial or emotional stakes in events Some applications are controversial (e.g. terrorism futures ) 42

43 Forecasting U.S. Open men s tennis champion via a betting market Outright winner price quotes on Tradesports.com at 5:00pm EDT, August 30: Roger Federer s estimated probability of winning is around 65% Forecast for women s champion Maria Sharapova s probability of winning was estimated to be 20% 43

44 Forecasting the baseball world series champion Boston Red Sox are at around 20% Recap of today s topics Course introduction Forecasting tools & principles How to obtain data & move it around Statistical graphics Forecasts and confidence intervals: the simplest case (mean model) Betting markets 44