MFx Macroeconomic Forecasting

Transcription

1 MFx Macroeconomic Forecasting IMFx This training material is the property of the International Monetary Fund (IMF) and is intended for use in IMF Institute for Capacity Development (ICD) courses. Any reuse requires the permission of the ICD. EViews is a trademark of IHS Global Inc.

2 Module 9 (Optional): Combination Forecasts Pat Healy & Carl Sandberg

3 Roadmap M9_Intro. Introduction, Motivations & Outline M9_S1. Forecast Combination Basics M9_S2. Solving the Theoretical Combination Problem & Implementation Issues M9_S3. Methods to Estimate Weights (M<T) M9_S4. Methods to Estimate Weights (M>T) M9_S5. EViews 1: EViews Workshop 1 M9_S6. EViews 2: Forecast Combination Tools in EViews M9_Conclusion. Conclusions

4 Module 9 M9_Intro: Introduction, Motivations & Outline

5 Introduction Generally speaking, multiple forecasts are available to decision makers before they make a policy decision Key Question: Given the uncertainty associated with identifying the true DGP, should a single (best) forecast be used? Or should we (somehow) average over all the available forecasts?

6 Motivations: One vs. Many There are several disadvantages of using only one forecasting model: Model misspecifications of an unknown form Implausible that one statistical model would be preferable to others at all points of the forecast horizon Combining separate forecasts offers: A simple way of building a complex, more flexible forecasting model to explain the data Some insurance against breaks or other non-stationarities that may occur in the future

7 Outline 1. Forecast Combination Basics 2. Solving the Theoretical Combination Problem & Implementation Issues 3. Methods to Estimate & Assign Weights 4. Workshop 1: Combining Forecasts in EViews 5. Workshop 2: EViews Combination Tool 6. Wrap-Up

8 Module 9 Session 1, Part 1: Forecast Combination Basics

9 1. Forecast Combination Basics Learning Objectives: Look at a general framework and notation for combining forecasts Introduce the theoretical forecast combination problem

10 General Framework Today (say, at time T) we want to forecast the value that a variable of interest (Y) will take at time T+h We have a certain number (M) of forecasts available How can we pool, or combine these M forecasts into an optimal forecast? Is there any advantage of pooling instead of just finding and using the best one among the M available?

11 Some Notation y t is the value of Y at time t (today is T) x t,h,i is an unbiased (point) forecast of y t+h made at time t h is the forecasting horizon i = 1, M is the identifier of the available forecast M is the total number of forecasts

12 Some (more) Notation e t+h,i = y t+h - x t,h,i is the forecast (prediction) error 2 t+h,i = Var(e 2 t+h,i) is the inverse of precision t+h,i,j = Cov(e t+h,i e t+h,j ) w t,h,i = [w t,h,1, w t,h,m ] is a vector of weights L(e t,h ) is the loss from making a forecast error E [L(e t,h )] is the risk associated to a forecast

13 The Forecast Combination Problem A combined forecast is a weighted average of the M forecasts: M c yˆ T h wt, h, ixt, h, i i1 The forecast combination problem can be formally stated as: Problem: Choose weights w T,h,i to minimize M subject to,, i1 w T h i 1 c E[ L( et h)]

14 T What is the Problem Really About? Observations on a variable Y Observations of forecasts of Y with forecasting horizon 1 Forecasting errors Question: How much weight shall we give to the current forecast given past performance and knowing that there will be a forecasting error??

15 What is the Problem Really About? GDP A C B T T + 1 Time

16 Module 9 Session 1, Part 2: Combining Prediction Errors

17 Examples of Loss Functions Squared error loss: E e c 2 [( T h) ] Absolute error loss: c E[ et h ] Linex loss: c c E[exp( e ) e 1] T h T h We will focus on mean squared error loss c c 2 E[ L( e )] E[( e ) ] T h T h

18 Combining Prediction Errors Notice that because c e y yˆ c T h T h T h M i1 M i1 then: T, h, i T h T, h, i T, h, i T h, i T h T, h, i T, h, i T, h, i i1 i1 M c E LeT h E L wt, h, iet h, i i1 M w ( y x ) Hence, if weights sum to one: M i1 w w T, h, i e 1 M y w w x

19 Combination Problem with MSE Let w be the (M x 1) vector of weights, e the (M x 1) vector of forecast errors, u an (M x 1) vector of 1s, and the (M x M) variance covariance matrix of the errors It follows that M i1 w T, h, i uw ' Σ Eee' M c T h T, h, i T h, i i1 c e w e w'e e 2 T h c 2 Th E e E w'ee'w w' E ee' w w'σw w'ee'w Problem 1: Choose w to minimize w w subject to u w = 1.

20 Issues and Clarification (I) Must the weights sum to one? If forecasts are unbiased, this guarantees an unbiased combination forecast How restrictive is pooling the forecasts rather than information sets? Pooling information sets is theoretically better but practically difficult or impossible

21 Issues and Clarification (II) Are point forecasts the only forecasts that we can combine? We can also combine forecasts of distributions. Are Bayesian methods useful? Not entirely. Is there a difference between averaging across forecasts and across forecasting models? If you know the models and the models are linear in the parameters, there is no difference.

22 Broad Summary of Questions 1. What are the optimal weights in the population? 2. How can we estimate the optimal weights? 3. Are these estimates good?

23 Module 9 Session 2, Part 1: Solving the Theoretical Combination Problem

24 2. Solving the Theoretical Combination Learning Objectives: Problem Create a simple combination of forecasts using 2 forecasts Generalize to M forecasts Explore key takeaways from combining

25 Simple Combination For now, let s assume that we know the distribution of the forecasting errors associated to each forecast. What are the optimal (loss minimizing) weights, in population? T T + 1

26 Simple Combination with m=2 Consider two point forecasts: yˆ wx (1 w) x c 2 2 E[( e ) ] E[( we (1 w) e ) ] c T h T, h,1 T, h,2 T h T h,1 T h,2 = w 2 2 s T+h,1 +(1- w) 2 2 s T+h,2 The solution to the Problem (m=2) is: 2 w 1w * * T h,2 T h,1,2 2 2 T h,1 T h,2 2 T h,1,2 2 T h,1 T h,1,2 2 2 T h,1 T h,2 2 T h,1,2 + 2w(1- w)s T+h,1,2 weight of x T,h,1 weight of x T,h,2

27 Interpreting the Optimal Weights Consider: * w 1w 2 T h,2 T h,1,2 * 2 T h,1 T h,1,2 a larger weight is assigned to the more precise model the weights are the same (w* = 0.5) if and only if 2 T+h,1 = 2 T+h,2 if the covariance between the two forecasts increases, greater weight goes to the more precise forecast

28 General Result with M Forecasts: The choice of weights will reflect var-covar matrix of FE. Result: The vector of optimal weights w with M forecasts is -1 u'σ w ' T,h u'σ u -1 T,h

29 Takeaway 1: Combining Forecasts Decreases Risk Compute the expected loss (MSE) under the optimal weights E e c * 2 [( Th( w )) ] Result: (1 ) T h,1 T h,2 T h,1,2 2 2 T h,1 T h,2 2 T h,1,2 T h,1 T h,2 Suppose that 2 2 c T h,2 (1 T h,1,2) E[( e ( w )) ] Is the correlation coefficient * T h T h,1 T h,1 T h,1 T h,2 T h,1,2 T h,1 T h,2 E[( e ( w )) ] min{, } c * T h T h,1 T h,2 That is, the forecast risk from combining forecasts is lower than the lowest of the forecasting risk from the single forecasts

30 Takeaway 2: Bias vs. Variance Tradeoff The MSE loss function of a forecast has two components: the squared bias of the forecast the (ex-ante) forecast variance E[( y T h x,, ) ] T h i Bias T, h, i y Var( xt, h, i) 2 M M c 2 E ( yt h y ) th E wt, h, ibiast, h, i y wt, h, i x t, h, i E( xt, h, i ) i i M i M T, h, i T, h, i y T, h, i T. h. i i w bias w Var Result: Combining forecasts offers a tradeoff between increased overall bias vs. lower (ex-ante) forecast variance.

31 What is the Problem Really About? GDP A C B T T + 1 Time

32 Module 9 Session 2, Part 2: Implementation Issues

33 Issue: What if it is optimal for one weight to be<0? Consider again the case M = 2. The optimal weights are such that: * 2 w T h,2 T h,1,2 * 2 1w T h,1 T h,1,2 If T,h,1,2 > 0 and 2 T,h,2 < T,h,1,2 < 2 T,h,1 then w* < 0 Shall we impose the constraints that w T,h,i > 0?

34 Another Issue: Estimating Σ In reality, we do not know : we can only estimate the theoretical weights using the observed past forecast errors. e t,h,1 T e t,h,2 T

35 Questions when estimating Σ 1) Are the estimates of based on past errors unbiased? 2) Does the population depend on t? If not, estimates become better as T increases If it does, different issues: heteroskedasticity of any sort, serial correlation, etc. Do our estimates capture this dependence? 3) Does depend on past realizations of y?

36 Questions when estimating Σ 4) How good are our estimates of? If M is large relative to T, our estimates are poor! 5) Shall we just focus on weighted averages? Why not to consider the median forecast, or trim excess forecasts?

37 One More Issue: Optimality of Equal When does it make sense (in terms of minimum squared error) to use equal weights? when the variance of the forecast errors are the same and all the pair wise covariances across forecast errors are the same the loss function is symmetric Weights? Result: Equal weights tend to perform better than many estimates of the optimal weights (Stock and Watson 2004, Smith and Wallis 2009)

38 Module 9 Session 3: Methods to estimate the weights when M is low relative to T

39 Methods of Weighting (M<T) Learning Objectives: Decide when to combine and when not to combine Estimate weights using OLS Address Sampling Error

40 To combine or not to combine? We need to assess whether one set of forecasts encompasses all information contained in another set of forecasts Example, for 2 forecasting models, run the regression: y x x t 1,2,... T - h th T, h,0 T, h,1 t, h,1 T, h,2 t, h,2 th If you cannot reject: H : (,, ) (0,1,0) 0 T, h,0 T, h,1 T, h,2 H :(,, ) (0,0,1) 0 T, h,0 T, h,1 T, h,2 All other outcomes imply that there is some information in both forecasts that can be used to obtain a lower mean squared error

41 OLS estimates of the weights If we assume a linear-in-weights model, OLS can be used to estimate the weights that minimizes the MSE using data for t = 1, T h: y x x... x th T, h,0 T, h,1 t, h,1 T, h,2 t, h,2 T, h, M t, h, M th y x x... x th T, h,1 t, h,1 T, h,2 t, h,2 T, h, M t, h, M th Including a constant allows correcting for the bias in any one of the forecasts N st.. 1 i1, i

42 OLS estimates of the weights If weights sum to one, then previous equation becomes a regression of a vector of 0s over the past forecasting errors: 0 ( x y ) ( x y )... T, h,1 t, h,1 th T, h,2 t, h,2 th th 0 e e... T, h,1 th,1 T, h,2 th,2 th Problem 2: Choose w to minimize w'σw ˆ subject to u w = 1 and Th w i 0, where Σˆ et,he' t,h and e t,h is a vector that collects the t1 forecast errors of the M forecasts made in t.

43 Reducing the dependency on sampling errors Assume that estimates of reflect (in part) sampling error. Although optimal weights depend on, it makes sense to reduce the dependence of the weights on such an estimate One way to achieve this is to shrink the optimal weights towards equal weights (Stock and Watson 2004): s T, h, i T, h, i (1 )(1/ M ), max(0,1 ( M / ( T h M 1)))

44 Module 9 Session 4: Methods to estimate the weights when M is high relative to T

45 Methods of Weighting (M>T) Learning Objectives: Explore shortcomings of OLS weights Look at other parametric weights. Consider some non-parametric weights and techniques

46 Premise: problems with OLS weights The problem with OLS weights is that: If M is large relative to T h, the estimator loses precision and may not even be feasible (if M > T h) Even if M is low relative to T h, OLS estimation of weights is subject to sampling error.

47 Other Types of Weights Relative Performance Shrinking Relative Performance Recent Performance Adaptive Weights Non-parametric (trimming and indexing)

48 Relative performance weights A solution to the problem of OLS weights is to ignore the covariance across forecast errors and compute weights based on their relative performance over the past. Th 1 2 For each forecast compute MSET, h, i et, h, i T h 1 MSE weights (or relative performance weights) T, h, i M 1 MSE T, h, i 1 MSE i1 T, h, i t1

49 Shrinking relative performance Consider instead T, h, i 1 MSE T, h, i i1 T, h, i The parameter k allows attenuating the attention we pay to performance M 1 MSE If k = 1 we obtain standard MSE weights If k = 0 we obtain equal weights 1/M k k

50 Highlighting recent performance Consider computing a discounted 1 MSE e () t Th 2 T, h, i t, h, i #period with 0 t1 where (t) can be either one of the following 1 if t T h v () t 0 if t T h v () t T h t rolling window discounted MSE Computing MSE weights using either rolling windows or discounting allows paying more attention to recent performance

51 Adaptive weights Relative performance weights could be sensitive to adding new forecast errors. A possibility is to adapt previous weights by the most recently computed weights T, h, i MSE weight (with or without covariance) (1- ) * * T, h, i T 1, h, i T, h, i

52 Non parametric weights: Trimming Often advantageous to discard the models with the worst and best performance when combining forecasts Simple averages are easily distorted by extreme forecasts/forecast errors Aiolfi and Favero (2003) recommend ranking the individual models by R^2 and discarding the bottom and top 10 percent

53 Non parametric weights: Indexing Rank the models by their MSE. Let Ri be the rank of the i-th model, then: Index based-weights w index T, h, i M 1 j1 1 R i R j

54 Module 9 Session 5: Workshop on Combining Forecasts

55 Workshop 1: Combining Forecasts Have open MF_combination_forecasting.wf1 we ll be working on the CF_W1_Combined pagefile. Let s estimate 4 regression models:

56 Combining Forecasts Step One: Estimate each of the models using LS and Forecast using EViews. Note the RMSE of each. Step Two: Calculate a combined forecast using the misspecified ones using: Equal Weights Trimmed Weights Inverse MSE weights

57 Combining Forecasts Step Three: Compare the RMSE of the combined forecast models to that of the individual forecast models. Which is most accurate? Do all 3 combined ones outperform the individual ones? Step Four: Repeat Steps 1-3 for the true DGP: Then forecast , what is RMSE of this full model?

58 Module 9 Session 6 : Forecast Combination Tools in EViews

59 Workshop 2: Forecast Combination Techniques Have open MF_M9.wf1 we ll be working on the USA pagefile. Let s explore different forecast combination techniques! Note: This workshop will involve the use of multiple program files. Make sure to have the workfile open at all times.

60 W2: Forecast Combination Techniques Step One: First, remember to always inspect your data: The USA pagefile contains quarterly data for 1959q1-2010q4: rgdp = real GDP index (2005=100) in this case will use natural log ( lngdp ). growth = growth rate over different time spans (1, 2 and 4 quarters) fh_i_j = Growth forecast indexed by time horizon (i) and model number (j) Step Two: Produce combined forecast for GDP growth in 2004q1 we can compare this to actual GDP growth. Use programs to compute forecasts for rest of (tedious).

61 Workshop 2: Forecast Combination Techniques Have open MF_M9.wf1 we ll be working on the USA pagefile. Let s explore different forecast combination techniques! Note: This workshop will involve the use of multiple program files. Make sure to have the workfile open at all times.

62 Module 9 Session 7: Conclusions

63 Conclusions Numerous weighting schemes have been proposed to formulate combined forecasts. Simple combination schemes are difficult to beat; why this is the case is not fully understood. Simple weights reduce variability with relatively little cost in terms of overall bias Also provide diversity if pool of models is indeed diverse.

64 Conclusions Results are valid for symmetric loss function; may not be valid if sign of the error matters Forecasts based solely on the model with the best in-sample performance often yields poor out-of-sample forecasting performance. Reflects the reasonable prior that a preferred model is really just an approximation of true DGP, which can change each period. Combined forecasts imply diversification of risk (provided not all the models suffer from the same misspecification problem).

65 Appendix JVI

66 Appendix 1 Let e be the (M x 1) vector of the forecast errors. Problem 1: choose the vector w to minimize E[w ee w] subject to u w = 1. Notice that E[w ee w] = w E[ee ]w = w w. The Lagrangean is and the FOC is Using u w = 1 one can obtain L w'ee'w - λ[u'w -1] Σw - λu 0 w * = Σ -1 uλ * u'w = u'σ uλ 1= u'σ uλ = u'σ u 1 Substituting back one obtains 1 * -1-1 w =Σ u u'σ u JVI

67 Appendix 1 Let t,h be the variance-covariance matrix of the forecasting errors 2 T h,1 T h,1,2 Th, 2 T h,1,2 T h,2 Consider the inverse of this matrix T h,2 T h,1,2 Th, 2 det Th, T h,1,2 T h,1 Let u = [1, 1]. The two weights w* and (1 - w*) can be written as w 1w * * Σ u'σ u -1 T,h -1 T,h u JVI

68 Appendix 2 Notice that and that T h,1 T h,2 2 T h,1,2 E ( et h,1 et h,2) 0 (1 ) T h,1 T h,1,2 So, the following inequality holds (1 ) 2 2 T h,2 T h,1,2 2 2 T h,1 T h,2 2 T h,1,2 T h,1 T h,2 1 (1 ) T h,2 T h,1,2 T h,1 T h,2 T h,1,2 T h,1 T h, T h,1 T h,1,2 T h,1 T h,2 T h,2 T h,1,2 0 ( ) T h,1 T h,2 T h,1 2 JVI

69 Appendix 3 The MSE loss function of a forecast has two components: the squared bias of the forecast the (ex-ante) forecast variance E[( y x ) ] E[( E( y ) x ) ] 2 2 T h T, h, i T h y T, h, i E E y E x E x x 2 [( ( T h) y ( T, h, i) ( T, h, i) T, h, i) ] E Bias E x x 2 [( i y ( T, h, i) T, h, i) ] Bias Var( x ) 2 2 i y T, h, i JVI

70 Appendix 4 Suppose that x,, Py where P is an (m x T) matrix, y is a (T x 1) T h i vector with all y t, t = 1, T. Consider: ˆ 1 Th 2 T, h, i xt, h, i yt h T h t1 Th 1 x E[ y ] T h t1 t, h, i th y, th T h T h 2 y t h i t h y t h t h i t h T h t1 T h t1 2 2 x,, E[ y ], x,, E[ y ] JVI

71 Appendix 4 Consider: 1 2 ˆ [ ] [ ] T h T h 2 2 T, h, i y t, h, i t h y, t h t, h, i t h T h t1 T h t1 2 x E y x E y 2... ε'(py - E[y]) T h 2... ε'(pe[y]+ Pε - E[y]) T h 2... ε'pε ε'(i - P)E[y] T h 2 MSPET Eε'Pε T h JVI