Is there an optimal forecast combination? A stochastic dominance approach to forecast combination puzzle.

Transcription

1 Is there an optimal forecast combination? A stochastic dominance approach to forecast combination puzzle. Mehmet Pinar University of Guelph and Fondazione Eni Enrico Mattei Thanasis Stengos University of Guelph M. Ege Yazgan Istanbul Bilgi University August 31, 2011 Abstract Even though different optimal forecast combination weights are offered for static, dynamic, or time-varying situations, empirical findings support the simple average forecast combination outperforms more sophisticated weighting schemes and/or the best individual model. Using an approach that relies on consistent tests for stochastic dominance efficiency, an alternative optimal weighting scheme is proposed. These tests are considered for a given forecast combination (i.e. equal weighted average of forecasts) with respect to all possible forecast combinations constructed from a set of individual forecasts to obtain optimal or worst forecast combinations. In our empirical applications we find that equally weighted forecast combinations are neither optimal nor the worst forecast combination. For the optimal forecast combination, the best forecasting model, i.e. the model which assumes relatively more weight than other forecast models, differs with the variable being forecasted and for different forecast horizons. On the other hand, random walk is the model that consistently contributes with more than arbitrarily assigned equal weights for the worst forecast combination for all variables being forecasted and for all forecast horizons. JEL Classifications: C12; C13; C14; C15; G01 Key Words: Nonparametric Stochastic Dominance, Mixed Integer Programming; Forecast combinations; Forecast combination puzzle mehmet.pinar@feem.it tstengos@uoguelph.ca eyazgan@bilgi.edu.tr

2 1 Introduction Since the seminal work of Bates and Granger (1969), combining forecasts of different models, instead of relying on forecasts of individual models have come to be viewed as an effective way of improving the accuracy of predictions regarding a certain target variable. A significant number of theoretical and empirical studies, e.g. Timmerman (2006 ) and Stock and Watson(2004) have been able to show the superiority of combined forecasts over single-model model based predictions. The central question on which the literature focused on, naturally, is to determine the optimal weights used in the calculation of combined forecasts. In combined forecasts the weights attributed to each model depends on the the model s out of sample performance. As time moves on forecast errors used for the calculation of optimal weights changes, and thus the weights themselves vary over time. However, in empirical applications, numerous papers (Clemen (1989), Stock and Watson (1999, 2001, 2004), Hendry and Clemens (2004), Smith and Wallis (2009), Huang and Lee (2010), Aiolfi et al. (2010)) have found that the equal weighted forecast combination often outperforms estimated optimal forecast combinations. This finding is frequently reffered as forecast combination puzzle by Stock and Watson (2004) 1. Overall, even though different optimal forecast combination weights are offered for static, dynamic, or time-varying situations, empirical findings support the simple average forecast combination outperforms more sophisticated weighting schemes and/or the best individual model. In this paper, we will follow an approach for the combination of forecasts based on stochastic dominance (SD hereafter) analysis and we test whether simple average combination of forecasts outperforms the more complicated combination weights or not. Constructing an optimal (worst) forecast combination based on SD analysis has advantages since it provides an efficient index which minimizes (maximizes) the forecast errors and offers the least variation of forecast errors and relatively large data sets are available, so that nonparametric analysis can let the data speak for themselves. We examine whether equal weighted forecast combination is optimal or not. Instead of giving equal weights on each forecast, which is arbitrary weights on each forecast, we use stochastic dominance efficiency analysis to propose a weighting scheme that dominates the equally weighted forecast combination. In other words, we will construct an optimal forecast combination with those weights given to each forecast that will make it stochastically dominate 1 Smith and Wallis (2009) found that the finite sample error is the reason behind the forecast combination puzzle. On the other hand, Aiofli et al (2010) suggested that parameter or model instability is another reason why simple average forecast combination outperforms the best individual forecast model (see Diebold and Pauly (1987), Clements and Hendry (1998, 1999, 2006), Timmermann (2006) for further discussion of model instability and Elliot and Timmermann (2005) forecast combinations for time varying data 1

3 all other competitor forecast combinations 2. The main contribution of the paper is the derivation of an optimal (worst) forecast combination based on SD analysis with differential forecast weights. For the optimal (worst) forecast combination, this index will minimize (maximize) the forecasts errors by combining time-series model based forecasts for a given probability level and also be the least volatile over time among its set of competitors. By weighting each forecast differently, we will find the optimal (worst) forecast combination that do not rely on arbitrary weights. In our empirical applications we find that equally weighted forecast combinations are neither optimal nor the worst forecast combination. For the optimal forecast combination, the best forecasting model (i.e. the model which gets relatively more weight than other forecast models) differs with the variable being foretasted and for different forecast horizons. On the other hand, random walk (hereafter RW) is the model that consistently contributes with more than arbitrarily assigned equal weights for the worst forecast combination for all variables being foretasted and for all forecast horizons. The remainder of the paper is as follows. In section 2 we examine the main framework of analysis, define the notions of stochastic dominance and discuss the general hypothesis for stochastic dominance of any order. Section 3 offers the data, time-series forecasting models and forecast methods used in our paper and develops the empirical analysis. We use Scaillet and Topaloglou (2010) s, hereafter ST, methodology to find the optimal (worst) forecast combination for macroe- 2 Mostly, stochastic dominance comparisons are made pairwise in the literature. Barrett and Donald (2003) developed pairwise stochastic dominance comparisons that relied on Kolmogorov-Smirnov type tests developed within a consistent testing environment. This offers a generalization to Anderson (1996), Beach and Davidson (1983), Davidson and Duclos (2000) who have looked at second order stochastic dominance using tests that rely on pair-wise comparisons made at a fixed number of arbitrary chosen points. This is not a desirable feature since it introduces the possibility of a test inconsistency. Linton et al. (2005) propose a subsampling method which can deal with both dependent samples and dependent observations within samples. This is appropriate for conducting SD analysis for model sellection among many forecasts. In this context, comparisons were available for pairs where one can compare one forecast with respect to another forecast and conclude whether one forecast dominates the other one. In other words, one can find the best individual model by comparing all forecasts. Lately, multi-variate (multidimensional) comparisons, in our case forecast combinations, have become more popular. In an important application to optimal portfolio construction in finance, Scaillet and Topaloglou (2010), hereafter ST, use SD efficiency tests that can compare a given portfolio with an optimal diversified portfolio constructed from a set of assets. In a related paper, Pinar, Stengos and Topaloglou (2010) use a similar approach to construct an optimal Human Development Index. The same methodology is applied in Agliardi et al (2011), where an optimal country risk index is constructed following SD analysis with differential component weights, yielding an optimal hybrid index for economic, political, and financial risk indices that do not rely on arbitrary weights as rating institutions do. 2

4 conomic variables for different forecast horizons. Finally, section 4 concludes. 2 Hypothesis, Test Statistics and Asymptotic Properties Let ŷ i,t+h be the forecast of the ith forecasting model for y t generated at time t for the period of t+h (h 1). We have n different time-series forecasting models and we denote the absolute forecast error of the ith forecasting model as ˆε i,t+h = y t+h ŷ i,t+h. Equally weighted forecast combination can be obtained as the simple average of individual forecast, i.e. ŷt+h ew = 1 n n ŷi,t+h. The absolute forecast error of the equally weighted forecast combination is given by ˆε ew t+h = y t+h ŷ ew. Now consider an alternative weighing scheme as ŷt+h w = n λ iŷ i,t+h, where n λ i = 1. Therefore ˆε w t+h = yt+h ŷt+h w. The forecast combination literature asserts that there exists a combination of forecasts that delivers smaller ˆε w t+h than all of the individual constituent models ˆε i,t+h s, hence there exists a solution to the following optimization problem apart from any of λ i -s taking on value 1 with remaining λ i s equal to 0s, i.e. one of the i-s is the best forecaster. Arg Minˆε w t+h = (λ i) y t+h t+h n n λ i ŷ i,t+h s.t. λ i = 1 (1) On the other hand, forecast combination puzzle refers to a situation in which ˆε ew t+h constitutes the minimum (absolute) forecast error. Let us define Mean Absolute Forecast Errors (MAFEs) of different models as ˆε ew t+h = 1 n n ˆε i,t+h which is obviously identical to ˆε ew t+h defined above. Similarly defining Weighted Absolute Forecast Errros (WAFEs) as ˆε w t+h = n λ iˆε i,t+h which is identical to ˆε w t+h defined above. Hence the optimization problem Equation (1) can also be written as a minimization problem of the sum of WAFEs 3. Arg Minˆε w t+h = (λ i) n n λ i (y t+h ŷ i,t+h ), s.t. λ i = 1 The optimization problem above obtains the weights that minimizes the overall absolute forecast error which only gives emphasis to the first moment. However, stochastic dominance efficiency analysis allows for all moments where it analyzes the whole possible distribution function of absolute forecast errors. In this current paper, we test the efficiency of MAFEs by testing whether 3 The identity between (1) and (2) cannot be carried over when the loss function is replaced by Squared Forecast Error (RSE). Arg Minˆε w t+h (y = t+h (λ i ) ) n 2 λ i ŷ i,t+h 3

5 the cumulative distribution function of mean absolute forecast errors, ˆε ew t+h = 1 n n ˆε i,t+h, is stochastically efficient or not. We denote by F (ˆε t+h ), the continuous cdf of ε t+h = (ˆε 1,t+h,..., ˆε i,t+h,..., ˆε n,t+h ). Let us consider a forecast combination λ L where L := {λ R n + : e λ = 1} with e for a vector made of ones. This means that all the different absolute forecast errors from different models have positive weight and that the forecast combination weights sum to one. Let us denote by G(z, λ; F ) the cdf of the forecast combination λ ε t+h at point z given by G(z, λ; F ) := I{λ ε t+h z}df (ε t+h ). R n Let denote equally weighted forecast combination τ which is a special case of λ, being vector of 1 ns. Therefore G(z, τ ; F ) is the cdf of the equally equally weighted forecast combination, ˆεew t+h = 1 n n ˆε i,t+h, or τ ε t+h The general hypotheses for testing the stochastic dominance efficiency of order j of τ, hereafter SDE j, can be written compactly as: H j 0 :J j(z, τ ; F ) J j (z, λ; F ) for all z R and for all λ L, H j 1 :J j(z, τ ; F ) > J j (z, λ; F ) for some z R or for some λ L. where 4. 1 J j (z, λ; F ) = R (j 1)! (z λ ε t+h ) j 1 I{λ ε t+h z}df (ε t+h ) n Under the null hypothesis H j 0 there is no forecast combination λ constructed from the set of absolute forecast errors from the model based forecasts that dominates the equal weighted index τ (MAFEs) at order j. In this case, the function J j (z, τ ; F ) is always lower that the function J j (z, λ; F ) for all possible forecast combinations λ for any absolute forecast error level z. In this 4 Defining z R: and so on. J 1 (z, λ; F ) := G(z, λ; F ), z z J 2 (z, λ; F ) := G(u, λ; F )du = J 1 (u, λ; F )du, z u z J 3 (z, λ; F ) := G(v, λ; F )dvdu = J 2 (u, λ; F )du, From Davidson and Duclos (2000) Equation (2), we know that z 1 J j (z, λ; F ) = (j 1)! (z u)j 1 dg(u, λ, F ), which can be rewritten as above 4

6 case, equal weighted forecast combination offers the worst scenario 5. With equal weighed forecast combination, there is always higher absolute forecast error above all absolute forecast error level z with respect to all possible forecast combination weights. Under the alternative hypothesis H j 1, we can construct a forecast combination λ that for some absolute forecast error level z, the function J j (z, τ ; F ) is greater than the function J j (z, λ; F ). Thus, when j = 1, the equal weighted absolute forecast errors (MAFEs) τ is stochastically inefficient (i.e. not the worst scenario) at first order if and only if some other forecast combination λ dominates it at some absolute forecast error level z. Alternatively, the equally weighted forecast combination τ is stochastically efficient (i.e. the worst forecast combination) at first order if and only if there is no other forecast combination λ that dominates it at all absolute error levels. In particular we obtain SD at first and second order when j = 1 and j = 2, respectively. The hypothesis for testing SD of order j of the distribution of equal weighted forecast combination τ over the distribution of an alternative forecast combination λ takes analogous forms, but for a given λ instead of several of them. 2.1 Tests of the optimality of equal weighted forecast combination The distribution of the alternative forecast combination λ dominates the distribution of the equal weighted forecast combination (i.e. MAFEs) τ stochastically at first order (SD1) if, for any absolute forecast error level z, G(z, τ ; F ) G(z, λ; F ). If z denotes an absolute forecast error level, then the inequality in the definition means that the number of absolute forecast errors in distribution λ with value of absolute forecast error smaller than z is not larger than the number of absolute forecast errors in τ. In other words, there is at least as high a proportion of absolute error level in λ as in τ. If the forecast combination λ dominates the equal weighted forecast combination (i.e. MAFEs) τ at first order, then there is always less value of absolute forecast error in τ than in λ, so that λ suggest the worst forecast combination implying that equal weighted 5 Throughout the methodology, we examine the worst forecast combination. If an alternative weighting scheme dominates the equal weighted forecast combination, then there is alternative forecast combination that maximizes the absolute forecast errors. If there is no other alternative weighting that dominates the equal weighted forecast combination, then equal weighted forecast combination is the worst forecast combination. Since, in this case, equal weighted forecast combination will offer the highest forecast errors with respect to all possible forecast combination. One can keep the same methodology but multiply the absolute forecast errors with minus one (i.e. reversing the cumulative distribution functions) and therefore will convert the problem to find weights that minimizes the absolute forecast errors which offers the optimal forecast combination. 5

7 forecast combination is not the worst forecast combination 6. We maximize the following function: Max[G(z, τ ; F ) G(z, λ; F )] This maximization results in the worst forecast combination λ constructed from the set of forecast models in the sense that it reaches the maximum value of absolute forecast errors for a given probability, implying there are more absolute forecast errors above the given argument z than not only equal weighted but also any possible forecast combination. We test whether the equal weighted forecast combination is optimally constructed using arbitrary weights (i.e. equal weights to each forecast), or whether we can obtain an alternative forecast combination optimal (worst) weights for the different forecasts which offers the least (most) absolute forecast errors. Thus, we test whether the equal weighted forecast combination is optimal, or whether we can construct a forecast combination λ from the set of the forecasts that dominates the equal weighted forecast combination. The general hypotheses for testing the optimality of equal weighted forecast combination τ, can be written compactly as: H 0 :G(z, τ ; F ) G(z, λ; F ) for all z R and for all λ L, H 1 :G(z, τ ; F ) > G(z, λ; F ) for some z R or for some λ L. The empirical counterpart is simply obtained by integrating with respect to the empirical distribution ˆF of F, which yields: J j (z, λ; ˆF ) = 1 T h 1 T h (j 1)! (z λ ε t+h ) j 1 I{λ ε t+h z}, t=1 and can be rewritten more compactly for j 2 as: 2.2 Test Statistics J j (z, λ; ˆF ) = 1 T h 1 T h (j 1)! (z λ ε t+h ) j 1 +. t=1 We consider the weighted Kolmogorov-Smirnov type test statistic and a test based on the decision rule: Ŝ j := T h 1 [ T h sup J j (z, τ ; ˆF ) J j (z, λ; ˆF ] ), z,λ reject H j 0 if Ŝ j > c j, 6 As proposed earlier at footnote 1, one also has to test whether equal weighted forecast combination is optimal or not even though it is not the worst forecast combination. 6

8 where c j is some critical value. (The derivation of the test is given by ST (2010)). In order to make the result operational, we need to find an appropriate critical value c j. Since the distribution of the test statistic depends on the underlying distribution, this is not an easy task, and we decide hereafter to rely on a block bootstrap method to simulate p-values (See Appendix) 7. The test statistic Ŝ1 for first order stochastic dominance efficiency is derived using mixed integer programming formulations (See Appendix). 3 Empirical Analysis 3.1 Data, Forecasting Models, and Forecast Methodology To show practial use of stochastic dominance efficiency tests to have the optimal (worst) forecast combinations, we apply our methodology to U.S. inflation and U.S. Dollar/Japanese Yen exchange rate returns. We use log first differences of the CPI and exchange rate levels. While CPI series are measured at the monthly frequency and the sample period is 1959:1-2010:12, obtained from BLS, exchange rate series are captured at weekly frequency for the period of 1959:1-2010:52. The daily noon buying rates in New York City certified by the Federal Re- serve Bank of New York for customs and cable transfers purposes are obtained from the Chicago Federal Reserve Board ( The weekly series is generated by selecting Wednesdays series (if a Wednesday is a holiday then the following Thursday is used). The use of weekly data avoids the so-called weekend effect, as well as other biases associated with nontrading, bid-ask spread, asynchronous rates and so on, which are often present in higher frequency data. To initialize our parameter estimates we use data from 1959:1-2005:12 (CPI), 1959:1-2007:52 (exchange rate). We then generate pseudo out-of-sample forecasts from 2005:01 to 2010:12 (CPI), 2007: :52. Parameter estimates are updated recursively by expanding the estimation window by one observation each period. In our out-of-sample forecasting exercise we concentrate only on univariate models and consider three types of linear and 4 types of nonlinear univariate models. While linear models consist of random walk (RW), autoregressive (AR) and autoregressive moving average (ARMA) models, nonlinear models comprise logistic smooth transition autoregressive (LSTAR), self-exciting 7 The asymptotic distribution of ˆF is given by T h( ˆF F ) which tends weakly to a mean zero Gaussian process B F in the space of continuous functions on R n (see e.g. the multivariate functional central limit theorem for stationary strongly mixing sequences stated in Rio (2000)). ST (2010) derive the limiting behavior by using the Continuous Mapping Theorem (as in Lemma 1 of Barrett and Donald (2003)), see ST (2010) Lemma

9 threshold autoregressive (SETAR), markov switching autoregressive (MS-AR)and autoregressive neural networks (NNETTS). In RW model ŷ t+h is equal to the the value of y t at time t. In AR model, on the other hand, the forecast is given by p ŷ t+h = α + φ i ŷ t+h i (1) Where p is selected to minimize the AIC with a maximum of 24 lags. When h > 1, to obtain forecasts we iterate on a one-period forecasting model, by feeding the previous period forecasts as regressors into the model. An obvious alternative to iterating forward on a single-period model would be to tailor the forecasting model directly to the forecast horizon, i.e., ŷ t+h i s of the right hand side of Equation (3) are replaced by y 8 t i. Similarly to AR model, forecasts from ARMA model can be obtained through the following regression p q ŷ t+h = α + φ 1,i ŷ t+h i + φ 2,i ε t+h i (2) Where p and q, like in the all subsequent models are selected to minimize the AIC with a maximum of 24 lags. The LSTAR model is ( ) ) p q y t = α 1 + φ 1,i y t i + d t (α 2 + φ 2,i y t i + ε t (3) with d t = (1 + exp { γ(y t 1 c)}) 1 and ε t is regarded as i.i.d. A linear regression is used to estimate the αs and φs, and then γ and c are estimated by minimizing the sum of squared residuals, repeating until convergence. In LSTAR model, it is not possible to apply any iterative scheme to obtain multistep ahead forecasts as in the linear models. This impossibility follows from the general fact that conditional expectation of a nonlinear function is not necessarily equal to function of that conditional expectation, and one cannot iteratively derive the forecasts for h > 1 by plugging in previous 8 Which approach is best -the direct or the iterated- is an empirical matter since it involves trading off estimation efficiency against robustness to model misspecification, see Timmerman (2007). Marcellino, Stock and Watson (2006) address these points empirically using a data set of 170 US monthly macroeconomic time series. They find that the iterated approach generates the lowest MSE-values, particularly if long lags of the variables are included in the forecasting models and if the forecast horizon is long. 8

10 forecasts (see, for example, Clements et al, 1999) 9. Therefore we use Monte Carlo integration scheme suggested by Granger and Lin (1994)numerically calculate conditional expectations. When γ LSTAR model approaches two-regime SETAR model, which we use in our application. There are basically more than one approaches for forecasting from a SETAR model (see, for example, Clements et al, 1999). We employ a version of the normal forecasting error method suggested by Al-Qassam and Lane (1989). The two-regime MS-AR model that we consider here is p y t = α s + φ s,i y t i + ε t (4) where s t is a two-state discrete Markov chain with S = {1, 2} and ε t i.i.d. N(0, σ 2 ). We estimate MS-AR by using the maximum likelihood algorithm expectation-maximization. Although MS-AR models may encompass complex dynamics, point forecasting is less complicated in comparison to other non-linear models. The h-step forecasts from the MS-AR model is ( ) ( ) p p ŷ t+h = P (s t+h = 1) α s=1 + φ s=1,i ŷ t+h i + P (s t+h = 2) α s=2 + φ s=2,i ŷ t+h i where P is the smoothed transition probabilities (see, Hamilton 1996, for example). Hence, multistep forecasts can be obtained iteratively by plugging in 1, 2, 3,... period forecasts similar to the iterative forecasting method of AR processes. NNETTS model, the autoregressive single hidden layer feed-forward neural network model suggested in Terasvrita et al (2005), is defined as ( p h p ) y t = α + φ i y t i + λ j d γ i y t i c + ε t (5) j=1 where d is the logistic function defined above such that d = (1 + exp { x}) 1. (See van Dijk (2000) for reviews of feed-forward type neural network models). Estimation of an ARNN model may, in general, be computationally challenging. Here we follow QuickNet method, a kind of relaxed greedy algorithm, suggested by White (2006). Forecasting with NNETTS is similar to SETAR and LSTAR: 1-step ahead forecasts can be directly computed but there is no closed form formula for multistep ones. Furthermore, for applying simulation schemes to compute the conditional expectation raised by the multistep forecasting 9 Indeed, d t is convex in y t 1 whenever y t 1 < c and d t is convex whenever y t 1 > c. Therefore, by Jensen s inequality, naive estimation under-estimates d t if y t 1 < c and over-estimates if y t 1 > c. 9

11 equations, the probability density function of ε t. We again device a Monte Carlo simulation to numerically approximate the integral expressions of conditional expectations as we did in SETAR and LSTAR models. To obtain pseudo out-of-sample forecasts for a given horizon,h, the models are estimated by running regressions with data up through the date t 0 < T, where t 0 refers to the date where the estimation is initialized (2005:12 for CPI) and T to the final date in our data. The first h-horizon forecast is obtained by using the coefficient estimates from this first regression. Next, the time subscript is advanced, and the procedure is repeated for t 0 + 1, t 0 + 2,...,T h to obtain N f = T t 0 h 1 distinct h-step forecasts 10. For each of h-step forecasts, we calculate N f absolute forecast errors for each our models that we use in our applications. 3.2 Results for the efficiency of MAFEs This section presents our findings of the test for SD1 efficiency of the equal weighted forecast combination. We find that equal weighted forecast combination is neither optimal nor the worst forecast combination. Simple average of forecast combination is a sub-optimal forecast combination. We offer the best and worst forecast combinations of the model based forecasts for U.S. CPI forecasts and U.S. dollar/japanese yen exchange rate forecasts. We compute the weighting scheme on each forecast model which offers the optimal (worst) forecast combination that minimizes (maximizes) the absolute forecast errors Inflation application First, we start our empirical analysis with monthly seasonally-adjuested U.S. CPI forecasts for different forecast horizons. We proceed with testing whether the equal weighted forecast combination of the forecasting models for different horizons is the worst forecast combination or there are alternative weights on forecast models that stochastically dominate the equal weighted forecast combination τ, in the first order sense (e.g. for which G(z, τ; F ) > G(z, λ; F )) which maximizes the absolute forecast errors. Three panels of Table 1 offer the results for the pessimistic scenario (i.e. forecast combination of models that maximizes the absolute forecast error) for different forecast horizons. In panel A, B and C of Table 1, we present the results for the monthly (i.e. 1 to 12 steps ahead), quarterly (i.e. 15 to 36 steps ahead) and semi-annually (i.e. 42 to 60 step ahead) forecast horizons respectively. In panel A of Table 1, for one step ahead forecast horizon (i.e. h=1), each 10 N f differs between 207 and 57 for different h values between 1 and

12 time-series model have 180 one step ahead forecasts which offers absolute forecast errors for each model and average of absolute forecast errors offer MAFEs. There are 179 alternative forecast combinations that dominate the equal weighted forecast combination (i.e. MAFEs) in the first order sense and the first row in panel A of Table 1 summarizes the results. This table presents the average weights of the 179 alternative forecast combinations that dominate the equal weighted forecast combination. We find that the RW model has the highest contribution with 59.9% weight. On the other hand, NNETTS, ARMA, AR, MS and LSTAR contribute with weights of 16.4%, 14.6%, 8%, 0.9% and 0.2% respectively. We replicate the same exercise for monthly (step 1 to 12), quarterly (step 15 to 36) and semi-annually (step 42 to 60) forecast horizons in panel A, B and C respectively. Models which get more than the arbitrarily assigned equal weights are highlighted in all tables 11. Overall, we find that RW model consistently contributes to the worst forecast combination with weights more than and also gets the highest weight relative to other models for the majority of the forecast horizons. Moreover, ARMA, MS and NNETTS models contribute to the worst forecast combination with weights that are more than for some forecast horizons and for some other forecast horizons these models contribute less than a weight of Finally, AR, LSTAR and SETAR models always contribute with weights that are less than Overall, for all forecast horizons, we found that equal weighted forecast combination is not the worst forecast combination. There are many other forecast combinations that dominates the equal weight forecast combination for all different forecast horizons that maximizes the absolute forecast error. Now using the same monthly seasonally adjusted U.S. CPI data, we proceed with testing whether the equal weighted forecast combination of the forecasting models for different horizons is the optimal forecast combination or there exist alternative weights on forecasting models that stochastically dominate the equal weighted forecast combination τ, in the first order sense (e.g. for which G(z, τ; F ) > G(z, λ; F )) which minimizes the absolute forecast errors 12. Three panels of Table 2 offer the results for the optimistic scenario (i.e. forecast combination of models that minimizes the absolute forecast error) for different forecast horizons. In panel A, B and C of Table 2, we present the results for the monthly (i.e. 1 to 12 steps ahead), quarterly (i.e. 15 to 36 steps ahead) and semi-annually (i.e. 42 to 60 step ahead) forecast horizons respectively. For example, for 1 step ahead forecast, there are 180 given MAFEs and there exist 163 alternative forecast combination that dominate the equal weighted forecast combination in the first order 11 In our application, there are 7 time-series forecasting models and in order to obtain the mean absolute forecast errors, each model gets equal weights, a weight of approximately (i.e. 1/7). 12 As proposed earlier at footnote 1, we multiply each absolute forecast error with minus one to test whether equal weighted forecast combination is optimal or not. 11

13 sense. Table 2 summarizes the results. This table presents the average weights of the alternative forecast combinations that dominate the equal weighted forecast combination. For 1 step ahead forecast, SETAR is the model with the highest contribution with a weight of 65.7%. On the other hand, MS, AR, NNETTS, ARMA, LSTAR and RW models take weights of 13.5%, 11.%, 4%, 3%, 1.1% and 0.9% respectively. Overall, different models, expect ARMA and RW, contribute with more than equal weights (i.e. a weight of 0.143) for the construction of the optimal forecast combination at different forecast horizons. However, ARMA and RW models always the least contributors for the optimal forecast combination with weights that are less than equal weight (i.e. a weight of 0.143) for all forecast horizons. Overall, for the application to monthly seasonally-adjusted U.S. CPI forecasts, we find that equally weighted forecast combination is neither the optimal nor the worst forecast combination. We find that different models contribute with differential weights for the optimal forecast combination at different forecast horizons. Some of the models have robust contributions to the optimal (worst) forecast combination. RW model always contribute with more (less) than equal weights to the worst (optimal) forecast combination. Moreover, ARMA model contributes with more than equal weights to the worst forecast combination at some forecast horizons but never contributes more than equal weights to the optimal forecast combination at all forecast horizons. On the other hand, LSTAR and SETAR models contributes more than equal weight to the optimal forecast combination for some forecast horizons but never contributes more than equal weight to the worst forecast combination Foreign exchange rate application In this subsection, we find the optimal (worst) forecast combination for the foreign exchange rate of U.S dollar/japanese yen forecasts for different time horizons. There are many alternative forecast combinations that stochastically dominate the equal weighted forecast combinations. We proceed with testing whether the equal weighted forecast combination of the forecasting models for different horizons is the worst (optimal) forecast combination or there are alternative weights on forecast models that stochastically dominate the equal weighted forecast combination τ, in the first order sense (e.g. for which G(z, τ; F ) > G(z, λ; F )) which maximizes (minimizes) the absolute forecast errors. Three panels of Table 3 and 4 offer the results for the worst and optimal forecast combinations for different forecast horizons respectively. In panel A, B and C of Table 3 and 4, we present the results for the weekly (i.e. 1 to 24 steps ahead), monthly (i.e. 28 to 52 steps ahead) and quarterly (i.e. 65 to 104 step ahead) forecast horizons respectively. Table 3 and 4 presents the 12

14 average weights of the alternative forecast combinations that dominate the equal weighted forecast combination for the pessimistic (i.e. forecast combination that maximizes the absolute forecast errors) and optimistic (i.e. forecast combination that minimizes the absolute forecast errors) cases respectively. For the pessimistic case, for all forecast horizons, RW model contributes the most with at least a weight of 63.5%. NNETTS and SETAR models contributes to the worst forecast contribution more than equal weights for some forecast horizons but these models contributes less than equal weights to the worst forecast combination for the majority of the forecast horizons. On the other hand, for the worst forecast combination case, AR, ARMA, LSTAR and MS models always contribute with a weight that is less than 10%. For the optimal forecast combination (i.e. findings at Table 4), NNETTS and SETAR models always contribute more than equal weights (i.e. a weight of 0.143) for all horizons. Moerover, ARMA contributes more than equal weights to the optimal forecast combination for some forecast horizons and LSTAR also contributes more than equal weights to the the optimal forecast combination for only one forecast horizon (i.e. 36 step ahead). Finally, AR, MS and RW models contribute less than equal weights to the optimal forecast combination for all forecast horizons. Overall, for the application to weekly U.S. dollar/japanese yen exchange rate forecasts, we find that equally weighted forecast combination is neither the optimal nor the worst forecast combination. We find that different NNETTS, SETAR models contribute with weights more than equal weights for the optimal forecast combination at all forecast horizons and ARMA model contributes with more than equal weight to the optimal forecast combination for some forecast horizons. On the other hand, RW model always contribute with the highest weights to the worst forecast combination. Moreover, AR, MS and LSTAR (with one expectation) models do not contribute more than equal weights both to the optimal and worst forecast combination. 4 Conclusion This paper presents stochastic dominance efficiency tests at any order for time dependent data. We study tests for stochastic dominance efficiency of an equal weighted forecast combinations with respect to all possible forecast combinations constructed from a set of time-series model forecasts. We proceed to test whether stochastic dominance efficiency confirms or contradicts with the use of the equal weighted forecast combination. The results from the empirical analysis indicate that equal weighted forecast combination is neither the optimal nor the worst forecast combination for different forecast horizons. We can construct many alternative forecast combinations that domi- 13

15 nate the equal weighted forecast combinations which offer the least (most) absolute forecast errors. Moreover, we construct the optimal (worst) forecast combination for different forecast horizons for monthly seasonally-adjusted U.S. CPI and U.S. dollar/japanese yen foreign exchange rate forecasts. We found that the RW model is the main contributor to the worst forecast combinations for both applications. On the other hand, LSTAR and SETAR models contributes more than equal weights to the optimal forecast combination of the monthly seasonally-adjusted U.S CPI forecasts and NNETTS and SETAR models contributes more than equal weights to the optimal forecast combination of the weekly U.S. dollar/japanese yen exchange rate forecasts. Overall, there is an agreement for both applications that SETAR model contributes more than equal weight for the optimal forecast combinations. The implications of these results are important and also this paper offers a possible future applications. We find that the equally weighted forecast combination is neither optimal nor the worst forecast combination. Some time-series models getting more weights can lead to an improvement on forecasting. We should mention that we only applied stochastic dominance efficiency analysis with seven time-series models to monthly seasonally-adjusted U.S. CPI and U.S. dollar/japanese yen exchange rate data. Therefore, one should further can test whether equal weighted forecast combination is optimal or not by using other variables of interest and also by using different time-series model forecasts. In this paper, we only test equal weighted forecast combinations with time-series model forecasts and another possible future application is to test whether the average of survey based forecast combinations are optimal or not. One may expect that there may be some cases where equal weighted forecast combination outperforms the all possible forecast combinations. Therefore, we note that the stochastic dominance efficiency of the optimal forecast combination should be tested periodically. 14

16 References [1] Abadie, A., 2002, Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models, Journal of the American Statistical Association, 97, [2] Agliardi, E., Agliardi, R., Pinar, M. Stengos, T. and N. Topalaglou, 2010, Economic, political and financial risk in emerging markets: a stochastic dominance approach for a country risk index, mimeo [3] Aiolfi, M., Capistran C., and A. Timmermann, 2010, Forecast Combinations, In Forecast Handbook, edited by M. Clements and D. Hendry. Oxford, Oxford University Press. [4] Anderson, G., 1996, Nonparametric tests of stochastic dominance in income distributions, Econometrica, 64, [5] Barrett, G., and S. Donald, 2003, Consistent tests for stochastic dominance, Econometrica, 71, [6] Bates, J.M. and C.W.J. Granger, 1969, The Combination of Forecasts, Operations Research Quarterly, 20, [7] Beach, C., and R. Davidson, 1983, Distribution-free statistical inference with Lorenz curves and income shares, Review of Economic Studies, 50, [8] Christoffersen, P. and F.X. Diebold, 1997, Optimal prediction under asymmetrical loss, Econometric Theory, 13, [9] Clemen, R.T., 1989, Combining Forecasts: A Review and Annotated Bibliography. International Journal of Forecasting, 5, [10] Clements, M.P. and D.F. Hendry, 1998, Forecasting Economic Time Series, Cambridge University Press. [11] Clements, M.P. and D.F. Hendry, 1999, Forecasting Non-stationary Economic Time Series, The MIT Press. [12] Clements, M.P. and D.F. Hendry, 2006, Forecasting with Breaks, In Elliott, G., C.W.J. Granger and A. Timmermann (eds.), Handbook of Economic Forecasting, 1, North- Holland. 15

17 [13] Davidson, R., and J.-Y. Duclos, 2000, Statistical inference for stochastic dominance and for the measurement of poverty and inequality, Econometrica, 68, [14] Diebold, F.X. and P. Pauly, 1987, Structural Change and the Combination of Forecasts, Journal of Forecasting,6, [15] Elliott, G., and A. Timmermann, 2005, Optimal Forecast Combination under Regime Switching, International Economic Review, 46(4), [16] Granger, C.W.J. and M.H. Pesaran, 2000, Economic and statistical measures of forecast accuracy, Journal of Forecasting, 19, [17] Granger, C.W.J. and R. Ramanathan, 1984, Improved Methods of Combining Forecasts, Journal of Forecasting, 3, [18] Hendry, D.F. and M.P. Clements, 2004, Pooling of Forecasts, Econometrics Journal, 7, [19] Huang, H. and T.-H. Lee, 2010, To combine forecasts or to combine information?, Econometric Reviews, 29(5), [20] Linton, O., Maasoumi, E. and Whang, Y.-J., 2005, Consistent testing for stochastic dominance under general sampling schemes, Review of Economic Studies 72, [21] Patton, A. and A. Timmermann, 2007, Properties of optimal forecasts under asymmetric loss and nonlinearity, Journal of Econometrics, 140 (2), [22] Pinar, M., Stengos, T. and N. Topaloglou, 2010, On the Construction of the Human Development Index: A Stochastic Efficiency Approach, University of Guelph, manuscript. [23] Rio, E., 2000, Theorie Asymptotique des Processus Aleatoires Faiblement Dependants, Mathematiques et Applications, 31, Springer-Verlag, Berlin [24] Scaillet, O., and N. Topaloglou, 2010, Testing for stochastic dominance efficiency, Journal of Business and Economic Statistics, 28 (1), [25] Smith, J., K.F. Wallis, 2009, A simple explanation of the forecast combination puzzle, Oxford Bulletin of Economics and Statistics, 71, [26] Stock, J. H., M.W. Watson, 1999, Forecasting inflation, Journal of Monetary Economics, 44,

18 [27] Stock, J.H. and M.W. Watson, 2001, A Comparison of Linear and Nonlinear Univariate Models for Forecasting Macroeconomic Time Series, in R.F. Engle and H. White, eds., Festschrift in Honour of Clive Granger. Cambridge University Press, [28] Stock, J.H. and M.W. Watson, 2004, Combination Forecasts of Output Growth in a Seven Country Data Set, Journal of Forecasting, 23, [29] Timmermann A., 2006, Forecast combinations. In Handbook of Economic Forecasting, Elliott G, Granger CWJ, Timmermann A (eds). North-Holland: Amsterdam;

19 Tables Table 1 The worst forecast combination for inflation Panel A: Pessimistic case for inflation application (Monthly forecast horizons) Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes h= h= h= h= h= h= h= h= h= h= h= h= Panel B: Pessimistic case for inflation application (3 months, quarterly forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h= h= h= h= h= Panel C: Pessimistic case for inflation application (6 months, semi-annually forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons (N_f) Number of dominating weighting schemes h= h= h= h=

20 Table 2 The optimal forecast combination for inflation Panel A: Optimistic case for inflation application (Monthly forecast horizons) Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes h= h= h= h= h= h= h= h= h= h= h= h= Panel B: Optimistic case for inflation application (3 months, quarterly forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h= h= h= h= h= Panel C: Optimistic case for inflation application (6 months, semi-annually forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h=

21 Table 3 The worst forecast combination for exchange rate returns Panel A: Pessimistic case for exchange rate application (Weekly forecast horizons) Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= Panel B: Pessimistic case for exchange rate application (4 weeks, monthly forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h= h= h= h=

22 Table 3 (continued) Panel C: Pessimistic case for exchange rate application (13 weeks, quarterly forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h=

23 Table 4 The optimal forecast combination for exchange rate returns Panel A: Optimistic case for exchange rate application (Weekly forecast horizons) Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= h= Panel B: Pessimistic case for exchange rate application (4 weeks, monthly forecast horizons) AR ARMA LSTAR MS NNETS RW SETAR Average of dominating weighting schemes Step ahead forecasts (h) Number of observati ons ( ) Number of dominating weighting schemes h= h= h= h= h= h= h=