Pitfalls of Post-Model-Selection Testing: Experimental quantification

Transcription

1 Pitfalls of Post-Model-Selection Testing: Experimental quantification Matei Demetrescu a, Uwe Hassler a, and Vladimir Kuzin b a Goethe University Frankfurt, b DIW Berlin 31st July 2008 Abstract A careful look at data does not always improve inference. We demonstrate by means of computer experiments when and how severely datadriven model selection can destroy the size properties of subsequent tests. The investigated models are representative of typical macroeconometric and microeconometric workhorses. The model selection procedures include information criteria as well as sequences of significance tests ( general-to-specific ). We find that size distortions can be particularly large when competing models are close, with closeness being defined relatively to the sample size. Keywords: Pretest estimator, automatic model selection, empirical size JEL classification: C12 (Hypothesis Testing), C51 (Model Construction and Estimation), C52 (Model Evaluation and Selection) An earlier version was presented at the first meeting of the European Time-Series Econometrics Research Network (ETSERN) in Frankfurt, June 17, We wish to thank the participants for many helpful comments. Part of the research for this paper was carried out while the first author was a Max Weber Fellow at the European University Institute in Florence, whose hospitality is gratefully acknowledged. Corresponding Author: Matei Demetrescu, Applied Econometrics, Goethe University Frankfurt, Gräfstr. 78 / PF 76, D Frankfurt, Germany. deme@wiwi.uni-frankfurt.de. 1

2 1 Introduction Let us recall how econometrics or statistics are typically taught. The analysis consists of two steps, A: Given that the true model is M, the optimal estimation α n from a sample of size n and inference about the parameter α of economic interest should proceed along the lines spelled out in our favorite textbook, and B: Given that the true model M is not known a priori, we have to select and specify a model relying on the same observed sample, M n. But when we actually practise econometrics, the order of the two steps has to be reversed. As the (optimality) properties of our inferential methods in step A crucially hinge on working with a correct or at least appropriate model, step B has to come first and requires consistent data-driven model selection; in other words, the model selection should be such that M n M as n. Consequently, α n from step A is in practice a post-modelselection estimator (often also called pretest estimator in the econometric literature). It was common belief or hope that consistent model selection ensures that the effect of this first research step may be ignored without damage for subsequent inference. Put differently, it was believed and hoped that post-model-selection estimators behaved as if the true model was known a priori at least asymptotically. Unfortunately, this does not hold true in general. See Leeb and Pötscher [LP] (2005) for a highly readable exposition to the topic. Post-model-selection effects have been discussed for a long time in the 2

3 econometric literature. Nakamura and Nakamura (1978), King and Giles (1984) and Griffiths and Beesley (1984) studied the effect of pretesting regression error autocorrelation; see also Kapetanios (2001). A more general specification search has been criticized and discussed in Lovell (1983) and Hoover and Perez (1999), see also more recently Danilov and Magnus (2004). In particular, White (2000) and Romano and Wolf (2005) attacked to formalize data snooping in form of multiple testing. Our work follows the avenue of a series of papers by Pötscher (1991), Kabaila (1995), Pötscher (1995), Pötscher and Novák (1998), Leeb and Pötscher (2003), Leeb (2005), and LP (2005), which proved that even consistent data-driven model selection may completely ruin inference in a second step. The mathematical reason for this rather negative result is that convergence is not uniform but pointwise only. Consequently, consistent model selection procedures fail to capture the true model when the competing models are close, where closeness is relative to the sample size. At first glance such non-uniformity may appear like a mathematical subtlety; but a closer look reveals that conventional model selection may be totally detrimental for subsequent inference. The papers following Pötscher (1991) examine this effect from a theoretical point of view. We substantiate their findings and quantify the actual size distortion of parameter tests in situations relevant for applied econometrics and empirical economics. The next section provides the framework of our treatment of post-modelselection inference. The third section is dedicated to a typical macroeconomic 3

4 dynamic regression where lagged endogenous variables have to be added to the set of explanatory variables. In this situation model selection means lag length determination. The second example addressed in Section 4 treats a discrete choice model with probit estimation as it is often encountered in applied microeconometric work. The general findings are in short: Distortions in post-model-selection inference have to be expected as long as explanatory variables are correlated, and they are particularly large when the competing models are close. The final section draws some conclusions and formulates recommendations for applied work. 2 Model selection We assume a sequence of nested models or specifications. The smaller model is embedded in the more general one. Here, model selection actually is specification search. To simplify the exposition we consider the bivariate linear regression model in the notation of LP (2005), y i = α x i + β z i + ε i, i = 1,..., n, (1) where ε i satisfies textbook assumptions like being identically and independently distributed with mean 0. The parameter of economic interest is assumed to be α, and the ultimate goal of the econometric exercise is to test hypotheses like e.g. H 0 : α = 0 or H 0 : α = 1. The applied worker, however, 4

5 is undecided whether to include the second variable z i or not. He or she is aware that a misspecified model will result in inefficient estimation of α, or even erroneous inference with respect to this parameter. Therefore, one tries to specify the econometric model up to the state of the art in a first step. In a second step the post-model-selection inference about α is drawn. Here we assume that the true model is contained in (1) either with β = 0 or β 0, i.e. the important topic of model approximation is not addressed (or only briefly in Section 3.1). Finally, the regressors are such that 1 n n i=1 x iz i 1 n n i=1 x2 1 n i n i=1 z2 i ρ 0 (2) as n, where the convergence is in probability in case of stochastic variables. In case of no correlation between the regressors, the pretest from the first step will not affect inference from the second step. Pitfalls in postmodel-selection inference are only lurking if ρ 0, see LP (2005). The typical model selection procedures, or specification search strategies, are minimizing information criteria or significance testing schemes ( generalto-specific ). Information criteria minimize a penalized objective function building on the sum of squared residuals. In this paper we focus on the Bayes criterion by Schwarz (1978) [SIC], the Hannan-Quinn criterion by Hannan and Quinn (1979) [HQ], and the original proposal by Akaike (1974) []. While the first two are known to be consistent, the latter is conservative in that it tends to choose too large models. LP (2005), however, argue that 5

6 pitfalls in post-model-selection inference do not hinge on the (in)consistency of information criteria. Further, they stress that hypothesis testing schemes amount to usage of consistent information criteria as long as the significance level decreases as n grows. We apply sequences of individual t-tests at the and 10% level to implement the widespread general-to-specific strategy, which is sometimes labelled LSE methodology after the London School of Economics, see e.g. Hoover and Perez (1999) for comments on this label. Now, let α n denote the post-model-selection estimator computed after data-driven specification in a first step. In contrast, α n stands for the estimator computed from the correct specification; this hypothetical estimator is infeasible in practice since we do not know whether β = 0 or β 0 a priori. Given true parameter values α or β the feasible estimator α n and the hypothetical one α n converge to the same limiting distribution in that lim P β( α n = α n ) = 1. (3) n Such a result is not restricted to our classroom model (1), but holds in a much more general setting allowing for nonlinear models and other estimation methods than least squares or maximum likelihood. The kind of convergence like in (3) was given for instance in Pötscher (1991, Lemma 1). It forms the basis for the belief and common practice that the first step of model selection can be safely ignored for subsequent inference at least asymptotically. The 6

7 argument behind (3) is for fixed parameter values lim P α,β( M n = M(β)) = 1, (4) n where M(β) is the true model (the unrestricted equation (1) if β 0 or the restricted version thereof for β = 0), and M n is the selected model based on n observations. However, it must be noted that convergence is pointwise only and not uniform with respect to α and β. This non-uniformity has been stressed repeatedly in Pötscher (1991, Section 4, remark iii), Kabaila (1995), Pötscher (1995), Pötscher and Novák (1998), Leeb and Pötscher (2003), and in LP (2005). It is the key to understanding the pitfalls in post-modelselection estimation and inference. It seems nevertheless that the difference between uniform and pointwise convergence, where uniform convergence implies pointwise convergence, but the converse does not hold in general, has not been appreciated by some applied econometricians. We now present a simple example illustrating the consequences of lack of uniformity. Example 1 Let f n (β) be a family of deterministic functions defined on [0, 1] for n = 1, 2,... with n β, 0 β 1 n f n (β) =. 1 0, n < β 1 Obviously f n (β) represents a sequence of straight lines with slope n that are 7

8 cut off at β = 1 n. As n we have pointwise convergence to zero, because for any β (0, 1] there exists N(β) such that β 1 N(β). Hence, for all n N(β) is holds f n (β) = 0. Since f n (0) = 0, this establishes pointwise convergence. At the same time convergence is not uniform in that no N exists such that for all β (0, 1] and all n N it holds f n (β) = 0. Consequently, for two arguments getting arbitrarily close with n the values f n may diverge. E.g. for β n = n 0.5 we obtain f n (β n ) f n (0) = n 0, which is unbounded in n notwithstanding that β n = n 0.5 is close to zero, with closeness being relative to n. While this may be counter-intuitive at first glance, it is simply a consequence of non-uniformity. Let us return to post-model-selection inference. LP (2005) show that for true values β n = cn the non-uniformity in (4) implies that the true model is NOT estimated with probability one, i.e. Mn and M(β n ) are not getting arbitrarily close. Consequently, when the competing models are close (i.e. β is small in absolute value relative to the sample size), consistent model selection procedures are not reliable any more. What is more, the pretest distribution of n( α n α) will depend on β, see Leeb and Pötscher 8

9 (2003) and Leeb (2005), which ruins subsequent inference. Some people may believe that the misspecification effect for small values of β cannot be very detrimental. But the contrary is often true. Massive size distortions are quantified in the following sections when the competing models are close, where closeness is relative to the sample size. 3 Dynamic regressions and lag length selection In analogy to (1) we now consider a time series regression with lagged endogenous regressors, y t = α x t + p β l y t l + ε t, t = p + 1,..., n. (5) l=1 The use of such framework is widespread in econometric practice. For instance, one can test for cointegration in a single-equation framework, or test the null hypothesis that some observed series ξ t is integrated of order d, H 0 : ξ t I(d), by suitably choosing regressand and regressors in (5). Our experiments focus on testing integration of order d. The test statistic is the usual t-statistic t α=0 testing for α = 0. The inno- 9

10 vations ε t are drawn independently from a standard normal distribution. We computed frequencies of rejection of t α=0. All results rely on replications at each point. Being interested in actual size properties we work under the null hypothesis. 3.1 ALM test The first test considered is constructed from the Lagrange Multiplier (LM) principle. It allows for any fractional d. The data are differenced under H 0. We focus on the augmented regression-based LM test (ALM) proposed by Demetrescu, Kuzin and Hassler (2008). With y t := d ξ t it relies on OLS estimation of (5) with t 1 x t := yt 1 := j 1 y t j. The particular choice of x t = yt 1 is anchored in the LM principle, see Demetrescu, Kuzin and Hassler (2008) or Breitung and Hassler (2002). Asymptotically, the regressor yt 1 is a stationary process, which is correlated with the short memory differences y t l. Demetrescu, Kuzin and Hassler (2008) assume the data generating process of y t to be unknown and allow for a general linear process, which can be approximated by an AR(p) process with some order p, where p is a function of the sample size n. Demetrescu, Kuzin and Hassler (2008) showed that t α=0 is asymptotically standard normally distributed under the null hypothesis as long as p diverges with an appropriate j=1 10

11 rate. But the devil is in the detail: how does one set the lag length p in practice? We investigate deterministic and stochastic (data-driven) rules of lag length selection. The deterministic one is the popular rule of thumb due to Schwert (1989), p K = [ K(T/100) 1/4], (6) where [ ] denotes the largest integer part of a real number. Schwert (1989) suggested K = 4 and K = 12. This is also our choice of the maximum lag length when employing information criteria or general-to-specific sequential testing procedures. We performed two-sided tests, i.e. compare t α=0 with standard normal percentiles, and further set without loss of generality d = 0. To explore the impact of lag selection on size we employ an AR(1) process with zero mean as y t : y t = β y t 1 + ε t, β {±0.95, ±0.9,..., 0}, (7) where the autoregressive parameter varies over the region of stationarity. Figure 1 contains graphically the frequencies of rejection of tests at nominal level (two-sided alternatives) depending on n, K in (6), and lag selection criteria (, SIC, HQ,, 10%, where the last two denote sequential significance testing). The corresponding autoregressive coefficients β are on the x-axis. The left graphs are for n = 100, while the case n = 500 is displayed on the right-hand side. As the true lag length p in (5) is 0 or 1, 11

12 p4 10% SIC HQ n = p4 10% SIC HQ n = p12 p12 n = % 10% n = 500 SIC SIC 0.4 HQ 0.4 HQ Figure 1. Percentage of rejection of true null hypothesis (t α=0, nominal ) for AR(1), β from (7) on the x-axis; p4 (upper panel) and p12 (lower panel) denote p K from (6), K = 4 and K = 12. Schwert s rule p K from (6) does a good job in terms of size. The bad news is, however, that under data-driven lag selection the test is always oversized. This size distortion is not a small sample problem, it shows up for n = 500 to the same degree as for n = 100. In particular, very serious distortions are observed for small autocorrelations in absolute value. We get, for example, two maxima of distortions around the white noise case (β = 0). For larger n the distortions are more symmetric around β = 0. But the problem does not disappear with growing sample size: the two peaks left and right to the 12

13 white noise case are moving closer but do not get lower. The issue seems to be of asymptotic nature as in the framework of LP (2005): with larger sample sizes, smaller autocorrelations cause huge size distortions a very alarming result given the fact that is not very likely to find a pure white noise process in practice. The next experiment adds the aspect of model approximation, because we assume an MA(1) process with zero mean as y t : y t = ε t + ψ ε t 1, ψ {±0.95, ±0.9,..., 0}, (8) with moving-average parameters varying over the range of invertibility. This implies p = in (5) as invertibility allows an AR( ) representation, y t = ( 1) l 1 ψ l y t l + ε t. l=1 Therefore, any choice of p in (5) will only approximate the true model. From Figure 2 we observe that deterministic lag length selection methods following Schwert s rule from (6) manage to keep the experimental size close to the nominal one. Serious problems emerge only in a neighborhood of ψ = 1, which is not surprising since y t is I( 1) for ψ = 1, which violates the null hypothesis. Generally, p 4 is inferior to p 12 because the true lag length is infinite. More importantly, data-driven lag selection results always in oversized tests. Again, this is not a small sample problem, but for n =

14 p4 10% SIC HQ p4 10% SIC HQ 0.6 n = n = p12 p12 10% 10% 0.8 SIC HQ 0.8 SIC HQ 0.6 n = n = Figure 2. MA(1) with ψ from (8) on the x-axis; for further comments see Figure 1. at least as severe as for n = 100. We observe two local maxima of distortions around the white noise case (ψ = 0) that move closer with growing n without getting any lower. This illustrates again the effect of nonuniform convergence. 3.2 ADF test The DF test due to Dickey and Fuller (1979) was discussed for the general case of lag augmentation (ADF) by Said and Dickey (1984). The null hypothesis specializes to d = 1, and y t = ξ t = ξ t ξ t 1. The regressor x t from 14

15 (5) becomes t 1 x t := ξ t 1 = y t j. j=1 The ADF test is one-sided and rejects for too small values. The limiting distribution of t α=0 is not normal, however. The asymptotic critical value from MacKinnon (1991) is Note that x t = ξ t 1 is nonstationary under H 0, while the lagged differences are all stationary. Consequently, y t l and x t are asymptotically uncorrelated, which violates (2). Consequently, no pitfalls in post-model-selection tests are to be expected. In fact, Ng and Perron (1995) provided a theoretical justification for data-driven model selection to determine the appropriate lag length in ADF tests. As we expect from the uncorrelatedness of y t l and x t, the size distortions displayed in Figure 3 are very moderate, and they vanish as the sample size increases from 100 to 500. This nicely illustrates that size distortions of post-model-selection tests do not necessarily always arise. 4 Probit estimation The problems highlighted in the previous section are not specific to the timeseries framework. This section is dedicated to the analysis of a binary response (or discrete choice) model estimated in the probit framework. Such a setup involves a latent variable ỹ i, which is generated as in (1) ỹ i = α x i + β z i + ε i, i = 1,..., n, 15

16 0.10 p4 n = p4 n = p12 p12 n = 100 n = Figure 3. Percentage of rejection of true null hypothesis (nominal ) for ADF with AR(1), for further comments see Figure 1. where ε i iin(0, 1), but only the sign of ỹ i is observed. We define y i = 1 if ỹ i > 0 and y i = 0 if ỹ i 0. In the probit framework, the binary response model becomes P(y i = 1) = P(ỹ i > 0) = P(α x i + β z i + ε i > 0) = Φ(α x i + β z i ), (9) where Φ( ) is the cumulative standard normal distribution function. The parameters are estimated via maximum likelihood method with asymptotic 16

17 standard errors. For the simulation exercise we assume α = 1 and correlation between the explanatory variables, z i = a x i + b ν i, x i, ν i iin(0, 1), (10) where x i and v i are independent such that the correlation coefficient amounts to ρ = E(x i z i ) E(x 2 i ) E(z 2 i ) = a a2 + b 2. The true value of β varies in [ 2, 2], and we assume α = 1 to be the null hypothesis of economic interest. First, we consider the case of a = 1 and b = 0.5 with ρ Figure 4 reports rejection frequencies of the true null at the asymptotic level as a function of β. The lines with solid squares and diamonds correspond to post-model-selection sizes obtained from and general-to-specific (), respectively. Like in Figure 1 dramatic size distortions are observed for small β in absolute value, where small is relative to the sample size. We get two maxima of distortions around the restricted model (β = 0). The two peaks move closer with growing sample size and get more symmetric, but they do not necessarily get lower. As a reference we plot a line with empty squares which is obtained from an unrestricted maximum likelihood estimation, where inference about α is drawn irrespective of the eventual insignificance of β. Comparing the last line with the first two clearly shows 17

18 0.4 unrestr. n = 100, ρ= unrestr. n = 500, ρ= Figure 4. Percentage of rejection of true null hypothesis (α = 1, nominal ) for the probit model (9) with β on the x-axis, where ρ 0.89 that the size distortion must be contributed to data-driven model selection. Next, we consider the cases of a = 0.5, b = 1 and a = b = 1 with ρ 0.45 and ρ 0.71, respectively. Figure 5 reports rejection frequencies of the true null at the asymptotic level as a function of β, where solid squares, diamonds and empty squares correspond to the procedures just described for Figure 4. Again, we observe considerable size distortions also for large n, although they are clearly less pronounced the weaker the correlation ρ is. 18

19 unrestr. n = 100, ρ= 0.45 unrestr. n = 500, ρ= unrestr. n = 100, ρ= 0.71 unrestr. n = 500, ρ= Figure 5. Probit model where ρ 0.45 and ρ 0.71, respectively; for further comments see Figure 4 5 Concluding remarks It seems widely acknowledged in theoretical econometrics that specification search or model selection prior to actual data analysis will affect the properties of post-model-selection inference even asymptotically. In this paper we quantified experimentally the size distortion attributable to model selection in a time-series context with lagged endogenous regressors and for a discrete choice (probit) model. One may characterize three situations. First, our results for the ADF test showed: If the variable of interest and 19

20 further eventual explanatory variables are uncorrelated, then post-modelselection inference is valid as if the true model was known, at least for large samples. Similarly, Hoover and Perez (1999) stressed the need of orthogonal regressors for the general-to-specific methodology. Second, in a stationary time-series context working with lagged endogenous regressors it will be difficult to find a parameterization ensuring orthogonality. For that reason, Demetrescu, Kuzin and Hassler (2008) recommended deterministic rules of thumb for lag-length selection. Our present findings support this view. Our preferred choice to determine the number of lags is p 4 with K = 4 in (6) because it provides good size properties and more power than the choice of more lags following e.g. p 12. Of course such a recommendation may be modified in practice. Working e.g. with monthly observations under likely seasonal autocorrelation, one might choose the number of lags as max(13, p 4 ), having a multiplicative autoregressive model, (1 ρ 1 L)(1 ρ 12 L 12 ), in mind. Third, size distortions in post-model-inference are not specific for dynamic models. They may just as well arise in a binary choice model with one variable of interest and a second variable where we are not sure whether to include it or not. We illustrate that it is not advisable to search the minimal model as this will ruin subsequent inference. Instead we recommend to work with the unrestricted model (at the prize of a loss in efficiency) guaranteeing properly sized tests, unless dependence between variables is expected to be low. 20

21 References Akaike, K. (1974), A New Look at the Statistical Model Identification; IEEE Transactions on Automatic Control AC-19, Breitung, J., Hassler, U. (2002), Inference on the Cointegration Rank in Fractionally Integrated Processes; Journal of Econometrics 110, Danilov, D., Magnus, J.R. (2004), On the Harm that Ignoring Pretesting can Cause; Journal of Econometrics 122, Demetrescu, M., Kuzin, V., Hassler, U. (2008), Long Memory Testing in the Time Domain; Econometric Theory 24, Dickey, D.A., Fuller, W.A. (1979), Distribution of the Estimators for Autoregressive Time Series with a Unit Root; Journal of the American Statistical Association 74, Griffiths, W.E., Beesley, P.A.A. (1984), The Small-sample Properties of some Preliminary Test Estimators in a Linear Model with Autocorrelated Errors; Journal of Econometrics 25, Hannan, E.J., Quinn, B.G. (1979), The Determination of the Order of an Autoregression; Journal of the Royal Statistical Society B41, Hoover, K.D., Perez, S.J. (1999), Data Mining Reconsidered: Encompassing and the general-to-specific approach to specification search; Econometrics Journal 2,

22 Kabaila, P. (1995), The Effect of Model Selection on Confidence Regions and Prediction Regions; Econometric Theory 11, Kapetanios, G. (2001), Incorporating Lag Order Selection Uncertainty in Parameter Inference for AR models; Economics Letters 72, King, M.L., Giles, D.E.A (1984), Autocorrelation Pre-testing in the Linear Model: Estimation, testing and prediction; Journal of Econometrics 25, Leeb, H. (2005), The Distribution of a Linear Predictor after Model Selection: Conditional finite-sample distributions and asymptotic approximations; Journal of Statistical Planning and Inference 134, Leeb, H., Pötscher, B.M. (2003), The Finite-Sample Distribution of Post- Model-Selection Estimators and Uniform versus Nonuniform Approximations; Econometric Theory 19, Leeb, H., Pötscher, B.M. (2005), Model Selection and Inference: Facts and fiction; Econometric Theory 21, Lovell, M.C. (1983), Data Mining; The Review of Economics and Statistics 65, MacKinnon, J.G. (1991), Critical Values for Co-Integration Tests; in: Engle, R.F., Granger, C.W.J. (eds.), Long-Run Economic Relationships, Oxford University Press,

23 Nakamura, A., Nakamura, M. (1978), On the Impact of the Tests for Serial Correlation upon the Test of Significance for the Regression Coefficient; Journal of Econometrics 7, Ng, S., Perron, P. (1995), Unit Root Tests in ARMA Models With Data- Dependent Methods for the Selection of the Truncation Lag; Journal of the American Statistical Association 90, Pötscher, B.M., (1991), Effects of Model Selection on Inference; Econometric Theory 7, Pötscher, B.M., (1995), Comment on Effects of Model Selection on Confidence Regions and Prediction Regions by P. Kabaila; Econometric Theory 11, Pötscher, B.M., Novák, A.J. (1998), The Distribution of Estimators after Model Selection: Large and small sample results; Journal of Statistical Computation and Simulation 60, Romano, J.P., Wolf, M. (2005), Stepwise Multiple Testing as Formalized Data Snooping; Econometrica 73, Said, S.E., Dickey, D.A. (1984), Testing for Unit Roots in ARMA(p,q)- Models with Unknown p and q; Biometrika 71, Schwarz, G. (1978), Estimating the Dimension of a Model; Annals of Statistics 6,

24 Schwert, G. W. (1989), Tests for Unit Roots: A Monte Carlo investigation; Journal of Business and Economic Statistics 7, White, H. (2000), A Reality Check for Data Snooping; Econometrica 68,