Size Distortion and Modi cation of Classical Vuong Tests

Size Distortion and Modi cation of Classical Vuong Tests Xiaoxia Shi University of Wisconsin at Madison March 2011 X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 1 / 30

Vuong Test (Vuong, 1989) Data fx i g n i=1. Two competing parametric models: f (x, θ), θ 2 Θ vs. g (x, β), β 2 B. Evaluate the relative t: H 0 : LR max θ2θ E [log f (X i, θ)] max E [log g (X i, β)] = 0 β2b Likelihood ratio statistic: LR n = n 1 n log f i=1 Xi, ˆθ n log g X i, ˆβ n. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 2 / 30

Vuong Test (Vuong, 1989) If the two models are nonnested, under H 0 : p nlrn! p N 0, ω 2 where ω 2 = E [log f (X i, θ ) log g (X i, β )] 2. One-Step Test: ( ˆω 2 n: sample version of ω 2 ) p Reject H 0 if nlrn ˆω n > z α/2. Two-step Test: reject H 0 if n ˆω 2 n > c n (1 α) and p nlrn ˆω n > z α/2. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 3 / 30

Approximation Quality of Normal (n=1000) 0.4 n 1/2 LR n /ω n N(0,1) 0.3 f(x) 0.2 0.1 0 4 3 2 1 0 1 2 3 4 x X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 4 / 30

About the Graph From the comparison of two normal regression models with 10 and 2 regressors respectively. Data generated under H 0. ω 2 > 0, and the variance test n ˆω 2 n rejects almost all the time. Rejection probability of a 5% test: 7.3%. AIC, BIC corrections mentioned in Vuong (1989), but they do not move the red curve to the right place. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 5 / 30

Approximation Quality of Normal (n=1000) 0.4 0.3 n 1/2 LR n /ω n N(0,1) T mod n f(x) 0.2 0.1 0 4 3 2 1 0 1 2 3 4 x X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 6 / 30

Outline Bias in LR n Over-rejection of the Vuong tests Modi ed Test Examples Extensions to GMM Models X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 7 / 30

Bias in LRn p nlrn = n 1/2 n log f i=1 Xi, ˆθ n log g X i, ˆβ n = n 1/2 n [log f (X i, θ ) log g (X i, β )] i=1 1 2 p n p n ˆφ n 0 p φ A n ˆφ n φ + op n 1 LR1 n n 1/2 LR2 n + o p n 1. Under H 0, E [LR1 n ] = 0, but E [LR2 n ] 6= 0 X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 8 / 30

Bias in LRn n 1/2 E [LR2 n ] is the higher-order bias in p nlr n. How in uential is the higher-order bias? X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 9 / 30

Bias in LRn n 1/2 E [LR2 n ] is the higher-order bias in p nlr n. How in uential is the higher-order bias? It depends on the relative magnitude of LR1 n and n 1/2 LR2 n X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 9 / 30

Bias in LRn n 1/2 E [LR2 n ] is the higher-order bias in p nlr n. How in uential is the higher-order bias? It depends on the relative magnitude of LR1 n and n 1/2 LR2 n is important if n 1/2 LR2 n X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 9 / 30

Bias in LRn - Asymptotic Form of E[LR2n] Let Λ i (φ) = log f (X i, θ) log g (X i, β); φ = θ 0, β 0 0. p n ˆφ n φ!d A 1 Z φ A 1 N (0, B), where A = E 2 Λ i (φ ) Λi (φ φ φ 0, B = E ) φ Λ i (φ ) φ 0. Lemma Under standard conditions, LR2 n! d Z 0 φ A 1 Z φ 2 X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 10 / 30

Bias in LRn - Asymptotic Form of Bias E [LR2 n ] = trace A 1 B 2 = trace A 1 1 B 1 2 trace A2 1 B 2, where A j and B j are respectively the Hessian and the outer-product versions of the information matrix of model j. Special case: under mild or no misspeci cation: bias= d θ d β /2. It can be quite large (relative to nω 2 ), and it favors the model with more parameters. AIC and BIC correct too much and result in an opposite bias. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 11 / 30

Outline Bias in LR n Over-rejection of the Vuong tests Modi ed Test Examples Extensions to GMM Models X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 12 / 30

Over-rejection of the Vuong Tests (Mainly) due to the bias in LR n, the Vuong tests can over-reject the null. The over-rejection can be arbitrarily large (close to 1 than illustrated in previous graph. α) far worse The over-rejection can be captured asymptotically by considering a drifting sequence of null DGPs fp n g nω 2 P n! σ 2 2 [0, ], A Pn! A, B Pn! B, and ρ P n = E Pn Λ i (φ ) Λ i (φ )! ρ φ X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 13 / 30

Over-rejection of the Vuong Tests Lemma Under fp n g and standard MLE conditions nlrn σz n ˆω 2! 0 2 1 Z1 0VZ 1 d n σ 2 2σρVZ 1 + Z1 0V 2 Z 1. where [Q, V ] = eig ρ = Q 0 Ω 1/2 + ρ. A 1 B, (Z 0, Z 1 ) N (0, [1, ρ 0 ; ρ, I ]) and p nlrn / ˆω n is close to N (0, 1) if σ is large relative to trace (V ) the bias dominates if trace (V ) is large relative to σ X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 14 / 30

Over-rejection of the Vuong Tests Theorem Under fp n,k gn,k=1 such that H 0 holds and (i) for all k, (nω 2 P n,k, A Pn,k, B Pn,k, ρ Pn,k )! (σ 2 k, A k, B, ρ) (ii) then tr (V k ) σ k!, If in addition, tr (V k ) q k ) tr(vk 2 )!, and tr(v 4! 0 [tr(vk 2 )] 2 p lim lim Pr nlrn > z k! n! ˆω α/2 = 1. n σ 2 k!, then we also have tr(vk 2 ) lim lim Pr n ˆω n > c n (1 k! n! α) & p nlrn > z ˆω α/2 = 1 n X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 15 / 30

Over-rejection of the Vuong Tests Implications of the Theorem: by increasing the number of parameters of one model, one can always make the Vuong tests pick this model, even if this model is no better than the other. "no better than" can be replaced with "worse". What about AIC and BIC corrections (suggested by various authors)? correct too much By increasing the number of parameters of one model, one can always make the Vuong tests reject this model, even if this model is no worse than the other OK if objective is forecasting; not OK if want to take Vuong tests as hypothesis tests seriously. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 16 / 30

Outline Bias in LR n Over-rejection of the Vuong tests Modi ed Test Examples Extensions to GMM Models X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 17 / 30

Modi ed Test Modi cation contains three parts: modi ed LR n : modi ed ˆω 2 n: LR mod n = LR n + tr ˆV n /(2n), 2 ˆω mod n = ˆω 2 n + n 1 tr ˆV n 4 /tr ˆV n 2, modi ed critical value (discussed later): z mod α/2. Modi cation to LR n removes most of the over-rejection, But tr ˆV n /(2n) introduces slight new over-rejection when ˆV n has one dominating element solved by the modi cation of ˆω 2 n, p nlr mod n / ˆω mod n has little bias and is close to N (0, 1), but still not exactly N (0, 1) fortunately we know what it is (asymptotically). X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 18 / 30

Asymptotic Distribution of Modi ed Statistic Lemma Under fp n g and standard MLE conditions n 1/2 LRn mod ˆω mod n! d J σ,ρ,v = p σ 2 σz 0 2 1 (Z1 0VZ 1 tr (V )) 2σρVZ 1 + Z1 0V 2 Z 1 + tr (V 4 ) /tr (V 2 ). Modi ed critical value: zα/2 mod = sup Quantile(jJ σ,ˆρn, ˆV n j, 1 σ2[0, ) α). where ˆρ n, ˆV n are consistent estimators of ρ, V, σ cannot be consistently estimated. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 19 / 30

Modi ed Test Modi ed Test: reject H 0 if T mod n n1/2 LRn mod ˆω mod n > z mod α/2. Theorem For a set of null DGPs H 0, suppose the standard MLE conditions hold uniformly over the set, then lim sup n! sup Pr P P 2H 0 n 1/2 LR mod n ˆω mod n > zmod α/2 α. In words: the asymptotic size of the modi ed test is less than or equal to α. In other words: the null rejection probability is uniformly well-controlled. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 20 / 30

Discussion of the Critical Value z mod α/2 is in a sense a worst-case critical value. How conservative is it? in the scenario when the classical Vuong tests over-rejection is the worst, z mod α/2 = z α/2. in other cases, zα/2 mod could be bigger, but not much bigger. For example z0.05/2 mod is up to around z 0.01/2. in the later cases, the modi ed test is much more powerful than the two-step Vuong test, and does not over-reject as the one-step Vuong test. How di cult is the computation? fast (because only maximizing over a scalar) convenient (because ˆρ n and ˆV n can be easily obtained from the maximum likelihood routines). X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 21 / 30

Outline Bias in LR n Over-rejection of the Vuong tests Modi ed Test Examples Extensions to GMM Models X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 22 / 30

Example 1 - Normal Regression M1. Y = β 0 + d 1 1 j=1 β j X 1,j + v, v N(0, σ 2 2 ). M2. Y = θ 0 + d 2 1 j=1 θ jx 2,j + u, u N(0, σ 2 1 ); DGP: Y = 1 + a 1 d 1 1 j=1 X 1,j p d1 1 + a 2 d 2 1 j=1 X 2,j p d2 1 + ε (X 1,1,..., X 1,d1 1, X 2,1,..., X 2,d2 1, ε) N (0, I ) Null: a 1 = a 2 = 0.25; Alterative: a 1 = 0, a 2 = 0.25 Base case: d 1 = 10, d 2 = 2, n = 250. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 23 / 30

Table 1. Rej. Prob. of Original and Modi ed Tests (α = 0.05) Original Tests Modi ed Test 2-Step 1-Step Var. Test Sel. Prob Max cn mod Null DGP Base (.087,.004) (.088,.004).949 (.015,.022) 2.00 d 1 = 20 (.205,.000) (.283,.000).680 (.015,.014) 2.00 d 1 = 5 (.037,.010) (.037,.010).990 (.018,.018) 2.04 n = 500 (.067,.005) (.067,.005) 1 (.020,.019) 1.98 n = 100 (.051,.000) (.136,.001).276 (.012,.013) 2.17 Alternative DGP (M2 true) Base (.000,.032) (.000,.032).625 (.000,.281) 2.00 d 1 = 20 (.001,.000) (.001,.000).249 (.000,.187) 2.00 d 1 = 5 (.000,.204) (.000,.204).830 (.000,.336) 2.10 n = 500 (.000,.315) (.000,.315).971 (.000,.724) 2.00 n = 100 (.003,.001) (.004,.001).109 (.001,.051) 2.10 X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 24 / 30

Example 2 - Joint Normal Location Model M1. (Y 1, Y 2 ) N ((θ 1, 0), I 2 ), θ 1 2 R; M2. (Y 1, Y 2 ) N ((0, θ 2 ), I 2 ), θ 2 2 R. DGP: Y1 Y 2 N θ1,0 θ 2,0 25 0, 0 1 LR = θ 2 1,0 θ 2 2,0. nominal size α = 0.05. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 25 / 30

0 0.5 1 1.5 2 2.5 Null Rej. Prob. of the Two Step Test Null Rej. Prob. of the One Step Test Null Rej. Prob. of the Modified Test 0. 12 n = 100 n = 250 0. 12 n = 100 n = 250 0. 12 n = 100 n = 250 0.1 n = 500 n = 750 0.1 n = 500 n = 750 0.1 n = 500 n = 750 Rejection Probability 0. 08 0. 06 0. 04 n = 1000 5% l i ne Rejection Probability 0. 08 0. 06 0. 04 n = 1000 5% l i ne Rejection Probability 0. 08 0. 06 0. 04 n = 1000 5% l i ne 0. 02 0. 02 0. 02 0 0 0 0.5 1 1.5 2 2.5 0 0 0.5 1 1.5 2 2.5 θ 2,0 ( = θ 1,0 ) 1 0.9 Power of the Two Step Test n = 100 n = 250 1 0.9 Power of the One Step Test 1 0.9 Power of the Modified Test 0.8 0.7 n = 500 n = 750 n = 1000 0.8 0.7 0.8 0.7 0.6 5% l i ne 0.6 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 1 1.5 2 2.5 0.5 0.4 n = 100 0.3 n = 250 n = 500 0.2 n = 750 0.1 n = 1000 5% l i ne 0 0 0.5 1 1.5 2 2.5 0.5 0.4 n = 100 0.3 n = 250 n = 500 0.2 n = 750 0.1 n = 1000 5% l i ne 0 0 0.5 1 1.5 2 2.5 θ 2,0 ( θ 1,0 = 0) X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 26 / 30

Outline Bias in LR n Over-rejection of the Vuong tests Modi ed Test Examples Extensions to GMM Models X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 27 / 30

GMM Models and GEL Criteria GMM models (or moment condition models): M1 : Em f (x, ψ f ) = 0 for some ψ f 2 Ψ f R d ψ f, M2 : Em g x, ψ g = 0 for some ψ g 2 Ψ g R d ψ g, (1) where m f and m g are known moment functions and ψ f and ψ g are unknown parameters. Generalized Empirical Likelihood criteria: H 0 : GELR max min E κ γ ψ f 2Ψ f γ f 0 m f (X i, ψ f ) f h i max min E κ γg 0 m g X i, ψ ψ g 2Ψ g g = 0. γ g EL: κ (v) = log (1 v), ET (exponential tilting): κ (v) = e v. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 28 / 30

General Framework In previous analysis, replace log f (x, θ) and log g (x, β) with κ γf 0 m f (X i, ψ f ) and κ γg 0 m g X i, ψ g 0 0 replace θ and β with γf 0,, ψ0 f, and γg 0,, ψ0 g, then everything go through. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 29 / 30

Summary Discover the higher-order bias in the Vuong test statistic Show that the bias cause (sometimes severe) over-rejection Propose a uniformly valid modi ed Vuong test Modi ed Vuong test is easy to compute and has good power. X. Shi (UW-Mdsn) H 0 : LR = 0 IUPUI 30 / 30