The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification Brett Presnell Dennis Boos Department of Statistics University of Florida and Department of Statistics North Carolina State University The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.1/27
The Problem Question: Given data Y 1,..., Y n, independent, with predictors x 1,..., x n, and a parametric model f(y i ; x i, θ) for the density of Y i, with θ a p-vector of unknown model parameters, how do we test for model misspecification? Answer: It depends. Are the data i.i.d.? (no x i s) Are there replications? Are the Y i s continuous or categorical? Univariate or multivariate? Is f of some specific form that I have a test for?... The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.2/27
Some Answers Kolmogorov-Smirnov, Cramér-von Mises, Anderson-Darling, Shapiro-Wilks, Lillefors,... Pearson chi-sq, deviance, power divergence,... Mardia s skewness and kurtosis tests,... Find (or contrive) a bigger model (but we are NOT trying to do variable selection here). However, there is often no obvious answer, esp. if we have x i s and no replication. Might try: Graphical methods. Information matrix test? The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.3/27
Motivation θ = MLE of θ. θ (i) = MLE of θ if ith obs. deleted from sample. f(y i ; x i, θ (i) ) measures how well the model predicts the ith observation. Compare to f(y i ; x i, θ ). f(y i ; x i, θ (i) ) f(y i ; x i, θ ) always: n l j ( θ ) l i ( θ ) n l j ( θ (i) ) l i ( θ (i) ) n l j ( θ ) l i ( θ (i) ) j=1 j=1 j=1 If f(y i ; x i, θ (i) ) << f(y i ; x i, θ ), then fitted model must shift appreciably to accomodate ith obs. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.4/27
Compare the in-sample and out-of-sample likelihoods as a global measure of lack-of-fit: IOS = log ( n i=1 f(y i; x i, θ) ) n i=1 f(y i; x i, θ (i) ) An Idea = n { l(yi ; x i, θ) l(y i ; x i, θ (i) ) }, i=1 The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.5/27
Connections The cross-validated (out-of-sample) log-likelihood (Stone, 1977) or Bayesian analogues (Geisser and Eddy, 1979) are popular in model selection. Geisser (1990) uses a Bayesian s version of the individual terms of IOS to test for outliers (discordancy). Similar motivation for Cook s distance. Asymptotic form of IOS related to information matrix test (White, 1982). In simple examples, IOS approximates well known, intuitive test statistics. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.6/27
For model X 1,..., X n i.i.d. Poisson(λ), IOS s2 Example: IID Poisson Obvious comparison of moments (Fisher, 1973). IOS P 1 if model correctly specified. Asymptotically normal. Y The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.7/27
For model X 1,..., X n i.i.d.exp(µ), Example: IID Exponential IOS s2 Y 2. Obvious comparison of moments again. IOS P 1 if model correctly specified. Asymptotically normal. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.8/27
For model X 1,..., X n i.i.d.n(µ, σ 2 ), IOS 1 ( ) µ4 2 σ 4 + 1, where µ 4 is sample 4th central moment. IOS is approximately a kurtosis test. IOS P 2 if model correctly specified. Asymptotically normal. Example: IID Normal The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.9/27
Example: Normal Regression For model Y i indep N(x T i β, σ2 ), let β be the LSE. Then IOS n i=1 h iê 2 i σ 2 + µ 4 2 σ 4 1 2, where h i = x T i (XT X) 1 x i, the leverage of the ith obs., e i = Y i x T i β, σ 2 = n 1 n i=1 e 2 i, µ 4 = n 1 n i=1 e 4 i. Looks for heterogeneity of variance and kurtosis. IOS P p = dim(β) + 1 if model correctly specified. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.10/27
Approximate Form For theory, regularity conditions are similar to those for consistency and asymptotic normality of MLEs. Details are provided for the i.i.d. case. IOS IOS A = o p (n 1/2 ), where (i.i.d. case) IOS A = 1 n with n i=1 l(y i ; θ) T Â( θ) 1 l(yi ; θ) = tr { Â( θ) 1 B( θ) }, Â( θ) = n 1 n { l(y i ; θ)}, B( θ) = n 1 n l(y i ; θ) l(y i ; θ) T i=1 i=1 The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.11/27
IOS P IOS = E { l(y1 ; θ 0 ) T A(θ 0 ) 1 l(y1 ; θ 0 ) } Some Theory where = tr { A(θ 0 ) 1 B(θ 0 ) } H = 0 p I(θ) = E{ l(y 1 ; θ)}, B(θ) = E{ l(y 1 ; θ) l(y 1 ; θ) T }. n 1/2 (IOS IOS ) is asymptotically normally distributed (but the asymptotic variance is complicated and convergence is slow). In i.i.d. location-scale models, the null hypothesis distributions of IOS and IOS A do not depend on parameter values. Parametric bootstrap p-values exact up to MC. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.12/27
Discussion of IOS A IOS A /n is the trace term appearing in model selection criteria based on Kullback-Leibler discrepancy (Linhart and Zucchini, 1986)). This is related to the fact that AIC can be viewed as an approximation to the out-of-sample log-likelihood in which the trace term is replaced by p/n (Stone, 1977). IOS A is the trace of the ratio of the observed Fisher information matrix, Â( θ), a model-dependent estimate of cov{ l(y 1 ; θ 0 )}, and B( θ), the sample covariance of l(y i ; θ), a model-free estimate. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.13/27
Information Matrix Test White s 1982 (668) information matrix (IM) test is based on a quadratic form in the vector of differences between elements of B( θ) and Â( θ). Similarities: Both IOS A and IM test looking at something important, e.g., should we use a sandwich estimator of var( θ), or can we trust the model and use inverse Fisher information? Asymptotic distributions not very useful for either IOS/IOS A or IM. Use parametric bootstrap instead. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.14/27
IOS vs IM Consistency: IM test consistent against any alternative with A(θ 0 ) B(θ 0 )? IOS suggests how to look for alternatives that IOS will miss. Can I use it this afternoon? IM requires a lot of prior analysis/programming for each type of problem, even if we bootstrap. IOS is automatic if we bootstrap. IOS A requires much less analysis than IM. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.15/27
Larsen and Marx (2001) Example: Hurricane Rainfall Max 24 hour precipitations for 36 hurricanes. Fit a gamma distribution (p = 2). shape = 2.2 ŝcale = 3.3 IOS = 3.6. (Largest obs. contributes 1.73.) Bootstrap p-value Test All data Drop largest Drop 2 largest IOS.028.061.101 IOS A.022.053.099 A-D.044.164.154 K-S.274.380.430 The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.16/27
Hurricane Rainfall Data 0 10 20 30 Max Rainfall (inches) The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.17/27
Johnson and Wichern (1998) Example: Board Stiffness Four measurements of stiffness on 30 boards. Model: 4-dimensional normal (p = 14). IOS = 30.7 p-value boot =.002 J&W identify two outliers (which contribute 5.1 and 13.5 to IOS). After deletion: IOS = 27.2 p-value boot =.006 For testing i.i.d. multivariate normal model: IOS with parametric bootstrap is exact. IOS is very much like Mardia s multivariate kurtosis test. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.18/27
Feigl and Zelen (1965) Example: Leukemia Survival Survival times of 33 leukemia patients. Predictors: WBC = log of white blood count AG = a binary factor (17 AG pos., 16 AG neg.) Models include WBC:AG interaction (p = 5 for both): Gamma GLM with log link. IOS = 15.7, p-value boot =.03 Linear regression w/ log(survival) as response. IOS = 7.29, p-value boot =.22 The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.19/27
Leukemia Survival (ctd) Does it matter which model we use? Interaction non-significant in gamma model, marginally significant in lognormal model. Important scientific finding in original paper was that WBC slope for AG negative group not significantly different from zero. Simulations suggest that.05 level tests maintain nominal size for both models. If testing lognormal model, power against gamma model is only about.20. Not great, but maybe not bad for n = 33, p = 5. Where s the competition? The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.20/27
Agresti (1996) Example: Horseshoe Crabs No. of satellites counted for 173 nesting females. Model: Poisson GLM with log link Carapace width as predictor (p = 2) Agresti tries Pearson chi-sq and deviance tests: Must pool data over ranges of carapace width. Finds no evidence of lack of fit. Later finds other evidence of overdispersion. IOS = 5.6 p-value boot 0 Negbinom model (p = 3): IOS = 2.66, p-val =.91. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.21/27
Slaton, Piegorsch and Durham (2000) 107 rat litters, 4 dose levels. Example: Toxicology Model: Heckman-Willis. Beta-binomial regression with implicit logit link. α = exp(a 0 + a 1 x), β = exp(b 0 + b 1 x). p = 4. Slaton, et al tested HW model against a larger (p = 6) model and found no evidence against HW. Larger model allows intralitter correlation to vary freely between the 4 dose levels, BUT still has beta-binomial response, and implicit logit link. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.22/27
For HW model IOS = 6.34 p-value boot =.04 IOS A = 5.19 p-value boot =.03 Toxicology (ctd) Further analysis (using IOS and other more standard tests) suggests that: Logit link inappropriate. Response at three lowest doses adequately modeled as binomial with same p for all three doses. Response at highest dose NOT adequately modeled by beta-binomial. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.23/27
Toxicology (ctd) Overall Proportion Dead 0.00 0.05 0.10 0.15 0.20 0.25 0.0 0.1 0.2 0.3 0.4 Dose The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.24/27
IOS is computer intensive (IOS A less so); automatic and easy to employ; and applicable to a variety of problems, where sometimes IM is the only obvious competitor. Other things to work on: dependent data (time series; spatial data). censored data. models without a fully specified likelihood. Conclusions The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.25/27
Some References AGRESTI, A. (1996). An Introduction to Categorical Data Analysis. Wiley, New York. FEIGL, P. and ZELEN, M. (1965). Estimation of exponential survival probabilities with concomitant information. Biometrics 21 826 838. FISHER, R. A. (1973). Statistical Methods for Research Workers. 14th ed. Hafner, New York. GEISSER, S. (1990). Predictive approaches to discordancy testing. In Bayesian and likelihood methods in statistics and econometrics: Essays in honor of George A. Barnard (S. Geisser, J. S. Hodges, S. J. Press and A. Zellner, eds.). North-Holland Publishing Co., Amsterdam. GEISSER, S. and EDDY, W. F. (1979). A predictive approach to model selection. J. Am. Statist. Ass. 74 153 160. JOHNSON, R. A. and WICHERN, D. W. (1998). Applied Multivariate Statistical Analysis. 4th ed. Prentice Hall, Upper Saddle River, NJ. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.26/27
More References LARSEN, R. J. and MARX, M. L. (2001). An Introduction to Mathematical Statistics and Its Applications. 3rd ed. Prentice-Hall Inc., Englewood Cliffs, NJ. LINHART, H. and ZUCCHINI, W. (1986). Model Selection. Wiley, New York. SLATON, T. L., PIEGORSCH, W. W. and DURHAM, S. D. (2000). Estimation and testing with overdispersed proportions using the beta-logistic regression model of Heckman and Willis. Biometrics 56 125 133. STONE, M. (1977). An asymptotic equivalence of choice of model by cross-validation and Akaike s criterion. J. R. Statist. Soc. B 39 44 47. WHITE, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica 50 1 26. The In-and-Out-of-Sample (IOS) Likelihood Ratio Test for Model Misspecification p.27/27