Model Assessment and Comparisons

Transcription

1 Model Assessment and Comparisons Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department of Forestry & Department of Geography, Michigan State University, Lansing Michigan, U.S.A. October 30,

2 The role of checking and comparing models First two stages: 1 Construct a reasonable probability model; 2 Compute the posterior distribution of model parameters typically by drawing samples from it. Third stage: Checking the quality of the model s fit. This is crucial Prior-to-Posterior inferences involve the whole structure (with hierarchies) of the Bayesian model and can produce spurious inference if the model is poor. Sensitivity Analysis: How much do posterior inferences change when other probability models are used in place of the present model? 2 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

3 The role of checking and comparing models Three critical questions Do the inferences from the model make sense? Is the model consistent with the data? How can we compare and, perhaps, rank different plausible models in their order of preference with respect to a given data set? 3 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

4 Posterior predictive distribution for replicates Replicating data sets using the posterior predictive distribution Let y = (y 1, y 2,..., y n ) be the observed data and θ be the collection of all parameters (including all hyperparameters) for a model p(θ) p(y θ). Let y rep = (y rep,1, y rep,2,..., y rep,n ) be the replicated data that we would see if the experiment that produced y today were replicated with the same model and the same value of θ that produced the observed data. Replicated data y rep, like predictions ỹ, has two components of uncertainty: 1 The fundamental variability of the model, represented by the posited variability in the data; 2 The posterior uncertainty in the estimation of θ 4 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

5 Posterior predictive distribution for replicates The distribution of y rep is the posterior predictive distribution: p(y rep y) = p(y rep θ)p(θ y)dθ We do not evaluate the above integral, but sample from p(y rep y): 1 Draw θ (j) p(θ y), j = 1, 2,..., M 2 Draw y (j) rep p(y rep θ (j) ), j = 1, 2,..., M. 5 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

6 Posterior predictive distribution for replicates Usually full inferential output for Bayesian inference comprises a table comprising both samples from the posterior distribution of θ and the posterior predictive distribution of replicated data sets. Sample θ 1 θ 2... θ p y rep,1 y rep,2... y rep,n 1 θ (1) 1 θ (1) 2... θ p (1) y (1) rep,1 y (1) rep,2... y rep,n (1) 2 θ (2) 1 θ (2) 2... θ p (2) y (2) rep,1 y (2) rep,2... y rep,n (2). M.... θ (M) 1 θ (M) 2... θ p (M). y (M) rep,1... y (M) rep,2... y rep,n (M) 6 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

7 Example: Linear regression model Example: linear regression model Recall the Bayesian linear regression model with non-informative priors: y i µ i, σ 2 ind N(µ i, σ 2 ); i = 1, 2,..., n; µ i = β 0 + β 1 x i1 + + β p x ip = x iβ; β = (β 0, β 1,..., β p ); β, σ 2 p(β, σ 2 ) = 1 σ 2. Unknown parameters include the regression parameters and the variance, i.e. θ = {β, σ 2 }. Obtain posterior samples: θ (j) = {β (j), σ 2(j) }, j = 1,..., M. 7 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

8 Example: Linear regression model For each sampled parameter vector θ (j) = {β (j), σ 2(j) }, we replicate n data points: y (j) rep,i N(x iβ (j), σ 2(j) ), j = 1,..., M and i = 1,..., n. ( y (j) rep = y (j) rep,1, y(j) rep,2 rep,n),..., y(j) is the j-th sample from the posterior predictive distribution p(y rep y). Remark: The number of posterior samples, M, represents post-convergence (i.e. after burn-in) posterior samples. There is no need to consider pre-convergence samples for drawing the posterior predictive samples. 8 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

9 Example: Linear regression model We distinguish between the replicated data, y rep, and the predictive outcomes, ỹ. The variable ỹ is any future observable value of the outcome. For example, in a linear regression model ỹ can have its own set of explanatory variables X. On the other hand, y rep must have the same explanatory variables X as those used in the model for the observed data y. In this sense, y rep is similar to predicting the observed data. 9 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

10 Classical and Bayesian p-values Lack of fit of the data with respect to the posterior predictive distribution can be measured by the tail-area probability, or p-value, of a test statistic. Recall the classical p-value for a test statistic T (y): p C = P ( T (y rep ) T (y) θ ), where the probability is taken over the distribution of y rep with θ fixed (usually at a value specified by a null hypothesis). In classical statistics, the test statistic T (y) does not depend upon model parameters. 10 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

11 Classical and Bayesian p-values In Bayesian inference, a test statistic can be a function of the parameters and the data because the test measure is evaluated over draws from the posterior distribution of the unknown parameters. We call T (y; θ) a test measure. The p-value is computed using the posterior samples of θ and y rep. 11 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

12 Bayesian p-value Does our model represent our data adequately? Choose a discrepancy measure or test measure, say T (y; θ) = T (y; β, σ 2 ) = = n i=1 (y i x i β)2 σ 2 n i=1 (y i E[y i θ]) 2 var(y i θ) Compute T (y, θ (j) ) and the set of of T (y (j) rep, θ (j) ) and obtain Bayesian p-values : p B = P (T (y rep, θ) > T (y, θ) y) = 1 M 1[T (y (j) rep M, θ(j) ) > T (y, θ (j) )]. j=1 12 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

13 Bayesian p-value Bayesian p-values close to 0 or 1 signifies lack of fit of the model with respect to the test measure. On the other hand, values of p B close to 0.5 indicate very good fit. Estimates of p B may be sensitive to choice of the test measure. Unlike p C, we should not interpret p B with regard to significance levels of a test. Instead it should be used as a diagnostic to see if the model adequately fits the data. Bayesian p-values are not concerned with Type-I error rates. Hence, there is no need to consider adjusting p B for multiple comparisons (in case we use several test measures). 13 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

14 Model comparisons Model comparisons using replicated data Compute the posterior predictive mean and variance for each observation: µ rep,i = E[y rep,i y] = 1 M y (j) rep,i, i = 1,..., n; M σ 2 rep,i = var[y rep,i y] = 1 M j=1 M j=1 (y (j) rep,i µ rep,i) 2. Goodness of fit measure G and expected mean-square predictive error P : n n G = (y i µ rep,i ) 2 ; P = σrep,i; 2 D = G + P i=1 i=1 D is a model comparison metric (lower values better). 14 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop

15 Model comparisons Model comparisons using the DIC A general choice for the test measure is the deviance: T (y; θ) = D(y; θ) = 2 log p(y θ). A better option for hierarchical models that does not require replicated data (saves computation time): D(y) = E[D(y; θ) y] = 1 M M D(y; θ (j) ); j=1 p D = D(y) D(y; θ), where θ = E[θ y] = 1 M DIC = D(y) + p D = 2 D(y) D(y; θ). M θ (j) ; j=1 15 LWF, ZWFH, DVFFA, & DR IBS: Biometry workshop