Generalized, Linear, and Mixed Models

Transcription

1 Generalized, Linear, and Mixed Models CHARLES E. McCULLOCH SHAYLER.SEARLE Departments of Statistical Science and Biometrics Cornell University A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS, INC. New York Chichester Weinheim Brisbane Singapore Toronto

2 List of Chapters PREFACE xix 1 INTRODUCTION 1 2 ONE-WAY CLASSIFICATION 28 3 SINGLE-PREDICTOR REGRESSION 71 4 LINEAR MODELS (LMs) GENERALIZED LINEAR MODELS (GLMs) LINEAR MIXED MODELS (LMMs) LONGITUDINAL DATA GLMMs PREDICTION COMPUTING NONLINEAR MODELS 286 APPENDIX M: SOME MATRIX RESULTS 291 APPENDIX S: SOME STATISTICAL RESULTS 300 REFERENCES 311 INDEX 321

3 Contents PREFACE xix 1 INTRODUCTION MODELS 1 a. Linear modeis (LM) and linear mixed modeis (LMM) 1 b. Generalized modeis (GLMs and GLMMs) FACTORS, LEVELS, CELLS, EFFECTS AND DATA FIXED EFFECTS MODELS 5 a. Example 1: Placebo and a drug 6 b. Example 2: Comprehension of humor 7 c. Example 3: Four dose levels of a drug RANDOM EFFECTS MODELS 8 a. Example 4: Clinics 8 b. Notation 9 - i. Properties of random effects in LMMs. 9 - ii. The notation of mathematical statistics 10 - iii. Variance of y 11 - iv. Variance and conditional expected values 11 c. Example 5: Ball bearings and calipers LINEAR MIXED MODELS (LMMs) 13 a. Example 6: Medications and clinics 13 b. Example 7: Drying methods and fabrics 13 c. Example 8: Potomac River Fever 14 d. Regression modeis 14 e. Longitudinal data 14 f. Model equations FIXED OR RANDOM? 16 a. Example 9: Clinic effects 16 vi

4 CONTENTS vii b. Making a decision INFERENCE 18 a. Estimation 20 - i. Maximum Hkelihood (ML) 20 - ii. Restricted maximum Hkelihood (REML) 21 - iii. Solutions and estimators 21 - iv. Bayes theorem 22 - v. Quasi-likelihood estimation 23 - vi. Generalized estimating equations b. Testing 23 - i. Likelihood ratio test (LRT) 24 - ii. Wald's procedure 24 c. Prediction COMPUTERSOFTWARE EXERCISES 25 2 ONE-WAY CLASSIFICATION NORMALITY AND FLXED EFFECTS 29 a. Model 29 b. Estimation by ML 29 c. Generalized likelihood ratio test 31 d. Confidence intervals 32 - i. For means 33 - ii. For differences in means 33 - iii. For linear combinations 34 - iv. For the variance 34 e. Hypothesis tests NORMALITY, RANDOM EFFECTS AND ML 34 a. Model 34 - i. Covariances caused by random effects ii. Likelihood 36 b. Balanced data 37 - i. Likelihood 37 - ii. ML equations and their Solutions iii. ML estimators 38 - iv. Expected values and bias 39 - v. Asymptotic sampling variances 40 - vi. REML estimation 42 c. Unbalanced data 42 - i. Likelihood 42

5 viii CONTENTS - ii. ML equations and their Solutions iii. ML estimators 43 d. Bias 44 e. Sampling variances NORMALITY, RANDOM EFFECTS AND REML 45 a. Balanced data 45 - i. Likelihood 45 - ii. REML equations and their Solutions iii. REML estimators 46 - iv. Comparison with ML 47 - v. Bias 47 - vi. Sampling variances 48 b. Unbalanced data MORE ON RANDOM EFFECTS AND NORMALITY a. Tests and confidence intervals 48 - i. For the overall mean, fi 48 -ii. Fora iii. For a\ 49 b. Predicting random effects 49 - i. A basic result 49 - ii. In a 1-way Classification BERNOULLI DATA: FIXED EFFECTS 51 a. Model equation 51 b. Likelihood 51 c. ML equations and their Solutions 52 d. Likelihood ratio test 52 e. The usual chi-square test 52 f. Large-sample tests and intervals 54 g. Exact tests and confidence intervals 55 h. Example: Snake strike data BERNOULLI DATA: RANDOM EFFECTS 57 a. Model equation 57 b. Beta-binomial model 57 - i. Means, variances, and covariances ii. Overdispersion 59 - iii. Likelihood 60 - iv. ML estimation 60 - v. Large-sample variances 61 - vi. Large-sample tests and intervals... 62

6 CONTENTS ix - vii. Prediction 63 c. Logit-normal model 64 - i. Likelihood 64 - ii. Calculation of the likelihood 65 - iii. Means, variances, and covariances iv. Large-sample tests and intervals v. Prediction 67 d. Probit-normal model COMPUTING EXERCISES 68 3 SINGLE-PREDICTOR REGRESSION INTRODUCTION NORMALITY: SIMPLE LINEAR REGRESSION 72 a. Model 72 b. Likelihood 73 c. Maximum likelihood estimators 73 d. Distributions of MLEs 74 e. Tests and confidence intervals 75 f. Illustration NORMALITY: A NONLINEAR MODEL 76 a. Model 76 b. Likelihood 76 c. Maximum likelihood estimators 76 d. Distributions of MLEs TRANSFORMING VERSUS LINKING 78 a. Transforming 78 b. Linking 79 c. Comparisons RANDOM INTERCEPTS: BALANCED DATA 79 a. The model 80 b. Estimating \i and ß 82 - i. Estimation 82 - ii. Unbiasedness 84 - iii. Sampling distributions 84 c. Estimating variances 85 - i. When ML Solutions are estimators ii. When an ML Solution is negative d. Tests of hypotheses - using LRT 88 - i. Using the maximized log likelihood l*(0) 88

7 x CONTENTS - ii. Testing the hypothesis Ho: a\ = iii. Testing H 0 : ß = 0 90 e. Illustration 91 f. Predicting the random intercepts RANDOM INTERCEPTS: UNBALANCED DATA 94 a. Themodel 95 b. Estimating fj. and ß when variances are known i. ML estirnators 96 - ii. Unbiasedness 99 - iii. Sampling variances 99 - iv. Predicting Oj BERNOULLI - LOGISTIC REGRESSION 100 a. Logistic regression model 100 b. Likelihood 102 c. ML equations. 103 d. Large-sample tests and intervals BERNOULLI - LOGISTIC WITH RANDOM INTERCEPTS 106 a. Model 106 b. Likelihood 108 c. Large-sample tests and intervals 108 d. Prediction 109 e. Conditional Inference EXERCISES LINEAR MODELS (LMs) A GENERAL MODEL A LINEAR MODEL FOR FIXED EFFECTS MLE UNDER NORMALITY SUFFICIENT STATISTICS MANY APPARENT ESTIMATORS 118 a. General result 118 b. Mean and variance 119 c. Invariance properties 119 d. Distributions ESTIMABLE FUNCTIONS 120 a. Introduction 120 b. Definition 121 c. Properties 121 d. Estimation A NUMERICAL EXAMPLE 122

8 CONTENTS xi 4.8 ESTIMATING RESIDUAL VARIANCE 124 a. Estimation 124 b. Distribution of estimators COMMENTS ON 1- AND 2-WAY CLASSIFICATIONS a. The 1-way Classification 126 b. The 2-way Classification TESTING LINEAR HYPOTHESES 128 a. Using the likelihood ratio *-TESTS AND CONFIDENCE INTERVALS UNIQUE ESTIMATION USING RESTRICTIONS EXERCISES GENERALIZED LINEAR MODELS (GLMs) INTRODUCTION STRUCTURE OF THE MODEL 137 a. Distribution of y 137 b. Link function 138 c. Predictors 138 d. Linear modeis TRANSFORMING VERSUS LINKING ESTIMATION BY MAXIMUM LIKELIHOOD 139 a. Likelihood 139 b. Some useful identities 140 c. Likelihood equations 141 d. Large-sample variances 143 e. Solving the ML equations 143 f. Example: Potato flour dilutions TESTS OF HYPOTHESES 147 a. Likelihood ratio tests 147 b. Wald tests 148 c. Illustration of tests 149 d. Confidence intervals 149 e. Illustration of confidence intervals MAXIMUM QUASI-LIKELIHOOD 150 a. Introduction 150 b. Definition EXERCISES LINEAR MIXED MODELS (LMMs) A GENERAL MODEL 156

9 xii CONTENTS a. Introduction 156 b. Basic properties ATTRIBUTING STRUCTURE TO VAR(y) 158 a. Example 158 b. Taking covariances between factors as zero c. The traditional variance components model i. Customary notation ii. Amended notation 161 d. An LMM for longitudinal data ESTIMATING FIXED EFFECTS FOR V KNOWN ESTIMATING FIXED EFFECTS FOR V UNKNOWN a. Estimation 164 b. Sampling variance 164 c. Bias in the variance 166 d. Approximate F-statistics PREDICTING RANDOM EFFECTS FOR V KNOWN PREDICTING RANDOM EFFECTS FOR V UNKNOWN. 170 a. Estimation 170 b. Sampling variance 170 c. Bias in the variance ANOVA ESTIMATION OF VARIANCE COMPONENTS a. Balanced data 172 b. Unbalanced data MAXIMUM LIKELIHOOD (ML) ESTIMATION 174 a. Estimators 174 b. Information matrix 175 c. Asymptotic sampling variances RESTRICTED MAXIMUM LIKELIHOOD (REML) a. Estimation 176 b. Sampling variances ML OR REML? OTHER METHODS FOR ESTIMATING VARIANCES APPENDIX 178 a. Differentiating a log likelihood i. A general likelihood under normality ii. First derivatives iii. Information matrix 179 b. Differentiating a generalized inverse c. Differentiation for the variance components model 182

10 CONTENTS xiii 6.13 EXERCISES LONGITUDINAL DATA INTRODUCTION A MODEL FOR BALANCED DATA 188 a. Prescription 188 b. Estimating the mean 188 c. Estimating Vo A MIXED MODEL APPROACH 189 a. Fixed and random effects 190 b. Variances PREDICTING RANDOM EFFECTS 191 a. Uncorrelated subjects 192 b. Uncorrelated between, and within, subjects c. Uncorrelated between, and autocorrelated within, subjects 193 d. Correlated between, but not within, subjects ESTIMATING PARAMETERS 195 a. The general case 195 b. Uncorrelated subjects 196 c. Uncorrelated between, and within, subjects d. Uncorrelated between, and autocorrelated within, subjects 199 e. Correlated between, but not within, subjects UNBALANCED DATA 202 a. Example and model 202 b. Uncorrelated subjects i. Matrix V and its inverse ii. Estimating the fixed effects iii. Predicting the random effects 204 c. Uncorrelated between, and within, subjects i. Matrix V and its inverse ii. Estimating the fixed effects iii. Predicting the random effects 205 d. Correlated between, but not within, subjects AN EXAMPLE OF SEVERAL TREATMENTS GENERALIZED ESTIMATING EQUATIONS A SUMMARY OF RESULTS 212 a. Balanced data i. With some generality 212

11 xiv CONTENTS - ii. Uncorrelated subjects iii. Uncorrelated between, and within, subjects iv. Uncorrelated between, and autocorrelated within, subjects v. Correlated between, but not within, subjects 214 b. Unbalanced data i. Uncorrelated subjects ii. Uncorrelated between, and within, subjects iii. Correlated between, but not within, subjects APPENDIX 215 a. For Section 7.4a 215 b. For Section 7.4b 215 c. For Section 7.4d EXERCISES GLMMs INTRODUCTION STRUCTURE OF THE MODEL 221 a. Conditional distribution of y CONSEQUENCES OF HAVING RANDOM EFFECTS a. Marginal versus conditional distribution 222 b. Mean of y 222 c. Variances 223 d. Covariances and correlations ESTIMATION BY MAXIMUM LIKELIHOOD 225 a. Likelihood 225 b. Likelihood equations i. For the fixed effects parameters ii. For the random effects parameters MARGINAL VERSUS CONDITIONAL MODELS OTHER METHODS OF ESTIMATION 231 a. Generalized estimating equations 231 b. Penalized quasi-likelihood 232 c. Conditional likelihood 234 d. Simpler modeis TESTS OF HYPOTHESES 239

12 CONTENTS xv a. Likelihood ratio tests 239 b. Asymptotic variances 240 c. Wald tests 240 d. Score tests ILLUSTRATION: CHESTNUT LEAF BLIGHT 241 a. A random effects probit model i. The fixed effects ii. The random effects iii. Consequences of having random effects iv. Likelihood analysis v. Results EXERCISES PREDICTION INTRODUCTION BEST PREDICTION (BP) 248 a. The best predictor 248 b. Mean and variance properties 249 c. A correlation property 249 d. Maximizing a mean 249 e. Normality BEST LINEAR PREDICTION (BLP) 250 a. BLP(u) 250 b. Example 251 c. Derivation 252 d. Ranking LINEAR MIXED MODEL PREDICTION (BLUP) 254 a. BLUE(Xß) 254 b. BLUP(t'X + s'u) 255 c. Two variances 256 d. Other derivations REQUIRED ASSUMPTIONS ESTIMATED BEST PREDICTION HENDERSON'S MIXED MODEL EQUATIONS 258 a. Origin 258 b. Solutions 259 c. Use in ML estimation of variance components i. ML estimation ii. REML estimation APPENDIX 260

13 xvi CONTENTS a. Verification of (9.5) 260 b. Verification of (9.7) and (9.8) EXERCISES COMPUTING INTRODUCTION COMPUTING ML ESTIMATES FOR LMMs 263 a. The EM algorithm i. EMfor ML ii. EM (a variant) for ML 265 -in. EMforREML 265 b. UsingE[u y] 266 c. Newton-Raphson method COMPUTING ML ESTIMATES FOR GLMMs 269 a. Numerical quadrature i. Gauss-Hermite quadrature ii. Likelihood calculations iii. Limits of numerical quadrature 273 b. EM algorithm 274 c. Markov chain Monte Carlo algorithms i. Metropolis ii. Monte Carlo Newton-Raphson 277 d. Stochastic approximation algorithms 278 e. Simulated maximum likelihood PENALIZED QUASI-LIKELIHOOD AND LAPLACE EXERCISES NONLINEAR MODELS INTRODUCTION EXAMPLE: CORN PHOTOSYNTHESIS PHARMACOKINETIC MODELS COMPUTATIONS FOR NONLINEAR MIXED MODELS EXERCISES 290 APPENDIX M: SOME MATRIX RESULTS 291 M.l VECTORS AND MATRICES OF ONES 291 M.2 KRONECKER (OR DIRECT) PRODUCTS 292 M.3 A MATRIX NOTATION 292 M.4 GENERALIZED INVERSES 293 a. Definition 293

14 CONTENTS xvii b. Generalized inverses of X'X 294 c. Two results involving XCX'V-^J-X'V d. Solving linear equations 296 e. Rank results 296 f. Vectors orthogonal to columns of X 296 g. A theorem for K' with K'X being null 296 M.5 DIFFERENTIAL CALCULUS 297 a. Definition 297 b. Sealars 297 c. Vectors 297 d. Inner produets 297 e. Quadratic forms 298 f. Inverse matrices 298 g. Determinants 299 APPENDIX S: SOME STATISTICAL RESULTS MOMENTS 300 a. Conditional moments 300 b. Mean of a quadratic form 301 c. Moment generating funetion NORMAL DISTRIBUTIONS 302 a. Univariate 302 b. Multivariate 302 c. Quadratic forms in normal variables i. The non-central x ii. Properties of y'ay when y ~ Af(ß, V) EXPONENTIAL FAMILIES MAXIMUM LIKELIHOOD 304 a. The likelihood funetion 304 b. Maximum likelihood estimation 305 c. Asymptotic variance-covariance matrix 305 d. Asymptotic distribution of MLEs LIKELIHOOD RATIO TESTS MLE UNDER NORMALITY 307 a. Estimation of ß 307 b. Estimation of variance components 308 c. Asymptotic variance-covariance matrix 308 d. Restricted maximum likelihood (REML) i. Estimation ii. Asymptotic variance 310

15 xec TTC SXM3XN0D xaami saonanadiah!!! AX