Generalized, Linear, and Mixed Models

Generalized, Linear, and Mixed Models CHARLES E. McCULLOCH SHAYLER.SEARLE Departments of Statistical Science and Biometrics Cornell University A WILEY-INTERSCIENCE PUBLICATION JOHN WILEY & SONS, INC. New York Chichester Weinheim Brisbane Singapore Toronto

List of Chapters PREFACE xix 1 INTRODUCTION 1 2 ONE-WAY CLASSIFICATION 28 3 SINGLE-PREDICTOR REGRESSION 71 4 LINEAR MODELS (LMs) 113 5 GENERALIZED LINEAR MODELS (GLMs) 135 6 LINEAR MIXED MODELS (LMMs) 156 7 LONGITUDINAL DATA 187 8 GLMMs 220 9 PREDICTION 247 10 COMPUTING 263 11 NONLINEAR MODELS 286 APPENDIX M: SOME MATRIX RESULTS 291 APPENDIX S: SOME STATISTICAL RESULTS 300 REFERENCES 311 INDEX 321

Contents PREFACE xix 1 INTRODUCTION 1 1.1 MODELS 1 a. Linear modeis (LM) and linear mixed modeis (LMM) 1 b. Generalized modeis (GLMs and GLMMs)... 2 1.2 FACTORS, LEVELS, CELLS, EFFECTS AND DATA... 2 1.3 FIXED EFFECTS MODELS 5 a. Example 1: Placebo and a drug 6 b. Example 2: Comprehension of humor 7 c. Example 3: Four dose levels of a drug 8 1.4 RANDOM EFFECTS MODELS 8 a. Example 4: Clinics 8 b. Notation 9 - i. Properties of random effects in LMMs. 9 - ii. The notation of mathematical statistics 10 - iii. Variance of y 11 - iv. Variance and conditional expected values 11 c. Example 5: Ball bearings and calipers 12 1.5 LINEAR MIXED MODELS (LMMs) 13 a. Example 6: Medications and clinics 13 b. Example 7: Drying methods and fabrics 13 c. Example 8: Potomac River Fever 14 d. Regression modeis 14 e. Longitudinal data 14 f. Model equations 16 1.6 FIXED OR RANDOM? 16 a. Example 9: Clinic effects 16 vi

CONTENTS vii b. Making a decision 17 1.7 INFERENCE 18 a. Estimation 20 - i. Maximum Hkelihood (ML) 20 - ii. Restricted maximum Hkelihood (REML) 21 - iii. Solutions and estimators 21 - iv. Bayes theorem 22 - v. Quasi-likelihood estimation 23 - vi. Generalized estimating equations... 23 b. Testing 23 - i. Likelihood ratio test (LRT) 24 - ii. Wald's procedure 24 c. Prediction 24 1.8 COMPUTERSOFTWARE 25 1.9 EXERCISES 25 2 ONE-WAY CLASSIFICATION 28 2.1 NORMALITY AND FLXED EFFECTS 29 a. Model 29 b. Estimation by ML 29 c. Generalized likelihood ratio test 31 d. Confidence intervals 32 - i. For means 33 - ii. For differences in means 33 - iii. For linear combinations 34 - iv. For the variance 34 e. Hypothesis tests 34 2.2 NORMALITY, RANDOM EFFECTS AND ML 34 a. Model 34 - i. Covariances caused by random effects. 35 - ii. Likelihood 36 b. Balanced data 37 - i. Likelihood 37 - ii. ML equations and their Solutions... 37 - iii. ML estimators 38 - iv. Expected values and bias 39 - v. Asymptotic sampling variances 40 - vi. REML estimation 42 c. Unbalanced data 42 - i. Likelihood 42

viii CONTENTS - ii. ML equations and their Solutions... 42 - iii. ML estimators 43 d. Bias 44 e. Sampling variances 44 2.3 NORMALITY, RANDOM EFFECTS AND REML 45 a. Balanced data 45 - i. Likelihood 45 - ii. REML equations and their Solutions.. 46 - iii. REML estimators 46 - iv. Comparison with ML 47 - v. Bias 47 - vi. Sampling variances 48 b. Unbalanced data 48 2.4 MORE ON RANDOM EFFECTS AND NORMALITY... 48 a. Tests and confidence intervals 48 - i. For the overall mean, fi 48 -ii. Fora 2 49 - iii. For a\ 49 b. Predicting random effects 49 - i. A basic result 49 - ii. In a 1-way Classification 50 2.5 BERNOULLI DATA: FIXED EFFECTS 51 a. Model equation 51 b. Likelihood 51 c. ML equations and their Solutions 52 d. Likelihood ratio test 52 e. The usual chi-square test 52 f. Large-sample tests and intervals 54 g. Exact tests and confidence intervals 55 h. Example: Snake strike data 56 2.6 BERNOULLI DATA: RANDOM EFFECTS 57 a. Model equation 57 b. Beta-binomial model 57 - i. Means, variances, and covariances... 58 - ii. Overdispersion 59 - iii. Likelihood 60 - iv. ML estimation 60 - v. Large-sample variances 61 - vi. Large-sample tests and intervals... 62

CONTENTS ix - vii. Prediction 63 c. Logit-normal model 64 - i. Likelihood 64 - ii. Calculation of the likelihood 65 - iii. Means, variances, and covariances... 65 - iv. Large-sample tests and intervals... 66 - v. Prediction 67 d. Probit-normal model 67 2.7 COMPUTING 68 2.8 EXERCISES 68 3 SINGLE-PREDICTOR REGRESSION 71 3.1 INTRODUCTION 71 3.2 NORMALITY: SIMPLE LINEAR REGRESSION 72 a. Model 72 b. Likelihood 73 c. Maximum likelihood estimators 73 d. Distributions of MLEs 74 e. Tests and confidence intervals 75 f. Illustration 75 3.3 NORMALITY: A NONLINEAR MODEL 76 a. Model 76 b. Likelihood 76 c. Maximum likelihood estimators 76 d. Distributions of MLEs 78 3.4 TRANSFORMING VERSUS LINKING 78 a. Transforming 78 b. Linking 79 c. Comparisons 79 3.5 RANDOM INTERCEPTS: BALANCED DATA 79 a. The model 80 b. Estimating \i and ß 82 - i. Estimation 82 - ii. Unbiasedness 84 - iii. Sampling distributions 84 c. Estimating variances 85 - i. When ML Solutions are estimators... 85 - ii. When an ML Solution is negative... 87 d. Tests of hypotheses - using LRT 88 - i. Using the maximized log likelihood l*(0) 88

x CONTENTS - ii. Testing the hypothesis Ho: a\ = 0... 89 - iii. Testing H 0 : ß = 0 90 e. Illustration 91 f. Predicting the random intercepts 92 3.6 RANDOM INTERCEPTS: UNBALANCED DATA 94 a. Themodel 95 b. Estimating fj. and ß when variances are known. 96 - i. ML estirnators 96 - ii. Unbiasedness 99 - iii. Sampling variances 99 - iv. Predicting Oj 99 3.7 BERNOULLI - LOGISTIC REGRESSION 100 a. Logistic regression model 100 b. Likelihood 102 c. ML equations. 103 d. Large-sample tests and intervals 105 3.8 BERNOULLI - LOGISTIC WITH RANDOM INTERCEPTS 106 a. Model 106 b. Likelihood 108 c. Large-sample tests and intervals 108 d. Prediction 109 e. Conditional Inference 109 3.9 EXERCISES 111 4 LINEAR MODELS (LMs) 113 4.1 A GENERAL MODEL 114 4.2 A LINEAR MODEL FOR FIXED EFFECTS 115 4.3 MLE UNDER NORMALITY 116 4.4 SUFFICIENT STATISTICS 117 4.5 MANY APPARENT ESTIMATORS 118 a. General result 118 b. Mean and variance 119 c. Invariance properties 119 d. Distributions 120 4.6 ESTIMABLE FUNCTIONS 120 a. Introduction 120 b. Definition 121 c. Properties 121 d. Estimation 122 4.7 A NUMERICAL EXAMPLE 122

CONTENTS xi 4.8 ESTIMATING RESIDUAL VARIANCE 124 a. Estimation 124 b. Distribution of estimators 125 4.9 COMMENTS ON 1- AND 2-WAY CLASSIFICATIONS... 126 a. The 1-way Classification 126 b. The 2-way Classification 127 4.10 TESTING LINEAR HYPOTHESES 128 a. Using the likelihood ratio 129 4.11 *-TESTS AND CONFIDENCE INTERVALS 130 4.12 UNIQUE ESTIMATION USING RESTRICTIONS 131 4.13 EXERCISES 132 5 GENERALIZED LINEAR MODELS (GLMs) 135 5.1 INTRODUCTION 135 5.2 STRUCTURE OF THE MODEL 137 a. Distribution of y 137 b. Link function 138 c. Predictors 138 d. Linear modeis 139 5.3 TRANSFORMING VERSUS LINKING 139 5.4 ESTIMATION BY MAXIMUM LIKELIHOOD 139 a. Likelihood 139 b. Some useful identities 140 c. Likelihood equations 141 d. Large-sample variances 143 e. Solving the ML equations 143 f. Example: Potato flour dilutions 144 5.5 TESTS OF HYPOTHESES 147 a. Likelihood ratio tests 147 b. Wald tests 148 c. Illustration of tests 149 d. Confidence intervals 149 e. Illustration of confidence intervals 150 5.6 MAXIMUM QUASI-LIKELIHOOD 150 a. Introduction 150 b. Definition 151 5.7 EXERCISES 154 6 LINEAR MIXED MODELS (LMMs) 156 6.1 A GENERAL MODEL 156

xii CONTENTS a. Introduction 156 b. Basic properties 157 6.2 ATTRIBUTING STRUCTURE TO VAR(y) 158 a. Example 158 b. Taking covariances between factors as zero... 158 c. The traditional variance components model... 160 - i. Customary notation 160 - ii. Amended notation 161 d. An LMM for longitudinal data 162 6.3 ESTIMATING FIXED EFFECTS FOR V KNOWN... 162 6.4 ESTIMATING FIXED EFFECTS FOR V UNKNOWN... 164 a. Estimation 164 b. Sampling variance 164 c. Bias in the variance 166 d. Approximate F-statistics 167 6.5 PREDICTING RANDOM EFFECTS FOR V KNOWN... 168 6.6 PREDICTING RANDOM EFFECTS FOR V UNKNOWN. 170 a. Estimation 170 b. Sampling variance 170 c. Bias in the variance 171 6.7 ANOVA ESTIMATION OF VARIANCE COMPONENTS.. 171 a. Balanced data 172 b. Unbalanced data 173 6.8 MAXIMUM LIKELIHOOD (ML) ESTIMATION 174 a. Estimators 174 b. Information matrix 175 c. Asymptotic sampling variances 176 6.9 RESTRICTED MAXIMUM LIKELIHOOD (REML)... 176 a. Estimation 176 b. Sampling variances 177 6.10 ML OR REML? 177 6.11 OTHER METHODS FOR ESTIMATING VARIANCES... 178 6.12 APPENDIX 178 a. Differentiating a log likelihood 178 - i. A general likelihood under normality.. 178 - ii. First derivatives 179 - iii. Information matrix 179 b. Differentiating a generalized inverse... 181 c. Differentiation for the variance components model 182

CONTENTS xiii 6.13 EXERCISES.... 184 7 LONGITUDINAL DATA 187 7.1 INTRODUCTION 187 7.2 A MODEL FOR BALANCED DATA 188 a. Prescription 188 b. Estimating the mean 188 c. Estimating Vo 188 7.3 A MIXED MODEL APPROACH 189 a. Fixed and random effects 190 b. Variances 190 7.4 PREDICTING RANDOM EFFECTS 191 a. Uncorrelated subjects 192 b. Uncorrelated between, and within, subjects... 192 c. Uncorrelated between, and autocorrelated within, subjects 193 d. Correlated between, but not within, subjects.. 193 7.5 ESTIMATING PARAMETERS 195 a. The general case 195 b. Uncorrelated subjects 196 c. Uncorrelated between, and within, subjects... 197 d. Uncorrelated between, and autocorrelated within, subjects 199 e. Correlated between, but not within, subjects.. 201 7.6 UNBALANCED DATA 202 a. Example and model 202 b. Uncorrelated subjects 203 - i. Matrix V and its inverse 203 - ii. Estimating the fixed effects 204 - iii. Predicting the random effects 204 c. Uncorrelated between, and within, subjects... 204 - i. Matrix V and its inverse 204 - ii. Estimating the fixed effects 205 - iii. Predicting the random effects 205 d. Correlated between, but not within, subjects.. 206 7.7 AN EXAMPLE OF SEVERAL TREATMENTS 206 7.8 GENERALIZED ESTIMATING EQUATIONS 208 7.9 A SUMMARY OF RESULTS 212 a. Balanced data 212 - i. With some generality 212

xiv CONTENTS - ii. Uncorrelated subjects 213 - iii. Uncorrelated between, and within, subjects 213 - iv. Uncorrelated between, and autocorrelated within, subjects 213 - v. Correlated between, but not within, subjects 214 b. Unbalanced data 214 - i. Uncorrelated subjects 214 - ii. Uncorrelated between, and within, subjects 214 - iii. Correlated between, but not within, subjects 214 7.10 APPENDIX 215 a. For Section 7.4a 215 b. For Section 7.4b 215 c. For Section 7.4d 215 7.11 EXERCISES 218 8 GLMMs 220 8.1 INTRODUCTION 220 8.2 STRUCTURE OF THE MODEL 221 a. Conditional distribution of y 221 8.3 CONSEQUENCES OF HAVING RANDOM EFFECTS... 222 a. Marginal versus conditional distribution 222 b. Mean of y 222 c. Variances 223 d. Covariances and correlations 224 8.4 ESTIMATION BY MAXIMUM LIKELIHOOD 225 a. Likelihood 225 b. Likelihood equations 227 - i. For the fixed effects parameters 227 - ii. For the random effects parameters... 228 8.5 MARGINAL VERSUS CONDITIONAL MODELS 228 8.6 OTHER METHODS OF ESTIMATION 231 a. Generalized estimating equations 231 b. Penalized quasi-likelihood 232 c. Conditional likelihood 234 d. Simpler modeis 238 8.7 TESTS OF HYPOTHESES 239

CONTENTS xv a. Likelihood ratio tests 239 b. Asymptotic variances 240 c. Wald tests 240 d. Score tests 240 8.8 ILLUSTRATION: CHESTNUT LEAF BLIGHT 241 a. A random effects probit model 242 - i. The fixed effects 242 - ii. The random effects 243 - iii. Consequences of having random effects 243 - iv. Likelihood analysis 244 - v. Results 245 8.9 EXERCISES 246 9 PREDICTION 247 9.1 INTRODUCTION 247 9.2 BEST PREDICTION (BP) 248 a. The best predictor 248 b. Mean and variance properties 249 c. A correlation property 249 d. Maximizing a mean 249 e. Normality 250 9.3 BEST LINEAR PREDICTION (BLP) 250 a. BLP(u) 250 b. Example 251 c. Derivation 252 d. Ranking 253 9.4 LINEAR MIXED MODEL PREDICTION (BLUP) 254 a. BLUE(Xß) 254 b. BLUP(t'X + s'u) 255 c. Two variances 256 d. Other derivations 256 9.5 REQUIRED ASSUMPTIONS 256 9.6 ESTIMATED BEST PREDICTION 257 9.7 HENDERSON'S MIXED MODEL EQUATIONS 258 a. Origin 258 b. Solutions 259 c. Use in ML estimation of variance components.. 259 - i. ML estimation 259 - ii. REML estimation 260 9.8 APPENDIX 260

xvi CONTENTS a. Verification of (9.5) 260 b. Verification of (9.7) and (9.8) 261 9.9 EXERCISES 262 10 COMPUTING 263 10.1 INTRODUCTION 263 10.2 COMPUTING ML ESTIMATES FOR LMMs 263 a. The EM algorithm 263 - i. EMfor ML 265 - ii. EM (a variant) for ML 265 -in. EMforREML 265 b. UsingE[u y] 266 c. Newton-Raphson method 267 10.3 COMPUTING ML ESTIMATES FOR GLMMs 269 a. Numerical quadrature 269 - i. Gauss-Hermite quadrature 270 - ii. Likelihood calculations 272 - iii. Limits of numerical quadrature 273 b. EM algorithm 274 c. Markov chain Monte Carlo algorithms 275 - i. Metropolis 276 - ii. Monte Carlo Newton-Raphson 277 d. Stochastic approximation algorithms 278 e. Simulated maximum likelihood 280 10.4 PENALIZED QUASI-LIKELIHOOD AND LAPLACE... 281 10.5 EXERCISES 284 11 NONLINEAR MODELS 286 11.1 INTRODUCTION 286 11.2 EXAMPLE: CORN PHOTOSYNTHESIS 286 11.3 PHARMACOKINETIC MODELS 289 11.4 COMPUTATIONS FOR NONLINEAR MIXED MODELS. 290 11.5 EXERCISES 290 APPENDIX M: SOME MATRIX RESULTS 291 M.l VECTORS AND MATRICES OF ONES 291 M.2 KRONECKER (OR DIRECT) PRODUCTS 292 M.3 A MATRIX NOTATION 292 M.4 GENERALIZED INVERSES 293 a. Definition 293

CONTENTS xvii b. Generalized inverses of X'X 294 c. Two results involving XCX'V-^J-X'V- 1... 295 d. Solving linear equations 296 e. Rank results 296 f. Vectors orthogonal to columns of X 296 g. A theorem for K' with K'X being null 296 M.5 DIFFERENTIAL CALCULUS 297 a. Definition 297 b. Sealars 297 c. Vectors 297 d. Inner produets 297 e. Quadratic forms 298 f. Inverse matrices 298 g. Determinants 299 APPENDIX S: SOME STATISTICAL RESULTS 300 5.1 MOMENTS 300 a. Conditional moments 300 b. Mean of a quadratic form 301 c. Moment generating funetion 301 5.2 NORMAL DISTRIBUTIONS 302 a. Univariate 302 b. Multivariate 302 c. Quadratic forms in normal variables 303 - i. The non-central x 2 303 - ii. Properties of y'ay when y ~ Af(ß, V) 303 5.3 EXPONENTIAL FAMILIES 304 5.4 MAXIMUM LIKELIHOOD 304 a. The likelihood funetion 304 b. Maximum likelihood estimation 305 c. Asymptotic variance-covariance matrix 305 d. Asymptotic distribution of MLEs 306 5.5 LIKELIHOOD RATIO TESTS 306 5.6 MLE UNDER NORMALITY 307 a. Estimation of ß 307 b. Estimation of variance components 308 c. Asymptotic variance-covariance matrix 308 d. Restricted maximum likelihood (REML) 309 - i. Estimation 309 - ii. Asymptotic variance 310

xec TTC SXM3XN0D xaami saonanadiah!!! AX