Chapter 5 Computational Methods for Mixed Models

Transcription

1 Chapter 5 Comptational Methods for Mixed Models In this chapter we describe some of the details of the comptational methods for fitting linear mixed models, as implemented in the lme4 package, and the theoretical development behind these methods. We also provide the basis for later generalizations to models for non-gassian responses and to models in which the relationship between the conditional mean, µ, and the linear predictor, γ = Xβ + Zb = ZΛ + Xβ, is a nonlinear relationship. This material is directed at those readers who wish to follow the theory and methodology of linear mixed models and how both can be extended to other forms of mixed models. Readers who are less interested in the how and the why of fitting mixed models than in the reslts themselves shold not feel obligated to master these details. We begin by reviewing the definition of linear mixed-effects models and some of the basics of the comptational methods, as given in Sect Definitions and Basic Reslts As described in Sect. 1.1, a linear mixed-effects model is based on two vectorvaled random variables: the q-dimensional vector of random effects, B, and the n-dimensional response vector, Y. Eqation (1.1) defines the nconditional distribtion of B and the conditional distribtion of Y, given B = b, as mltivariate Gassian distribtions of the form (Y B = b) N (Xβ + Zb,σ 2 I) B N (0,Σ ). The q q, symmetric, variance-covariance matrix, Var(B) = Σ, depends on the variance-component parameter vector,, and is positive semidefinite, which means that b T Σ b 0, b 0. (5.1) 83

2 84 5 Comptational Methods for Mixed Models (The symbol denotes for all.) The fact that Σ is positive semidefinite does not garantee that Σ 1 exists. We wold need a stronger property, b T Σ b > 0, b 0, called positive definiteness, to ensre that Σ 1 exists. Many comptational formlas for linear mixed models are written in terms of Σ 1. Sch formlas will become nstable as Σ approaches singlarity. And it can do so. It is a fact that singlar (i.e. non-invertible) Σ can and do occr in practice, as we have seen in some of the examples in earlier chapters. Moreover, dring the corse of the nmerical optimization by which the parameter estimates are determined, it is freqently the case that the deviance or the REML criterion will need to be evalated at vales of that prodce a singlar Σ. Becase of this we will take care to se comptational methods that can be applied even when Σ is singlar and are stable as Σ approaches singlarity. As defined in (1.2) a relative covariance factor, Λ, is any matrix that satisfies Σ = σ 2 Λ Λ T. According to this definition, Σ depends on both σ and and we shold write it as Σ σ,. However, we will blr that distinction and contine to write Var(B) = Σ. Another technicality is that the common scale parameter, σ, can, in theory, be zero. We will show that in practice the only way for its estimate, σ, to be zero is for the fitted vales from the fixed-effects only, X β, to be exactly eqal to the observed data. This occrs only with data that have been (incorrectly) simlated withot error. In practice we can safely assme that σ > 0. However, Λ, like Σ, can be singlar. Or comptational methods are based on Λ and do not reqire evalation of Σ. In fact, Σ is explicitly evalated only at the converged parameter estimates. The spherical random effects, U N (0,σ 2 I q ), determine B as B = Λ U. (5.2) Althogh it may seem more intitive to write U as a linear transformation of B, we cannot do that when Λ is singlar, which is why (5.2) is in the form shown. We can easily verify that (5.2) provides the desired distribtion for B. As a linear transformation of a mltivariate Gassian random variable, B will also be mltivariate Gassian. Its mean and variance-covariance matrix are straightforward to evalate, and E[B] = Λ E[U ] = Λ 0 = 0 (5.3) Page: 84 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

3 5.2 The Conditional Distribtion (U Y = y) 85 [ Var(B) = E (B E[B])(B E[B]) T] = E [BB T] (5.4) [ ] = E Λ U U T Λ T = Λ E[U U T ]Λ T = Λ Var(U )Λ T = Λ σ 2 I q Λ T = σ 2 Λ Λ T = Σ and have the desired form. Jst as we concentrate on how determines Λ, not Σ, we will concentrate on properties of U rather than B. In particlar, we now define the model according to the distribtions (Y U = ) N (ZΛ + Xβ,σ 2 I n ) U N (0,σ 2 I q ). (5.5) To allow for extensions to other types of mixed models we distingish between the linear predictor γ = ZΛ + Xβ (5.6) and the conditional mean of Y, given U =, which is µ = E[Y U = ]. (5.7) For a linear mixed model µ = γ. In other forms of mixed models the conditional mean, µ, can be a nonlinear fnction of the linear predictor, γ. For some models the dimension of γ is a mltiple of n, the dimension of µ and y, bt for a linear mixed model the dimension of γ mst be n. Hence, the model matrix Z mst be n q and X mst be n p. 5.2 The Conditional Distribtion (U Y = y) In this chapter it will help to be able to distingish between the observed response vector and an arbitrary vale of Y. For this chapter only we will write the observed data vector as y obs, with the nderstanding that y withot the sbscript will refer to an arbitrary vale of the random variable Y. The likelihood of the parameters,, β, and σ, given the observed data, y obs, is the probability density of Y, evalated at y obs. Althogh the nmerical vales of the probability density and the likelihood are identical, the interpretations of these fnctions are different. In the density we consider the parameters to be fixed and the vale of y as varying. In the likelihood we consider y to be fixed at y obs and the parameters,, β and σ, as varying. The natral approach for evalating the likelihood is to determine the marginal distribtion of Y, which in this case amonts to determining the marginal density of Y, and evalate that density at y obs. To follow this corse Page: 85 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

4 86 5 Comptational Methods for Mixed Models we wold first determine the joint density of U and Y, written f U,Y (,y), then integrate this density with respect to create the marginal density, f Y (y), and finally evalate this marginal density at y obs. To allow for later generalizations we will change the order of these steps slightly. We evalate the joint density fnction, f U,Y (,y), at y obs, prodcing the nnormalized conditional density, h(). We say that h is nnormalized becase the conditional density is a mltiple of h f U Y ( y obs ) = R h() q h()d. (5.8) In some theoretical developments the normalizing constant, which is the integral in the denominator of an expression like (5.8), is not of interest. Here it is of interest becase the normalizing constant is exactly the likelihood that we wish to evalate, L(,β,σ y obs ) = h()d. (5.9) Rq For a linear mixed model, where all the distribtions of interest are mltivariate Gassian and the conditional mean, µ, is a linear fnction of both and β, the distinction between evalating the joint density at y obs to prodce h() then integrating with respect to, as opposed to first integrating the joint density then evalating at y obs is not terribly important. For other mixed models this distinction can be important. In particlar, generalized linear mixed models, described in Sect.??, are often sed to model a discrete response, sch as a binary response or a cont, leading to a joint distribtion for Y and U that is discrete with respect to one variable, y, and continos with respect to the other,. In sch cases there isn t a joint density for Y and U. The necessary distribtion theory for general y and is welldefined bt somewhat awkward to describe. It is mch easier to realize that we are only interested in the observed response vector, y obs, not some arbitrary vale of y, so we can concentrate on the conditional distribtion of U given Y = y obs. For all the mixed models we will consider, the conditional distribtion, (U Y = y obs ), is continos and both the conditional density, f U Y ( y obs ), and its nnormalized form, h(), are well-defined. 5.3 Integrating h() in the Linear Mixed Model The integral defining the likelihood in (5.9) has a closed form in the case of a linear mixed model bt not for some of the more general forms of mixed models. To motivate methods for approximating the likelihood in more general sitations, we describe in some detail how the integral can be evalated sing the sparse Cholesky factor, L, and the conditional mode, Page: 86 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

5 5.3 Integrating h() in the Linear Mixed Model 87 ũ = argmax f U Y ( y obs ) = argmax h() = argmax f Y U (y obs ) f U () (5.10) The notation argmax means that ũ is the vale of that maximizes the expression that follows. In general, the mode of a continos distribtion is the vale of the random variable that maximizes the density. The vale ũ is called the conditional mode of, given Y = y obs, becase ũ maximizes the conditional density of U given Y = y obs. The location of the maximm can be determined by maximizing the nnormalized conditional density becase h() is jst a constant mltiple of f U Y ( y obs ). The last part of (5.10) is simply a re-expression of h() as the prodct of f Y U (y obs ) and f U (). For a linear mixed model these densities are ( ) 1 f Y U (y ) = y Xβ ZΛ 2 (5.11) with prodct 1 h() = (2πσ 2 ) (2πσ 2 ) 1 f U () = (2πσ 2 ) (n+q)/2 exp On the deviance scale we have ( n/2 exp q/2 exp ) ( 2 2σ 2 2σ 2 y obs Xβ ZΛ σ 2 ) (5.12). (5.13) 2log(h()) = (n + q)log(2πσ 2 ) + y obs Xβ ZΛ σ 2. (5.14) Becase (5.14) describes the negative log density, ũ will be the vale of that minimizes the expression on the right of (5.14). The only part of the right hand side of (5.14) that depends on is the nmerator of the second term. Ths ũ = argmin y obs Xβ ZΛ (5.15) The expression to be minimized, called the objective fnction, is described as a penalized residal sm of sqares (PRSS) and the minimizer, ũ, is called the penalized least sqares (PLS) soltion. They are given these names becase the first term in the objective, y obs Xβ ZΛ 2, is a sm of sqared residals, and the second term, 2, is a penalty on the length,, of. Larger vales of (in the sense of greater lengths as vectors) incr a higher penalty. The PRSS criterion determining the conditional mode balances fidelity to the observed data (i.e. prodcing a small residal sm of sqares) against simplicity of the model (small ). We refer to this type of criterion as Page: 87 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

6 88 5 Comptational Methods for Mixed Models a smoothing objective, in the sense that it seeks to smooth ot the fitted response by redcing model complexity while still retaining reasonable fidelity to the observed data. For the prpose of evalating the likelihood we will regard the PRSS criterion as a fnction of the parameters, given the data, and write its minimm vale as r,β 2 = min y obs Xβ ZΛ (5.16) Notice that β only enters the right hand side of (5.16) throgh the linear predictor expression. We will see that ũ can be determined by a direct (i.e. non-iterative) calclation and, in fact, we can minimize the PRSS criterion with respect to and β simltaneosly withot iterating. We write this minimm vale as r 2 = min,β y obs Xβ ZΛ (5.17) The vale of β at the minimm is called the conditional estimate of β given, written β. 5.4 Determining the PLS Soltions, ũ and β One way of expressing a penalized least sqares problem like (5.16) is by incorporating the penalty as psedo-data in an ordinary least sqares problem. We extend the response vector, which is y obs Xβ when we minimize with respect to only, with q responses that are 0 and we extend the predictor expression, ZΛ with I q. Writing this as a least sqares problem prodces ũ = argmin [ yobs Xβ 0 ] [ ] ZΛ I q 2 (5.18) with a soltion that satisfies (Λ T ZT ZΛ + I q ) ũ = Λ T ZT (y obs Xβ) (5.19) To evalate ũ we form the sparse Cholesky factor, L, which is a lower trianglar q q matrix that satisfies L L T = Λ T ZT ZΛ + I q. (5.20) The actal evalation of sparse Cholesky factor, L, often incorporates a fill-redcing permtation, which we describe next. Page: 88 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

7 5.4 Determining the PLS Soltions, ũ and β The Fill-redcing Permtation, P In earlier chapters we have seen that often the random effects vector is reordered before L is created. The re-ordering or permtation of the elements of and, correspondingly, the colmns of the model matrix, ZΛ, does not affect the theory of linear mixed models bt can have a profond effect on the time and storage reqired to evalate L in large problems. We write the effect of the permtation as mltiplication by a q q permtation matrix, P, althogh in practice we apply the permtation withot ever constrcting P. That is, the matrix P is only a notational convenience only. The matrix P consists of permted colmns of the identity matrix, I q, and it is easy to establish that the inverse permtation corresponds to mltiplication by P T. Becase mltiplication by P or by P T simply re-orders the components of a vector, the length of the vector is nchanged. Ths, P 2 = 2 = P T 2 (5.21) and we can express the penalty in (5.17) in any of these three forms. The properties of P that it preserves lengths of vectors and that its transpose is its inverse are smmarized by stating that P is an orthogonal matrix. The permtation represented by P is determined from the strctre of Λ T ZT ZΛ + I q for some initial vale of. The particlar vale of does not affect the reslt becase the permtation depends only the positions of the non-zeros, not the nmerical vales at these positions. Taking into accont the permtation, the sparse Cholesky factor, L, is defined to be the sparse, lower trianglar, q q matrix with positive diagonal elements satisfying L L T = P ( Λ T ZT ZΛ + I q )P T. (5.22) Note that we now reqire that the diagonal elements of Λ be positive. Problems 5.1 and 5.2 indicate why we can reqire this. Becase the diagonal elements of Λ are positive, its determinant, Λ, which, for a trianglar matrix sch as Λ, is simply the prodct of its diagonal elements, is also positive. Many sparse matrix methods, inclding the sparse Cholesky decomposition, are performed in two stages: the symbolic phase in which the locations of the non-zeros in the reslt are determined and the nmeric phase in which the nmeric vales at these positions are evalated. The symbolic phase for the decomposition (5.22), which incldes determining the permtation, P, need only be done once. Evalation of L for sbseqent vales of reqires only the nmeric phase, which typically is mch faster than the symbolic phase. The permtation, P, serves two prposes. The first and most important prpose is to redce the nmber of non-zeros in the factor, L. The factor is potentially non-zero at every non-zero location in the lower triangle of the Page: 89 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

8 90 5 Comptational Methods for Mixed Models matrix being decomposed. However, as we saw in Fig. 2.4 of Sect , there may be positions in the factor that get filled-in even thogh they are known to be zero in the matrix being decomposed. The fill-redcing permtation is chosen according to certain heristics to redce the amont of fill-in. We se the approximate minimal degree (AMD) method described in Davis [1996]. After the fill-redcing permtation is determined, a post-ordering is applied. This has the effect of concentrating the non-zeros near the diagonal of the factor. See Davis [2006] for details. The psedo-data representation of the PLS problem, (5.18), becomes ũ = argmin [ yobs Xβ 0 ] [ ZΛ P T ] P P T 2 (5.23) and the system of linear eqations satisfied by ũ is L L T Pũ = P ( Λ T ZT ZΛ + I q )P T Pũ = PΛ T ZT (y obs Xβ). (5.24) Obtaining the Cholesky factor, L, may not seem to be great progress toward determining ũ becase we still mst solve (5.24) for ũ. However, it is the key to the comptational methods in the lme4 package. The ability to evalate L rapidly for many different vales of is what makes the comptational methods in lme4 feasible, even when applied to very large data sets with complex strctre. Once we evalate L it is straightforward to solve (5.24) for ũ becase L is trianglar. In Sect. 5.6 we will describe the steps in determining this soltion. First, thogh, we shold show that the soltion, ũ, and the vale of the objective at the soltion, r,β 2, do allow s to evalate the deviance The Vale of the Deviance and Profiled Deviance After evalating L and sing that to solve for ũ, which also prodces r 2 β,, we can write the PRSS for a general as y obs Xβ ZΛ = r 2,β + LT ( ũ) 2 (5.25) which finally allows s to evalate the likelihood. We plg the right hand side of (5.25) into the definition of h() and apply the change of variable z = LT The determinant of the Jacobian of this transformation, ( ũ). (5.26) σ Page: 90 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

9 5.4 Determining the PLS Soltions, ũ and β 91 dz d = L T σ = L σ q (5.27) is reqired for the change of variable in the integral. We se the letter z for the transformed vale becase we will rearrange the integral to have the form of the integral of the density of the standard mltivariate normal distribtion. That is, we will se the reslt Rq e z 2 /2 dz = 1. (5.28) (2π) q/2 Ptting all these pieces together gives L(,β,σ) = h()d R q ( 1 = R q (2πσ 2 exp r2,β + ) LT ( ũ) 2 )(n+q)/2 2σ 2 d ( ) exp r2,β ( 2σ 2 1 = (2πσ 2 ) n/2 exp LT R q (2π) q/2 ( ) exp r2,β 2σ 2 = (2πσ 2 ) n/2 L ( ) exp r2,β 2σ 2 = (2πσ 2 ) n/2 L. Rq e z 2 /2 dz (2π) q/2 ) ( ũ) 2 L d 2σ 2 L σ q (5.29) The deviance can now be expressed as d(,β,σ y obs ) = 2log(L(,β,σ y obs )) = nlog(2πσ 2 ) + 2log L + r2 β, σ 2, as stated in (1.6). The maximm likelihood estimates of the parameters are those that minimize this deviance. Eqation (1.6) is a remarkably compact expression, considering that the class of models to which it applies is very large indeed. However, we can do better than this if we notice that β affects (1.6) only throgh r 2 β,, and, for any vale of, minimizing this expression with respect to β is jst an extension of the penalized least sqares problem. Let β be the vale of β that minimizes the PRSS simltaneosly with respect to β and and let r 2 be the PRSS at these minimizing vales. If, in addition, we set σ 2 = r 2 /n, which is the vale of σ 2 that minimizes the deviance for a given vale of r 2, then the profiled deviance, which is a fnction of only, becomes Page: 91 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

10 92 5 Comptational Methods for Mixed Models [ ( d( y 2πr 2 )] obs ) = 2log L + n 1 + log. (5.30) n Nmerical optimization (minimization) of d( y obs ) with respect to determines the MLE,. The MLEs for the other parameters, β and σ, are the corresponding conditional estimates evalated at Determining r 2 and β To determine ũ and β simltaneosly we rearrange the terms in (5.23) as [ ] ũ = argmin β,β [ yobs 0 ] [ ZΛ P T X P T 0 ][ P β ] 2. (5.31) The PLS vales, ũ and β, are the soltions to [ ( P Λ T Z T ) ZΛ + I q P T PΛ TZT X X T ZΛ P T X T X ][ Pũ β ] = [ PΛ T Z T ] y obs X T y obs. To evalate these soltions we decompose the system matrix as [ ( P Λ T Z T ) ZΛ + I q P T PΛ T ] [ ][ ] ZT X L 0 L T X T ZΛ P T X T = R ZX X 0 R X R T ZX RT X (5.32) (5.33) where, as before, L, the sparse Cholesky factor, is the sparse lower trianglar q q matrix satisfying (5.22). The other two matrices in (5.33): R ZX, which is a general q p matrix, and R X, which is an pper trianglar p p matrix, satisfy L R ZX = PΛ T ZT X (5.34) and R T X R X = X T X R T ZXR ZX (5.35) Those familiar with standard ways of writing a Cholesky decomposition as either LL T or R T R (L is the factor as it appears on the left and R is as it appears on the right) will notice a notational inconsistency in (5.33). One Cholesky factor is defined as the lower trianglar fractor on the left and the other is defined as the pper trianglar factor on the right. It happens that in R the Cholesky factor of a dense positive-definite matrix is retrned as the right factor, whereas the sparse Cholesky factor is retrned as the left factor. One other technical point that shold be addressed is whether X T X R T ZX R ZX is positive definite. In theory, if X has fll colmn rank, so that X T X is positive definite, then the downdated matrix, X T X R T ZX R ZX, mst also be positive definite (see Prob. 5.4). In practice, the downdated matrix can Page: 92 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

11 5.5 The REML Criterion 93 become comptationally singlar in ill-conditioned problems, in which case an error is reported. The extended decomposition (5.33) not only provides for the evalation of the profiled deviance fnction, d(), (5.30) bt also allows s to define and evalate the profiled REML criterion. 5.5 The REML Criterion The so-called REML estimates of variance components are often preferred to the maximm likelihood estimates. ( REML can be considered to be an acronym for restricted or residal maximm likelihood, althogh neither term is completely accrate becase these estimates do not maximize a likelihood.) We can motivate the se of the REML criterion by considering a linear regression model, Y N (Xβ,σ 2 I n ), (5.36) in which we typically estimate σ 2 as σ R 2 = y obs X β 2 n p (5.37) even thogh the maximm likelihood estimate of σ 2 is σ L 2 = y obs X β 2. (5.38) n The argment for preferring σ 2 R to σ 2 L as an estimate of σ 2 is that the nmerator in both estimates is the sm of sqared residals at β and, althogh the residal vector, y obs X β, is an n-dimensional vector, the residal at satisfies p linearly independent constraints, X T (y obs X β) = 0. That is, the residal at is the projection of the observed response vector, y obs, into an (n p)-dimensional linear sbspace of the n-dimensional response space. The estimate σ 2 R takes into accont the fact that σ 2 is estimated from residals that have only n p degrees of freedom. Another argment often pt forward for REML estimation is that σ R 2 is an nbiased estimate of σ 2, in the sense that the expected vale of the estimator is eqal to the vale of the parameter. However, determining the expected vale of an estimator involves integrating with respect to the density of the estimator and we have seen that densities of estimators of variances will be skewed, often highly skewed. It is not clear why we shold be interested in the expected vale of a highly skewed estimator. If we were to transform to a more symmetric scale, sch as the estimator of the standard deviation or the estimator of the logarithm of the standard deviation, the REML estimator Page: 93 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

12 94 5 Comptational Methods for Mixed Models wold no longer be nbiased. Frthermore, this property of nbiasedness of variance estimators does not generalize from the linear regression model to linear mixed models. This is all to say that the distinction between REML and ML estimates of variances and variance components is probably less important that many people believe. Nevertheless it is worthwhile seeing how the comptational techniqes described in this chapter apply to the REML criterion becase the REML parameter estimates R and σ R 2 for a linear mixed model have the property that they wold specialize to σ R 2 from (5.37) for a linear regression model, as seen in Sect Althogh not sally derived in this way, the REML criterion (on the deviance scale) can be expressed as d R (,σ y obs ) = 2log L(,β,σ y obs)dβ. R p (5.39) The REML estimates R and σ 2 R minimize d R(,σ y obs ). To evalate this integral we form an expansion, similar to (5.25), of r 2,β abot β r 2,β = r2 + R X(β β ) 2. (5.40) In the same way that (5.25) was sed to simplify the integral in (5.29), we can derive ( ) ( ) exp r2,β exp r2 2σ 2 R p (2πσ 2 ) n/2 L dβ = 2σ 2 (2πσ 2 ) (n p)/2 (5.41) L R X corresponding to a REML criterion on the deviance scale of d R (,σ y obs ) = (n p)log(2πσ 2 ) + 2log( L R X ) + r2 σ 2. (5.42) Plgging in the conditional REML estimate, σ 2 R = r 2 /(n p), provides the profiled REML criterion [ ( d 2πr 2 )] R ( y obs ) = 2log( L R X ) + (n p) 1 + log n p The REML estimate of is R = argmin (5.43) d R ( y obs ), (5.44) and the REML estimate of σ 2 is the conditional REML estimate of σ 2 at R, σ 2 R = r2 R /(n p). (5.45) Page: 94 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

13 5.6 Step-by-step Evalation of the Profiled Deviance 95 It is not entirely clear how one wold define a REML estimate of β becase the REML criterion, d R (,σ y), defined in (5.42), does not depend on β. However, it is cstomary (and not nreasonable) to se β R = β as the R REML estimate of β. 5.6 Step-by-step Evalation of the Profiled Deviance As we have seen, an object retrned by lmer contains an environment, accessed with the env extractor. This environment contains several matrices and vectors that are sed in the evalation of the profiled deviance. In this section we se these matrices and vectors from one of or examples to explicitly trace the steps in evalating the profiled deviance. This level of detail is provided for those whose style of learning is more of a hands on style and for those who may want to program modifications of this approach. Consider or model fm8, fit as > fm8 <- lmer(reaction ~ 1 + Days + (1 + Days Sbject), sleepstdy, + REML = 0, verbose = TRUE) 0: : : : : : : : The environment of the model contains the converged parameter vector, (theta), the relative covariance factor, Λ (Lambda), the sparse Cholesky factor, L (L), the matrices R ZX (RZX) and R X (RX), the conditional mode, ũ (), and the conditional estimate, β (fixef). The permtation represented by P is contained in the sparse Cholesky representation, L. Althogh the model matrices, X (X) and Z T (Zt), and the response vector, y obs (y), are available in the environment, many of the prodcts that involve only these fixed vales are precompted and stored separately nder the names XtX (X T X), Xty, ZtX and Zty. To provide easy access to the objects in the environment of fm8 we attach it to the search path. > attach(env(fm8)) Please note that this is done here for illstration only. The practice of attaching a list or a data frame or, less commonly, an environment in an R session is oversed, somewhat dangeros (becase of the potential of forgetting to detach it later) and discoraged. The preferred practice is to se the with fnction to gain access by name to components of sch composite objects. For this section of code, however, sing with or within wold qickly become very tedios and we se attach instead. Page: 95 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

14 96 5 Comptational Methods for Mixed Models To pdate the matrix Λ to a new vale of we need to know which of the non-zeros in Λ are pdated from which elements of. Recall that the dimension of is small (3, in this case) bt Λ is potentially large (18 18 with 54 non-zeros). The environment contains an integer vector Lind that maps the elements of theta to the non-zeros in Lambda. Sppose we wish to recreate the evalation of the profiled deviance at the initial vale of = (1,0,1). We begin by pdating Λ and forming the prodct U T = Λ T ZT > str(lambda) Formal class 'dgcmatrix' [package "Matrix"] with 6 slots..@ i : int [1:54] @ p : int [1:37] @ Dim : int [1:2] @ Dimnames:List of 2....$ : NULL....$ : NULL..@ x : nm [1:54] @ factors : list() > str(lind) int [1:54] > Lambda@x[] <- c(1,0,1)[lind] > str(lambda@x) nm [1:54] > Ut <- crossprod(lambda, Zt) The Cholesky factor object, L, can be pdated from Ut withot forming U T U+ I explicitly. The optional argment mlt to the pdate method specifies a mltiple of the identity to be added to U T U > L <- pdate(l, Ut, mlt = 1) Then we evalate RZX and RX according to (5.34) and (5.35) > RZX <- solve(l, solve(l, crossprod(lambda, ZtX), sys = "P"), sys = "L") > RX <- chol(xtx - crossprod(rzx)) Solving (5.32) for ũ and β is done in stages. Writing c and c β for the intermediate reslts that satisfy [ ][ ] [ L 0 c PΛ T = Z T ] y obs c β X T (5.46) y obs. we evalate R T ZX RT X > c <- solve(l, solve(l, crossprod(lambda, Zty), sys = "P"), sys = "L") > cbeta <- solve(t(rx), Xty - crossprod(rzx, c)) Page: 96 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

15 5.7 Generalizing to Other Forms of Mixed Models 97 The next set of eqations to solve is [ ][ ] [ ] L T R ZX Pũ cu = 0 R X β c β (5.47) > fixef <- as.vector(solve(rx, cbeta)) > <- solve(l, solve(l, c - RZX %*% fixef, sys = "Lt"), sys = "Pt") We can now create the conditional mean, m, the penalized residal sm of sqares, prss, the logarithm of the sqare of the determinant of L, ldl2, and the profiled deviance, which, fortitosly, eqals the vale shown earlier. > m <- gamma <- as.vector(crossprod(ut, ) + X %*% fixef) > prss <- sm(c(y - m, as.vector())^2) > ldl2 <- 2 * as.vector(determinant(l)$mod) > (deviance <- ldl2 + nobs * (1 + log(2 * pi * prss/nobs))) [1] The last step is detach the environment of fm8 from the search list > detach() to avoid later name clashes. In terms of the calclations performed, these steps describe exactly the evalation of the profiled deviance in lmer. The actal fnction for evalating the deviance, accessible as fm8@setpars, is a slightly modified version of what is shown above. However, the modifications are only to avoid creating copies of potentially large objects and to allow for cases where the model matrix, X, is sparse. In practice, nless the optional argment compdev = FALSE is given, the profiled deviance is evalated in compiled code, providing a speed boost, bt the R code can be sed if desired. This allows for checking the reslts from the compiled code and can also be sed as a template for extending the comptational methods to other types of models. 5.7 Generalizing to Other Forms of Mixed Models In later chapters we cover the theory and practice of generalized linear mixed models (GLMMs), nonlinear mixed models (NLMMs) and generalized nonlinear mixed models (GNLMMs). Becase qite a bit of the theoretical and comptational methodology covered in this chapter extends to those models we will cover the common aspects here. Page: 97 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

16 98 5 Comptational Methods for Mixed Models Descriptions of the Model Forms We apply the name generalized to models in which the conditional distribtion, (Y U = ), is not reqired to be Gassian bt does preserve some of the properties of the spherical Gassian conditional distribtion (Y U = ) N (ZΛ + Xβ,σ 2 I n ) from the linear mixed model. In particlar, the components of Y are conditionally independent, given U =. Frthermore, affects the distribtion only throgh the conditional mean, which we will contine to write as µ, and it affects the conditional mean only throgh the linear predictor, γ = ZΛ + Xβ. Typically we do not have µ = γ, however. The elements of the linear predictor, γ, can be positive or negative or zero. Theoretically they can take on any vale between and. Bt many distribtional forms sed in GLMMs pt constraints on the vale of the mean. For example, the mean of a Bernolli random variable, modeling a binary response, mst be in the range 0 < µ < 1 and the mean of a Poisson random variable, modeling a cont, mst be positive. To achieve these constraints we write the conditional mean, µ as a transformation of the nbonded predictor, written η. For historical, and some theoretical, reasons the inverse of this transformation is called the link fnction, written η = g(µ), (5.48) and the transformation we want is called the inverse link, writted g 1. Both g and g 1 are determined by scalar fnctions, g and g 1, respectively, applied to the individal components of the vector argment. That is, η mst be n-dimensional and the vector-valed fnction µ = g 1 (η) is defined by the component fnctions µ i = g 1 (η i ), i = 1,...,n. Among other things, this means that the Jacobian matrix of the inverse link, dµ dη, will be diagonal. Becase the link fnction, g, and the inverse link, g 1, are nonlinear fnctions (there wold be no prpose in sing a linear link fnction) many people se the terms generalized linear mixed model and nonlinear mixed model interchangably. We reserve the term nonlinear mixed model for the type of models sed, for example, in pharmacokinetics and pharmacodynamics, where the conditional distribtion is a spherical mltivariate Gassian (Y U = ) N (µ,σ 2 I n ) (5.49) bt µ depends nonlinearly on γ. For NLMMs the length of the linear predictor, γ, is a mltiple, ns, of n, the length of µ. Like the map from η to µ, the map from γ to µ has a diagonal property, which we now describe. If we se γ to fill the colmns of an n s matrix, Γ, then µ i depends only on the ith row of Γ. In fact, µ i is determined by a Page: 98 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

17 5.7 Generalizing to Other Forms of Mixed Models 99 nonlinear model fnction, f, applied to the i row of Γ. Writing µ = f(γ) based on the component fnction f, we see that the Jacobian of f, dµ dγ, will be the vertical concatenation of s diagonal n n matrices. Becase we will allow for generalized nonlinear mixed models (GNLMMs), in which the mapping from γ to µ has the form we will se (5.50) in or definitions. γ η µ, (5.50) Determining the Conditional Mode, ũ For all these types of mixed models, the conditional distribtion, (U Y = y obs ), is a continos distribtion for which we can determine the nscaled conditional density, h(). As for linear mixed models, we define the conditional mode, ũ as the vale that maximizes the nscaled condtional density. Determining the conditional mode, ũ, in a nonlinear mixed model is a penalized nonlinear least sqares (PNLS) problem ũ = argmin y obs µ (5.51) which we solve by adapting the iterative techniqes, sch as the Gass- Newton method [Bates and Watts, 1988, Sect ], sed for nonlinear least sqares. Starting at an initial vale, (0), (the bracketed sperscript denotes the iteration nmber) with conditional mean, µ (0), we determine an increment δ (1) by solving the penalized linear least sqares problem, where δ (1) = argmin δ [ yobs µ (0) 0 (0) U (0) = dµ d (0) ] [ ] U (0) δ I q 2 (5.52). (5.53) Natrally, we se the sparse Cholesky decomposition, L (0), satisfying ( L (0) L (0) ) [ ( = P U (0)) ] T U (0) + I q P T (5.54) to determine this increment. The next iteration begins at (1) = (0) + kδ (1) (5.55) Page: 99 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

18 100 5 Comptational Methods for Mixed Models where k is the step factor chosen, perhaps by step-halving [Bates and Watts, 1988, Sect ], to ensre that the penalized residal sm of sqares decreases at each iteration. Convergence is declared when the orthogonality convergence criterion [Bates and Watts, 1988, Sect ] is below some prespecified tolerance. The Laplace approximation to the deviance is d(,β,σ y obs ) = nlog(2πσ 2 ) + 2log L,β + r2,β σ 2, (5.56) where the Cholesky factor, L,β, and the penalized residal sm of sqares, r,β 2, are both evalated at the conditional mode, ũ. The Cholesky factor depends on, β and for these models bt typically the dependence on β and is weak. 5.8 Chapter Smmary The definitions and the comptational reslts for maximm likelihood estimation of the parameters in linear mixed models were smmarized in Sect A key comptation is evalation of the sparse Cholesky factor, Λ, satisfying eqn. 5.22, L L T = P ( Λ T ZT ZΛ + I q )P T. where P represents the fill-redcing permtation determined dring the symbolic phase of the sparse Cholesky decomposition. An extended decomposition (eqn. 5.33) provides the q p matrix R ZX and the p p pper trianglar R X that are sed to determine the conditional mode ũ, the conditional estimate β and the minimm penalized residal sm of sqares, r 2 from which the profiled deviance [ ( d( y 2πr 2 )] obs ) = 2log L + n 1 + log. n or the profile REML criterion [ ( d 2πr 2 )] R ( y obs ) = 2log( L R X ) + (n p) 1 + log n p can be evalated and optimized (minimized) with respect to. Page: 100 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23

19 5.8 Chapter Smmary 101 Exercises Unlike the exercises in other chapters, these exercises establish theoretical reslts, which do not always apply exactly to the comptational reslts Show that the matrix A = PΛ T ZT ZΛ P T + I q is positive definite. That is, b T Ab > 0, b (a) Show that Λ can be defined to have non-negative diagonal elements. (Hint: Show that the prodct Λ D where D is a diagonal matrix with diagonal elements of ±1 is also a Cholesky factor. Ths the signs of the diagonal elements can be chosen however we want.) (b) Use the reslt of Prob. 5.1 to show that the diagonal elements of Λ mst be non-zero. (Hint: Sppose that the first zero on the diagonal of Λ is in the ith position. Show that there is a soltion x to Λ T x = 0 with x i = 1 and x j = 0, j = i + 1,...,q and that this x contradicts the positive definite condition.) 5.3. Show that if X has fll colmn rank, which means that there does not exist a β 0 for which Xβ = 0, then X T X is positive definite Show that if X has fll colmn rank then [ ZΛ P T ] X P T 0 also mst have [ ] fll colmn rank. (Hint: First show that mst be zero in any vector satisfying β [ ZΛ P T ][ ] X P T 0 β = 0. Use this reslt and (5.33) to show that [ ][ ] L 0 L T R ZX R T ZX RT X 0 R X is positive definite and, hence, R X is non-singlar.) Page: 101 job: lmmwr macro: svmono.cls date/time: 17-Feb-2010/14:23