Approximations to the Loglikelihood Function in the Nonlinear Mixed Effects Model

Transcription

1 Approxmatons to the Lolkelhood Functon n the Nonlnear Mxed Effects Model José C. Pnhero and Doulas M. Bates Department of Statstcs Unversty of Wsconsn, Madson Nonlnear mxed effects models have receved a reat deal of attenton n the statstcal lterature n recent years because of the flexblty they offer n handln unbalanced repeated measures data that arse n dfferent areas of nvestaton, such as pharmacoknetcs and economcs. Several dfferent methods for estmatn the parameters n nonlnear mxed effects model have been proposed. We concentrate here on two of them: maxmum lkelhood and restrcted maxmum lkelhood. A rather complex numercal ssue for (restrcted) maxmum lkelhood estmaton s the evaluaton of the lolkelhood functon of the data, snce t nvolves the evaluaton of a multple nteral that n most cases does not have a closed form expresson. We consder here four dfferent approxmatons to the lolkelhood, comparn ther computatonal and statstcal propertes. We conclude that the alternatn approxmaton suested by Lndstrom and Bates (1990), the Laplacan approxmaton, and Gaussan quadrature centered at the condtonal modes of the random effects are qute accurate and computatonally effcent. Gaussan quadrature centered at the expected value of the random effects s qute naccurate for a smaller number of abscssas and computatonally neffcent for a larer number of abscssas. Importance sampln s accurate but qute neffcent computatonally. Keywords: Nonlnear mxed effects models, maxmum lkelhood estmaton, Laplacan approxmaton, Gaussan quadrature, mportance sampln. 1 Introducton. Several dfferent nonlnear mxed effects models and estmaton methods for ther parameters have been proposed n recent years (Shener and Beal, 1980; Mallet, Mentre, Stemer and Lokek, 1988; Lndstrom and Bates, 1990; Vonesh and Carter, 199; Davdan and Gallant, 199; Wakefeld, Smth, Racne-Poon and Gelfand, 1994). We consder here a slhtly modfed verson of the model proposed n Lndstrom 1

2 and Bates (1990). Ths model can be vewed as a herarchcal model that n some ways eneralzes both the lnear mxed effects model of Lard and Ware (198) and the usual nonlnear model for ndependent data (Bates and Watts, 1988). In the frst stae the jth observaton on the th cluster s modeled as y j = f ( j ; x j ) + j ; = 1; : : :; M; j = 1; : : :; n (1) where f s a nonlnear functon of a cluster-specfc parameter vector j and the predctor vector x j, j s a normally dstrbuted nose term, M s the total number of clusters, and n s the number of observatons n the th cluster. In the second stae the cluster-specfc parameter vector s modeled as j = A j + B j b ; b N (0; D); where s a p-dmensonal vector of fxed populaton parameters, b s a q-dmensonal random effects vector assocated wth the th cluster (not varyn wth j), A j and B j are desn matrces for the fxed and random effects respectvely, and D s a (eneral) varance-covarance matrx. It s further assumed that observatons made on dfferent clusters are ndependent and that the j are..d. N (0; ) and ndependent of the b. We consder estmaton of the model s parameters by ether maxmum lkelhood, or restrcted maxmum lkelhood, based on the marnal densty of y Z p(y j ; D; ) = p(y j b; ; D; ) p(b) db () In eneral ths nteral does not have a closed-form expresson when the model functon f s nonlnear n b so dfferent approxmatons have been proposed for estmatn t. Some of these methods consst of takn a frst order Taylor expanson of the model functon f around the expected value of the random effects (Shener and Beal, 1980; Vonesh and Carter, 199), or around the condtonal (on D) modes of the random effects (Lndstrom and Bates, 1990). Others have proposed the use of Gaussan quadrature rules (Davdan and Gallant, 199). We consder here four dfferent approxmatons to the lolkelhood (): Lndstrom and Bates (1990) s alternatn method, a modfed Laplacan approxmaton (Terney and Kadane, 1986), mportance sampln (Geweke, 1989), and Gaussan quadrature (Davdan and Gallant, 199). We compare

3 them based on ther computatonal and statstcal propertes, usn both real data examples and smulaton results. Secton contans a descrpton of the dfferent approxmatons to the lolkelhood as appled to the nonlnear mxed effects model (1). Secton 3 presents a comparson of the dfferent approxmatons based on real and smulated data. Our conclusons and suestons for further nvestaton are ven n secton 4. Approxmatons to the Lolkelhood In ths secton we descrbe four dfferent approxmatons to the lolkelhood of y n the nonlnear mxed effects model (1). We show that there exsts a close relaton between the Laplacan approxmaton, mportance sampln and a Gaussan quadrature rule centered around the condtonal modes of the random effects b..1 Alternatn Approxmaton Lndstrom and Bates (1990) propose an alternatn alorthm for estmatn the parameters n model (1). Based on the current estmates of D (the scaled varance-covarance matrx of the random effects), the condtonal modes of the random effects b and the condtonal estmates of the fxed effects are obtaned by mnmzn a penalzed nonlnear least squares (PNLS) objectve functon where [f (; b )] j = f MX =1 ky? f (; b )k + b T D?1 b j ; x j ; = 1; : : :; M; j = 1; : : :; n. To update the estmate of D at the wth teraton, Lndstrom and Bates use a frst order Taylor expanson of the model functon around the current estmates of and the condtonal modes of the (w) (w) random effects b, whch we wll denote by b and b respectvely. Lettn T ; d X b;b T ; and bw (w) = y? f (b (w) ; b b (w) ) + c X (w) b (w) (w) + Z b bb (w) (3) 3

4 the approxmate lolkelhood used for the estmaton of D s? `A ; ; D j y =? 1 MX =1 lo +? bw (w) (w) (w) I + Z b D Z b (w)? X c T T I + b Z (w) (4) ) T (w) (w) D Z b? X c?1 bw (w) Ths lolkelhood s dentcal to that of a lnear mxed effects (LME) model (Lard and Ware, 198) n whch the response vector s ven by bw (w) and the fxed and random effects desn matrces are ven by c X (w) and b Z (w). Usn the results n Lndstrom and Bates (1988) one can express the optmal values of and as functons of D and work wth the profle lolkelhood of D, reatly smplfyn the optmzaton problem. Lndstrom and Bates (1990) have also proposed an approxmate restrcted lolkelhood for the estmaton of D? `RA ; ; D j y =? 1 MX =1 lo (w) c X T (w) (w) I + Z b D Z b T cx (w)? + `A ; ; D j y (5) Ther estmaton alorthm alternates between the PNLS and LME steps untl some converence crteron s met. Such alternatn alorthms tend to be more effcent when the estmates of the varancecovarance components (D and ) are not hhly correlated wth the estmates of the fxed effects (). Pnhero and Bates (1993) have demonstrated that, n the lnear mxed effects model, the (restrcted) maxmum lkelhood estmates of D and are asymptotcally ndependent of the (restrcted) maxmum lkelhood estmates of. These results have not yet been extended to the nonlnear mxed effects model (1). It can be shown that the maxmum lkelhood estmate of and the condtonal modes of the random effects b correspondn to the approxmate lolkelhood (4) are the values obtaned n the frst teraton of the Gauss-Newton alorthm used to mnmze the PNLS objectve functon (3). Therefore, at the convered value of c D, the estmates of and b obtaned from the LME and PNLS steps concde. We wll use `A when comparn the dfferent approxmatons at the optmal values n secton 3, but we do note that n Lndstrom and Bates (1990) approxmaton (4) s used only to update the estmates of D and not for estmatn. 4

5 . Laplacan Approxmaton Laplacan approxmatons are frequently used n Bayesan nference to estmate marnal posteror denstes and predctve dstrbutons (Terney and Kadane, 1986; Leonard, Hsu and Tsu, 1989). These technques can also be used for the nteraton consdered here. The nteral that we want to estmate for the marnal dstrbuton of y n model (1) can be wrtten as p(y j ; D; ) = Z??(n +q)= jdj?1= exp?(; D; y ; b )= db ; where (; D; y ; b ) = ky? f (; b )k + b T D?1 b Let bb = b b (; D; y ) = ar mn (; D; y ; b ) b 0 (; D; y ; b ) D; y ; b 00 (; D; y ; b ) (; D; y ; T and consder a second order Taylor expanson of around b b (; D; y ; b ) ' ; D; y ; b b + 1 h b? b b T 00 ; D; y ; b b hb? b b (6) where the lnear term of the approxmaton vanshes snce 0 (; D; y ; b b ) = 0. approxmaton s defned as p? y j ; D; '? "?N= jdj?m= exp? 1 Z? ( q= exp =??N= jdj?m= M Y =1? 1 MX =1 MX =1 ; D; y ; b b # h b? b b T ; D; y ; b b?1= exp The Laplacan ; D; y ; b b h b? b b ) db h? ; D; y ; b b = where N = P M =1 n. Now we consder an approxmaton to 00 smlar to the one used n Gauss-Newton optmzaton. 5

6 We have 00 ; D; y ; b b f (; T b=bb At b b, the contrbuton f (; b T that b )=@b T b=bb 00 ; D; y ; b b h y? f (; b b ) (; b T b ) + b=bb h y? f (; b b ) s usually nelble compared to (; b )=@b j b=bb (Bates and Watts, 1980) so we use the approxmaton ' G (; D; y ) (; b T (; b ) + b=bb Ths has the advantae of requrn only the frst order partal dervatves of the model functon wth respect to the random effects, whch are usually avalable from the estmaton of b b. Ths estmaton of bb s a penalzed least squares problem, for whch standard and relable code s avalable. The modfed Laplacan approxmaton to the lolkelhood of model (1) s then ven by? `LA ; D; j y = (? 1 N lo? + M lo jdj + MX =1 lo [G (; D; y )] +? M X =1 ; D; y ; b b ) (7) Snce b b does not depend upon, for ven and D the maxmum lkelhood estmate of (based upon `LA ) s b = b (; D; y) = MX =1 ; D; y ; b b =N We can profle `LA on to reduce the dmenson of the optmzaton problem, obtann `LAp =? 1 ( ) N 1 + lo () + lo? b + M lo jdj + M X =1 lo [G (; D; y )] (8) We note that f f s lnear n b then the modfed Laplacan approxmaton s exact because the second order Taylor expanson n (6) s exact when f (; b) = f () + Z () b.. There does not yet seem to be a strahtforward eneralzaton of the concept of restrcted maxmum lkelhood (Harvlle, 1974) to nonlnear mxed effects models. The dffculty s that restrcted maxmum lkelhood depends heavly upon the lnearty of the fxed effects n the model functon, whch does not occur n nonlnear models. Lndstrom and Bates (1990) crcumvented that problem by usn an 6

7 approxmaton to the model functon f n whch the fxed effects occur lnearly. Ths cannot be done for the Laplacan approxmaton, unless we consder yet another Taylor expanson of the model functon, what would lead us back to somethn very smlar to Lndstrom and Bates approach. We wll return to ths topc later n secton 4..3 Importance Sampln Importance sampln provdes a smple and effcent way of performn Monte Carlo nteraton. The crtcal step for the success of ths method s the choce of an mportance dstrbuton from whch the sample s drawn and the mportance wehts calculated. Ideally ths dstrbuton corresponds to the densty that we are tryn to nterate, but n practce one uses an easly sampled approxmaton. For the nonlnear mxed effects model the functon that we want to nterate s, up to a multplcatve constant, equal to exp [? (; D; y ; b ) = ]. As shown n subsecton., by takn a second order Taylor expanson of (; D; y ; b ) around b b the nterand s, up to a multplcatve constant, approxmately equal to a N bb ; [G(; D; y )]?1 densty. Ths ves us a natural choce for the mportance dstrbuton. Let N IS denote the number of mportance samples to be drawn. In practce one such sample can be enerated by selectn a vector z wth dstrbuton N (0; I) and calculatn the sample of random effects as b = b b + [G (; D; y )]?1= z, where [G (; D; y )]?1= denotes the nverse of the Cholesky factor of G (; D; y ). The mportance sampln approxmaton to the lolkelhood of y s then defned as? " `IS ; D; j y =? 1? N lo # MX + M lo jdj + lo jg (; D; y )j =1 8 9 MX < NX IS h = + lo exp? ; D; y : ; b j = + kz j k = =N IS ; =1 j=1 (9) Note that we cannot n eneral obtan a closed form expresson for the MLE of for fxed and D, so that profln on s no loner reasonable. As n the modfed Laplacan approxmaton, mportance sampln ves exact results when the 7

8 model functon s lnear n b because n ths case? p(y j b ; ; D; ) p(b ) = p y j ; D; N bb ; [G (; D; y?1 )] so that the mportance wehts are equal to p (y j ; D; )..4 Gaussan quadrature Gaussan quadrature s used to approxmate nterals of functons wth respect to a ven kernel by a wehted averae of the nterand evaluated at pre-determned abscssas. The wehts and abscssas used n Gaussan quadrature rules for the most common kernels can be obtaned from the tables of Abramowtz and Steun (1964) or by usn an alorthm proposed by Golub (1973) (see also Golub and Welsch (1969)). Gaussan quadrature rules for multple nterals are known to be numercally complex (Davs and Rabnowtz, 1984), but usn the structure of the nterand n the nonlnear mxed effects model we can transform the problem nto successve applcatons of smple one dmensonal Gaussan quadrature rules. Lettn z j ; w j j = 1; : : :; N GQ denote respectvely the abscssas and the wehts for the (one dmensonal) Gaussan quadrature rule wth N GQ ponts based on the N (0; 1) kernel, we et Z ( )?q= jdj?1= exp h? ky? f (; b )k = exp?b T D?1 b= Z = ' where z j 1;:::;j q = NX GQ ()?q= exp j 1=1 NX GQ j q=1 exp? y? f? y? f ; D T= z = ; D T= z j 1;:::;j q = exp? kz k = q Y k=1 w jk db (10) z j 1 ; : : :; z j q T. The correspondn approxmaton to the lolkelhood functon s dz? `GQ ; D; j y = (11) 8 MX < NX GQ 9? N lo( )= + lo exp? y :? f ; D T= z j Y q = = w jk ; =1 j k=1 where j = (j 1 ; : : :; j q ) T. The Gaussan quadrature rule n ths case can be vewed as a determnstc verson of Monte Carlo nteraton n whch random samples of b are enerated from the N (0; D) dstrbuton. The samples 8

9 (z j ) and the wehts (w j) are fxed beforehand, whle n Monte Carlo nteraton they are left to random choce. Snce mportance sampln tends to be much more effcent than smple Monte Carlo nteraton, we also consder the equvalent of mportance sampln n the Gaussan quadrature context, whch we wll denote by adaptve Gaussan quadrature. In ths approach the rd of abscssas n the b scale s centered around the condtonal modes b b rather than 0, as n (10). Another modfcaton s the use of G (; D; y ) nstead of D n the scaln of the z. The adaptve Gaussan quadrature s then ven by Z ( )?q= jdj?1= exp h? ky? f (; b )k = exp?b T D?1 b= Z = ' ()?q= jg (; D; y ) Dj?1= exp + kz k = exp? kz k = dz NX GQ j 1=1 NX GQ j q=1 exp?? db n; D; y ;b b + [G (; D; y )]?1= z o = n; D; y ; b b + [G (; D; y )]?1= z j 1;:::;j q o The correspondn approxmaton to the lolkelhood s then? `AGQ ; D; j y =? MX X + =1 lo 4 NGQ j " N lo? + M lo jdj + MX =1 = + lo jg (; D; y )j n exp? ; D; y ;b b + [G (; D; y )]?1= z j # o = + z Y q j 1;:::;j q = = (1) z j = qy The adaptve Gaussan quadrature approxmaton very closely resembles that obtaned for mportance sampln. The basc dfference s that the former uses fxed abscssas and wehts, whle the latter allows them to be determned by a pseudo-random mechansm. It s also nterestn to note that the one pont (.e. N GQ = 1) adaptve Gaussan quadrature approxmaton s smply the modfed Laplacan approxmaton (8), snce n ths case z 1 = 0 and w 1 = 1. The adaptve Gaussan quadrature also ves the exact lolkelhood when the model functon s lnear n b, but that s not true n eneral for the Gaussan quadrature approxmaton (10). Lke the mportance sampln approxmaton, the Gaussan quadrature approxmaton cannot be profled on to reduce the dmensonalty of the optmzaton problem. k=1 w jk 3 5 k=1 w jk 9

10 3 Comparn the Approxmatons In ths secton we present a comparson of the dfferent approxmatons to the lolkelhood of model (1) descrbed n secton. Two real data examples, the orane trees and Theophyllne data sets, and smulaton results are used to compare the statstcal and computatonal aspects of the varous approxmatons. 3.1 Orane Trees The data are presented on Fure 1 and consst of seven measurements of the trunk crcumference (n mllmeters) on each of fve orane trees, taken over a perod of 1600 days. These data were ornally presented n Draper and Smth (1981, p. 54) and were descrbed n Lndstrom and Bates (1990). Tree crcumference Days Fure 1: Trunk crcumference (n mllmeters) of fve orane trees: Data and ndvdual ftted curves from maxmum lkelhood estmaton usn the exact lolkelhood. The dashed lne represents the mean curve. The lostc model y = 1 = f1 + exp [? (t? ) = 3 ] seems to ft the data well. Lndstrom and Bates (1990) concluded n ther analyss that only the asymptotc crcumference 1 needs a random 10

11 effect to account for tree to tree varaton and suested the follown nonlnear mxed effects model y j = 1 + b exp [? (t j? ) = 3 ] + " j (13) where y j represents the jth crcumference measurement on the th tree, t j represents the day correspondn to the jth measurement on the th tree, b 1 ; = 1; : : :; 5 are..d. N (0; D), and " j ; = 1; : : :; 5; j = 1; : : :; 7 are..d. N (0; ) and ndependent of the b 1. Note that the snle random effect occurs lnearly n (13) and therefore the modfed Laplacan (8), the mportance sampln (9), and the adaptve Gaussan quadrature (1) approxmatons are all exact. Table 1 presents the results of estmaton usn the alternatn approxmaton, Gaussan quadrature wth 10 and 00 abscssas, and the exact lolkelhood. Snce only the alternatn approxmaton provdes a verson of restrcted maxmum lolkelhood, we wll just consder maxmum lkelhood estmaton n ths and the next subsecton. The subscrpt on Gaussan refers to the number of abscssas used n the approxmaton and the scalar L s p D, the square root of the scaled varance of the random effects. In eneral ths s a matrx but there s only one random effect here. Table 1: Estmaton Results Orane Trees Data Approxmaton lo(l) 1 3 lo( ) ` Alternatn Gaussan Gaussan Exact The estmaton results n Table 1 ndcate that the dfferent approxmatons produce smlar fts. The Gaussan approxmaton wth only 10 abscssas ves the worst approxmaton, n terms of the value of the lolkelhood, but even that s not far from the exact value. The Gaussan quadrature wth 00 abscssas s almost dentcal to the exact lolkelhood. The alternatn approxmaton s also very close to the exact value. Another mportant ssue reardn the dfferent approxmatons s how well they behave n a nehborhood of the optmal value, snce ths behavor s often used to assess the varablty of maxmum lkelhood estmates. Fure dsplays the profle traces and contours (Bates and Watts, 1988) for the exact lolkelhood and the alternatn approxmaton. Ths plot could not be obtaned for the Gaussan approxmaton because the objectve functon presented several local optma durn the profln alo- 11

12 rthm. We beleve that ths s related to the fact that the Gaussan approxmaton s centered at b = 0 and not at the condtonal modes of the random effects, where the nterand n () takes ts hhest values lo(l) lo(s) beta beta beta Fure : Profle traces and profle contour plots for the orane trees data based on the exact lolkelhood (sold lne) and the alternatn approxmaton (dashed lne). Plots below the daonal are n the ornal scale and plots above the daonal are n the zeta scale (Bates and Watts, 1988). Interpolated contours correspond approxmately to jont confdence levels of 68%, 87%, and 95%. It can be seen from Fure that the alternatn method ves a ood approxmaton to the lolkelhood n a nehborhood of the optmal values. It s nterestn to note that the profle traces for the varance-covarance components (D and ) and the fxed effects () meet almost perpendcularly. Ths ndcates a local uncorrelaton between the varance-covarance components and the fxed effects, whch explans why the alternatn method was so successful n approxmatn the lolkelhood. The same pattern was observed n several other data sets that we have analyzed, leadn us to conjecture that the asymptotc uncorrelaton between the estmators of the varance-covarance components and the fxed 1

13 effects verfed n the lnear mxed effects model also holds, at least approxmately, for the nonlnear mxed effects model. To compare the computatonal effcency of the dfferent approxmatons we consder the number of functon evaluatons needed untl converence. For the alternatn approxmaton there are two dfferent functons ben evaluated durn the teratons: the objectve functon (3) wthn the PNLS step and the approxmate lolkelhood `A (4) wthn the LME step. We wll use here the total number of evaluatons of ether (3) or `A, multpled by the number of clusters. For the other approxmatons we wll use the total number of calls to (; D; y ; b ). Even thouh the number of functon evaluatons used for the alternatn approxmaton s not drectly comparable to the number of functon evaluatons of the remann approxmatons, t ves a ood dea of the relatve computatonal effcency of ths alorthm. Table presents the number of functon evaluatons for the dfferent approxmatons n the orane trees example. The Gaussan quadrature approxmatons are consderably less effcent than ether the alternatn approxmaton or the exact lolkelhood. As expected the alternatn approxmaton s the most computatonally effcent. Table : Number of Functon Evaluatons to Converence Orane Trees Data Approxmaton Functon Evaluatons Alternatn 00 Exact 40 Gaussan 10 8,150 Gaussan , Theophyllne Knetcs The data consdered here are courtesy of Dr. Robert A. Upton of the Unversty of Calforna, San Francsco. Theophyllne was admnstered orally to 1 subjects whose serum concentratons were measured at 11 tmes over the next 5 hours. Ths s an example of a laboratory pharmacoknetc study characterzed by many observatons on a moderate number of ndvduals (clusters). Fure 3 dsplays the data and the ndvdual fts obtaned throuh maxmum lkelhood usn the adaptve Gaussan approxmaton wth 10 abscssas. A common model for such data s a frst order compartment model wth absorpton n a perpheral 13

14 Tme (hrs) Concentraton (m/l) a a a a a a a a a a a b b b b b b b b b b b c c c c c c c c c c c d d d d d d d d d d d e e e e e e e e e e e f f f f f f f f f f f h hh h h h h h h h h j j j j j j j j j j j k k k k k k k k k k k l l l l l l l l l l l Fure 3: Theophyllne concentratons (n m/l) of twelve patents: Data and ndvdual ftted curves from maxmum lkelhood estmaton usn the adaptve Gaussan approxmaton. compartment C t = DKk a Cl(k a? K) [exp (?Kt)? exp (?k at)] (14) where C t s the observed concentraton at tme t (m/l), t s the tme (hr), D s the dose (m/k), Cl s the clearance (L/k), K s the elmnaton rate constant (1/hr), and k a s the absorpton rate constant (1/hr). In order to ensure postvty of the rate constants and the clearance, the loarthms of these quanttes were used n the ft. Analyss of the Theophyllne data usn model (14) ndcated that only lo(cl) and lo(k a ) needed random effects to account for the patent-to-patent varablty. The nonlnear mxed effects model used for the Theophyllne data s C t = D exp [? ( 1 + b 1 ) + ( + b ) + 3 ] exp ( + b )? exp ( 3 ) fexp [? exp ( 3 ) t]? exp [? exp ( + b ) t] (15) Table 3 presents the estmaton results from the varous approxmatons to the lolkelhood. Only maxmum lkelhood estmaton s consdered. The subscrpts on Gaussan and on Adap. Gaussan 14

15 refer to the number of abscssas used n the Gaussan and adaptve Gaussan approxmatons, whle the subscrpt on Imp. Sampln refers to the number of mportance samples used n ths approxmaton. L denotes the vector wth elements ven by the upper tranular half of the Cholesky decomposton of D, stacked by columns. Table 3: Estmaton Results Theophyllne Data Approxmaton lo(l 1 ) L lo(l 3 ) 1 3 lo( ) ` Alternatn Laplacan Imp. Sampln Gaussan Gaussan Gaussan Adap. Gaussan Adap. Gaussan We can see from Table 3 that the alternatn approxmaton, the Laplacan approxmaton, the mportance sampln approxmaton, and the adaptve Gaussan approxmaton all ve smlar estmaton results. The Gaussan approxmaton only approaches the other approxmatons when the number of abscssas s ncreased consderably. Note that the actual number of ponts used n the rd that defnes the Gaussan approxmaton for ths example s the square of the number of abscssas. The adaptve Gaussan approxmatons for 1 (Laplacan), 5, and 10 abscssas ve smlar results, ndcatn that just a few ponts are needed for ths approxmaton to be accurate. The mportance sampln approxmaton caused some numercal dffcultes for the optmzaton alorthm (the ms() functon n S (Chambers and Haste, 199)) used to obtan the maxmum lkelhood estmates, snce the stochastc varablty assocated wth dfferent mportance samples overwhelmed the numercal varablty of the lolkelhood for small chanes n the parameter values (used to calculate numercal dervatves). We ended up havn to keep the random number enerator seed fxed durn the optmzaton process, thus usn the same mportance samples throuhout the calculatons. Snce the results obtaned usn mportance sampln were very smlar to those of the adaptve Gaussan approxmaton, we concluded that the latter s to be preferred for ts reater smplcty and computatonal effcency. Table 4 ves the number of functon evaluatons untl converence for the dfferent approxmatons. The alternatn approxmaton s the most effcent, followed by the Laplacan and adaptve Gaussan approxmatons. Gaussan quadrature wth 5 abscssas s effcent compared to the adaptve Gaussan, but s qute naccurate. The more relable Gaussan approxmaton wth 100 abscssas takes about

16 tmes more functon evaluatons than the adaptve Gaussan wth 10 abscssas. The mportance sampln approxmaton had the worst performance n terms of functon evaluatons. Table 4: Number of Functon Evaluatons to Converence Theophyllne Data Approxmaton Functon Evaluatons Alternatn 1,51 Laplacan 7,683 Adap. Gaussan 5 30,00 Adap. Gaussan 10 96,784 Gaussan 5 47,700 Gaussan ,000 Gaussan ,00,000 Imp. Sampln ,11,84 Next we consder the approxmatons n a nehborhood of the optmal value. We wll restrct ourselves here to the alternatn, the Laplacan, and the adaptve Gaussan approxmaton, as the Gaussan approxmaton for a moderate number of abscssas s not relable, and both the Gaussan approxmaton wth a larer number of abscssas and the mportance sampln approxmaton are very neffcent computatonally and ve results qute smlar to the adaptve Gaussan approxmaton. We used fve abscssas for the adaptve Gaussan quadrature, as ths ves rouhly the same precson as the ten-abscssa quadrature rule. The alternatn approxmaton ves results very smlar to the adaptve Gaussan quadrature. As n the orane trees example, the profle traces of the varance-covarance components and the fxed effects meet almost perpendcularly, ndcatn a local uncorrelaton between these estmates. The Laplacan and the adaptve Gaussan approxmatons ve vrtually dentcal plots (not ncluded here). Ths suests there s lttle to be aned by ncreasn the number of abscssas past one n the quadrature rule. The major an n precson s obtaned by centern the rd at the condtonal modes and scaln t usn the approxmate Hessan. 3.3 Smulaton Results In ths secton we nclude a comparson of the approxmatons to the lolkelhood n model (1) usn smulaton. We restrct ourselves to the alternatn, the Laplacan, and the (fve-abscssa) adaptve Gaussan approxmatons as these seem to be more accurate and/or more effcent than the Gaussan and the mportance sampln approxmatons. Two models were used n the smulaton analyss: a 16

17 lo(l1) L lo(l3) lo(s) lo(cl) lo(ka) lo(k) Fure 4: Profle traces and profle contour plots for the Theophyllne data based on the adaptve Gaussan approxmaton wth 5 abscssas (sold lne) and the alternatn approxmaton (dashed lne). Plots below the daonal are n the ornal scale and plots above the daonal are n the zeta scale (Bates and Watts, 1988). Interpolated contours correspond approxmately to jont confdence levels of 68%, 87%, and 95%. lostc model smlar to the one used for the orane trees data and a frst order open compartment model smlar to the one used for the Theophyllne example. For both models 1000 samples were enerated and maxmum lkelhood (ML) estmates based on the dfferent approxmatons obtaned. For the alternatn approxmaton, restrcted maxmum lkelhood (RML) estmates were also obtaned. 17

18 3.3.1 Lostc Model A lostc model smlar to (13), but wth two random effects nstead of one, was used to enerate the data. The model s ven by y j = 1 + b exp f? [t j? ( + b )] = 3 + " j; = 1; : : :; M; j = 1; : : :; n (16) where the b are..d. N (0; D), and the " j are..d. N (0; ) and ndependent of the b. We used M = 15, n = 10; ; : : :; 15, = 5, = (00; 700; 350) T, and D = 6 4 4?? 5 Table 5 summarzes the smulaton results for the varance-covarance components (MSE denotes the mean square error of the estmators). The dfferent approxmatons to the lolkelhood ve smlar smulaton results for all the parameters nvolved. The cluster specfc varance ( ) s estmated wth more relatve precson than the elements of the scaled varance-covarance matrx of the random effects (D). Ths s probably because the precson of the estmate of (as well as the estmates of ) s determned by the total number of observatons, whle the precson of the estmates of D s determned by the number of clusters. We can also see a tendency for the restrcted maxmum lkelhood to ve postvely based estmates of D 11 and D, whle the other approxmatons ve neatvely based estmates. The ratonale for restrcted maxmum lkelhood s to reduce bas n estmatn varance components. It does not seem to do so n ths case; t just chanes ts drecton. Table 5: Smulaton results for the varance-covarance components n the lostc model D 11 D 1 Approxmaton Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan D Approxmaton Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan Fure 5 presents the scatter plots of the varance-covarance component ( and D) estmates for 18

19 the alternatn RML, the alternatn ML, and the Laplacan approxmatons versus the adaptve Gaussan approxmaton. We see that, except for the alternatn RML approxmaton, all methods lead to very smlar estmates. In eneral the alternatn RML approxmaton ves larer values for the estmates of the varance components (especally D 11 and D ) than the other methods. The hher mean square error for D 1 from the alternatn ML and RML methods s vsble n the plot, as each of the panels comparn these estmates to those from the adaptve Gaussan method has a vertcal clump of ponts at the true value. s s s Alternatn - RML Alternatn - ML Laplacan D D D11 Alternatn - RML Alternatn - ML Laplacan D D D1 Alternatn - RML Alternatn - ML Laplacan Alternatn - RML D Alternatn - ML D Laplacan D Fure 5: Scatter plots of varance-covarance components estmates for the alternatn (RML and ML), Laplacan, and adaptve Gaussan approxmatons n the lostc model (16). The dashed lnes ndcate the true values of the parameters. Table 6 presents the smulaton results for the fxed effects estmates. The results are very smlar for all approxmatons consdered. We also note that the relatve varablty of the fxed effects estmates 19

20 s much smaller than those of the estmates of the elements of D. There s very lttle, f any, bas n the fxed effects estmates. Table 6: Smulaton results for the fxed effects n the lostc model 1 3 Approxmaton Mean Bas MSE Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan Fure 6 presents the scatter plots of the fxed effects estmates for the alternatn RML, alternatn ML, and Laplacan approxmatons versus the adaptve Gaussan approxmaton. Aan we observe a stron areement n the estmates obtaned throuh the varous approxmatons. The alternatn approxmatons tend to ve estmates slhtly smaller than the Laplacan and adaptve Gaussan, but the dfferences are mnor Frst Order Compartment Model The model used n the smulaton s dentcal to (15). As n the Theophyllne example we set M = 1 and n = 11; = 1; : : :; 1. The parameter values used were = 0:5, = (?3:0; 0:5;?:5) T, and D = 6 4 0: Table 7 summarzes the smulaton results for the varance-covarance components estmates. As n the lostc model analyss, we observe that the elements of D are estmated wth less relatve precson than. The alternatn ML, Laplacan, and adaptve Gaussan approxmatons seem to lead to slhtly downward based estmates of D 11 and D, whle the alternatn RML approxmaton appears to ve unbased estmates (thus achevn ts man purpose). Note however that the unbasedeness of the RML estmates does not translate nto smaller mean square error all four estmaton methods lead to smlar MSE, for all parameters. Fure 7 presents the scatter plots of the varance-covarance estmates for the alternatn RML, alternatn ML, and Laplacan approxmatons versus the adaptve Gaussan approxmaton. The alternatn RML approxmaton tends to ve larer values for D 11 and D, and larer absolute values for D 1, whle the remann approxmatons lead to very smlar estmates. There was one sample for whch the alternatn approxmatons apparently convered to a dfferent soluton than the Laplacan and 0

21 Alternatn - RML Alternatn - RML beta beta Alternatn - ML Alternatn - ML beta beta Laplacan Laplacan beta beta beta beta beta3 Alternatn - RML Alternatn - ML Laplacan Fure 6: Scatter plots of fxed effects estmates for the alternatn (RML and ML), Laplacan, and adaptve Gaussan approxmatons n the lostc model (16). The dashed lnes ndcate the true values of the parameters. adaptve Gaussan. Overall there were no major dfferences between the approxmatons n estmatn the varance-covarance components. Table 6 ves the smulaton results for the fxed effects estmates. All four approxmatons ve vrtually dentcal results for the estmaton of the fxed effects. They all show very lttle bas and smaller relatve varablty when compared to the estmates of the varance-covarance components. The scatter plots of the fxed effects estmates, not ncluded here, show practcally dentcal results for the alternatn RML and ML, the Laplacan, and the adaptve Gaussan approxmatons. 1

22 Table 7: Smulaton results for the varance-covarance components n the frst order compartment model D 11 D 1 Approxmaton Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan D Approxmaton Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan Table 8: Smulaton results for the fxed effects n the frst order compartment model 1 3 Approxmaton Mean Bas MSE Mean Bas MSE Mean Bas MSE Alternatn RML Alternatn ML Laplacan Adap. Gaussan Conclusons The results of secton 3 ndcate that the alternatn approxmaton (4) to the lolkelhood functon n the nonlnear mxed effects model (1) proposed by Lndstrom and Bates (1990) ves accurate and relable estmaton results. The man advantaes of ths approxmaton are ts computatonal effcency (allown the use of lnear mxed effects technques to estmate the scaled varance-covarance matrx of the random effects D) and the avalablty of a restrcted lkelhood verson of t, whch s not yet defned for other approxmatons/estmaton methods. Wth reard to the restrcted maxmum lkelhood estmaton thouh, the results of secton 3 suest that the bas correcton ablty of ths method depends on the nonlnear model that s ben consdered: RML estmaton acheved ts purpose for the frst order compartment model (15), but t ncreased the bas n the lostc model (16). More research s needed n ths area. Snce t s smpler computatonally the alternatn approxmaton should be used to provde startn values for the more accurate approxmatons (e.. Laplacan and adaptve Gaussan) f they are preferred. The Gaussan quadrature approxmaton (11) only seems to ve accurate results for lare number of

23 s s s Alternatn - RML D11 Alternatn - ML D11 Laplacan D11 Alternatn - RML Alternatn - ML Laplacan D1 D1 D1 Alternatn - RML Alternatn - ML Laplacan D D D Alternatn - RML Alternatn - ML Laplacan Fure 7: Scatter plots of varance-covarance components estmates for the alternatn (RML and ML), Laplacan, and adaptve Gaussan approxmatons n the frst order compartment model (15). The dashed lnes ndcate the true values of the parameters. abscssa (> 100), what makes t very neffcent computatonally. The basc problem s that t centers the rd of abscssas at 0 (the expected value of the random effects) and scales t accordn to D, whle the hhest values of the nterand n () are concentrated around the posteror modes of the random effects (b b) and scaled accordn to 00 ; D; y; b b. The advantaes of ths approxmaton are that t does not requre the estmaton of the posteror modes of the random effects at each teraton and t admts closed form partal dervatves wth respect to the parameters of nterest (; D; and ), provded these are avalable for the model functon f (Davdan and Gallant, 199). We feel that these advantaes do not compensate for the naccuracy or computatonal neffcency of the Gaussan approxmaton. The mportance sampln approxmaton (9) ves relable estmaton results, comparable to those of 3

24 the adaptve Gaussan and Laplacan approxmatons, but s consderably less effcent computatonally than these approxmatons. Also, the stochastc varablty assocated wth the dfferent mportance samples may overwhelm the numercal varablty of the lolkelhood for small chanes n the parameter values, makn t dffcult to calculate numercal dervatves. The man advantae of the mportance sampln approxmaton s ts versatlty n handln dstrbutons other than the normal, for both the random effects and the cluster-specfc error term (). For example t would be rather strahtforward to adapt the mportance sampln nteraton to handle a multvarate t dstrbuton for the random effects, but that would not be a trval task for ether the alternatn, the Laplacan, or the adaptve Gaussan approxmatons. Wakefeld et al. (1994) use the smlar property of Gbbs sampler methods to check for outlers n nonlnear mxed effects models. If one s wlln to stck wth the normal dstrbuton for b and n the nonlnear mxed effects model (1) then the mportance sampln approxmaton s not the most effcent choce. Of all approxmatons consdered here, the Laplacan and adaptve Gaussan approxmatons probably ve the best mx of effcency and accuracy. The former can be rearded as a partcular case of the latter, where just one abscssa s used. Both approxmatons (and the mportance sampln approxmaton as well) ve the exact lolkelhood when the model functon f n (1) s a lnear functon of the random effects. In the examples that we analyzed not much was aned by on from a one-pont adaptve Gaussan quadrature (Laplacan) approxmaton to approxmatons wth a larer number of abscssas. It appears that the major an n adaptve Gaussan approxmatons s related to the centern and scaln of the abscssas. Increasn the number of ponts n the evaluaton rd only ves marnal mprovement. The Laplacan approxmaton has the addtonal advantae over the adaptve Gaussan approxmaton wth more than one abscssa of allown profln of the lolkelhood over, thus reducn the dmensonalty of the optmzaton problem. For statstcal analyss purpose we would recommend usn a hybrd scheme n whch the alternatn alorthm would be used to et ood ntal values for the more refned Laplacan approxmaton to the lolkelhood of model (1). Ths way the computatonal effcency of the alternatn alorthm would be combned wth the reater accuracy of the Laplacan approxmaton. Acknowledment Ths research was partally supported by Coordenação de Aperfeçoamento de Pessoal de Nível Superor, Brazl and NSF Grant DMS

25 References Abramowtz, M. and Steun, I. A. (1964). Handbook of Mathematcal Functons wth Formulas, Graphs, and mathematcal Tables, Dover, New York. Bates, D. M. and Watts, D. G. (1980). Relatve curvature measures of nonlnearty, Journal of the Royal Statstcal Socety, Ser. B 4: 1 5. Bates, D. M. and Watts, D. G. (1988). Nonlnear Reresson Analyss and Its Applcatons, Wley, New York. Chambers, J. M. and Haste, T. J. (eds) (199). Statstcal Models n S, Wadsworth, Belmont, CA. Davdan, M. and Gallant, A. R. (199). Smooth nonparametrc maxmum lkelhood estmaton for populaton pharmacoknetcs, wth applcaton to qundne, Journal of Pharmacoknetcs and Bopharmaceutcs 0: Davs, P. J. and Rabnowtz, P. (1984). Methods of Numercal Interaton, second edn, Academc Press, New York. Draper, N. R. and Smth, H. (1981). Appled Reresson Analyss, nd edn, Wley, New York. Geweke, J. (1989). Bayesan nference n econometrc models usn Monte Carlo nteraton, Econometrca 57: Golub, G. H. (1973). Some modfed matrx eenvalue problems, SIAM Revew 15: Golub, G. H. and Welsch, J. H. (1969). Calculaton of Gaussan quadrature rules, Math. Comp. 3: Harvlle, D. A. (1974). Bayesan nference for varance components usn only error contrasts, Bometrka 61: Lard, N. M. and Ware, J. H. (198). Random-effects models for lontudnal data, Bometrcs 38: Leonard, T., Hsu, J. S. J. and Tsu, K. W. (1989). Bayesan marnal nference, Journal of the Amercan Statstcal Assocaton 84:

26 Lndstrom, M. J. and Bates, D. M. (1988). Newton-Raphson and EM alorthms for lnear mxed-effects models for repeated-measures data, Journal of the Amercan Statstcal Assocaton 83: Lndstrom, M. J. and Bates, D. M. (1990). Nonlnear mxed effects models for repeated measures data, Bometrcs 46: Mallet, A., Mentre, F., Stemer, J.-L. and Lokek, F. (1988). Nonparametrc maxmum lkelhood estmaton for populaton pharmacoknetcs, wth applcatons to Cyclosporne, J. Pharmacokn. Bopharm. 16: Pnhero, J. C. and Bates, D. M. (1993). Asymptotc propertes of maxmum leklhood estmates n the eneral lnear mxed effects model. Submtted to Annals of Statstcs. Shener, L. B. and Beal, S. L. (1980). Evaluaton of methods for estmatn populaton pharmacoknetc parameters. I. mchaels-menten model: Routne clncal pharmacoknetc data, Journal of Pharmacoknetcs and Bopharmaceutcs 8(6): Terney, L. and Kadane, J. B. (1986). Accurate approxmatons for posteror moments and denstes, Journal of the Amercan Statstcal Assocaton 81(393): Vonesh, E. F. and Carter, R. L. (199). Mxed-effects nonlnear reresson for unbalanced repeated measures, Bometrcs 48: Wakefeld, J. C., Smth, A. F. M., Racne-Poon, A. and Gelfand, A. E. (1994). Bayesan analyss of lnear and nonlnear populaton models usn the Gbbs sampler, Appled Statstcs. Accepted for publcaton. 6