3. Models with Random Effects

Size: px

Start display at page:

Download "3. Models with Random Effects"

Bruce Phillips
5 years ago
Views:

1 3. Models wth Radom Effects 3. Error-Compoets/Radom-Itercepts model Model, Desg ssues, GLS estmato 3. Example: Icome tax paymets 3.3 Mxed-Effects models Lear mxed effects model, mxed lear model 3.4 Iferece for egresso coeffcet 3.5 Varace compoets estmato Maxmum lkelhood estmato, Newto-Raphso ad Fsher scorg, restrcted maxmum lkelhood (REML) estmato Appedx 3A REML calculatos

2 3. Error compoets model Samplg - Subjects may cosst of a radom subset from a populato, ot fxed subjects Iferece - I the fxed effects models, our ferece deals, part, wth subject-specfc parameters {α }. These parameters are based o the subjects our sample. We may wsh to make statemets about the etre populato. I the fxed effects model, because s typcally large, there are may usace parameters {α }.

3 Basc model The error compoets model s y t α + x t β + ε t. Ths porto of the otato s the same as the basc fxed model. However, ow the quattes α are assumed to be radom varables, ot fxed ukow parameters. We assume that α are depedetly ad detcally dstrbuted (..d) wth mea zero ad varace σ α. We assume that {α } are depedet of the error radom varables, {ε t }. We stll assume that x t s a vector of covarates, or explaatory varables, ad that β s a vector of fxed, yet ukow, populato parameters. I the error compoets model, we assume o seral correlato, that s, Var ε σ I. Thus, the varace of the th subject s Var y σ α J + σ I V

4 Tradtoal ANOVA set-up Wthout the covarates, ths s the tradtoal radom effects (oe way) ANOVA set-up. Ths model ca be terpreted as arsg from a stratfed samplg scheme. We draw a sample from a populato of subjects. We observe each subject over tme. Is there heterogeety amog subjects? Oe respose s to test the ull hypothess H 0 : σ α 0. Estmates of σ α are of terest but requre scalg to terpret. A more useful quatty to report s σ α /(σ α +σ ), the tra-class correlato.

5 Samplg The expermetal desg specfes how the subjects are selected ad may dctate the model choce. Selectg subjects based o a (stratfed) radom sample mples use of the radom effects model. Ths samplg scheme also suggests that the covarates are radom varables. Selectg subjects based o characterstcs suggests usg a fxed effects model. I the extreme, each represets a characterstc. Aother example s where we sample the etre populato. For example, the 50 states the US.

6 α α α 3 Fgure 3.. Two-stage radom effects samplg.

7 Error Compoets Model Assumptos E (y t α ) α + x t β. {x t,,..., x t,k } are ostochastc varables. Var (y t α ) σ. { y t } are depedet radom varables, codtoal o {α,, α }. y t s ormally dstrbuted, codtoal o {α,, α }. E α 0, Var α σ α ad {α,, α } are mutually depedet. {α } s ormally dstrbuted.

8 Observables Represetato of the Error Compoets Model E y t x t β. {x t,,..., x t,k } are ostochastc varables. Var y t σ + σ α ad Cov (y r, y s ) σ α, for r s. { y } are depedet radom vectors. {y } are ormally dstrbuted.

9 Structural Models What s the populato? A stadard defese for a probablstc approach to ecoomcs s that although there may be a fte umber of ecoomc ettes, there s a fte rage of ecoomc decsos. Accordg to Haavelmo (944) the class of populatos we are dealg wth does ot cosst of a fty of dfferet dvduals, t cossts of a fty of possble decsos whch mght be take wth respect to the value of y. See Nerlove ad Balestra s chapter a moograph edted by Mátyás ad Sevestre (996, Chapter ) the cotext of pael data modelg.

10 Iferece If you would lke to make statemets about a populato larger tha the sample, desg the sample to use the radom effects model. If you are smply terested cotrollg for subject-specfc effects (treatg them as usace parameters), the use the fxed model. I addto to samplg ad ferece, the model desg may also be flueced by a desre to crease the degrees of freedom avalable for parameter estmato. Degrees of freedom There are +K lear regresso parameters plus varace parameter the fxed effects model, compared to oly +K regresso plus varace parameters the radom effects model. Choose the radom effects models to crease the degrees of freedom avalable for parameter estmato.

11 Tme-costat varables If the prmary terest s testg for the effects of tme-costat varables, the, other thgs beg equal, desg the sample to use a radom effects model. A mportat example of a tme-costat varable s a varable that classfes subjects by groups: Ofte, we wsh to compare the performace of dfferet groups, for example, a treatmet group ad a cotrol group. I the fxed effects model, tme-costat varables are perfectly collear wth subject-specfc tercepts ad hece are estmable.

12 GLS estmato Expressg the model matrx form, we have E y X β ad Var y V σ α J + σ I. J s a T T matrx of oes, I s a T T detty matrx. Here, X s a T K matrx of explaatory varables, X (x, x,..., x T ). The geeralzed least squares (GLS) equatos are: Ths yelds the error-compoets estmator of β The varace of the error compoets estmator s: + T J I V σ σ σ σ α α y X V β X X V EC T T y J I X X J I X b + + σ σ σ σ σ σ α α α α Var + EC T X J I X b σ σ σ σ α α

13 Feasble geeralzed least squares Ths assumes that the varace parameters σ α ad σ are kow. Oe way to get a feasble geeralzed least squares estmate s: Frst ru a regresso assumg V I, ordary least squares. Use the resduals to determe estmates of σ α ad σ. Ths estmato procedure yelds estmates of σ α that ca be egatve, although ubased. Determe b EC usg the estmates of σ α ad σ. Ths procedure could be terated. However, studes have show that terated versos do ot mprove the performace of the oe-step estmators. There are may ways to estmate the varace parameters: Regardless of how the estmate s obtaed, use t the GLS estmates. See Secto 3.5 for more detals.

14 Poolg test Test whether the tercepts take o a commo value. That s, do we have to accout for subject-specfc effects? Usg otato, we wsh to test the ull hypothess H 0 : σ α 0. Ths s a exteso of a Lagrage multpler statstc due to Breusch ad Paga (980). Ths ca be doe usg the followg procedure: Ru the model y t x t β + ε t to get resduals e t. For each subject, compute a estmator of σ α T s T e et T ( T ) t Compute the test statstc, TS t Reject H 0 f TS exceeds a quatle from a χ (ch-square) dstrbuto wth oe degree of freedom. N s T ( T T ) e t

15 3.3 Mxed models The lear mxed-effects model s y t z t α + x t β + ε t. Ths s short-had otato for the model y t α z t α q z tq +β x t β K x tk + ε t The matrx form of ths model s y Z α + X β + ε The resposes betwee subjects are depedet, yet we allow for temporal correlato through Var ε R. Further, we ow assume that the subject-specfc effects {α } are radom wth mea zero ad varace-covarace matrx D. We assume E α 0 ad Var α D, a q q (postve defte) matrx. Subject-specfc effects ad the ose term are assumed to be ucorrelated, that s, Cov (α, ε ) 0. Thus, the varace of each subject ca be expressed as Var y Z DZ + R V (τ).

16 Observables Represetato of the Lear Mxed Effects Model E y X β. {x t,,..., x t,k } ad {z t,,..., z t,q } are ostochastc varables. Var y Z DZ + R V (τ) V. { y } are depedet radom vectors. {y } are ormally dstrbuted.

17 Repeated measures desg Ths s a specal case of the lear mxed effects model. Here we have,..., subjects. A respose for each subject s measured based o each of T treatmets. The order of treatmets s radomzed. The mathematcal model s: respose 443 y t radom subject effect α fxed treatmet effect β error 3 ε The ma research questo of terest s H 0 : β β... β T, o treatmet dffereces. Here, the order of treatmets s radomzed ad o seral correlato s assumed. + t + t

18 Radom coeffcets model Here s aother mportat specal case of the pael data mxed model. Take z t x t. I ths case the pael data mxed model reduces to a radom coeffcets model, of the form y t x t (α + β) + ε t x t β + ε t, where {β } are radom varables wth mea β, depedet of {ε t }. Two-stage terpretato. Sample subject to get β. Sample observatos wth E(y β ) X β ad Var(y β ) R. Ths yelds E y X β ad Var y X DX + R V.

19 Varatos Take colums of Z to be a strct subset of the colums of X. Thus, certa compoets of β assocated wth Z are stochastc whereas the remag compoets that are assocated wth X but ot Z are ostochastc. Two-stage terpretato Use varables B such that E β B β. The, we have, E y X B β ad Var y R + X DX. Ths s the radom effects model replacg X by X B ad Z by X

20 More specal cases Icluso of group effects. Take q ad z t ad cosder: y t α + δ g + x gt β + ε gt, for g,..., G groups,,..., g subjects each group ad t,..., T g observatos of each subject. Here, {α } represet radom, subject-specfc effects ad {δ g } represet fxed dffereces amog groups. Ths model s ot estmable whe {α } are fxed effects. Tme-costat varables. We may splt the explaatory varables assocated wth the populato parameters to those that vary by tme ad those that do ot (tme-varat). Thus, we ca wrte our pael data mxed model as y t z t α + x β + x t β + ε t Ths model s a geeralzato of the group effects model. Ths model s ot estmable whe {α } are fxed effects. Sec Chapter 5 o multlevel models

21 Mxed Lear Models Not all models of terest ft to the lear mxed effects model framework, so t s of terest to troduce a geeralzato, the mxed lear model. Ths model s gve by y Z α + X β + ε. Here, for the mea structure, we assume E (y α) Z α + X β ad E α 0, so that E y X β. For the covarace structure, we assume Varε R,Varα D ad Cov (α, ε ) 0. Ths yelds Var y Z D Z + R V. Ths model does ot requre depedece betwee subjects. Much of the estmato ca be accomplshed drectly terms of ths more geeral model. However, the lear mxed effects model provdes a more tutve platform for examg logtudal data.

22 Mxed lear model: Specal cases Lear mxed effects model Take y (y,..., y ), ε (ε,..., ε ), α (α,..., α ), X (X,..., X ) ad Z block dagoal (Z,..., Z ). Wth these choces, the model y Z α + X β + ε s equvalet to the model y Z α + X β + ε The two-way error compoets model s a mportat pael data model that s ot a specfc type of lear mxed effects model although t s a specal case of the mxed lear model. Ths model ca be expressed as y t α + λ t + x t β + ε t Ths s smlar to the error compoets model but we have added a radom tme compoet, λ t.

23 3.4 Regresso coeffcet ferece The GLS estmator of β takes the same form as the error compoets model wth a more geeral varace covarace matrx V. The GLS estmator of β s bgls X V X X V Recall V V (τ) Z DZ + R. The varace s: Var b GLS X V X Iterpret b GLS as a weghted average of subject-specfc gls estmators. b GLS W, GLS W, GLSb, GLS b,gls s the least squares estmator based solely o the th subject b,gls (X V - X ) - X V - y, W,GLS X V - X y

24 Matrx verso formula To smplfy the calculatos, here s a formula for vertg V (τ). Ths matrx has dmeso T T. V (τ) - (R + Z DZ ) - R - - R - Z (D - + Z R - Z ) - Z R - Ths s easer to compute f the temporal covarace matrx R has a easly computable verse ad the dmeso q s smaller tha T. Because the matrx (D - + Z R - Z ) - s oly a q q matrx, t s easer to vert tha V (τ), a T T matrx. For the error compoets model, ths s: σ V ( ) α σ I + σ αzz I J σ Tσ α + σ

25 Maxmum lkelhood estmato The log-lkelhood of a sgle subject s l ( β, τ) T + V τ + y Xβ l(π ) ldet ( ) ( ) V ( τ) ( y X β) Thus, the log-lkelhood for the etre data set s L(β, τ ) Σ l (β, τ ). The values of β, τ that maxmze L(β, τ ) are the maxmum lkelhood estmators. The score vector s the vector of dervatves wth respect to the parameters. For otato, let the vector of parameters be θ (β, τ ). Wth ths otato, the score vector s L ( β,. τ ) /( θ ) If ths score has a root, the the root s the maxmum lkelhood estmator.

26 Computg the score vector To compute the score vector, we frst take dervatves wth respect to β ad fd the root. That s, Ths yelds That s, for fxed covarace parameters τ, the maxmum lkelhood estmators ad the geeralzed least squares estmators are the same. l ), ( ), L( τ β β τ β β ( ) ( ) ) ( X β y τ V X β y β ( ) ) ( β X y τ X V GLS MLE b y τ X V X τ X V b ) ( ) (

27 Robust estmato of stadard errors A alteratve, weghted least squares estmator, s where the weghtg matrx W,RE depeds o the applcato at had. If W,RE V -, the b W b GLS. Basc calculatos show that t has varace Thus, a robust estmator of the stadard error s: RE RE W y X W X X W b,,,,,, Var RE RE RE RE W X X W X V W X W X X W b,,,,, ) ( RE RE RE RE th j W of elemet dagoal j b se X X W X W e e X W X X W

28 Testg hypotheses The terest may be testg H 0 : β j β j,0, where the specfed value β j,0 s ofte (although ot always) equal to 0. Use: t statstc b j, GLS se( b β j, GLS Two varats: Oe ca replace se(b j,gls ) by se(b j,w ) to get so-called robust t- statstcs. Oe ca replace the stadard ormal dstrbuto wth a t- dstrbuto wth the approprate umber of degrees of freedom SAS default s the cotamet method. We typcally wll have large umber of observatos ad wll be more cocered wth potetal heteroscedastcty ad seral correlato ad thus wll use robust t-statstcs. j,0 )

29 Lkelhood rato test procedure Usg the ucostraed model, calculate maxmum lkelhood estmates ad the correspodg lkelhood, deoted as L MLE. For the model costraed usg H 0 : C β d, calculate maxmum lkelhood estmates ad the correspodg lkelhood, deoted as L Reduced. Compute the lkelhood rato test statstc, LRT (L MLE -L Reduced ). Reject H 0 f LRT exceeds a percetle from a χ (chsquare) dstrbuto wth p degrees of freedom. The percetle s oe mus the sgfcace level of the test. See Appedx C.7 for more detals o the lkelhood rato test.

30 3.5 Varace compoet estmato Maxmum Lkelhood Iteratve estmato:newto-raphso ad Fsher Scorg Restrcted maxmum lkelhood (REML) Startg values: Swamy s method Rao s MIVQUE estmators

31 Maxmum lkelhood estmato The cocetrated log-lkelhood s L( b GLS, τ) T l(π ) + l det V ( τ) + ( Error SS ) τ Here, the error sum of squares s ( Error SS) τ y X b V ( τ) y X b ( ( )) ( ) ( ) ( ) GLS I some cases, oe ca obta closed forms solutos. I geeral, ths must be maxmzed teratvely. GLS

32 Varace compoets estmato Thus, we ow maxmze the log-lkelhood as a fucto of τ oly. The we calculate b MLE (τ) terms of τ. Ths ca be doe usg ether the Newto-Raphso or the Fsher scorg method. Newto-Raphso. Let L L(b MLE (τ), τ ), ad use the teratve method: L L τ NEW τold τ τ τ τ τ Here, the matrx OLD L /( τ τ ) s called the sample formato matrx. Scorg. Defe the expected formato matrx H(τ) E ( L/ τ τ ) ad use ( ) τ NEW τ OLD H ( τ ) OLD L τ τ τ OLD

33 Motvato for REML Maxmum lkelhood ofte produces based estmator of varace compoets. To llustrate, cosder the basc cross-sectoal regresso model: Lety x β + ε,,..., N, where β s a p vector, {ε } are..d. N(0, σ ). Themleof σ s (Error SS)/ N, where Error SS s the error sum of squares from the model ft. Ths estmate has expectato σ (N /(N -p)) ad thus s a based estmate of σ.

34 Further motvato for REML As aother example, cosder our basc fxed effects pael data model: y t α + x β + ε t, where β s a K vector, {ε t } are..d. N(0, σ ). As above, the mle of σ s (Error SS)/N, where Error SS s the error sum of squares from the model ft. Ths estmator has expectato σ (N-(+K))/ N ad thus s a based estmate of σ. The bas s ot asymptotcally eglgble. To llustrate, the balaced desg case, we have NT ad bas σ (T-(+K))/ (T) - σ - σ (+K)/(T) - σ /T, for large.

35 REML The acroym REML stads for restrcted maxmum lkelhood. The dea s to cosder oly lear combatos of resposes {y} that do ot deped o the mea parameters. To llustrate, cosder the followg geerc stuato: the resposes are deoted by the vector y, are ormally dstrbuto ad have mea E y X β ad varacecovarace matrx Var y V(τ). The dmeso of y s N ad the dmeso of X s N p. Suppose that we wsh to estmate the parameters of the varace compoet, τ.

36 REML estmato Defe the projecto matrx Q I - X (X X) - X. If X has dmeso N p, the the projecto matrx Q has dmeso N N. Cosder the lear combato of resposes Q y. Some straghtforward calculato show that ths has mea 0 ad varace-covarace matrx Var y Q V Q. Because () Q y s ormally dstrbuted ad () the mea ad varace do ot deped o β, ths meas that the etre dstrbuto of Q y does ot deped o β. We could also use ay lear trasform of Q, such as A Q. That s, the dstrbuto of A Q y s also ormally dstrbuted wth wth a mea ad varace that does ot deped o β.

37 Modfed lkelhood These observatos led Patterso ad Thompso (97) ad Harvlle (974) to modfy our lkelhood calculatos by cosderg the restrcted maxmum log-lkelhood : L REML ( b GLS ( τ), τ) ldet( ( )) ldet( ( ) ) ( )( ), V τ + X V τ X + Error SS τ a fucto of τ. Here, the error sum of squares s ( Error SS)( τ) ( y Xb ( τ)) V( τ) ( y Xb ( τ)). GLS For comparso, the usual log-lkelhood s : L( b GLS ( τ), τ) [ ldet( V( τ)) + ( Error SS)( τ) ], The oly dfferece s the term l det(x V(τ) X ) ; thus, methods of maxmzato are the same (that s, usg Newto- Raphso or scorg). GLS

38 Propertes of REML estmates For the case V σ I, the the REML estmate yelds the ubased estmate of σ. Whe p, the umber of regressors s small, the MLE ad REML estmates of varace compoets are smlar. Whe p, the umber of regressors s large, REML estmates ted to outperform MLE estmates. The addtoal term for the logtudal data mxed model s l det X V ( τ ) X ( )

39 Startg Values Both Swamy ad Rao s procedures provde useful, orecursve, varace compoets estmates Rao s MIVQUE estmators are avalable for a larger class of models (hadlg seral correlato, for example) A verso of MIVQUE s the default opto SAS PROC MIXED for startg values.

40 REML versus MLE Both are lkelhood based estmators They appled to a wde varety of models They rely o a parametrc specfcato For lkelhood rato tests, oe should ot use REML. Use stead maxmum lkelhood estmators Appedx 3A.3 demostrates the potetally dsastrous cosequeces of usg REML estmators for lkelhood rato tests.

TESTS BASED ON MAXIMUM LIKELIHOOD

ESE 5 Toy E. Smth. The Basc Example. TESTS BASED ON MAXIMUM LIKELIHOOD To llustrate the propertes of maxmum lkelhood estmates ad tests, we cosder the smplest possble case of estmatg the mea of the ormal