Florida State University Libraries

Transcription

1 Flora State Unversty Lbrares Electronc Theses, Treatses an Dssertatons The Grauate School 04 Wthn Stuy Depenence n Meta-Analyss: Comparson of GLS Metho an Multlevel Approaches Seungjn Lee Follow ths an atonal works at the FSU Dgtal Lbrary. For more nformaton, please contact lb-r@fsu.eu

2 FLORIDA STATE UNIVERSITY COLLEGE OF EDUCATION WITHIN STUDY DEPENDENCE IN META-ANALYSIS: COMPARISON OF GLS METHOD AND MULTILEVEL APPROACHES By SEUNGJIN LEE A Dssertaton submtte to the Department of Eucatonal Psychology an Learnng Systems n partal fulfllment of the requrements for the egree of Doctor of Phlosophy Degree Aware: Fall Semester, 04

3 Seungjn Lee efene ths ssertaton on November 6, 04. The members of the supervsory commttee were: Betsy Jane Becker Professor Drectng Dssertaton Fre Huffer Unversty Representatve Insu Paek Commttee Member Yanyun Yang Commttee Member The Grauate School has verfe an approve the above name commttee members, an certfes than the ssertaton has been approve n accorance wth unversty requrements.

4 I ecate my ssertaton to my mother, Sukja Jung an my father who s n heaven, Keehyuk Lee I love you Also, I express my sncere apprecaton to my bg sster, Yongman Lee

5 ACKNOWLEDGMENTS My avsor, Dr. Betsy Jane Becker, I coul not magne my lfe n USA wthout her support an encouragement. She has been wth me as my best acaemc avsor an she has been wth me as my counsellor for my lfe. An t s my turn to be wth her as her prou stuent an fren. Frst of all, I coul not forget forever her warmth to show me n a meetng at her ktchen table on November If I have a chance, I woul lke to raw the scene but I woul not forget to a my lttle fren Tash on there. Dr. Becker, I coul not have complete my ssertaton wthout your help an avce, I love you. I woul lke to express my grattue to my ssertaton commttee members, Dr Yanyun Yang, Dr. Fre Huffer, an Dr. Insu Paek. I am very thankful for ther valuable avce, comments, an suggestons for my progress n my ssertaton. An my fren Fath Orcan who s n Turkey now, I coul not complete my ssertaton wthout hs contrbuton on my ssertaton as well. He s my man partner to evelop the R an SAS coes to smulate ata n ths ssertaton. Also, he was my best fren who share most of my aly lfe n Measurement an Statstcs. We are plannng to work together n the future. Thank you so much, Fath. I woul lke to exten my thanks to my famly, my sster Yongha Lee, Bomoon Lee, an my brother Taehoon Lee, an my frens Jyeo Yun, Kyunghwa Cho, Raesun Km, Abullah Algham, Bern Wess, Chrstopher Thompson, an Jyoun Km. Also, I exten my love to my lttle lovely fren Tash. v

6 TABLE OF CONTENTS LIST OF TABLES... v LIST OF FIGURES... x ABSTRACT... x CHAPTER : INTRODUCTION... CHAPTER : LITERATURE REVIEW... 5 Stanarze-Mean-Dfference Effect Sze... 5 Moels to Analyze Effect Szes... 6 Unvarate Meta-analyss... 8 Tratonal Meta-Analyss Moel... 8 Unvarate Two-Level Meta-Analyss Moel... Multvarate Meta-analyss... GLS Metho... 3 Multvarate Two-Level Moel... 8 Unvarate Three-Level Moel... Purpose of Ths Stuy... 3 Overall Effects n GLS an Three-Level Approaches... 3 Outcome-Specfc Effects n GLS an Three-level Approaches CHAPTER 3: METHODOLOGY Smulaton Contons v

7 Data Generaton Proceure Level (Stuy Level) Level (Partcpant Level) Data Analyss Relatve Percentage Bas ANOVA Tests CHAPTER 4: RESULTS... 4 Convergence... 4 Bas Parameter Estmates SE of Parameter Estmate ANOVA Tests CHAPTER 5: CONCLUSIONS AND DISCUSSION APPENDICES A. R-CODES FOR DATA GENERATION B. R-CODES FOR PARAMETER ESTIMATE (GLS) C. SAS-CODES FOR PARAMETER ESTIMATE (THREE-LEVEL) D. WITHOUT AND WITH REPLICATIONS THAT DID NOT CONVERGE REFERENCES v

8 BIOGRAPHICAL SKETCH... 6 v

9 LIST OF TABLES Table.: Comparson of the Two-Level Multvarate Moels Table.: Comparson of the Three-Level Moels Table.3: Comparsons of Estmates Between Three-Level an GLS Metho Approaches Table 3.: Smulaton Contons Table 4.: Partal Eta Squares for Dfference n SE Bas Between Two Approaches n the Ranom-Effects Moel v

10 LIST OF FIGURES Fgure 4.: Relatve percentage bas of the SE n the overall effects for two outcomes n the ranom-effects moel Fgure 4.: Relatve percentage bas of the SE n the overall effects for fve outcomes n the ranom-effects moel Fgure 4.3: Relatve percentage bas of the SE of the outcome-specfc effects for two outcomes n the ranom-effects moel Fgure 4.4: Relatve percentage bas of the SE n the outcome-specfc effects for fve outcomes n the ranom-effects moel x

11 ABSTRACT Multvarate meta-analyss methos typcally assume the epenence of effect szes. One type of expermental-esgn stuy that generates epenent effect szes s the multple-enpont stuy. Whle the generalze least squares (GLS) approach requres the sample covarance between outcomes wthn stues to eal wth the epenence of the effect szes, the unvarate three-level approach oes not requre the sample covarance to analyze such multvarate effectsze ata. Conserng that t s rare that prmary stues report the sample covarance, f the two approaches prouce the same estmates an corresponng stanar errors, the unvarate threelevel moel approach coul be an alternatve to the GLS approach. The man purpose of ths ssertaton was to compare these two approaches uner the ranom-effects moel for syntheszng stanarze mean fferences n multple-enponts expermental esgns usng a smulaton stuy. Two ata sets were generate uner the ranomeffects moel: one set wth two outcomes an the other set wth fve outcomes. The smulaton stuy n ths ssertaton foun that the unvarate three-level moel yele the approprate parameter estmates an ther stanar errors corresponng to those n the multvarate metaanalyss usng the GLS approach. x

12 CHAPTER INTRODUCTION Meta-analyss s a statstcal metho to combne results from stues on the same or relate topcs, whch s wely use n mecne an the socal scences. The results from stues are often ntegrate usng statstcal ncators or effect-sze estmates (Heges, 007). Common effect-sze estmates are correlaton coeffcents, stanarze mean fferences, an os ratos. Two types of meta-analyss have been evelope to synthesze effect szes across stues: unvarate an multvarate meta-analyss. Unvarate meta-analyss methos typcally assume the nepenence of effect szes an thus each estmate effect sze oes not affect the other estmate effect szes n ther rectons an magntues. However, t s common that stues prouce multple effect szes, an they mght be correlate (Gleser & Olkn, 994; Heges & Olkn, 985; Rosenthal & Rubn, 986). For example, f a stuy prouces multple effect szes for multple outcome measures wth common partcpants, the effect szes n ths stuy are not nepenent of each other. Multvarate meta-analyss methos, thus, account for the epenence of effect szes wthn the same stuy. Gleser an Olkn (994) scusse two types of expermental-esgn stues that generate epenent effect szes wthn the same stuy: multple-treatment stues an multple-enpont stues. In multple treatment stues, effect szes are estmate wth one common control group an multple treatment groups. Even wth only one outcome measure the effect szes usng the common control group are correlate (Km & Becker, 00). Multple-enpont stues may use

13 sngle treatment an control groups to estmate effect szes for multple outcome measures, an effect szes from the common partcpants are epenent on each other. There are three approaches for ealng wth the epenence of effect szes wthn stues n meta-analyss (Becker, 000; Lttell, Corcoran, & Plla, 008). Frst, a researcher mght gnore the epenence of effect szes wthn stues an treat all effect szes as nepenent. Secon, the epenence coul be avoe. For example, f a stuy has three outcome measures for the same partcpants, a researcher mght perform three separate meta-analyses, one for each nvual outcome measure across stues. Thr, a researcher coul statstcally moel the epenence. Three statstcal multvarate meta-analyss approaches for multple-enpont stues are explane n ths ssertaton: the multvarate meta-analyss moel wth generalze least squares (GLS) estmaton, the multvarate two-level moel, an the unvarate three-level moel estmate usng herarchcal moelng methos. Rauenbush, Becker, an Kalaan (988) apple the GLS metho to evelop the multvarate fxe-effects regresson moel an use t to examne the effect of the hours of coachng on effect szes for SAT-Math an SAT-Verbal scores. Gleser an Olkn (994) also escrbe the GLS metho for multvarate fxe-effects effect-sze ata. Both sets of authors use the sample varance-covarance matrx of effect szes wthn the same stuy to hanle the epenence of effect szes. The tratonal meta-analyss moel s a specal case of a unvarate two-level moel. In the two-level moel, the sperson of effect szes s explane by two ranom varance components: the partcpant-level (level ) samplng error an the stuy-level (level-) varaton. Kalaan an Rauenbush (996) extene the multvarate two-level fxe-effects regresson

14 moel (Rauenbush, Becker, & Kalaan, 988) to a multvarate two-level mxe-effects regresson moel to examne the varance an covarance of effect szes between stues, base on the sample varances an covarances of effect szes wthn stues. They also nvestgate whether hours of coachng woul explan the fferences among effect szes from SAT coachng stues. The between-stues varances an covarances were estmate usng restrcte maxmum lkelhoo (REML). The estmates of regresson coeffcents (.e., the hours of coachng) an nvual effect szes were base on GLS methos an a Bayesan approach, respectvely. Fnally, Van Den Noortgate, Lopez-Lopez, Marn-Martnez, an Sanchez-Meca (0) propose the unvarate three-level moel as an alternatve approach to multvarate metaanalyss, base on nverse-varance weghtng, the REML metho, an an emprcal Bayesan approach. The unvarate three-level moel was an extenson of the tratonal two-level moel wth the ncluson of an atonal level, the outcome level, to account for the epenence of effect szes. Ths three-level moel, thus, accounts for three ranom varance components to explan the sperson of effect szes over three levels: the partcpant-level samplng error (level ), the outcome-level varaton (level ), an the stuy-level varaton (level 3). Van en Noortgate et al. (0) nvestgate how the between-stues varance n the unvarate threelevel moel reflecte the epenence of effect szes. Unlke the multvarate fxe approach moel base on the GLS metho an the multvarate two-level moel approach, the unvarate three-level approach oes not requre the sample covarance between outcomes wthn stues to analyze the multvarate effect-sze ata. Conserng that t s rare that prmary stues report the sample covarance, the unvarate threelevel moel s both applcable an relatvely smple n comparson to the other two approaches. 3

15 If the three approaches prouce the same estmates an corresponng stanar errors, the unvarate three-level moel approach coul be an alternatve to the other two approaches. Van en Noortgate et al. (0) foun that the parameter estmates an the corresponng stanar errors n the untvarate three-level moel were not base n comparson to those from a multvarate two-level moel wth a smulaton stuy. Thus, the man purpose of ths ssertaton s to nvestgate whether the unvarate three-level moel woul be a vable alternatve to multvarate meta-analyss wth GLS methos. To ths en, I generate a multvarate ata set base on a multple-enponts expermental esgn. The generate ata was use to compute effect szes for the corresponng outcome measures. Two multvarate metaanalyss moels were examne wth the effect szes: the unvarate three-level moel approach an the multvarate meta-analyss moel usng GLS approach. 4

16 CHAPTER LITERATURE REVIEW There are two types of meta-analyss: unvarate meta-analyss an multvarate metaanalyss. Unvarate meta-analyss analyzes stues whch prouce effect szes to measure a sngle outcome varable an assumes the nepenence of effect szes across stues. On the other han, multvarate meta-analyss analyzes stues that may have multple effect szes for the multple outcomes whch are epenent on each other. Thus, the epenence of effect szes wthn the same stuy s accounte for by the analyss n multvarate meta-analyss. The man purpose of ths chapter s to compare the multvarate meta-analyss moel estmate usng the GLS metho an the mult-level meta-analyss moel for the unvarate three-level case. To ths en, ths chapter brefly begns wth a escrpton of the stanarzemean-fference effect sze an two types of unvarate meta-analyss moels: tratonal metaanalyss an meta-analyss wth a unvarate two-level mult-level moel. Stanarze-Mean-Dfference Effect Sze Meta-analyss nvolves combnng effect szes (e.g., correlaton coeffcents, stanarze mean fferences, os ratos) from nvual stues ealng wth the same topc, an askng whether the effect szes are homogenous across stues (Heges, 007). The estmate stanarze mean fference (effect sze) between the treatment group an the control group for one outcome measure s compute as 5

17 YC YT g =, () S where YC an T Y are the means of the outcome varable n the control group an treatment group n stuy, respectvely, an S s the poole stanar evaton for the effect sze n stuy, whch s calculate as follows S = ( n T ) ST + ( n ) S C, () T C ( n ) + ( n ) C T C where n an n are group sample szes, an S an S T C are the corresponng group varances. Heges (98) showe that effect-sze estmates from small samples are base an ths small sample bas s correcte by E[g ] = δ /c(m ) where c(m )=-3/(4m -), an m = C n + n. Effect-sze estmates =g c(m j ) are approxmately normally strbute wth a mean of the true T C effect sze δ = ( µ µ ) / σ where σ s the populaton poole stanar evaton for the effect sze n stuy an a varance of T δ σ = + +. (3) T C T C n n ( n + n ) Typcally, σ s estmate by substtutng for δ n Equaton 3. Moels to Analyze Effect Szes Meta-analyss stues use statstcal methos (.e., homogenety tests) to examne whether effect szes estmate the same populaton effect. Two types of moels can be use to analyze effect szes: the fxe-effects moel an the ranom-effects moel. In fxe-effects moels, the 6

18 sperson of effect szes aroun the populaton mean s vewe as resultng from the partcpantlevel samplng error alone, an each effect sze estmates the common populaton value. If each effect sze oes not estmate a common populaton mean, the varablty of effect szes s etermne to contan true fferences between stues n aton to the partcpant-level samplng error, whch leas to the ranom-effects moel. Effect szes are estmate from stues of fferent szes, thus the samplng varances for each effect sze are not equal. Thus, n analyses each effect sze s weghte by the nverse varance. In the fxe-effects moel, the source of varaton for effect szes s samplng error an each effect sze s weghte by the nverse samplng varance. However, uner the ranomeffects moel, the varablty of the effect szes comes from samplng error an true fferences n effects across stues. Thus, each effect s weghte by the nverse of the sum of the samplng varance an the between-stues varance n the ranom-effects moel. In the ranom-effects moel, the true fferences between stues can be characterze by three moels: fxe-effects ANOVA or regresson moels or mxe-effects moels (Heges, 98; Heges & Olkn, 985). In the fxe-effects ANOVA or regresson moels, the observe strbutons of effect szes are more varable than for the smplest fxe-effects moels. The ae varablty s not ranom, but has a systematc porton whch can be explane by moerators (stuy characterstcs) to fferentate stues. Fnally, the mxe-effects moel for ANOVA or regresson as another ranom component to the fxe-effects ANOVA or regresson moels for the further fferences n effect szes across stues; ths reflects a leftover porton whch s not explane by moerators. 7

19 The man purpose of ths ssertaton s to compare the multvarate meta-analyss moel base on the GLS approach an the multvarate meta-analyss moel for the unvarate threelevel case uner the ranom-effects moels wthout moerators. Unvarate Meta-analyss Unvarate meta-analyss assumes the nepenence of effect szes for a sngle outcome measure across stues. Two types of unvarate meta-analyss approaches are escrbe: tratonal meta-analyss moels an meta-analyss wth a unvarate two-level moel. Tratonal Meta-Analyss Moel Homogenety test. The Q statstc test (.e., homogenety test) examnes whether the varablty among the effect szes s greater than the varance expecte from the partcpant-level samplng error. The null hypothess for the Q statstc s that all effect-sze estmates arse from a common populaton effect (.e., H 0 : δ... = δ = δ ). Q s strbute as a ch-square wth k- = k egrees of freeom where k s the number of effect szes. The formula s Q = k ( ) σ, (4) where s the effect sze estmate n stuy for =,, k, an σ s the varance of the effect sze n stuy. Also, s the fxe-effects weghte mean effect-sze estmate over k effect szes (Heges & Olkn, 985), whch s expresse as = [ / σ ], [/ σ ] (5) 8

20 where each effect sze s weghte wth an nverse varance σ. The varance of the common populaton mean effect s [/ ] σ an the stanar evaton of the mean effect s the square root of the varance. Whle meta-analysts may choose to aopt the ranom-effects moel as a matter of prncple, n other cases, the Q test can be use to ece whether the fxe or ranomeffects moel apples. Fxe-effects moel. If Q s smaller than the crtcal value of χ wth k- egrees of freeom, we can conclue that all effect szes are from one common populaton effect, whch leas to the moel (Lpsey & Wlson, 00) = δ + ε, (6) where s an observe effect sze n stuy, δ s the corresponng true or populaton effect sze, an ε s an error term n stuy. The ranom effect ε s approxmately normally strbute wth a mean zero an a varance σ. Thus, the estmate effect sze s normally strbute about the true effect sze δ (.e., wth the mean of δ ) wth a varance of σ. The varance σ s treate as a known varance an s typcally estmate usng Equaton 3. In the fxe-effects moel, all varaton arses from samplng error wthn stues alone, an all effect szes estmate the common populaton effect sze δ. Ranom-effects moel. Uner the ranom-effects moel, effect szes o not estmate a common populaton effect. The ranom-effects moel assumes that true effect szes δ vary an another ranom component, the between-stues varance τ, s nclue to explan the true 9

21 fferences n effect szes. The varaton n effect szes comes from true fferences n effect szes across stues n aton to samplng error wthn stues, specfcally = δ + ε or = δ + ε + u (7) where s an observe effect sze n stuy, δ s the corresponng populaton effect sze, δ s an overall effect (.e., the average of the populaton effect szes) across stues an u s a ranom effect showng between-stues varaton aroun the overall effect sze. The ranom effects u are normally strbute wth a mean zero an a varance τ. The overall effect sze s fxe, an thus the varance of the ranom effects ( u ) s also the varance of the true effect szes ( δ ). Thus, the true effect szes δ can be vewe as beng normally strbute aroun the overall effect sze δ wth a varance τ. The overall populaton mean effect δ (the average effect sze) s estmate as ˆ δ = /[ σ + ˆ] τ. /[ σ + ˆ] τ (8) Two metho-of-moments estmators of the between-stues varance (τ ) are avalable (Rauenbush, 994). Whle one approach s not weghte, the other approach s weghte. That s, the between-stues varance s compute usng a typcal sample varance of the observe effect szes n the frst approach. In the secon approach, the between-stues varance s estmate as Q ( k ) ˆ =, w ( w / w ) τ (9) 0

22 where Q s the value of the homogenety test n Equaton 4, k s the number of effect szes an w s the nverse samplng varance for stuy. Unvarate Two-Level Meta-Analyss Moel Effect szes are neste wthn stues n the meta-analyss ata structure. Thus, multlevel moelng s a useful framework for conuctng meta-analyss, accountng for varaton n all levels. Fxe- an ranom-effects moels. The tratonal smplest ranom-effects moel n Equaton 7 typcally follows a two-level structure usng the one-way ANOVA moel (Hox, 00; Konstantopoulos, 0). The two-level ranom-effects moel accounts for both wthn-stuy an between-stues varances n level an level, respectvely. The frst level moel (wthn-stuy level) s = δ + ε (0), the secon-level moel (between-stues level) s δ = δ +, () u an the combne moel wth wthn-stuy an between-stues levels leas to = δ + + ε. () u All terms n Equatons 0,, an were efne extensvely for Equaton 7. The true effect szes δ are estmate usng a Bayesan approach base on the estmates gven n Equatons an 3.

23 If the effect szes are not sgnfcantly fferent across stues an the between-stues varance τ s not sgnfcantly fferent from zero, the effect szes across stues are consere homogeneous an the fxe-effects moel ( = δ + ε ) can be aopte. If the between-stues varance τ s sgnfcantly fferent from zero, effect szes are treate as heterogeneous, an the ranom-effects moel ( = δ + + ε ) woul be aopte. The overall populaton effect across u stues δ s then estmate usng a weght that nclues both the samplng varance an between-stues varance. Homogenety test. The between-stues varance τ s estmate usng REML an s teste (.e., H : τ 0 ) to examne whether effect szes estmate the common populaton effect sze 0 = usng a ch-square test, whch correspons to the Q test n the tratonal meta-analyss approach. Multvarate Meta-analyss Several types of stues wth multvarate ata structures can lea to epenence of effect szes n meta-analyss: multple-treatment stues, multple-enpont stues, an multple tmepont stues. Gleser an Olkn (994) prove such covarance matrces for two types of expermental esgns that may generate epenent effect szes n the same stuy. Frst, when a stuy has multple treatments an one common control group gven a epenent varable, the effect szes from ths stuy are epenent on each other; such stues calle multple-treatment stues. Secon, n an expermental stuy that measures multple outcomes wth sngle treatment an control groups, the effect szes are not nepenent; ths s calle a multple-enpont stuy. Ths ssertaton focuses on multple-enpont stues but smlar results are expecte to hol for other epenence structures. It s esrable that each stuy has the exact same set of

24 outcome measures n multvarate meta-analyss. However, t s rare that all stues have the same set of effect szes. Thus, the multvarate meta-analyss methos scusse n ths ssertaton allow fferent stues to have fferent subsets of effect szes or outcome varables. Three types of multvarate meta-analyss approaches to estmate mean effects across stues are escrbe: Frst, the multvarate fxe-effects meta-analyss approach wth GLS methos, next the multvarate two-level approach, an thr, the unvarate three-level approach. The frst approach estmates mean effects usng GLS methos. In the secon approach, the mean effects are estmate usng GLS methos, gven the REML estmates of the varances an covarances across stues. The mean effects are estmate usng the nverse varances base on REML n the unvarate three-level approach. GLS Metho The GLS estmaton metho accounts for the epenence of effect szes wth a sample varance-covarance matrx (Berkey, Anerson, & Hoagln, 996; Berkey, Hoagln, Antczak- Bouckoms, Mosteller, & Coltz, 998; Gleser & Olkn, 994; Rauenbush, Becker, & Kalaan, 988). In a multple-enponts expermental esgn (Gleser & Olkn, 994; Rauenbush et al., 988), the populaton varances ( σ ) an covarances ( σ ) of effect szes n the same stuy j j' are estmate, respectvely, as j T C n ˆ + n j ψ j = ˆ σ j + ; j =,..., p ; =,..., k, (3) T C T C n n ( n + n ) an T C n + n j j ' r jj ' ψ ˆ jj ' = ˆ σ r ', ' T C jj + (4) j j T C n n ( n + n ) 3

25 where T n an C n are the sample szes of the treatment group an control group n stuy, respectvely, j an j' are effect-sze estmates for outcomes j an sample correlaton between the two outcome varables Y j an Y j'. ' j n stuy, an r ' s the jj Each stuy has a complete vector of true effect szes, δ = δ,..., δ )' for the corresponng ( p outcome measures. Dfferent stues coul report fferent subsets of the complete effect szes but I conser the case where each stuy has ts own vector of all effect-sze estmates, =. Each stuy prouces ts own estmate varance-covarance matrx ψ ˆ ( ) wth ' (,..., p) menson p p whch accounts for the epenence of effect szes n the stuy. Varances an covarances n the matrx are compute usng Equatons 3 an 4, respectvely. The combne estmate column vector of effect szes for k prmary stues ' ( = (,..., k ) ) has menson p where p = p. The corresponng combne estmate k varance-covarance matrx wll be a block agonal matrx ψˆ of mensonalty p p. In the matrx ψˆ, the man agonal blocks are the varance an covarance matrces ψ ˆ ( ) from the nvual prmary stues, an off-agonal blocks are zero whch ncates the nepenence of stues. Ths s represente as ψ() ˆ 0 0 ψˆ = 0 0. (5) 0 0 ψ(k) ˆ The multvarate fxe-effects moel. The multvarate fxe-effects moel (wth homogeneous effect szes over stues) s 4

26 = Xδ + ε, (6) Where s the combne estmate column vector of effect szes for k prmary stues wth menson p. The esgn matrx X has menson p p. The values of the esgn matrx are ummy varables. If an effect sze s estmate for a corresponng outcome measure n stuy, ts X matrx element s equal to, an equal to 0 otherwse. δ s a column vector whch contans the common effect szes δ,..., δ )' for the corresponng outcome measures across ( p stues. The common effects ( δˆ j ) for the p enponts across stues are estmate by ˆ δ = p ˆ ˆ ( δ,..., δ )' = ( X'ψˆ X) Xψ ' ˆ, (7) where the matrx ( X ' ψˆ X) s the estmate varance-covarance matrx of common effect szes. The stanar errors of the p common effects are the square roots of the agonal elements n the varance-covarance matrx ( X ' ψˆ X). Fnally, the vector of errors ε has menson p. The errors ε n stuy are assume to have a multvarate normal strbuton, whch s expresse as ε ~ N (0, ψ( )). For example, when stuy reports two effect szes for outcomes an, the varance an covarance matrx ψ() of the two effect szes s σ σ σ σ, (8) where σ an σ are the samplng varances of effect szes for outcomes an, respectvely, an σ s the covarance of the two effect szes n the stuy. 5

27 When each stuy coul have two effect szes (p = ) for two outcome measures over k stues, the estmate multvarate regresson GLS moel n Equaton 6 s expresse n matrx notaton as 3 k k 0 0 ε ε 0 ε δ = 0 + ε δ 0 ε k 0 ε k 3. (9) The esgn matrx X has ummy varables n the frst an secon columns. The frst column has a ummy varable that s equal to when the effect sze s estmate for the frst outcome measure n stuy, an equal to 0, otherwse. In the secon column, the ummy varable takes value when the secon effect sze (for the secon outcome measure) s reporte, 0, otherwse. Ths matrx shows that the frst stuy reporte two effect szes for two outcomes whereas the secon stuy reporte only the effect sze for the frst outcome. The multvarate ranom-effects moel. The multvarate ranom-effects moel s * = Xδ + ε. (0) In the ranom-effects moel, the varaton of effect szes s compose of the true fferences n stues (between-stues varance) an the wthn-stues varance, an each true effect sze s * sperse from the average populaton mean effect sze δ j. * δ s a column vector whch * * contans the average effect szes ( δ,..., δ ) for the corresponng outcome measures across ˆ p stues. The average effects ( δ ) for the p enponts across stues are estmate by * j 6

28 7, ˆ ' ) ˆ ( )' ˆ,..., ˆ ( ˆ * * * * * ψ X X X'ψ δ = = p δ δ () an the matrx * ) ˆ ' ( X ψ X s the estmate varance an covarance matrx of the average effect szes. In aton to the samplng varances an covarances among effect szes ) ( ψ wthn stuy, the varance-covarance among effect szes across stues, τ, s nclue to capture all varaton n effect szes n the multvarate ranom-effects moel. The total varance-covarance among effect szes n a stuy, * () ψ, s thus obtane by the sum of wthn-stuy-varancecovarance ) ( ψ an between-stues-varance-covarance, τ. ) ( ) ( + = + = τ τ τ τ σ σ σ σ τ ψ ψ () Thus, the stanar errors for the corresponng overall effects are compute by takng the square roots of the agonal elements of the varance-covarance matrx among the overall effects ˆ ) ˆ ( = X ψ X' v δ. All other terms were efne for Equatons 0 an, or earler. The multvarate ranom-effects moel for two effect szes (p=) across k stues s ε Xδ + = * or * * 3 + = k k k k ε ε ε ε ε ε δ δ (3) Homogenety test. The Q statstc for the homogenety test s compute as Xδ δ Xψ ψ ' ' ˆ ˆ ˆ ˆ ' = Q, (4)

29 whch follows a ch-square strbuton wth egrees of freeom equal to the menson p mnus p. The Q statstc tests the null hypothess that all effect szes across all outcomes an all stues arse from the same populaton, whch s enote as H δ... = δ =, (5) 0 : = kp δ where δ s the common populaton effect sze mean across all outcomes an stues. The matrx notaton for the homogenety of effect sze across k stues an two outcomes s represente as Xδ + ε δ = or = [ ] +, k k ε ε ε εk ε k (6) where all terms were efne n Equatons an, except for the esgn matrx X whch s a column vector of s. The common populaton effect sze δ s estmate as ˆ δ = ( X'ψˆ X) X' ψˆ. (7) The stanar errors of the overall common effects are the square roots of the agonal elements of ( ˆ X ' ψ X) n Equaton 7. Thus, the overall populaton effect n the ranom-effects moel ˆ δ s estmate by δ = ( X' ψˆ * X) X' ψˆ *. Multvarate Two-Level Moel Kalaan an Rauenbush (996) evelope a two-level multvarate moel to conuct multvarate mxe-effects analyses n multple-enponts stues. The frst level moel (for wthn the stuy) represents the relatonshp between the true effect szes an the corresponng 8

30 effect-sze estmates n each stuy. The secon level (between-stues) s for the strbuton of these true effect szes aroun ther populaton mean. expresse as Multvarate two-level ranom-effects moel. The level (wthn stuy) moel s p = δ X + e : =,..., k; X = 0 or (8) j j j= j j j where j an δ j are the effect-sze estmate an the corresponng true effect sze for outcome j n stuy. The varable X j ncates the presence of each effect sze for the corresponng outcome j n stuy. Each stuy s assume to have a complete vector of true effect szes, δ = δ,..., δ )' for the p outcome measures. Each stuy coul report a fferent subset of effect ( p szes an stuy has own vector of sample effect szes, = ' (,..., p). For example, n twoenpont stues, f stuy reports two effect-sze estmates, the vector estmates the effect sze for outcome, the value Thus, the equaton becomes for j = s (, ) '. If stuy X s equal to, an equal to 0, otherwse. = δ X + δ X + e = δ ) + δ (0) + ε. (9) ( For j =, = δ X + δ X + e = δ 0) + δ () + ε. (30) ( If we express ths n matrx notaton, 0 δ = Xδ + e, or. 0 + e = δ e (3) 9

31 The errors e j are assume to have a multvarate normal strbuton e ~ N(0, ψˆ ( )), where ψ ˆ ( ) s a p p varance-covarance matrx n stuy. In ths matrx ˆ ψ ( ), the varances ( ) an covarances ( σ j j' ) of effect szes are estmate, usng Equatons 3 an 4, respectvely, so that σ j e e 0 N, 0 σ σ ~ σ σ. (3) The between-stues moel (level ) s δ = δ + u, (33) j j j where δ j s the true effect sze n stuy, δ j s the effect sze mean across stues for the corresponng outcome measure. The ranom effects u j are evatons of the true effect szes from the populaton means δ j, an are assume to have a multvarate normal strbuton wth a mean of zero an a p p varance an covarance matrxτ, whch s expresse as u ~ N ( 0, τ), u u 0 ~ N, 0 τ τ τ. τ (34) The combne moel nclung the wthn-stuy moel n Equaton 8 an the between-stues moel n Equaton 33 s j = p j= j (δ + u ) X + e. The ranom components n the varance an j j j * covarance matrx τ are estmate usng REML an the estmates of the overall effects ˆ δ j are estmate on the GLS metho, gven the estmate varance an covarance n τ. Multvarate two-level fxe-effects moel. If the effect szes are not sgnfcantly fferent across stues, an there are no true fferences among effect szes n stues, the combne 0

32 moel s j = p j= (δ ) X + e, whch leas to the multvarate fxe-effects moel. Ths fxe-effects j j j moel s comparable to the moel n the GLS metho n Equaton 0. The common-effects estmates for each outcome δˆ j n j = p j= (δ ) X + e are thus comparable to the corresponng j elements n the effect-sze column vector δ ( δ δ )' = ( X'ψˆ X) Xψ ' ˆ, n Equaton. j j ˆ ˆ ˆ =,..., p Homogenety test. The between-stues varances an covarances n τ are teste usng a ch-square strbuton to examne whether effect szes estmate the common populaton effect 0 ; j = jj' = sze ( H τ 0; τ 0 ), whch correspons to the Q test n the multvarate fxe-effects metaanalyss approach wth GLS methos (Equaton 4). Unvarate Three-Level Moel A major lmtaton of the multvarate meta-analyss approach for both the GLS metho an the two-level moel s that they requre the sample covarance matrx of the effect szes wthn a stuy. However, recently Van en Noortegate et al. (03) have propose a unvarate three-level moel whch oes not requre a sample covarance matrx for the multvarate effectsze ata. The unvarate three-level moel was extene from the tratonal two-level moel wth the ncluson of an atonal level (.e., the outcome level) to account for the epenence of effect szes. Ths moel contans three types of ranom varance components over three levels: the partcpant-level samplng error (level ), the outcome-level varaton (level ), an the stuy-level varaton (level 3). Thus, the effect szes vary over partcpants, outcomes, an stues. Van en Noortgate et al. (0) smulate a multvarate two-level ata set to nvestgate how the between-stues varance n the unvarate three-level moel reflects the epenence of effect szes n the multvarate two-level moel. They foun that the parameter estmates an the

33 corresponng stanar errors n the unvarate three-level moel were not base n comparson to those from a multvarate two-level moel wth a smulaton stuy. The multvarate two-level moel to generate ata. Van en Noortegate et al. (03) smulate multvarate two-level ata for two outcome measures n multple-enpont stues for 300 stues wth a sample sze of 50 per group (the treatment group an the control group). The raw ata were use to examne four moels (summarze n Tables. an.). All parameter values (.e., true values) to generate ata are escrbe n Table.. In the multvarate two-level moel for two outcomes, the partcpant-level (level-) moels for two outcomes are Y = δ + δ T + e an s 0 s s Ys = δ 0 + δts + es, =,..., p; s =,... n; j =,..., p (35) where Y sj s an outcome value for partcpant s for outcome j n stuy an T s a ummy treatment factor varable. If a partcpant s s assgne nto the treatment group, hs or her T s value s, an the value T s s 0 for control-group members. The coeffcent δ 0 j s the expecte value (.e., ntercept) for the control group an δ j s the fference n the expecte values between the treatment group an the control group (.e., slope for the treatment effect or effect sze) for outcome j n stuy. In a stuy, each partcpant prouces two scores (for two outcomes), an the errors e sj n the two outcomes are multvarate normally strbute as e e s s σ e ~ N 0, σ ee σ σ ee e. (36)

34 The stuy-level (level-) moels for outcome from Equaton 35 are 0 = δ 00 + u0 δ an = δ 0 + u. δ (37) For outcome, 0 = δ 00 + u0 δ an = δ 0 + u δ, (38) where δ 00 an δ 00 are the means of expecte values (ntercepts) across stues for outcomes an, respectvely. The parameters δ 0 an δ 0 are the means of the treatment effects across stues for each outcome. Level has four ranom components: two ranom effects from the expecte values n the control groups, u 0 an u 0 treatment effects n the treatment group ( u an u, an the two ranom effects from the ). The error u gj where g s the group (0=control group an =treatment group) s multvarate normally strbute as u ~ N (0, τ). τ s a varance-covarance matrx for the resuals of the two treatment effects an the two expecte values for the corresponng outcomes, specfcally, u u u u 0 0 τ ~ 0, τ N τ τ τ τ τ 00 0 τ 0 0 τ τ 0 0 τ τ τ τ τ 0 0 τ. (39)or The generate ata were use to examne four moels. 3

35 Moel. In moel, the two separate expecte values ( δ 00 an δ 00 ) for each outcome were aggregate an the overall expecte value ( δ 000 ) was estmate. In the same way, one overall treatment effect ( δ 00 ) across stues was estmate, whch was an aggregaton of the two separate treatment effects ( δ 0 an δ 0 ). The moels for the expecte values for the stuy level (Equatons 36 an 37) were mofe nto 0 = δ u0 δ an 0 = δ u0, δ (40) where δ 000 s the overall expecte value an u 0 an u 0 are the evatons from the overall expecte value δ 000 (.e., not from the separate expecte values δ 00 an δ 00 corresponng outcome measures n Equatons 37 an 38. ) for the For treatment effects, δ an = δ 00 + u δ = δ + u, (4) 00 where δ 00 s the overall treatment effect n the stuy level an u an u are the stuy- specfc evatons from the overall treatment effect δ means ( δ 0 an δ , not from the separate treatment-effect ). Thus, when the treatment effects truly ffer by outcome, the varaton of the treatment effects aroun the overall treatment effect ( δ varaton about the separate treatment effect means ( δ an δ ncrease the between-stues varances ( σ u an 0 00 ) s lkely to be greater than the 0 ). Ths fference woul σ u ) for each treatment effect, an ecrease the

36 covarance between the two effects ( σ u u ) across stues. For example, the estmate overall treatment effect δ 00 = 0.09 was n the mle of the two separate treatment effect means (0.09 an 0.30) from moel, as expecte. In moel, n the control group wth n partcpants, the expecte values for outcome j have two sources of errors, the samplng varaton σ e j n (.e., the error for the expecte values) n level an the between-stues varance σ n level. The multlevel metho assumes the u 0 j nepenence of resuals across levels (Rauenbush & Bryk, 00), an the total varance n the σ e j expecte value s the sum of the samplng varaton an the between-stues varance + σ u. n 0 j Smlarly, the total covarance between the expecte values over two levels s the sum of the sample covarance ( σ e je j ' n ) n level an the covarance between expecte values ( σ ) σ e je j ' between the two expecte values n level as σ +. u0 ju0 j ' n u 0 j u 0 j ' The treatment effect (.e., effect sze) s the mean fference between the control an treatment groups. The expecte mean fferences between two groups vary wthn stues an across stues because of the samplng varaton an the between-stues varance, respectvely. Specfcally, E( Y jt ) = ( δ + δ + e ) ( δ + e ) (4) = δ 0 j 0 j Y + u jc j j + e jt jt e jc 0j. jc 5

37 Unlke the expecte value for the control group, for the expecte mean fferences between two groups, two sources of samplng varaton are nvolve for the effects: samplng varaton n the control group e j C (.e., the error for the expecte values n the control group) an samplng varaton n the treatment group e j T (.e., the error for the expecte value n the treatment group) n Equaton 4. If we assume that the samplng varance s the same n both groups, the total σ e je j ' covarance of the expecte treatment effects s σ +. u ju j' n Table.: Comparson of the Two-Level Multvarate Moels. Parameter Values Moel δˆ (SE) Moel δˆ (SE) Fxe effects Intercept -.09 (.00) -.09 (.00) Outcome ( δ 0).000 Outcome ( δ 0).000 Treatment effect.09 (.0).09 (.0) Outcome ( δ 0).00 Outcome( δ ).300 Ranom effects Level Intercept Treatment effect τ τ Level τ 0 τ τ σ τ σ e ee 3 σ e Moel. Moel was evelope to nvestgate how the covarance of the expecte treatment effects for two outcomes n the stuy level reflects the total covarance of the expecte 6

38 treatment effects over two levels; the partcpant level an the stuy level. For ths, moel esj assume that the sample covarance was zero, specfcally ~ N(0, σe ) e sj or σ e N 0, 0 0. σ e Van en Noortegate et al. (0) foun that the covarance between the treatment effects n level n moel was equal to the total covarance of the expecte treatment effects over two levels, σ e je j ' σ + (whch was u ju j ' n = ) n moel. Therefore they argue that the 50 gnore samplng covarance between outcomes n level appears as part of the overestmate covarances between the expecte treatment effects for the two outcomes n level. Moel 3. Moel 3 ncorporates an atonal level (the outcome level as a wthn-stues level) nto moel to account for the epenence of outcomes. Levels,, an 3 are the partcpant level, the outcome level wthn stues, an the stuy level, respectvely. Each level accounts for a fferent source of errors: partcpant-level samplng error, between-outcomes varance, an between-stues varance, respectvely (see Table ). The partcpant-level (level-) moel s Y = δ + δ T + e, e ~ N (0, σ ): =,..., k; j =,..., p; s,...,. (43) sj 0 j j s sj sj e = n Y sj s a value of outcome j of partcpant s n stuy. T s a ummy treatment factor (= treatment group an 0= control group). δ 0 j an δ j are the expecte values n the control group an the treatment effects for outcome j n stuy, respectvely. The outcome level (level ) moel s δ 0 j = δ 00 + v 0 j 7

39 δ j = δ 0 + v j, wth v v 0 j j 0 ~ V0 V N 0,. V 0 V (44) For the stuy level (level 3), δ 00 = δ u00 δ = δ + u, wth u ~ τ τ N 0,, (45) u0 τ 0 τ where δ 00 an δ 0 are the expecte value of the mean n the control group, an the treatment effect mean over outcomes wthn stues, an δ 000 an δ 00 refer to the overall expecte value an the overall treatment effect across stues an outcomes. The errors v j an u are assume to have multvarate normal strbutons wth mean vectors of zeros an varancecovarance matrces between the expecte values an the treatment effects. These are V an τ n the outcome level an the stuy level, respectvely. Thus v ~ N ( 0, V) an u ~ N ( 0, τ). The expecte value an the treatment effect vary over outcomes an stues n ths moel. The varances of the expecte values for the two outcomes n moel are restrbute nto the varances over the outcome level (level ) an the stuy level (level 3) n moel 3, the between-outcomes varance an the between-stues varance n the stuy level. Thus, the total varance of the expecte values was ( ) from levels an 3 n moel 3 was mway between the two expecte value varances (0.08 an 0.) for outcome an outcome n the stuy level n moel n Table.. Smlarly, the total varance of the treatment effects 8

40 over the two levels was = n moel 3. Ths value was between the two stuylevel treatment effect varances (0.098 an 0.00) for outcomes an n moel. The between-stues varances for the expecte values an the expecte treatment effects n the stuy level (level 3), an n moel 3 were equal to the covarances among the expecte values an the expecte treatment effect for two outcomes (level ) n moel. Also, conserng that the covarances at the stuy level were the total covarance of the expecte treatment effects an the expecte values over two levels the partcpant level an the stuy level. The overall expecte value, the overall treatment effect across stues, an the corresponng stanar errors were not change from those n moels an. Moel 4. Whle Moel 3 was the three-level moel for raw ata, Moel 4 was the unvarate three-level moel evelope for effect szes. That s, the generate ata were use to compute effect szes per stuy. The effect szes were use to analyze the unvarate thee-level moel. Thus, the expecte value for the control group was exclue n the unvarate three-level moel, j j 0 = δ + e j 0 00 j δ = δ + v δ = δ j + u 0 wthe wthv j j wthu ~ N(0, σ ), ~ N(0, σ ), 0 e j v ~ N(0, σ ) : =,..., k; j =,..., p. u (46) Equaton 46 states that the sperson of effect szes conssts of the samplng error, betweenoutcomes varance, an between-stues varance. Here, j s the observe effect sze for outcome j n stuy, estmate usng Equatons an, δ j s the true effect sze for the corresponng observe effect sze ( j ), an e j s a resual whch s asymptotcally normally strbute wth a mean of zero an a varance σ e j. The varance σ e j s replace by the samplng 9

41 varance σ ˆ e j, estmate usng Equaton 3. The estmate effect sze, j s thus normally strbute wth mean δ j an varance σ. Also δ 0 s the effect-sze mean over outcomes e j wthn stues. The ranom effect v j s the evaton of δ j from the mean effect δ 0 wthn stuy. The true effect szes δ j are normally strbute wth mean δ 0 an varance σ v. Fnally, δ 00 s the overall mean effect sze across stues. The ranom effects u 0 are resuals from the overall effect δ 00 for each stuy. The effect-sze means δ 0 over outcomes are normally strbute wth a mean of δ 00 an the error varance of the ranom effects u 0 equal to σ u. In moel 4 the estmate overall treatment effect δ 00 across stues, the corresponng stanar error, between-outcomes varance σ v, an the between-stues varance σ u whch reflecte the total covarance of the expecte treatment effects an the expecte values over two levels were almost entcal to the estmates n moel 3. If the effect szes are not sgnfcantly fferent across outcomes an stues, an the between-outcomes varance (level ) an the between-stues varance (level 3) are zero 0 u = v = ( H ; σ 0; σ 0 ), the effect szes across stues are consere homogeneous an the fxeeffects moel ( = δ + e ) can be aopte. If the varances are sgnfcantly fferent from zero j j j the ranom-effects moel ( = δ + v + u + ε ) woul be aopte. The overall populaton j effects are estmate wth weghts that ncorporate both the samplng varance an betweenstues varance. Homogenety test n the unvarate three-level moel. The between-outcomes an between-stues varances are estmate usng REML an are teste wth a ch-square test to examne whether effect szes estmate a common populaton effect sze. 30

42 Table.: Comparson of the Three-Level Moels. Moel 3 Moel 4 Fxe Effects Intercept 0.003(.0) Treatment Effect 0.0(0.0) 0.0(0.0) Ranom effects Level Level Level * * The samplng varances were compute before analyzng moel 5. Purpose of Ths Stuy A major lmtaton of multvarate meta-analyss conucte usng the GLS approach an the multvarate two-level moel s that they requre the sample covarance matrx of effect szes wthn a stuy. The covarance of the two effect szes nvolves the sample correlaton between two outcomes. However, t s rare that all prmary stues report the sample covarance or correlaton between outcomes. Van en Noortegate et al. (0) evelope the unvarate threelevel approach whch oes not requre the sample covarance matrx of effect szes to account for the epenence of effect szes. They foun wth a smulaton stuy that the unvarate three-level moel yele approprate parameter estmates an stanar errors corresponng to those from the multvarate two-level approach. Thus, the man purpose of ths ssertaton s to compare the corresponng parameter estmates an the stanar errors n the multvarate meta-analyss usng the GLS metho an the unvarate three-level moel approach wth a smulaton stuy. For ths, two multvarate ata sets were generate uner the ranom-effects moel: one set wth two outcomes an the other set wth fve outcomes. Each generate ata set was use to estmate the overall effect over 3

43 outcomes, an the specfc-outcome effects assocate wth the overall effect. For example, three parameters were estmate from the ata set wth two outcomes: the overall effect across two outcomes an the two specfc outcome effects. The escrpton of the parameter estmates an ther stanar errors whch I ntene to compare n both approaches follows. Overall Effects n GLS an Three-Level Approaches GLS approach. The sperson of effect szes s explane by two ranom varance components n the multvarate ranom-effects moel: the samplng varance an covarance among effect szes ψ () wthn stuy an the varance-covarance among effect szes across stues τ.thus, the total varance-covarance among effect szes s obtane by the sum of wthn-stuy-varance-covarance an between-stues-varance-covarance σ σ τ τ ψ ( ) = ψ( ) + τ = +. (48) σ σ τ τ The overall mean effect-sze estmate (δˆ ) across outcomes, across all stues s estmate as ˆ δ = ( X'ψˆ * X) X' ψˆ *, escrbe n Chapter. The corresponng stanar error s compute as the square root of v ˆ. All terms were efne n secton.4.. ˆ = ( X'ψ * X) δ effect follows Three-level approach. The unvarate three-level moel to estmate the overall populaton j j 0 = δ + e j 0 00 j δ = δ + v δ = δ j + u 0 wth e j wth v j wth u ~ N(0, σ ), 0 e ~ N(0, σ ), v j ~ N(0, σ ) : =,..., k; j =,..., p. u (47) 3

44 The coeffcent δ 00 s the overall effect across outcomes an stues. All other terms were efne for Equaton 46 or above. Outcome-Specfc Effects n GLS an Three-level Approaches GLS approach. The overall effect for each outcome s estmate by ˆ ˆ = ( δ, δ )' = ( X' ψˆ X ) X' ψˆ. The stanar errors for the corresponng overall effects ˆ δ are compute as the square roots of the agonal elements of the varance-covarance matrx among the overall effects, v δ ˆ. ˆ = ( X' ψ X ) Three-level moel. The unvarate three-level moel to estmate the overall mean effect for separate outcomes s δ j j = δ + e j δ = δ X δ = δ = δ 0 0 j + u + δ X + u () () + v j wth e j wth v j wth u wth u ~ N (0, σ ), ~ N(0, σ ), ej v ~ N (0, σ u() ~ N (0, σ ), an u() ), (49) where j s the observe effect sze for outcome j ( j =,..., p ) n stuy ( =,..., k ) an s normally strbute wth a mean of the true effect sze δj an varance σ. Effects δ an δ e j are the true mean effects for outcomes an, respectvely. The ummy varable X j shows the presence of an effect sze for the j th outcome. For example, f stuy estmates the effect sze for outcome ( ), the corresponng X value for the slope δ s equal to, an equal to 0, otherwse. The ranom effect v j s the resual wthn stuy. The coeffcents δ 0 an δ 0 are the overall mean effects across stues for outcomes an, respectvely. The ranom effects u an u are the evatons from each overall effect δ 0 an δ 0. Thus, the true treatment effect 33

45 for each outcome ( δ or δ ) s normally strbute wth mean δ 0 or δ 0 an varance σ or u () σ u (), respectvely. In summary, the overall populaton effect δ 00 an the specfc-outcome effects assocate wth the overall effect, δ 0 an δ are comparable to the ˆ δ = ( X'ψˆ * 0 X), an ther stanar errors n the unvarate three-level moel X' ψˆ * an the corresponng elements n the commoneffects vector δˆ ˆ ˆ = ( δ, δ )' = ( X' ψˆ X ) X' ψ, respectvely (Table.3). ˆ Table.3: Comparsons of Estmates Between Three-Level an GLS Metho Approaches. The mean effect sze n the ranom-effects moel The two mean effect szes n the ranom-effects moel Three-level estmates GLS metho estmates ˆ δ 00 ˆ δ = ( X'ψˆ X) Xψ ' ˆ ˆ δ 0 an ˆ δ 0 δˆ ˆ ˆ = ( δ, δ )' = ( X' ψˆ X ) X' ψ ˆ 34