ESTIMATES OF VARIANCE COMPONENTS IN RANDOM EFFECTS META-ANALYSIS: SENSITIVITY TO VIOLATIONS OF NORMALITY AND VARIANCE HOMOGENEITY

Transcription

1 ESTIMATES OF VARIANCE COMPONENTS IN RANDOM EFFECTS META-ANALYSIS: SENSITIVITY TO VIOLATIONS OF NORMALITY AND VARIANCE HOMOGENEITY Jeffrey D. Kromrey and Krstne Y. Hogarty Department of Educatonal Measurement and Research, Unversty of South Florda, Tampa, FL 3360 KEY WORDS: Meta-analyss, Varance Components, Random Effects Models Meta-analyss s a statstcal technque through whch nformaton obtaned from a collecton of ndependent studes s analyzed and syntheszed, potentally provdng a more complete representaton of the phenomenon under nvestgaton. Models for meta-analyss may be roughly dvded nto those based upon fxed effects and those based upon random effects (Feld, 00; Hedges, 994; Hedges & Vevea, 998; Raudenbush, 994). The prmary goal of meta-analyss s to obtan estmates of populaton effect szes and confdence bands around those estmates. For example, recent research n the area of educatonal reform related to the Natonal Scence Foundaton s Systemc Intatves (Kromrey et al., 00) has nvestgated the use of effect szes as ndces of the effectveness of systemc change. These efforts have been drected towards syntheszng student outcome dfferences across subject areas, grade levels, and years of partcpaton. Because sample effect szes obtaned for a metaanalyss typcally present dfferent magntudes of estmaton error, weghted means and varances are used to obtan the estmates of populaton effect szes and confdence bands. For fxed effects models, these weghts are gven by v = σ where σ = the estmaton varance of the th effect sze. In contrast, for random effects models, the weghts used are gven by w = σ τ + where τ = σ = the populaton varance n effect szes. Because τ s not known, t must be estmated from the observed sample effect szes. Three estmators of τ have been suggested n the lterature. One estmator s derved from the observed varance of the sample effect szes and the average of the estmaton varances. s σ! σ = k where s = the ordnary sample varance of effect szes, and k = the number of studes n the metaanalyss. A second estmator, and the most frequently employed n meta-analyss, s derved from the obtaned value of Q (the frequently used test of homogenety of effect szes, Hedges & Olkn, 985). ( k ) Q = c v where c = v. v Fredman (000) presented the analytcal dervaton of the varance of these estmators under the assumpton of normalty and demonstrated that although both estmators are unbased, the varance of s less than that of σ! when τ s close to zero, and hence s a more effcent estmator. Conversely, when τ s large, σ! s a more effcent estmator. A maxmum lkelhood approach to the estmaton of τ was presented by Bggerstaff and Tweede (997) based on the lkelhood equaton: k k k ( y ) ;, = exp = σ + τ = σ + τ ( ) ( ) l y τ π where the y are the observed sample effect szes and = the populaton mean effect sze. The estmaton of τ by maxmum lkelhood (.e., τ ) also provdes an estmate of the varance of τ whch allows the constructon of confdence bands around τ. Further, Bggerstaff and Tweede (997) derved the varance of, and provded a method for constructng confdence bands around ths frequently used estmator. Although the samplng errors of and τ have been derved, an assumpton of normalty was requred. The assumpton of normalty for many educatonal varables s often a tenuous one (Mccer, 989). Further, prevous research (Hogarty & Kromrey, 00) has suggested that commonly used 963

2 effect sze estmates themselves (e.g., standardzed mean dfferences) become based under condtons of non-normalty or heterogeneous varances n the prmary studes. PURPOSE OF THE STUDY The behavor of the three estmators of τ (the σ!,, and τ ; and the nterval pont estmates estmates constructed usng andτ ) under condtons of non-normalty and varance heterogenety cannot be determned analytcally. For such problems, Monte Carlo methods must be appled for the nvestgaton of ther samplng behavors. The current study provdes estmates of the bas and samplng error of these statstcs under condtons n whch the assumptons of normalty and homogenety of varance have been volated. METHOD Ths research was a Monte Carlo study desgned to smulate meta-analyses. The use of smulaton methods allows the control and manpulaton of research desgn factors and the ncorporaton of samplng error nto the analyses. Observatons n prmary studes were generated under known populaton condtons, then prmary studes were combned to smulate meta-analyses. Each prmary study conssted of two groups of observatons (e.g., a treatment and control group). For each prmary study, the Hedges g effect sze was calculated based on the smulated data (Hedges & Olkn, 985). The values of Hedges g obtaned from the samples n each meta-analyss were then used to estmate τ va the three methods descrbed above (.e., σ!,, and τ ). The Monte Carlo study ncluded sx factors n the desgn. These factors were (a) populaton value of τ ( 0,.0,.33,.50, and.00), (b) the number of prmary studes n each meta- analyss (5, 0, 50, and 00), (c) the sample szes of the two groups n each prmary study (wth sample szes rangng from 5 n each group to 00 n each group, as well as unbalanced condtons), (d) group varances n the prmary studes (varance ratos of :, :4, and :8, as well as a homogeneous varance condton), (e) populaton dstrbuton shape n the prmary studes (condtons wth skewness and kurtoss values, respectvely, of 0,0 (.e., normal dstrbuton);,3;.5,5;,6; and 0,5), and (f) the magntude of the populaton mean effect sze ( = 0,.0,.50,.80). Ths research was conducted usng SAS/I verson 8.. Condtons for the study were run under Wndows 98. Normally dstrbuted random varables were generated usng the RANNOR random number generator n SAS. A dfferent seed value for the random number generator was used n each executon of the program. For condtons nvolvng non-normal populaton dstrbutons, the non-normal data were produced by transformng the normal random varates obtaned from RANNOR usng the technque descrbed by Fleshman (978). The program code was verfed by hand-checkng results from benchmark datasets. For each condton examned n the Monte Carlo study, 5000 meta-analyses were smulated. The use of 5000 replcatons provdes a maxmum 95% confdence nterval wdth around an observed proporton that s ± 0.04 (Robey & Barckowsk, 99). The relatve effectveness of the estmators was evaluated n terms of the emprcally estmated statstcal bas and standard errors for the pont estmators, and n terms of confdence band coverage and mean confdence band wdth for the nterval estmates. RESULTS Statstcal Bas and Standard Error The dstrbutons of the statstcal bas estmates for the three pont varance estmators across all condtons examned n ths study are dsplayed as box and whsker plots n Fgure. Examnaton of ths fgure revealed a host of condtons n whch all three methods evdenced postve bas, although the observed varance ndcator appeared to exhbt both greater average bas and somewhat more varablty n bas across the condtons examned. Bas n Estmate Observed Varance Q Based Maxmum Lkelhood Varance Estmator Fgure. Dstrbutons of Statstcal Bas Across Condtons Examned n the Monte Carlo Study Fgure llustrates the dstrbuton of the standard errors of the pont estmators across the condtons examned. The standard errors for all three estmates were reasonably smlar wth each 964

3 estmator evdencng errors of consderable magntude for a number of the condtons examned. These errors were somewhat more pronounced for the observed varance estmator. Evdence of statstcal bas and the assocated samplng error, were further nvestgated n relaton to the central desgn factors n the study. Standard Error Observed Varance Q Based Maxmum Lkelhood Varance Estmator Fgure. Dstrbutons of Standard Error Across Condtons Examned n the Monte Carlo Study Performance wth normal dstrbutons and homogeneous varances. The estmates of bas and standard errors across condtons n whch these two crtcal assumptons are met suggested that the observed varance estmator σ! evdenced the least statstcal bas, but the largest standard errors. Across ths set of condtons, the bas of σ! dd not exceed.07. Further, bas was greatest under condtons wth small sample szes n the prmary studes and large values of τ. Wth large samples, the bas n ths estmator was essentally zero, regardless of the number of studes ncluded n the meta-analyss. In contrast, wth τ > 0, the estmators and τ evdenced substantal bas under condtons of small k or small n. However, the standard errors of these estmators were notably smaller than that of the observed varance estmator. In general, the samplng error of τ was slghtly smaller than that of. The bas of the maxmum lkelhood estmator reached as large as -.38, wth τ =, under small samples, whle that of reached -.3 n these condtons. Wth large k and large n, the maxmum lkelhood estmator evdenced very lttle bas, but the estmator remaned substantally based wth large values of τ. For example, wth τ =, k = 00 and n = n = 00, the bas of was 0.6. Performance wth normal dstrbutons and heterogeneous varances. Under condtons of normal dstrbutons and heterogeneous varances n the prmary studes, the unbased nature of the observed varance estmator was no longer evdent as substantal bas was seen n these estmates, especally under condtons wth small samples n the prmary studes (wth bas reachng as large as.06, wth k = 5, n = 6, 4 and a varance rato of :8). Wth large values of n, however, ths approach provded reasonably unbased estmates f sample szes were equal. In contrast, the and τ estmators evdenced substantal bas n these condtons as the magntude of τ ncreased, although the bas of was typcally smaller than that of τ. A notable excepton occurred wth large values of k and large, equal values of n. In these condtons, the bas of τ dd not exceed.0 n absolute value, whle the bas of reached as large as -.6 wth τ =. Performance wth non-normal dstrbutons and homogeneous varances. For the condtons smulated wth non-normal dstrbutons and homogeneous varances n the prmary studes, the performance of the observed varance estmator depended upon the sample szes of the prmary studes (wth small samples, substantal postve bas was evdent as τ ncreased; wth large samples, relatvely unbased results were obtaned). For condtons smulated wth skewness = and kurtoss = 6, for example, the estmated bas reached as hgh as.69 wth τ = and n = n = 5, but dd not exceed.03 wth large samples. For small samples n the prmary studes, both and τ evdenced superor performance, wth the latter showng slghtly less bas than the former n most condtons. However, wth large n and small k, τ showed notably more bas than, whle wth both k and n large, τ evdenced less bas. For example, wth τ =, k = 5 and n = n = 00, the bas of τ was -.9, whle that of was only In contrast, for the same condton but k = 00, the bas of τ was.0, whle that of was -.4. Performance wth non-normal dstrbutons and heterogeneous varances. The results for heterogeneous varance and non-normal dstrbutons suggested that under condtons of small n, the observed varance estmator performed very poorly regardless of the number of studes n the metaanalyss, wth the estmated bas exceedng.00 for 965

4 the most extreme condtons of varance heterogenety. However, the bases of the other estmators were also substantal under these condtons, especally when heterogeneous varances were pared wth unequal sample szes. As expected, the bas ncreased as the degree of varance heterogenety ncreased. Wth large samples szes, however, the observed varance estmator evdenced the least bas wth equal sample szes or wth postve parngs of sample sze wth populaton varance (wth the bas not exceedng.09 n these condtons). In contrast, wth negatve parngs of sample sze and varance, the bas of ths estmator reached as hgh as.36 n the most extreme condtons of heterogenety. For these condtons, the maxmum lkelhood method performed best wth small values of k (wth bas not exceedng.0), whle performed best wth large k (wth bas not exceedng.08). Confdence Band Coverage and Band Wdth After conductng an analyss of the relatonshps among the pont estmators and the central research desgn factors, we turned our attenton to an examnaton of these relatonshps wth regard to the two confdence nterval estmaton methods. Ths analyss ncluded an examnaton of both the estmated confdence band coverage and the estmated confdence band wdth. The dstrbutons of 95% confdence band coverage across the condtons examned n ths study are dsplayed as box and whsker plots n Fgure 3. As s evdent n ths fgure, both varance estmators provded poor band coverage for a vast number of the condtons examned, wth the maxmum lkelhood estmaton method evdencng both poorer average coverage and consderably more varablty n coverage probablty. Confdence Band Coverage Probablty Q Based Maxmum Lkelhood Varance Estmator Fgure 3. Dstrbutons of Confdence Band Coverage Probablty Across Condtons Examned n the Monte Carlo Study The dstrbutons of confdence band wdths across the condtons examned n ths study are presented n Fgure 4. As s evdent n ths fgure, the Q-based method dsplayed more varablty n the wdths of the confdence ntervals. Mean Confdence Band Wdth Q Based Maxmum Lkelhood Varance Estmator Fgure 4. Dstrbutons of Confdence Band Wdths Across Condtons Examned n the Monte Carlo Study As wth the examnaton of statstcal bas and standard errors of the pont estmates, the confdence band coverage probabltes and confdence band wdths were further nvestgated n relaton to the central desgn factors n the study. Performance wth normal dstrbutons and homogeneous varances. The estmates of confdence band coverage and wdth under selected condtons examned wth normal dstrbutons and homogeneous varances suggested that the confdence bands constructed usng showed excellent coverage of the parameter τ across prmary study sample szes and true values of the parameter for condtons wth small k. However, the confdence bands that provded ths coverage were exceptonally large (reachng as wde as.0 wth the smallest samples, n = 5). Wth larger values of k, the confdence band wdth decreased substantally, but the qualty of the coverage probablty deterorated as well, droppng as low as.6 wth n = 5 and τ =. The confdence ntervals constructed usng τ showed poor performance wth small k and wth small n. Wth large values of k and n the coverage probabltes exceeded.90, but dd not reach the nomnal.95 wth τ > 0. Performance wth normal dstrbutons and heterogeneous varances. The confdence band coverage and wdth for normal dstrbutons wth heterogeneous varances suggested smlar patterns. 966

5 As wth the homogeneous varance condtons, provded confdence bands wth excellent coverage under small k condtons, but the bands were panfully large. In contrast, τ evdenced very poor coverage wth small k and small n. Only under condtons wth both large k and large, balanced n dd the bands computed usng τ show a reasonable degree of coverage (rangng from.9 to.99 wth k = 00 and n = n = 00). Wth unbalanced samples, however, the coverage probabltes of the confdence bands computed usng τ deterorated, reachng as low as.36 under the most extreme condtons of varance heterogenety. Performance wth non-normal dstrbutons and homogeneous varances. The estmates of confdence band coverage and wdth under condtons examned wth non-normal dstrbutons and homogeneous varances suggested that provded good confdence band coverage wth small k, and wth large k coupled wth small n. However, wth large k and large n, the confdence band coverage deterorated. In contrast, τ provded relatvely poor confdence band coverage, except under the large k and large n condtons (where coverage probabltes ranged from.94 to.97). Performance wth non-normal dstrbutons and heterogeneous varances. Fnally, the results from non-normal populaton dstrbutons combned wth heterogeneous varances ndcated that the confdence band coverage of both methods was exceptonally poor n many crcumstances. For example, among the small k condtons, the confdence bands computed usng evdenced coverage probabltes as low as.70 (wth n = 0, 80, τ = 0, and a varance rato of :8). Smlarly, the coverage probabltes of confdence bands constructed usng τ reached as low as.65 (wth n = 80, 0, τ = 0, and a varance rato of :8). Wth condtons characterzed by large k and small n, the confdence band coverage of both methods was notably worse (becomng as low as.0 for confdence bands, and as low as.4 for τ confdence bands). Even wth the large k and large n condtons, confdence band coverage was very poor n many condtons. DISCUSSION Accordng to Freedman (000), under the assumpton of normalty, the Q-based estmator ( ) s a more effcent estmator when τ s close to 0, whereas the observed varance estmator ( σ! ) should provde more effcent estmates wth large values of the populaton value of τ. Clearly, ths pattern was not observed n our analyss of the behavor of the pont estmates under varance heterogenety and non-normalty. In general, the observed varance estmator evdenced notably poorer performance across all condtons examned. In contrast, the Q-based estmator and the maxmum lkelhood estmator provded smlar performance n terms of statstcal bas and standard errors, but showed notable dfference n confdence band coverage probabltes and confdence band wdths. Although both technques provded poorer band coverage under varance heterogenety n the prmary studes, the average performance of the Q- based estmator was consstently more accurate than that of the maxmum lkelhood estmator. The Q- based estmator also provded better band coverage under condtons wth small numbers of studes n the meta-analyss. However, ths superor confdence band coverage comes at a cost of wder confdence bands. The bands obtaned usng the Q-based estmator were consstently wder than those obtaned wth the maxmum lkelhood estmator, often beng several tmes the wdth. All of the estmators examned n ths study evdenced extreme senstvty to volatons of the assumptons of normalty and varance homogenety. Ths senstvty underscores the mportance of assessng the tenablty of these assumptons n prmary studes. However, the estmaton of τ s usually not the prmary goal of random-effects metaanalyss rather, ths statstc s used to obtan the weghts employed n the calculatons of means and varances of effect szes. The mpact of the bas n estmatng τ on these nferences (e.g., the estmaton of populaton mean effect szes) requres further nvestgaton. Meta-analyss has become ncreasngly mportant for the synthess of research results n a varety of felds, ncludng educaton, the behavoral scences and medcne. The accuracy of nferences derved from meta-analyss depends upon the approprate applcaton of statstcal tools. As the use of meta-analytc methods becomes more commonplace, researchers must reman mndful of the lmtatons of certan estmates. The results of ths study underscore the need to exercse cauton n the nterpretaton of the results obtaned from random effects models. The estmates of τ appear to be substantally affected by volatons of the assumptons of normalty and varance homogenety n prmary studes. Ths research furnshes valuable nformaton about the senstvty of estmates of the 967

6 random effect varance and provdes gudance regardng the choce among alternatve methods. Most mportantly, these results hghlght the need for the development of meta-analytc methods that are robust to volatons of these assumptons. ACKNOWLEDGEMENTS Ths work was supported, n part, by the Unversty of South Florda and the Natonal Scence Foundaton, under Grant No. REC The opnons expressed are those of the authors and do not reflect the vews of the Natonal Scence Foundaton or the Unversty of South Florda. REFERENCES Bggerstaff, B. J. & Tweede, R. L. (997). Incorporatng varablty n estmates of heterogenety n the random effects model n meta-analyss. Statstcs n Medcne, 6, Feld, A. P. (00). Meta-analyss of correlaton coeffcents: A Monte Carlo comparson of fxed- and random-effects methods. Psychologcal Methods, 6, Fleshman, A. I. (978). A method for smulatng non-normal dstrbutons. Psychometrka, 43, Fredman, L. (000). Estmators of random effects varance components n meta-analyss. Journal of Educatonal and Behavoral Statstcs, 5, -. Hedges, L. V. (994). Fxed effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthess (pp ). New York: Russell Sage Foundaton. Hedges, L. V. & Olkn, I. (985). Statstcal methods for meta-analyss. Orlando, FL: Academc Press. Hedges, L. V. & Vevea, J. L. (998). Fxed- and random-effects models n meta-analyss. Psychologcal Methods, 3, Hogarty, K. Y. & Kromrey, J. D. (00, Aprl). We ve been reportng some effect szes: Can you guess what they mean? Paper presented at the annual meetng of the Amercan Educatonal Research Assocaton, Seattle. Kromrey, J. D., Ferron, J. M, Parshall, C. G., Hogarty, K. Y., Grnnell, L., Hess, M. R., Lee, R., Romano, J., Sentovch, C., Watson, F., Dawkns, G. & Nles, J. (00, Aprl). Evdence of attanment: A comparson of methods for representng and communcatng student outcomes n systemc reform. Paper presented at the annual meetng of the Amercan Educatonal Research Assocaton, New Orleans. Mccer, T. (989). The uncorn, the normal curve, and other mprobable creatures. Psychologcal Bulletn, 05, Raudenbush, S. W. (994). Random effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthess (pp. 30-3). New York: Russell Sage Foundaton. Robey, R. R. & Barckowsk, R. S. (99). Type I error and the number of teratons n Monte Carlo studes of robustness. Brtsh Journal of Mathematcal and Statstcal Psychology, 45,