Katholieke Universiteit Leuven Belgium

Transcription

1 Using multilevel models for dependent effect sizes Wim Van den Noortgate Participants pp pp pp pp pp pp pp pp pp Katholieke Universiteit Leuven Belgium 5th Annual SRSM Meeting Cartagena, July / / Sample Sample1 Sample Sample k Subects Sub 1 Sub Sub 1 Sub 1 Sub Sub 3 Sub 4 - Subect characteristics? σ d n + δ E nc + n n ( n + n ) E C E C Sample Samp 1 Samp Samp 1 Sam 1 Sam Sam 3 Sam 4 Van den Noortgate & Onghena (003, SPQ; 003, BRMIC; 007, BAT; 008, EBCAI) 3 / 4 / 1

2 But independent samples assumed! Experiments Exp 1 Exp Exp 1 Exp 1 Exp Exp 3 Exp 4 - characteristics of exp.? What if multiple outcomes? Sample Samp 1 Samp Samp 1 Sam 1 Sam Sam 3 Sam 4 Van den Bussche, Van den Noortgate, & Reynvoet (009, Psychological Bulletin) 5 / 6 / Dealing with sample dependencies Sample Sample1 Sample Sample k σ σ d d d ' n + δ E nc + n n ( n + n ) E C E C δ δ ρ E + nc ' ' ρ ' 7 / nenc ( ne + n C ) n + - Randomly choose one effect size per study - Average effect sizes per study - Choose the most relevant outcome per study - Do separate meta-analysesanalyses But: loss of information + loss of efficiency - Perform a multivariate meta-analysis analysis But: where do we get the correlations? 8 /

3 What about using a three level model? Outcomes Out 1 Out Out 1 Out 1 Out Out 3 Out 4 Sample Samp 1 Samp Samp 1 Sam 1 Sam Sam 3 Sam 4 - characteristics of out.? 9 / Simulation design ( outcomes) Level 1: participants within studies Y = β ( ) 0 + β 1 Treatment + r Oi O O i Oi Level : (Between) study level β = u O0 O0 1 roi ~ N(0, ) 0/.4/. 8 1 = γ + u O1 O10 O1 u10.1 u N(0, Ωu ) : Ω u = u /. u /. β (for outcome O, person i, study ) 0 /.5 /.5 n 30 / 80 k 40 / 80 γ = = = 10 / O10 Analysis of raw data: Analysis Model 1: multivariate -level model Model : univariate 3-level model Model 3: univariate -level model (ignoring dependency) Analysis of effect sizes: Model 4: univariate 3-level model Model 5: univariate -level model Effect size analyses= raw data analyses (for effect = 0, n=30, k=80; but same pattern for other combinations) 11 / 1 / 3

4 13 / 14 / Effect sizes analysis= raw data analysis Control over Type I errors 15 / 16 / 4

5 Effect sizes analysis= raw data analysis Control over Type I errors Power 17 / 18 / Effect sizes analysis= raw data analysis Control over Type I errors Power Variances 19 / 0 / 5

6 Conclusion First results are promising for the three level approach for dependent samples! More simulation (& analytical work) is needed More outcome variables Studies reporting subsets of outcomes Varying covariance over pairs of outcomes Including indicators for (groups of) outcomes Thank you! 1 / / Van den Bussche, E., Van den Noortgate, W., Reynvoet, B. (009). Mechanisms of masked priming: A meta-analysis analysis. Psychological bulletin, 135,, Van den Noortgate, W., Onghena, P. (008). A multilevel meta-analysis analysis of single-subect subect experimental design studies. Evidence based Communication Assessment and Intervention,,, Van den Noortgate, W., Onghena, P. (007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8(), Van den Noortgate, W., Onghena, P. (003). Combining single-case experimental data using hierarchical linear models. School psychology quarterly, 18(3), Van den Noortgate, W., Onghena, P. (003). Multilevel meta-analysis: analysis: a comparison with traditional meta-analytical analytical procedures. Educational and psychological measurement, 63(5), Van den Noortgate, W., Onghena, P. (003). Hierarchical linear models for the quantitative integration of effect sizes in single-case research. Behavior research methods, instruments and computers, 35,, / 6