Meta-Analysis: An Introduction. Elizabeth Moorman Kim, PhD SRM Statistics and Research Methodology Series April 22, 2011

Transcription

1 Meta-Analyss: An Introducton Elzabeth Moorman Km, PhD SRM Statstcs and Research Methodology Seres Aprl 22, 20

2 OVERVIEW What s meta-analyss? When s t approprate to do a meta-analyss? What are the steps for conductng a metaanalyss? Problem formaton and lterature search Codng and analyss Interpretaton and presentaton of results

3 WHAT IS META-ANALYSIS? Lterature revew Theoretcal revew Research synthess (.e., research revew, systematc revew) Meta-analyss: quanttatve procedures for combnng results

4 WHEN TO DO A META-ANALYSIS? Hgh parent nvolvement n the chld s educaton w[as] assocated wth school success n terms of school records of achevement. ~Mlots, Sesma, & Masten, 999, p. The fndngs suggest that parental nvolvement does not ndependently mprove chldren's learnng. ~Domna, 2005, p. 223 Parental nvolvement n 3rd grade had a sgnfcant drect effect on achevement n 3rd grade. ~Englund, Luckner, Whaley, & Egeland, 2004, p.723 The results of the between-chld analyses suggested that hgher parent nvolvement s unrelated to average achevement across elementary school. ~El Nokal, Bachman, & Votruba-Drzal, 200, p. 00 What should we conclude from these fndngs?

5 Ctatons for the Years 998 to Meta- analyss Research synthess, systema<c revew, research revew, or lterature revew From Cooper, H. (200). Research synthess and meta-analyss. Thousand Oaks, CA, USA: Sage.

6 WHEN TO DO A META-ANALYSIS? Body of emprcal research Producng quanttatve fndngs Utlzes data typcally reported n research reports Comparable conceptually Smlar statstcal forms Apples and oranges

7 STEPS IN META-ANALYSIS. Specfyng the Problem 2. Searchng the Lterature 3. Report Codng 4. Data Analyss 5. Interpretaton and Presentaton of Results

8 . SPECIFYING THE PROBLEM Consder t carefully Breadth vs. Specfcty Does not need to be hghly detaled early on Research lterature Independent varables Dependent varables Type of relatonshp

9 2. SEARCHING THE LITERATURE How should I conduct my search? What outlets should I use for gatherng studes? What crtera should I use n my search? How do I go about retrevng the studes? How do I manage the yelded reports?

10 2. SEARCHING THE LITERATURE IRB Approval 98% of total Conducted the 98% study Dd not conduct the 2% study Research Progresson 95% of total 97% Study completed Study ncomplete 3% 76% of total 80% Data analyzed Data not analyzed 20% 66% of total Wrote up the 87% results Dd not wrte up the 3% results 35% of total Prepared for submsson 52% Dd not prepare for submsson 48% 24% of total Accepted n a 69% journal Not accepted n a journal 6% Publshed n a book 6% 9% In submsson From Fndng the mssng scence: The fate of studes submied for revew by a human subjects commiee, by H. Cooper, K. DeNeve, & K. Charlton, 997, Psychologcal Methods, 2,

11 2. SEARCHING THE LITERATURE Approaches Contact Mass Solctaton Conference Presentatons Journals References Reference Databases

12 2. SEARCHING THE LITERATURE Reference Databases Whch ones should I use? Consder publshed and unpublshed How do I decde on my search terms? Thesaurus

13 2. SEARCHING THE LITERATURE

14 2. SEARCHING THE LITERATURE Elgblty crtera Dstngushng Features Respondents Varables Desgn Culture/Language Tme Type of Publcaton

15 2. SEARCHING THE LITERATURE

16 2. SEARCHING THE LITERATURE Search terms Operators AND vs. OR Truncaton Feld

17 2. SEARCHING THE LITERATURE

18 2. SEARCHING THE LITERATURE Lterature Retreval Lbrary System Inter-Lbrary Loan Meta- Analysts Personal Contact Mantan Records the LIBRARY

19 2. SEARCHING THE LITERATURE Record Management Database program

20 2. SEARCHING THE LITERATURE

21 3. REPORT CODING Developng a codng protocol Plannng A pror Revew of lterature Frequency Content Study descrptors Moderators Effect szes Effect sze statstc Assocated sample sze

22 3. REPORT CODING Study Descrptors Bblographc Informaton e.g., type of report, publcaton year, authors, etc. Parental Involvement Research Synthess Study Descrptors Bblographc Informaton. Record Number EndNote Record Number 2. Study Authors Last Name, Frst Intal; etc. 3. Year Publcaton Year 4. Type of Report = Journal artcle 2 = Book or book chapter 3 = Government report 4 = Conference paper 9 = Other (please specfy) Code

23 3. REPORT CODING Study Descrptors Settng e.g., country, type of communty (urban, rural), type of school (publc, prvate), etc. Settng Informaton 5. Country = USA 2 = Canada 3 = Mexco 9 = Other (please specfy) 6. Communty type = Urban 2 = Suburban 3 = Rural 4 = Heterogeneous 9 = Other (please specfy) 7. Language research conducted n = Englsh 2 = Spansh 9 = Other (please specfy) 8. Type of school = Publc 2 = Prvate 9 = Other Code

24 3. REPORT CODING Study Descrptors Descrpton of the Sample e.g., age, ethncty, gender, etc. Descrpton of Sample 9. Number of chldren Specfy number of chldren partcpatng 0. Grade(s) n school - = Preschool 0 = Kndergarten = Frst grade 2 = 2 th grade. Socoeconomc status = Low SES 2 = Low- mddle SES 3 = Mddle SES 4 = Mddle- upper SES 5 = Upper SES 6 = Heterogeneous 9 = Other (please specfy) 2. Gender % Male % Female Code

25 3. REPORT CODING Study Descrptors Research Desgn e.g., unt of assgnment, type of assgnment, etc. Research Desgn Code 3. Unt of assgnment = Indvdual chldren/parents 2 = Small groups 3 = Classroom 4 = School 5 = Regon 9 = Other (please specfy) 4. Type of assgn = Completely randomzed 2 = Randomzed block 3 = Nonrandomzed 4 = Nonrandomzed block 9 = Other (please specfy) 5. Control group procedures = Busness- as- usual 2 = Attenton placebo 3 = Alternatve nterventon 4 = Wat lst/delayed nterventon 9 = Other (please specfy)

26 3. REPORT CODING Study Descrptors Independent Varables Independent Varables 6. Type of Involvement Home- based = Homework nvolvement 2 = Talkng about school 3 = Readng together 4 = Plannng 9 = Other home- based (please specfy) School- based 2 = Volunteerng 22 = Parent- teacher assocaton 23 = Parent- teacher conferences 24 = Open houses 29 = Other school- based (please specfy) 7. Reporter = Parent 2 = Chld 3 = Teacher 9 = Other (please specfy) Code

27 3. REPORT CODING Study Descrptors Dependent Varables Dependent Varables 8. Chld Academc Adjustment Grades = Grades: Combned Subjects 2 = Grades: Language Arts 3 = Grades: Math 9 = Grades: Other Standardzed Test Scores 2 = Test Scores: Combned 22 = Test Scores: Language Arts 23 = Test Scores: Math 29 = Test Scores: Other Academc Competence 3 = Academc Competence: Combned 32 = Academc Competence: Language Arts 33 = Academc Competence: Math 39 = Academc Competence: Other 9. Reporter = Records 2 = Parent 3 = Chld 4 = Teacher 9 = Other (please specfy) Code

28 3. REPORT CODING Effect Sze Codng Standardzed Mean Dfference Bas-Corrected s x x2 d s pooled = 3 ES sm ʹ = d N 4 9 x = observed mean of group x = observed mean of group 2 2 pooled = pooled standard devaton v Samplng Varance 2 n + n2 ( ESʹ sm) = + n n 2( n + n ) 2 2 n = sample sze n group n 2 = sample sze n group 2 N = total sample sze (Hedges, 98)

29 3. REPORT CODING Effect Sze Informaton Code 7. Treatment Group Mean 8. Treatment Group Standard Devaton 9. Treatment Group Sample Sze 20. Control Group Mean 2. Control Group Standard Devaton 22. Control Group Sample Sze 23. Effect Sze (calculated)

30 3. REPORT CODING Effect Sze Codng Correlaton Coeffcent Z-transformaton r xy σ 2 xy + r = ES z r = ln 2 r σ x σ y σ σ x σ 2 xy y = covarance between x and y = standard devaton of x = standard devaton of y r = correlaton coeffcent ln = natural logarthm Samplng Varance vz r = N 3 N = total sample sze

31 3. REPORT CODING Correlaton Coeffcent Informaton Code 26. Correlaton Coeffcent 27. Correlaton Sample Sze

32 3. REPORT CODING Coder Tranng Choosng coders Elaborated codng protocol Independent codng, group comparson and dscusson Double codng

33 4. DATA ANALYSIS Data Preparaton Convertng effect szes to a common metrc Drecton of effect Multple effect szes Random selecton Underrepresented areas Aggregaton Statstcal artfacts (Hunter & Schmdt, 2004) Samplng error Measurement unrelablty Range restrcton Dchotomzaton

34 4. DATA ANALYSIS Model Selecton ES =θ + ε Fxed effects model Fxed effects wth moderators Dfferences due to presence of moderators Random-effects.e., θ = θ = = 2 θ k Dfferences due to random heterogenety Mxed-effects Dfferences due to presence of moderators and random heterogenety

35 4. DATA ANALYSIS Model Selecton Model Moderators present Random heterogenety Fxed effects No No Fxed effects wth moderators Yes No Random effects No Yes Mxed effects Yes Yes From Roberts, B. W., Kuncel, N. R., Vechtbauer, W, & Bogg, T. (2007). Meta-analyss n personalty psychology: A prmer. (Ch. 36, pp ). In R. W. Robns, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of Research Methods n Personalty Psychology. New York, NY, USA: Gulford Press.

36 4. DATA ANALYSIS Standardzed Mean Dfference w Inverse Varance Weghted Average Effect Sze 2( n + n = 2( n + n ) ) n + n n n 2 2 d 2 n = number of data ponts n group n 2 = number of data ponts n group 2 d = effect sze 95% Confdence Interval CI d 95% = d ±.96 k = number of ndependent samples k = w d = k = k = d w w

37 4. DATA ANALYSIS Study n n 2 d w d w Weghted Average Effect Sze: 95% Confdence Interval: CI d d k = d = = = = k w w 95 % = d ±.96 =.3 ±.96 =.3 ±.09 = w k =.3 [.04,.22]

38 4. DATA ANALYSIS Correlaton Coeffcent (Z-transformed) z k = = k = ( n 3) ( n z 3) n = sample sze for th ndependent sample z = z-transformed correlaton Weghted Average Effect Sze 95% Confdence Interval CI z 95% = z ± k ( n 3) =.96

39 4. DATA ANALYSIS Study n r z n -3 (n -3)z Weghted Average Effect Sze: 95% Confdence Interval: CI z 95% = z ± k = z.96 k = ( n 3) ( n 3) = = = = k ( n 3) =.2± z =.2±.05 = [.6,.26]

40 4. DATA ANALYSIS Correlaton Coeffcent (untransformed) Samplng Varance v r = 2 ( r ) n 2 Inverse Varance w = v r n = sample sze for th ndependent sample r = correlaton coeffcent Weghted Average Effect Sze r = k = k = r w w 95% Confdence Interval CI r 95% = r ± k.96 = w

41 4. DATA ANALYSIS Study n r v w r w Weghted Average Effect Sze: 95% Confdence Interval: CI r 95% = r ± k = = = = = k r k.96 = w r w w =.2± =.2±.05 = [.6,.26]

42 4. DATA ANALYSIS Homogenety Is the varance observed dfferent than would be expected by samplng error only? Q statstc If so, what mght moderate ths effect?

43 4. DATA ANALYSIS Standardzed Mean Dfference k Q = wd = 2 k = k = w d w =.04 = = Compare to ch-square wth k- degrees of freedom Study n n 2 d w d w w d

44 4. DATA ANALYSIS Correlaton Coeffcent (Z-transformed) Q = k = ( n 3) z 2 k = k ( n 3) = ( n 3) 2 z = = = 5 Compare to ch-square wth k- degrees of freedom Study n r z n -3 (n -3)z (n -3)z

45 4. DATA ANALYSIS Publcaton Bas Fle drawer problem Sample Sze Funnel plot Effect sze 200 Fal-safe N (Rosenthal, 979) Sample Sze Trm and fll (Duval & Tweede, 2000) Effect sze

46 4. DATA ANALYSIS Programs SAS SPSS STATA R Stand-alone programs Comprehensve Meta-Analyss (Borensten, Hedges, Hggns, & Rothsten, 2005)

47 5. PRESENTATION OF RESULTS Graphcal Presentaton Error-Bar Chart Stem and Leaf Dsplay Effect Sze Elementary Mddle Hgh Developmental Level Meta-Analyss Reportng Standards (MARS; Publcaton Manual of the Amercan Psychologcal Assocaton (200))

48 HOW DO I GET STARTED? Resources Research team Supples Fundng Mechansms IES Goal : Exploraton NIH R2: Exploratory/Developmental

49 REFERENCES Borensten M, Hedges L, Hggns J, Rothsten H. (2005). Comprehensve Meta-analyss Verson 2. Englewood NJ, USA: Bostat. Cooper, H. (200). Research synthess and meta-analyss. Thousand Oaks, CA, USA: Sage. Cooper, H., DeNeve, K., & Charlton, K. (997). Fndng the mssng scence: The fate of studes submtted for revew by a human subjects commttee. Psychologcal Methods, 2, Duval, S., & Tweede, R. (2000). A nonparametrc trm and fll method of accountng for publcaton bas n meta-analyss. Journal of the Amercan Statstcal Assocaton, 95, Hedges, L. V. (98). Dstrbuton theory for Glass s estmator of effect sze and related estmators. Journal of Educatonal Statstcs, 6, Hunter, J. E., & Schmdt, F. L. Methods of meta-analyss: Correctng error and bas n research fndngs. Thousand Oaks, CA, USA: Sage. Lpsey, M. W., & Wlson, D. B. (200). Practcal meta-analyss. Thousand Oaks, CA, USA: Sage. Roberts, B. W., Kuncel, N. R., Vechtbauer, W, & Bogg, T. (2007). Meta-analyss n personalty psychology: A prmer. (Ch. 36, pp ). In R. W. Robns, R. C. Fraley, & R. F. Krueger (Eds.), Handbook of Research Methods n Personalty Psychology. New York, NY, USA: Gulford Press. Rosenthal, R. (979). The fle-drawer problem and tolerance for null results. Psychologcal Bulletn, 86,

50 Thank You! Questons? Elzabeth Moorman Km, PhD