Meta-Analysis What is it? Why is it important? How do you do it? What is meta-analysis? Good books on meta-analysis

Transcription

1 Meta-Analyss What s t? Why s t mportant? How do you do t? (Summer) What s meta-analyss? Meta-analyss can be thought of as a form of survey research n whch research reports are the unts surveyed (Lpsey and Wlson, 00, Practcal Meta-Analyss, Sage) Meta-analyss s the quanttatve ntegraton of research that s a specal form of systematc research synthess Meta-analyss can be thought of as an approach to the quanttatve analyss of replcatons Good books on meta-analyss Lpsey and Wlson, (00), Practcal Meta- Analyss, Sage. (Easy to read, very practcal) Glass, McGaw, and Smth, (98), Meta- Analyss n Socal Research, Sage. (A classc) Cooper and Hedges, (994), Handbook of Research Synthess, Russell Sage Foundaton. (Very comprehensve, techncal, a must for any meta-analyst)

2 What types of research questons can be addressed n a meta-analyss? Types of research questons addressed n meta-analyss What does the research n a partcular area tell us about.? Does cogntve-behavor therapy decrease depresson? (Gaffan, Tsaouss, and Kemp-Wheeler, Researcher allegance and meta-analyss: The case of cogntve therapy for depresson, (995), Journal of Consultng and Clncal Psychology, 63(6), ). Is there a relatonshp between beng sexually abused as a chld and later psychopathology? (Rnd, Tromovch, and Bauserman, A meta-analytc examnaton of assumed propertes of chld sexual abuse usng college samples, (998), Psychologcal Bulletn, 4(0), -53). Is there a relatonshp between partcpaton n vctm-offender medaton and subsequent delnquent behavor? (Nugent, Wllams-Hayes, and Umbret, n press, Research on Socal Work Practce). What study characterstcs moderate effect sze magntude? Substantve questons about some phenomena Questons about whch methodologcal characterstcs contrbute the varablty n outcomes

3 Why s Understandng Meta- Analyss Important The use of systematc research revews as a tool for dentfyng best practces s becomng more and more promnent. Meta-analyss s rapdly becomng a prncpal method for conductng systematc revews. How s Meta-Analyss Done? Steps n a meta-analyss Research queston/problem formulaton Retreval of research studes Effect sze selecton Identfcaton and codng of ndependent varables Data analyss Interpretng and understandng results Wrtng up results 3

4 Create a Lterature Search Record Include sources searched Include ctatons found; ctatons retreved and how; ctatons not retreved and methods used to get them Include personal contacts wth other researchers and results Include advertsements used Include how world wde web searched done and results Fve Lterature Search methods Footnote chasng References n nonrevew papers n journals References n revew papers References n books Topcal bblographes Consultaton Informal conversatons Communcaton wth fellow researchers Formal requests from other researchers General requests to government agences Searches n subject ndexes Manual search of abstract data bases Computer search of abstract data bases (eg., PsychInfo, ERIC, etc.) Browsng Browsng through lbrares Ctaton searches Manual search of ctaton ndex Computer search of ctaton ndex (eg., SSCI) 4

5 Varables nvolved n a metaanalyss Dependent one or more measures of effect sze Independent Varables study characterstcs methodologcal qualty; samplng methods; group formaton methodology; measurement; etc. subject characterstcs age; gender; ethncty; etc. treatment varables treatment type; type of comparson group (eg., placebo; notreatment; etc.) context varables locaton of study; type of supervson of therapst; etc. researcher characterstcs therapeutc allegance; experence; educaton level; etc. 5

6 Effect szes An effect sze s a statstc whch embodes nformaton about ether the drecton or magntude (or both) of quanttatve research fndngs (Lpsey & Wlson, 00) Effect szes used n a meta-analyss are consdered to be metrc free Just about any statstc can, n prncpal, be consdered as an effect sze Effect sze statstcs Sngle varable Two varable D-famly R-famly Odds-rato Sngle varable effect szes Statstcs that descrbe Angel 6

7 Sngle varable the mean m m s X n w m n s m Example X 3. s 357. n m w m Sngle varable - The logt l p ln p l + np n( p) w l np( p) l 7

8 Sngle varable the standard devaton sd ln( s ) + [ ( n ] ) sd ( n ) w ( sd n ) The d-famly two varable effect sze statstcs descrbng the dfference between groups Marmaduke Standardzed mean dfference IF N < 0 sm X X SD G G 3 ' SM 4N 9 SM 8

9 sm ng + n n n G G G ( ' sm ) + ( n + n ) G G w sm sm Computng sm from statstcal tests of sgnfcance sm t ng n + n n G G G sm F( ng + ng ) n n G G sm from a ph coeffcent sm r r 9

10 BY CONVENTION, WHEN TREATMENT AND CONTROL GROUPS ARE CONTRASTED, A + SIGN IS GIVEN TO AN EFFECT SIZE TO INDICATE THE TREATMENT GROUP DID BETTER THAN THE COMPARISON GROUP The r-famly of effect szes: Indces of correlatonal assocaton Pasley r r Zr + r ln r Zr wz r n 3 n 3 Zr 0

11 Computng r from t-test results r t t + df t t + ( n ) t r t + n + n Pont-bseral correlaton effect sze from sm rpb 4 + sm sm Effect sze statstcs for dchotomous outcomes

12 The odds-rato The odds-rato s a statstc that compares two groups n terms of the relatve odds of an event or outcome odds p p no recdvsm recdvated Control group A B Treatment group C D OR AD BC pa ( pc ) p ( p ) c a w LOR LOR LOR Natural log odds-rato ln ( ) n n n n OR a b c d LOR LOR pg pg ln ln pg p G

13 Data analyss methods Graphcal methods Plot of effect szes wth 95% confdence ntervals Effect sze represented as natural logarthm of rato of odds of VOM partcpants re-offendng to odds of non-partcpants re-offendng Study stes 3

14 Plot of effect szes versus group formaton methodologcal qualty Effect sze represented as natural logarthm of rato of odds of VOM partcpants re-offendng to odds of non-partcpants re-offendng Study stes Effect sze represented as natural logarthm of rato of odds of VOM partcpants re-offendng to odds of non-partcpants re-offendng Study stes Effect sze represented as natural log of rato of odds of VOM partcpants re-offendng to odds of non-partcpants re-offendng GFM scores n lower half of all GFM scores GFM scores n upper half of all GFM scores 4

15 VOM effect (VOM groups percentage of re-offenders mnus non-vom groups percentage) Narrow defnton Broad defnton The use of weghted least squares regresson Statstcal analyss methods Fxed effects models: have fxed parameters plus a sngle resdual term Random effects models: have two resdual terms Mxed models: have fxed parameters plus two resdual terms 5

16 Data analyss steps n analyzng a dstrbuton of effect szes Create set of ndependent effect szes Compute weghted mean, weghtng by nverse varance weghts Determne confdence nterval for mean Test for homogenety of dstrbuton If heterogeneous dstrbuton, conduct further analyses Weghted least squares regresson (fxed effects) HLM (random effects; mxed models) The mean effect sze and 95% confdence nterval ( w ) w w z ( ) L + z ( ) U ( α ) ( α ) Z.96 for alpha.05 Z.58 for alpha.0 Z test of mean effect sze z w w w 6

17 .. 3. w 4. w wes 5. w wessq 6. w wsq Compute the followng: 7. wes w 8. w 9. z 0. L 96. ( ) U ( ). ( wes ) Q wessq w (Q has k- df, where k number of Studes) A statstcally non-sgnfcant Q s consstent wth homogenous effect szes; varablty n effect szes s lkely due to samplng varablty assocated wth samplng of dfferent subjects n studes A statstcally sgnfcant Q s nterpreted to mean that varablty n effect szes s greater than would be expected from samplng varablty assocated wth dfferent persons n studes. Three possbltes exst: () there s systematc varablty n effect szes n addton to samplng error assocated wth dfferent subjects; () there s an addtonal random component assocated wth random varatons n studes that cannot be modeled; and (3) a combnaton of () and (). 7

18 If researcher chooses to model a random effects component, then an addtonal varance component must be added to the squared standard error of the effect sze statstc: Q k v ( ) θ wsq w w So, the new v s, and v * v + v θ w * v + v θ Then the terms wes wessq wsq are recomputed as are the assocated sums of these terms; then a new 95% confdence nterval for the mean effect sze s computed. Weghted regresson analyss Forrest Gump 8

19 . Conduct weghted least squares regresson, usng the nverse varance as the regresson weght.. Conduct homogenety tests of the regresson model and resdual varance by: a. Test of homogenety of effect szes Q Overall total sums-of-squares wth total df as the ch-square df b. Test of regresson model Q R regresson sums-of-squares, wth regresson model df as the ch-square df. c. Test of resdual varaton homogenety Q E resdual sums-of-squares, wth Resdual df as the ch-square df. d. Test statstcal sgnfcance of partal regresson coeffcents by: B z ' B where ' B B M and M mean square resdual for the regresson model A Fxed Effects Analyss 9

20 Test of homogenety overall: χ ( 8) SSTOTAL p <. 05 χ ( 3) MSREGRSION p <. 05 χ ( 5) SSRIDUAL 738. p <. 05 Tests of regresson coeffcents B. 006 delta. Coeffcent for delta delta M 449. Bdelta z Coeffcent for def (defnton) B def. 98 def. 645 M z delta 3. Coeffcent for score. 38 score z

21 A Random Effects Analyss Model Regresson Resdual Total ANOVA b,c Sum of Squares df Mean Square F Sg a a. Predctors: (Constant), nonvom, def, delta, score b. Dependent Varable: efs c. Ch-sq(4) 5.7, p <.0, Ch-sq(4) 5.48, p >.05 Model (Constant) delta def score nonvom a. Dependent Varable: efs Unstandardzed Coeffcents Coeffcents a,b Standardzed Coeffcents B Std. Error Beta Z Sg > < >.5 b. Weghted Least Squares Regresson - Weghted by newwght

22 Vote Countng Methods Nonparametrc approaches An applcaton of the sgn test. Set H 0 : p.50; H : p >.50. Count number of outcomes n desred drecton 3. Use bnomal probablty dstrbuton to obtan p-value for obtaned count Example: 5 of 9 outcomes n specfed drecton, so p. 84 And assocated p-value from bnomal table: Test of combned statstcal sgnfcance another nonparametrc approach. tppet s mnmum p : (a) arrange exact p-values from lowest to hghest; (b) set crtcal alpha by: ( / k ) α ( α *) (k n of studes or effect szes) where α * desred overall type I error rate; (c) compare mnmum obtaned exact p-value aganst alpha; (d) f mnmum obtaned exact alpha < set alpha, then reject null hypothess that all obtaned effect szes are zero. Example: Obtaned p-values (k 9) range from.000 to.945 ( / 9) α (. 05) mnmum p.000 <.0069; reject null hypothess