DrPH Seminar Session 3. Quantitative Synthesis. Qualitative Synthesis e.g., GRADE

Transcription

1 DrPH Semnar Sesson 3 Quanttatve Synthess Focusng on Heterogenety Qualtatve Synthess e.g., GRADE Me Chung, PhD, MPH Research Assstant Professor Nutrton/Infecton Unt, Department of Publc Health and Communty Medcne, Tufts School of Medcne Steps of Performng a Systematc Revew 1 Ask FORMULATE STUDY QUESTION ESTABLISH PROTOCOL Identfy LITERATURE SEARCH / RETRIEVAL Acqure CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL Apprase DATA EXTRACTION and QUALITY ASSESSMENT Synthesze ANALYSIS and INTERPRETATION WEIGHTED AVERAGE REGRESSION SENSITIVITY ANALYSIS 1

2 2 Meta-analyss 101 n The term meta-analyss typcally referred to meta-analyss of study-level, summary data n The term pooled analyss typcally referred to meta-analyss of ndvduallevel data across dfferent studes n Conventonal meta-analyss methods and software are desgned for combnng RCTs (comparsons of ES between two ndependent groups) 3 Reasons for Meta-analyss n Improve the power to detect a small dfference f the ndvdual studes are small n Improve the precson of the effect measure n Compare the effcacy of alternatve nterventons and assess consstency of effects across study and patent characterstcs n Gan nsghts nto statstcal heterogenety n Help to understand controversy arsng from conflctng studes or generate new hypotheses to explan these conflcts n Force rgorous assessment of the data 2

3 Prncples of Combnng Data for Basc Meta-analyses 4 n For each analyss, one study should contrbute only one treatment effect. n The effect estmate may be for a sngle outcome or a composte. n The outcome beng combned should be the same (or smlar-based on clncal plausblty) across studes. n Know your queston. The queston drves your study selecton, synthess, and nterpretaton of the results. Thngs to Know about the Data before Combnng 5 n Bologcal and clncal plausblty n Scale of effect measure n Studes wth small number of events do not gve relable estmates 3

4 6 True Assocatons May Dsappear When You Combne Data Inapproprately Apparent Assocaton May Be Seen When There s None 7 4

5 Changes n the Same Scale May Have Dfferent Meanng 8 C D Both A B and C D nvolve a change of one absolute unt. Effect of nterest A B A B change (1 to 2) represents a 100% relatve change. C D change (7 to 8) represents only a 14% relatve change. Varable of nterest Commonly Encountered Comparatve Effect Measures 9 Type of Data Contnuous Dchotomous Correspondng Effect Measure Mean dfference (e.g., mmol, mmhg) Standardze mean dfference (effect sze) Correlaton Odds rato, rsk rato, rsk dfference Tme to event Hazard rato 5

6 10 What Is the Average (Overall) Treatment-Control BP Dfference? Study N Mean dfference 95% CI mmhg A to -5.5 B to -5.2 C to 6.3 Smple Average 11 ( 6.2) + ( 7.7) + ( 0.1) 3 = 4.7 mmhg Study N Mean dfference 95% CI mmhg A to -5.5 B to -5.2 C to 6.3 X k = = 1 n x 6

7 Weghted Average 12 (554 x 6.2) + (304 x 7.7) + (39 x 0.1) = 6.4 mmhg Study N Mean dfference 95% CI mmhg A to -5.5 B to -5.2 C to 6.3 X k = 1 = k = 1 w x w General Formula: Weghted Average Effect Sze (d + ) 13 d k = = 1 + k = 1 w d w where: d = effect sze of the th study w = weght of the th study k = number of studes 7

8 14 Calculaton of Weghts n Generally the nverse of the varance of treatment effect (that captures both study sze and precson) n Dfferent formula for odds rato, rsk rato, rsk dfference n Readly avalable n books and software 15 Heterogenety (Dversty) n Is t reasonable (are studes characterstcs and effects suffcently smlar) to estmate an average effect? n Types of heterogenety Clncal dversty Methodologcal dversty Statstcal heterogenety 8

9 16 Clncal Dversty Are the studes of smlar treatments, populatons, settngs, desgn, etc., such that an average effect would be clncally meanngful? 17 Methodologcal Dversty Are the studes of smlar desgn and conduct such that an average effect would be meanngful? 9

10 18 Statstcal Heterogenety n Is the observed varablty of effects greater than that expected by chance alone? n Two statstcal measures are commonly used to assess statstcal heterogenety Cochran s Q-statstcs I 2 ndex Cochran s Q-Statstcs χ 2 (ch-square) Test for Homogenety 19 Q-statstcs measures between study varaton Q k 2 ( k 1) df = w + = 1 ( d ) 2 = χ d d = effect measure d + = weghted average 10

11 I 2 Index and ts Interpretaton 20 H 2 Q = max,1 k 1 I 2 = 2 H 1 2 H I 2 descrbes the percentage of total varaton n study estmates that s due to heterogenety rather than chance. The value of I 2 ndex ranges from 0% to 100%. A value of 25% s consdered as low heterogenety, 50% as moderate, and 75% as large. The I 2 ndex s ndependent of the number of studes n the meta-analyss. It could be compared drectly between metaanalyses. Hggns JPT, et al. Measurng nconsstency n meta-analyses. BMJ 2003;327: A Meta-analyss wth a Large Degree of Heterogenety 21 11

12 22 Cho et al. Developmental Fluorde Neurotoxcty: A Systematc Revew and Meta-Analyss Heterogenety n Dversty of studes n a meta-analyss n Typcally abundant n Arguably the most mportant role of metaanalytc methodologes s to quantfy, explore, and explan between-study heterogenety 12

13 Statstcal heterogenety Statstcal heterogenety exsts when the results of the ndvdual studes are not consstent among themselves Clncal heterogenety Methodologcal heterogenety Bases Chance Statstcal heterogenety Clncal vs. statstcal heterogenety n Clncal and methodologcal heterogenety s abundant. Our am s to explore t, and use these observatons to formulate nterestng hypotheses. n Often, but not always, clncal and methodologcal heterogenety wll result n a statstcally sgnfcant test n Chance, techncal ssues or bases may result n statstcally sgnfcant results n heterogenety tests 13

14 26 Dealng wth Heterogenety HETEROGENEOUS TREATMENT EFFECTS IGNORE ESTIMATE (nsenstve) INCORPORATE EXPLAIN FIXED EFFECT MODEL DO NOT COMBINE WHEN HETEROGENEITY IS PRESENT RANDOM EFFECTS MODEL SUBGROUP ANALYSES META- REGRESSION (control rate, covarates) Lau J, et al. Quanttatve synthess n systematc revew. Ann Intern Med 1997; 127:826. Fxed- vs. Random-Effects Meta-analyss 27 n Fxed Effect Model (FEM): Assumes a common treatment effect. n Random Effect Model (REM): In contrast wth the FEM, the REM accounts for between study varaton. 14

15 Weghts of the Fxed Effect and Random Effects Models 28 Fxed Effect Weght Random Effects Weght w = 1 v w * = v 1 + v* where: v = wthn study varance v * = between study varance Commonly used Statstcal Methods for Combnng 2x2 Tables 29 Odds Rato Rsk Rato Rsk Dfference Fxed Effect Model Random Effects Model Mantel-Haenszel Peto Exact Inverse varance weghted DerSmonan& Lard Mantel- Haenszel Inverse varance weghted DerSmonan& Lard Inverse varance weghted DerSmonan& Lard 15

16 Fgure 1. An example of a fxed-effects MA 30 Fgure. An example of a random-effects MA Borensten et al. A basc ntroducton to fxed-effect and random-effects models for meta-analyss Res. Syn. Meth. 2010, Dealng wth Heterogenety HETEROGENEOUS TREATMENT EFFECTS IGNORE ESTIMATE (nsenstve) INCORPORATE EXPLAIN FIXED EFFECTS MODEL DO NOT COMBINE WHEN HETEROGENEITY IS PRESENT RANDOM EFFECTS MODEL SUBGROUP ANALYSES META- REGRESSION (control rate, covarates) 16

17 RESPONSE SURFACE modelng ndvdual patent data META-REGRESSION modelng summary data Treatment effect OVERALL ESTIMATE combnng summary data Treatment effect SUBGROUP ANALYSES dfferentatng effects n subgroups Treatment effect Treatment effect varable of nterest varable 2 varable 1 Summary: Statstcal Models of Combnng 2x2 Tables 33 n Most meta-analyses of clncal trals combne treatment effects (rsk rato, odds rato, rsk dfference) across studes to produce a common estmate, usng ether a fxed effect or random effects model. n In practce, the results usng these two models are smlar when there s lttle or no heterogenety. n When heterogenety s present, the random effects model generally produces a more conservatve result (smaller Z-score) wth a smlar estmate but wth a wder confdence nterval. However, there are rare exceptons of extreme heterogenety where random effects model may yeld counterntutve results. 17

18 34 Caveats n Many assumptons are made n meta-analyses, care s needed n the conduct and nterpretaton. n Most meta-analyses are retrospectve exercses, sufferng from all the problems of beng an observatonal desgn. n We cannot make up mssng nformaton or fx poorly collected, analyzed, or reported data. 35 Summary n Basc meta-analyses can be easly carred out wth one of many readly avalable statstcal software. n Relatve measures are more lkely to be homogeneous across studes and s generally preferred. n Random effects model s the approprate statstcal model n most nstances. n Decson to do a meta-analyss should be based on a well-formulated queston, apprecaton of the heterogenety of the data, and understandng of how the results wll be used. 18

19 36 Q & A 37 Qualtatve Synthess n Also called, Gradng Strength of Evdence Dstnct from ratng qualty of artcles/ studes Many tools, e.g. GRADE n Qualtatve synthess s requred for a systematc revew Meta-analyss s optonal 19

20 38 GRADE WHO GRADE process WHO gudelnes and GRADE: An overvew summary (60 mnutes) ndex.html 39 Dvdng Homework n Sesson 4: Mock expert panel Jefferson T et al. Vaccnes for preventng nfluenza n healthy adults. Cochrane Database Syst Rev Jul 7; (7):CD do: / CD pub4. Revew. Update n: Cochrane Database Syst Rev. 2014;3:CD PubMed PMID: l Three outcomes : Three students (what a concdence!) 20