A little (more) about me.

Transcription

1 Comparative Effectiveness Research Methods Training Module 2: Research Designs J. Michael Oakes, PhD Associate Professor Division of Epidemiology University of Minnesota A little (more) about me. Image from cartoonbank.com removed. Image description: Depicts cat looking at mouse in a maze. Mouse is looking up at cat with caption Well, you don t look like an experimental psychologist to me. 1

2 (2001) (2010) (1963) (1979) Module #2 Outline 1. Review of Core Ideas 2. Experimental Designs 3. Observational Designs 4. Issues & Assumptions 5. Review 6. Questions 2

3 Module #2 Outline 1. Review of Core Ideas Key Distinction 2. Experimental Designs 3. Observational Designs 4. Issues & Assumptions 5. Review 6. Questions On being asked to talk on the principles of research, my first thought was to arise and say, Be careful, and to sit down. Cornfield ldj J. Principles i i l of f research Am J Ment Defic. 1959;64: Review of Core Ideas 3

4 Counterfactuals & Potential Outcomes 4

5 Potential Outcomes Condition Assigned Treatme nt Outcome if Treated, Y 1 Observed Outcome if not Treated, Y 0 Unobservable Counterfactual Control Unobservable Counterfactual Observed More formally, T has a causal effect on Y for person i if Y Y i T 0 i T 1 But we can only observe one of these outcome for any i At the population level, we use probabilities and assuming exchangeability, Prob[Y 1] Prob[Y 1] T 0 T 1 Causal inference is about finding the best counterfactual substitutes for the unobservable counterfactuals. 5

6 Effect Identification An effect is identifiable if it is theoretically possible to learn the true value of the parameter when the sample size approaches infinity. Imagining that your sample size is infinitely large eliminates all problems of statistical uncertainty, confidence intervals and p- values. Identification is about ruling out competing hypotheses or explanations. Apart from statistical imprecision, if more than one explanation for your effect (eg, difference b/w treatment and control) exists, then you have an identification problem. You cannot say X 1 caused Y because the cause might have been X 2, or X 3, or X n. 6

7 It s all always about data and assumptions Got quality data? Got plausible assumptions? You can draw causal conclusions from a cross-sectional study with a non-random sample! You can draw causal conclusions from a cross-sectional study with a non-random sample! You just need heroic assumptions! 7

8 Designs vs. Models Image from cartoonbank.com removed. Image description: Depict two men in suits standing at the base of a large main frame computer. One man is gesturing at input feed to computer. Caption reads: The machine then selects the likely equations from a complicated pattern of theoretical probables. It calculates these, and the correct answer is printed on a card. Then our Miss Swenson files them God knows where, and we can never find the damn things again. Most people learn some statistics, a few learn fancy statistical models, yet fewer still learn research design. Research Design is more important than fancy statistical models. Simple Design. Intense Content. 8

9 The quality and strength of evidence provided by a research study is determined largely by its design. Excellent methods of analysis will not salvage a poorly designed study Aspects of design include the framing of the questions and assessment of measures, bias, and precision An observational study that begins by examining an outcome variable is a formless, undisciplined investigation that lacks design Design anticipates analysis. Adapted from Rosenbaum 2010, p. ix Three Types of Validity Internal Validity External Validity Statistical Conclusion Validity Threats to Internal Validity Temporal Precedence Selection History (exogenous events) M i ( d ) Maturation (endogenous events) Regression to mean Attrition Testing and instrumentation 9

10 Threats to External Validity Interactions by units and/or contexts 2. Experimental Designs Image from despair.com removed. Image description: Sign reads Dear science: I beg you not to prove that any more of my pleasures will harm me. History of Experiments Emergence of scientific method, 16 th century (esp. Bacon) Galileo s telescope and prosecution Hill s randomized clinical trial for TB Income maintenance experiments 10

11 Key Epistemological Point Experiments are used to test our knowledge of the world and correct it as necessary. What is an Experiment? 1) Exogenous treatment/intervention 2) Randomly assigned to subjects What is an Experiment? 1) Exogenous treatment/intervention 2) Randomly assigned to subjects Find effects of (known) causes 11

12 Exogenous? Random? Random vs. Random Random sampling (external validity) Random assignment (internal validity) 12

13 Random Assignment Balances both measured and unmeasured confounders (ie, removes bias) Works in expectation only (ie, large numbers) Manipulation of Treatments? Better to have it Twist the lions tale to reveal his secrets Can learn w/out it, especially when phenomena don t react to measurement (eg, planetary orbits) Observational Study Differs how? Exogenous treatment/intervention NOT randomly assigned to subjects Find causes of (known) effects 13

14 Experiments vs. Obs Designs Experimental Designs Find effects of causes Manipulation of Tx Random assignment Rule out competing explanations by design Observational Designs Find causes of effects No manipulation of Tx No random assignment Rule out competing explanations by analysis Caution! If treatment is not exogenous but endogenous then estimation of treatment effects is virtually impossible. (eg, social capital) Research Design Diagrams O represents an observation/measurement X represents an intervention/treatment R indicates randomization NR indicates no randomization 14

15 Posttest only R X O R O Posttest only R X O R O Counterfactual Substitutes? Posttest only R X O R O Counterfactual Substitutes? Same outcome values but for the treatment? 15

16 Pretest-Posttest, with same pre-post subjects R O X O R O O Pretest-Posttest, with same pre-post subjects R O X O Counterfactual Substitutes? R O O Pretest-Posttest, with same pre-post subjects R O X O R O O Counterfactual Substitutes? 16

17 Double-Blind RCT Neither researchers nor subjects know if Tx or Cx is administered (placebos) Gold standard for FDA (pharm. studies) Not feasible for most social research Non-Blind RCT Researchers and/or subjects know if Tx or Cx is administered Typical experiment in social research Sometimes subjects can be blinded while investigators are not Cross-Over Experiments Subjects exposed to all conditions, flipflopping or switching between them Requires wash out period Rarely feasible in social research 17

18 Cross-Over Experiment R O X A O X B O R O X B O X A O Factorial Experiments Estimate independent effects of one or more treatments NOT factor analysis! Usually 2 levels per factor: 2 n conditions 4 factors each of 2 levels = 2 4 = 16 conditions Consider two treatments: A and B Factor B Level 1 Level 2 Level 1 A1+B1 A1+B2 Factor A Level 2 A2+B1 A2+B2 18

19 Group-Randomized Trial Randomize in-tact social groups to Tx or Cx Use when Tx is for whole groups Use when changing local infrastructure Typically low-powered: df = c(g-1) Natural Experiments Mother Nature (eg, earth quake) Human Lotteries (eg, military draft) Policy change??? Regression Discontinuity Designs 19

20 RDD Offer treatment at prespecified cut-off of assignment variable Developed in the late 1950s Nearly as powerful as RCT Shadish et al 2002 Shadish et al

21 3. Observational Designs Quasi-Experimental Designs without Control Group One-group Posttest X O 1 One-group Pretest-Posttest O 1 X O 2 One-group Pretest-Posttest, with Double Pretest O 1 O 2 X O 3 21

22 Quasi-Experimental Designs with Control Group Is Control Group Exchangeable with Treatment Group? Posttest only NR X O 1 NR O 2 Pretest-Posttest, Posttest with different pre-post post subjects NR O 1 X O 2 NR O 1 O 2 22

23 Pretest-Posttest, with same pre-post subjects NR O 1 X O 2 NR O 1 O 2 Pretest-Posttest, Posttest, same pre-post post subjects and 2 posttests NR O 1 X O 2 O 3 NR O 1 O 2 O 3 Y Policy Change 66 Time Y Policy Change 0 Time 23

24 Y Policy Change 1.5 Time Y Policy Change?? Time Y Policy Change 0? Time 24

25 Y Policy Change Time Y Policy Change? Time Pre-Post, same subjects, 2 pretests and 2 posttests NR O 1 O 2 X O 3 O 4 NR O 1 O 2 O 3 O 4 Pre-Post, Post, same subjects, 2 pretests, 2 posttests, switching NR O 1 X O 2 O 3 NR O 1 O 2 X O 3 25

26 Interrupted Time Series Designs Simple ITS O 1 O 2 O 3 O 4 O 5 O 6 X O 7 O 8 O 9 Shadish et al

27 Shadish et al Issues & Assumptions Experiments Usually tell us that some cause (known to us) yields an effect Rarely tells us how the effect came about (no insight into mechanism) A full explanation requires deep understanding of the mechanisms 27

28 Assumptions in Experiments Full compliance No attrition External validity? in Observational Studies Internal validity? Exchangeability Positivity Consistency Oh, what shall I do? Image from despair.com removed. Image description: Picture shows a scenic mountain sun rise in the background with river in the foreground. Caption of picture reads It s always darkest just before it goes pitch black. 28

29 1) Seek causal effects 2) Always think experiment 3) If you cannot randomize, imagine the experiment you wish you could conduct 4) Conduct analyses that mimic the desired experiment and rule out competing explanations Add this watermark Image recreated from: Freedman, DA. Oasis or mirage? CHANCE Magazine. 2008; 21: (1): Review I. Background Concepts a. Experiments correct knowledge b. Counterfactuals c. Effect identification d. Data and assumptions e. Design trumps analysis III. Observational Designs a. Exchangeability b. Positivity c. No control group d. Control group(s) e. Interrupted Time Series II. Experiments IV. Assumptions & Issues a. Randomization b. Double-blind RCT c. Factorial Experiments d. Group-randomized trial e. Natural experiments f. RDD a. Black-box mechanisms b. Experimental Fidelity c. Exchangeability 29

30 6. Questions 1. What are the two elements that make up an experiment? 2. What is a factorial experiment? 3. Why is selection such a problem for observational designs? 4. What does exogenous mean? 5. When should one conduct a group-randomized trial? References & Resources 1. Cornfield J. Principles of research. American Journal of Mental Deficiency. 1959; 64: (24): Freedman, DA. Oasis or mirage? CHANCE Magazine. 2008; 21: (1): Rosenbaum PR. Observational Studies. 2nd ed. New York, NY: Springer; Shadish, WR, Cook, TD, Campbell, DT. Experimental & Quasi Experimental Designs for 5. Generalized Casual Inference. Boston, New York: Houghton Mifflin Company; 2002:175, 183, 211,