New Weighting Methods for Causal Mediation Analysis

Transcription

1 New Weighting Methods for Causal Mediation Analysis Guanglei Hong, University of Chicago, Jonah Deutsch, Mathematica, Xu Qin, University of Chicago, NCME Training Session April 7, 2016 SREE Workshop, March 1, 2017

2 Concepts of causal mediation Brief review of existing methods Rationale of the RMPW strategy Parametric and nonparametric procedures Demonstration and hands-on practice with o Stata module (Jonah) o R package (Xu) o Stand-alone RMPW software Multisite causal mediation analysis (Xu) Web Access: 2

3 Concepts of Causal Mediation 3

4 Intervention theory: Hypothesis of a causal mechanism operating through a focal mediator Total effect = Indirect effect + Direct effect Indirect effect: Expected change in the outcome due to an intervention-induced change in the focal mediator Direct effect: Expected change in the outcome caused by the intervention through other unspecified mechanisms without affecting the mediator 4

5 Indirect and Direct Effects Mediator Treatment Outcome e.g., Treatment: Encouragement to study Mediator: Time spent on studying Outcome: Test score 5

6 Indirect and Direct Effects Mediator Treatment Outcome Explicate your theoretical hypotheses: o How would encouragement affect hours of study? o How would encouragement affect test score through changing hours of study? o Would encouragement affect test score without changing hours of study? o Would encouragement change the effectiveness of study? o Would encouragement change test-taking behavior? 6

7 Treatment-by-Mediator Interaction Mediator: Z If treated If control Treatment: A Outcome: Y The treatment may change (1) an individual s mediator values, and (2) the mediator-outcome relationship. (Judd & Kenny, 1981) 7

8 Change in Legislation Replacing AFDC with TANF in the late 1990s Employment-focused incentives and services Population Welfare applicants/recipients with preschool-age children (mostly single mothers, disproportionately depressed) Public Concern How would the policy affect maternal depression? What is the mediating role of employment? 8

9 NEWWS Riverside Experiment LLLLLL: AA = 11 CCCCCCCCCCCCCC: AA = 00 Labor force attachment program (LFA) vs. the Control condition 9

10 Features of the LFA program provided encouragement, support, and emphasis on seeking and securing jobs threatened sanctions no guaranteed employment 10

11 Under LFA Employment might boost self-efficacy and reduce depression Anticipated sanctions if unemployed might add stress and increase depression Under the control condition (with guaranteed cash assistance) Employment might not be as beneficial Unemployment might not be as detrimental to psychological well-being 11

12 Theoretical Hypotheses Employment Under LFA Under control Treatment Depression 12

13 Data LFA Control Total Sample Size Depressive symptoms measured at the 2-year follow-up A 12-item battery (CES-D) (e.g., I could not get going in the past week) Item responses on a scale from rarely to most of the time Summary score ranging from 0 to 34 Mean = 7.49; SD = 7.74 Quarterly employment data maintained by the state 13

14 Potential Mediators and Potential Outcomes under SUTVA Individual Unit Treatment Potential Mediators Potential Outcomes A Z(1) Z(0) Y(1) Y(0) Population Average E[Z(1)] E[Z(0)] E[Y(1)] E[Y(0)] 14

15 Intention-to-Treat (ITT) Effects Sample estimate of E[Y(1) Y(0)]: 0.11 (SE=0.64, t=0.18, p=0.86) Sample estimate of E[Z(1) Z(0)]: 0.26 (p <.05) LFA Control Employment Rate Null ITT Effect = No Mediation? Or Are There Counteracting Mechanisms? 15

16 Challenges to Causal Mediation Analysis The treatment may change the mediator-outcome relationship, key to understanding the causal mechanism How to decompose the total effect then? Individuals select mediator values in randomized experiments How to adjust for selection bias? 16

17 Notation Potential Outcomes: Y(1,Z(1)): depression score if assigned to LFA i.e., Y(1) Y(0,Z(0)): depression score if assigned to the control condition i.e., Y(0) Y(1,Z(0)): depression score if assigned to LFA yet counterfactually experiencing employment associated with the control condition Y(0,Z(1)): depression score if assigned to the control condition yet counterfactually experiencing employment associated with LFA 17

18 Potential Mediators and Potential Outcomes Unit Potential Treatment Potential Outcomes Mediators A Z(1) Z(0) Y(1,Z(1)) Y(0,Z(0)) Y(1,Z(0)) Y(0,Z(1)) Y(1,1) Y(0,1) Y(1,1) Y(0,1) Y(1,1) Y(0,0) Y(1,0) Y(0,1) Y(1,0) Y(0,0) Y(1,0) Y(0,0) Y(1,1) Y(0,1) Y(1,1) Y(0,1) Y(1,1) Y(0,0) Y(1,0) Y(0,1) Y(1,0) Y(0,0) Y(1,0) Y(0,0) Population Average E[Z(1)] E[Z(0)] E[Y(1,Z(1))] E[Y(0,Z(0))] E[Y(1,Z(0))] E[Y(0,Z(1))] 18

19 Treatment Effect Decomposition Total effect (Policy impact on depression) E [ Y ( 1, Z( 1) ) Y ( 0, Z( 0) )] Natural indirect effect (Policy impact on depression through changing employment) = E [ Y ( 1, Z( 1) ) Y ( 1, Z( 0) )] Natural direct effect (Policy impact on depression if failed to change employment) [ Y ( 1, Z( 0) ) Y ( 0, Z( 0) )] + E (Robins & Greenland, 1992; Pearl, 2001) 19

20 Further Decomposition Natural indirect effect (Employment effect on depression if treated) E [ Y ( 1, Z( 1) ) Y ( 1, Z( 0) )] Pure indirect effect (Employment effect on depression if untreated) = E [ Y ( 0, Z( 1) ) Y ( 0, Z( 0) )] Treatment-by-mediator interaction effect (Policy impact on depression through changing the mediator-outcome relationship) + E [{ Y ( 1, Z( 1) ) Y ( 1, Z( 0) )} { Y ( 0, Z( 1) ) Y ( 0, Z( 0) )}] 20

21 Brief Review of Existing Methods 21

22 Path Analysis and SEM Mediator Model Outcome Model Z Y = γ + αa + ε Z = γ + βz + λa + ε Y Z Y. αβ identifies the natural indirect effect while λ identifies the natural direct effect under the following assumptions (possibly within levels of covariates): Sequential Ignorability The treatment assignment is independent of all the potential mediators and the potential outcomes The mediator value assignment under each treatment is independent of all the potential outcomes There is no treatment-by-mediator interaction Both models are linear and are correctly specified 22

23 New Strategies Allowing for Treatment-by-Mediator Interaction Modified regression approaches (Pearl, 2010; Petersen, Sinisi, & van der Laan, 2006; Preacher, Rucker, & Hayes, 2007; Valeri & VanderWeele, 2013; VanderWeele, 2013; VanderWeele & Vansteelandt, 2009, 2010) Identification Assumptions: Sequential Ignorability (within levels of covariates) The treatment assignment is independent of all the potential mediators and the potential outcomes The mediator value assignment under each treatment is independent of all the potential outcomes Both the mediator model and the outcome model are correctly specified 23

24 New Strategies Allowing for Treatment-by-Mediator Interaction Resampling approach (Imai, Keele, and Yamamoto, 2010; Imai, Keele, & Tingley, 2010) Identification Assumptions: Sequential Ignorability (within levels of covariates) The treatment assignment is independent of all the potential mediators and the potential outcomes The mediator value assignment under each treatment is independent of all the potential outcomes with Both the mediator model and the outcome model are correctly specified 24

25 Rationale of the RMPW Strategy 25

26 Rationale for RMPW Obtain consistent estimates of E[Y(0,Z(0))] E[Y(1,Z(1))] E[Y(1,Z(0))] E[Y(0,Z(1))] 26

27 Rationale for RMPW E[Y(0,Z(0))] E[Y(1,Z(1))] Direct Effect : E[Y(1,Z(0))] E [ Y ( 1, Z( 0) ) Y ( 0, Z( 0) )] E[Y(0,Z(1))] 27

28 Rationale for RMPW E[Y(0,Z(0))] E[Y(1,Z(1))] Indirect Effect : E[Y(1,Z(0))] E [ Y ( 1, Z( 1) ) Y ( 1, Z( 0) )] E[Y(0,Z(1))] 28

29 Rationale for RMPW E[Y(0,Z(0))] E[Y(1,Z(1))] Pure Indirect Effect : E[Y(1,Z(0))] E [ Y ( 0, Z( 1) ) Y ( 0, Z( 0) )] E[Y(0,Z(1))] 29

30 Rationale for RMPW E[Y(0,Z(0))] E[Y(1,Z(1))] E[Y(1,Z(0))] Treatment - by - Mediator Interaction Effect : E {[ Y ( 1, Z( 1) ) Y ( 1, Z( 0) )] [ Y ( 0, Z( 1) ) Y ( 0, Z( 0) )]} E[Y(0,Z(1))] 30

31 How to Estimate E[Y(1, Z(0))]? 31

32 Hypothetical Sequential Randomized Experiment A = 0 Employment Rate: pr Z = 1 A = 0 =. ( ) 4 Unemployment Rate: pr ( Z = 0 A = 0) =. 6 [ ( 0, Z( 0) )] E Y 32

33 Hypothetical Sequential Randomized Experiment A = 0 A = 1 Employment Rate: pr Z = 1 A = 0 =. ( ) 4 Employment Rate: pr Z = 1 A = 1 =. ( ) 7 Unemployment Rate: pr ( Z = 0 A = 0) =. 6 Unemployment Rate: pr pr E Y ( Z1 = 1 A = 1) =.7 [( Z = A0 = 1, A Z= = 11) ] = E [ Y ( 0, Z( 0) )] E[ Y ( 1, Z( 1) )] 33

34 Hypothetical Sequential Randomized Experiment A = 0 A = 1 A = 1 Employment Rate: pr Z = 1 A = 0 =. ( ) 4 Employment Rate: pr Z = 1 A = 1 =. ( ) 7 ω = = 4 7 Employment Rate: pr ( Z = 1 A = 0) =. 4 Unemployment Rate: pr ( Z = 0 A = 0) =. 6 Unemployment Rate: pr ω = = 2 pr E Y 0.3 ( Z1 = 1 A = 1) =.7 [( Z = A0 = 1, A Z= = 11) ] = Unemployment Rate: pr ( Z = 0 A = 0) =. 6 E [ Y ( 0, Z( 0) )] E[ Y ( 1, Z( 1) )] ( Z = z A = 0) ( Z = z A = 1) pr θz ω = = pr θ [ ( 1, Z( 0) )] E Y Z ( 0) ( 1) 34

35 Outcome Model Unweighted Control Group AA = 0 Treatment (A) Duplicate (D1) RMPW Estimate E[Y(0,Z(0))] 35

36 Outcome Model Weighted LFA Group AA = 1 Treatment (A) Duplicate (D1) RMPW Estimate E[Y(0,Z(0))] θ Z θ 1 0 ( 0) Z ( 1) E[Y(1,Z(0))] Direct Effect 36

37 Outcome Model Unweighted LFA Group (AA = 1), a duplicate DDD = 1 Treatment (A) Duplicate (D1) RMPW Estimate E[Y(0,Z(0))] θ Z θ 1 0 ( 0) Z ( 1) E[Y(1,Z(0))] E[Y(1,Z(1))] Indirect Effect 37

38 Outcome Model Merge the original sample with a duplicate LFA group Treatment (A) Duplicate (D1) RMPW Estimate E[Y(0,Z(0))] θ Z θ 1 0 ( 0) Z ( 1) E[Y(1,Z(0))] E[Y(1,Z(1))] Direct Effect Indirect Effect Weighted outcome model (with cluster-robust standard errors): Y ( DE ) ( IE.1) = γ + γ A + D1 + e 0 γ Direct Effect Indirect Effect 38

39 Randomization of Treatment Only Assume sequential randomization within levels of X For an employed LFAunit, ω = pr pr ( Z = 1 A = 0, X = x) ( Z = 1 A = 1, X = x) = θ θ Z Z ( 0) = 1( x) ( 1) = 1( x) Propensity of employment under the control condition Propensity of employment under LFA For an unemployed LFAunit, ω = pr pr ( Z = 0 A = 0, X = x) ( Z = 0 A = 1, X = x) θz = θ Z ( 0) = 0( x) ( 1) = 0( x) Propensity of unemployment under the control condition Propensity of unemployment under LFA 39

40 86 Pretreatment Covariates X Psychological well-being at the time of randomization Personal history of employment and history of welfare dependence Human capital, employment status, earnings, income at randomization Preference to work, willingness to accept a low-wage job, ashamed to be on welfare, perceived social support, and perceived barriers to work Household composition (e.g., number of children, age of children, living with an extended family, living with a partner, marriage status; ever been a teen mother) Childcare arrangement, extra family burden (e.g., physical health or mental health problem of a family member), public housing residence, and residential mobility Demographic features (age and race/ethnicity) 40

41 Identification Assumptions Sequential Ignorability When only the treatment is randomized, we assume sequential randomization within levels of pretreatment covariates X = x, that is, No confounding of treatment-mediator relationship and treatment-outcome relationship No confounding of mediator-outcome relationship within a treatment and across treatments 41

42 We Do Not Assume Unlike the existing methods reviewed earlier, we do not assume: No treatment-by-mediator interaction The exclusion restriction Functional form of the outcome model 42

43 How to Estimate E[Y(0, Z(1))]? 43

44 Hypothetical Sequential Randomized Experiment A = 0 A = 1 A = 0 Employment Rate: pr Z = 1 A = 0 =. ( ) 4 Employment Rate: pr Z = 1 A = 1 =. ( ) 7 ω = = 7 4 Employment Rate: pr Z = 1 A = 1 =. ( ) 7 Unemployment Rate: pr ( Z = 0 A = 0) =. 6 Unemployment Rate: pr ( Z = 0 A = 1) =. 3 ω = = 1 2 Unemployment Rate: pr ( Z = 0 A = 1) =. 3 E [ Y ( 0, Z( 0) )] E [ Y ( 1, Z( 1) )] E[ Y ( 0, Z( 1) )] pr( Z = z A = 1) ω = pr( Z = z A = 0) 44

45 Outcome Model Weighted Control Group (AA = 0), a duplicate DD0 = 1 Treatment (A) Duplicate (D1) Duplicate (D0) RMPW Estimate E[Y(0,Z(0))] E[Y(0,Z(1))] θ Z θ ( 1) Z ( 0) Pure Indirect Effect E[Y(1,Z(0))] θ Z θ ( 0) Z ( 1) E[Y(1,Z(1))] 45

46 Outcome Model Weighted outcome model (with robust clustered standard errors): Y ( DE ) ( IE.1) ( IE.0) = γ + γ A + γ D1 + D0 0 γ + e Direct Effect Indirect Effect Pure Indirect Effect γ γ ( IE.1) ( IE.0) estimates the natural treatment - by - mediator interaction effect ( IE. 1) ( IE.0) ( IE.1) ( IE.0) withstandard error : Var( ˆ γ ) + Var( ˆ γ ) 2Cov( ˆ γ, ˆ γ ). 46

47 Parametric and Nonparametric Procedures 47

48 Analytic Procedure 1. Estimating/predicting the propensity scores 2. Parametrically or nonparametrically estimating the weight 3. Parametrically or nonparametrically checking balance 4. Estimating the causal effects under weighting 48

49 1. Estimation/Prediction of Propensity Scores of (Un)Employment For each participant in the LFA program, the propensity of being employed under LFA θ Z ( ) ( ( ) ) ( 1) = 1 x = pr Z 1 = 1 A = 1, X = x The propensity of being unemployed under LFA θθ ZZ 1 =0 xx = 1 θθ ZZ 1 =1 xx 49

50 1. Estimation/Prediction of Propensity Scores of (Un)Employment For each member of the control group, the propensity of being employed under the control condition : θ x = ( = = X = x) Z 0 = 1 pr Z 0 1 A 0, ( ) ( ) ( ) The propensity of being unemployed under the control condition θθ ZZ 0 =0 xx = 1 θθ ZZ 0 =1 xx. Apply this prediction function to members of the LFA group, we predict their propensity of employment (or unemployment) had they counterfactually been assigned in the control group. 50

51 2a. Parametric Estimation of RMPW Weights For an employed ˆ ω = ˆ θz ˆ θ Z ( 0) = 1( x) ( 1) = 1( x) LFA unit, For an unemployed ˆ ω = ˆ θz ˆ θ Z ( 0) = 0( x) ( 1) = 0( x) LFA unit, 51

52 2a. Parametric RMPW Adjustment Control LFA Unweighted LFA Weighted Never Employed Ever Employed

53 3a. Parametric Balance Checking IF the sequential ignorability assumption holds, and IF the propensity score models are correctly specified, propensity score-based parametric weighting (e.g., IPTW) is expected to balance the covariate distribution across the treatment-by-mediator combinations A=1 A=0 Standardized Difference Z=1 Z=0 Z=1 Z=0 Global Test X1 X

54 4. Weighted Estimation of Causal Effects Y ( DE ) ( IE.1) = γ + γ A + D1 + 0 γ e A two-step estimation procedure (equivalent to a GEE procedure in Stata) that estimates the asymptotic standard errors fully accounting for the fact that the weight is unknown and must be estimated. 54

55 Total Effect Decomposition Parametric RMPW Natural Direct Effect 1.29 SE = 0.87, t = 1.48, p =.11 Natural Indirect Effect SE = 0.47, t = -1.89, p =.06 55

56 Further Decomposition Parametric RMPW Natural Direct Effect 1.29 SE = 0.87, t = 1.48, p =.11 Natural Indirect Effect SE = 0.47, t = -1.89, p =.06 Pure Indirect Effect Natural Treatment-by-Mediator Interaction Effect 0.32 SE = 0.27, t = 1.48, p = SE = 0.53, t = -2.26, p <.05 56

57 2b. Nonparametric Estimation of RMPW Weights Each observation, whether in the LFA group or the control group, has θθ ZZ 1 =1 xx and θθ ZZ 0 =1 xx In the nonparametric procedure, i. Group observations into quartiles by their values of θθ ZZ 1 =1 xx. ii. Separately within each of these quartiles, further group observations into quartiles by their values of θθ ZZ 0 =1 xx iii. After steps 1 and 2, each observations has been grouped into 1 of 16 strata 57

58 2b. Nonparametric Estimation of RMPW Weights θ Z ( 0 ) = 1 θ Z ( 1 ) = 1 58

59 2b. Nonparametric Estimation of RMPW Weights Within each stratum, re-estimate each individual s propensity scores on the basis of the cell counts: A = 1 A = 0 Z(1) = 1 ˆ θ Z ( 1 ) = 1 =. 5 ˆ θ Z ( 0 ) = 1 =. 3 Z(0) = 1 ˆ ˆ Z(1) = 0 θ θ Z ( 0 ) = 0 =. 7 Z ( 1 ) = 0 =. 5 Z(0) = 0 59

60 2b. Nonparametric Estimation of RMPW Weights For an employed LFA unit: ωω = pppp(zz = 1 AA = 0, SS = ss) pppp(zz = 1 AA = 1, SS = ss) For an unemployed LFA unit: pppp(zz = 0 AA = 0, SS = ss) ωω = pppp(zz = 0 AA = 1, SS = ss) For example, pppp(zz = 1 AA = 0, SS = ss) is simply the proportion of the control group members in strata s who are employed, which re-estimates the propensity of employment under the control condition for individuals in this same stratum. 60

61 3b. Nonparametric Balance Checking IF the sequential ignorability assumption holds, propensity score-based nonparametric weighting (e.g., marginal mean weighting through stratification, MMWS, see Hong, 2010, 2012, 2015) is expected to balance the covariate distribution across the treatment-by-mediator combinations (Nonparametric weighting tends to have robust results despite misspecifications of functional forms of the propensity score models.) 61

62 Total Effect Decomposition Parametric RMPW Natural Direct Effect 1.29 SE = 0.87, t = 1.48, p =.11 Natural Indirect Effect SE = 0.47, t = -1.89, p =.06 Nonparametric RMPW 1.34 SE = 0.79, t = 1.70, p = SE = 0.38, t = -2.43, p <.05 62

63 Further Decomposition Parametric RMPW Natural Direct Effect 1.29 SE = 0.87, t = 1.48, p =.11 Natural Indirect Effect SE = 0.47, t = -1.89, p =.06 Nonparametric RMPW 1.34 SE = 0.79, t = 1.70, p = SE = 0.38, t = -2.43, p <.05 Pure Indirect Effect Natural Treatment-by- Mediator Interaction Effect 0.32 SE = 0.27, t = 1.48, p = SE = 0.53, t = -2.26, p < SE = 0.30, t = 1.50, p = SE = 0.49, t = -2.85, p <.01 63

64 Summary: Strengths of the RMPW Method Does not assume no treatment-by-mediator interaction Does not make assumptions about the functional form of the outcome model Unconstrained by the distribution of the outcome or the distribution of the mediator Directly estimates the natural direct and indirect effects and their standard errors Further decomposes the indirect effect or the direct effect and estimates the impact of treatment-induced change in the mediational process Easy to implement with standard software or the specialized RMPW software 64

65 Summary: Limitations of the RMPW Method Untestable sequential ignorability assumption!!! Unmeasured pretreatment confounders Unmeasured post-treatment confounders Correct specification of the propensity score models remains crucial especially when applying the parametric RMPW method 65

66 BREAK 66

67 Stata Module R Package Stand-Alone RMPW Software 67

68 Stata RMPW ado File 68

69 RMPW R Package Single-level mediation analysis Download the package: install.packages( rmpw ) Load the package: library(rmpw) The package includes one function, rmpw(). It decomposes the total treatment effect into: Natural direct and indirect effects; Pure direct effect, total and pure indirect effect, and treatmentby-mediator interaction effect; Pure indirect, total and pure direct effect, and treatment-bymediator interaction effect. Try the examples in the help file:?rmpw 69

70 RMPW R Package Multisite mediation analysis Download the package: install.packages( MultisiteMediation ) Load the package: library(multisitemediation) The package implements mediation analysis in multisite trials. It includes two functions: msmediate(): estimates population average and between-site variance of direct and indirect effects. vartest.mediate(): performs hypothesis testing for between-site variances. Try the examples in the help file: (1)?msmediate (2)?vartest.msmediate 70

71 Stand-Alone RMPW Software RMPW Click here to get RMPW. For a sample RMPW data set, Click here For details of features, please refer to RMPW program manual.pdf. Please report problems with the software by sending to Richard Congdon (richard@hlmsoft.net) and copying Guanglei Hong (ghong@uchicago.edu) 71

72 Data Set for Exercise Save RMPW_workshop_unimputed_Apr dta Save rmpw.exe Run rmpw.exe In the Data Selection and Preparation window, Click the radio button for Stata under Input Data Type Browse the data file on your computer 72

73 Variables for Exercise idnumber ID dep12sm2 outcome (depression score) treat treatment (treatment indicator) emp mediator (ever employed) pq51 precov (depressed past week) pq53 precov (lonely past week) nohsdip precov (no HS diploma/ged) teen_parent precov (ever been a teen parent) emp_prior precov (prior employment) race precov (race/ethnicity) AGE precov (age category) 73

74 Variables for Exercise idnumber Continuous dep12sm2 Continuous treat (0, 1) emp (0, 1) pq51 Continuous pq53 Continuous nohsdip (0, 1) Teen_parent (0, 1) emp_prior Describe scale: 0, 1, 2 AGE (0, 1, 2) race Describe scale: 0, 1, 2 Click Save or Save as to save your specifications Click Next 74

75 Multisite Causal Mediation Analysis 75

76 Job Corps: the largest federal training program Target population years old Disadvantaged Disconnected from school and work Services provided Focus: Academic and vocational training Supplementary services Residential services Health education and care Drug and alcohol treatment Social skills training Job placement services Program administered at centers 125 centers nationwide Average duration at centers: eight months 76

77 77

78 National Job Corps Study: nationally representative study Multi-site randomized trial From 1994 to 1996, applicants at each center were randomly assigned to Job Corps program Control group: could enroll in other training programs Survey data collected during the four years after random assignment Sample Total sample size: 15,386 (9,409 in Job Corps; 5,977 in control) Number of sites (centers): 103 Site size: 40 to 700, with an average of

79 Job Corps Program (Treatment: A) Weekly Earnings (Outcome: Y) Outcome: weekly earnings measured four years after treatment assignment How much is the population average program impact? Does the program impact vary significantly across sites? 79

80 Site 80

81 Job Corps: 40% vs Control: 22% Educational Attainment (Mediator: Z) Job Corps Program (Treatment: A) Weekly Earnings (Outcome: Y) Educational Attainment: whether obtained a GED diploma or vocational certificate 30 months after treatment assignment. (Indirect Effect) Supplementary services: residential living, social skill training, individual counseling, drug and alcohol treatment, and health care. (Direct Effect) 81

82 Educational Attainment (Mediator: Z) Job Corps Program (Treatment: A) Weekly Earnings (Outcome: Y) How much are the average direct and indirect effects? How do direct and indirect effects vary across sites? 82

83 Sequential ignorability: Within levels of the observed pretreatment covariates at each site, there is no confounding of treatment-mediator relationship and treatment-outcome relationship, i.e. a person is as if randomized to different treatment conditions. Within levels of the observed treatment status and observed pretreatment covariates at each site, there is no confounding of mediator-outcome relationship within a treatment and across treatments, i.e. a person is as if randomized to different mediator values. Selected pretreatment covariates in Job Corps: age, gender, race, criminal involvement, drug use, parental education, employment, educational attainment and earnings at the baseline. 83

84 1. Estimation/prediction of propensity scores 2. Balance checking 3. Estimate direct and indirect effects site by site 4. Estimate population average effects 5. Estimate standard errors of population average estimates 6. Estimate between-site variance of effects 7. Test between-site variance of effects 8. Sensitivity Analysis 84

85 Fit a multilevel logistic regression to each treatment group. Let pp aa aaiiii = PPPP(ZZ iiii = 1 AA iiii = aa, XX iiii = xx, SS iiii = jj), i for individual, j for site llllllllll (aa) pp aaiiii (aa) 1 pp aaiiii = XX aaiiii αα aa + SS aaiiii rr aajj, rr aajj ~NN 00, ΣΣ aa, aa = 0,1 XX: pretreatment covariates at the baseline. SS: vector of covariates with random effects rr jj Predict PPPP(ZZ iiii = zz AA iiii = 1, XX iiii = xx, SS iiii = jj) for each individual in the experimental group based on the model fitted to the experimental group, in which zz is the observed mediator value of individual i at site j. Predict PPPP(ZZ iiii = zz AA iiii = 0, XX iiii = xx, SS iiii = jj) for each individual in the experimental group based on the model fitted to the control group. 85

86 IF the sequential ignorability assumption holds, and IF the propensity score models are correctly specified, propensity score-based parametric weighting (e.g., IPTW) is expected to balance the covariate distribution across the treatment-by-mediator combinations at each site. A=0 A=1 Standardized Difference Z=0 Z=1 Z=0 Z=1 Global Test X1 X

87 Parameters Estimators E[Y ij (1,Z ij (1))] = E[Y ij A ij = 1, S ij = j] E[Y ij (1,Z ij (0))] = E[W ij Y ij A ij = 1, S ij = j] Mean outcome in the treated group at site j Weighted mean outcome in the treated group at site j Indirect Effect: ββ jj (II) WW ii = Pr(ZZ iiii = zz AA iiii = 0, XX iiii = xx, SS iiii = jj) Pr(ZZ iiii = zz AA iiii = 1, XX iiii = xx, SS iiii = jj) Direct Effect: ββ jj (DD) E[Y ij (0,Z ij (0))] = E[Y ij A ij = 0, S ij = j] Mean outcome in the control group at site j 87

88 Take a simple average over site-specific estimate estimates JJ γγ = 1 JJ jj=1 ββ jj in which γγ = γγ DD, γγ II, ββ jj = ( ββ jj (DD), ββ jj (II) ). 88

89 Sampling variability in weight cannot be ignored The sampling variability in weight, estimated through multilevel logistic regressions in step 1, needs to be reflected in the sampling variability of site-specific effect estimates. Solution: treat the two-step estimators as a one-step generalized method of moments (GMM) estimator, which solves the moment functions stacked from both steps. 89

90 Var ββ jj = Var ββ jj ββ jj + ββ jj = Var ββ jj ββ jj + Var ββ jj Site 1 Site 2 Site 3 ββ 1 (DD) ββ 3 (DD) ββ 3 (DD) ββ 1 (DD) ββ 2 (DD) ββ 2 (DD)

91 The test randomly permutes the site indices, based on the idea that all permutations of the site indices are equally likely under the null. With direct effect as an example, the algorithm is as follows: Step 1. Calculate the test statistic, QQ (DD) oooooo, for the original sample. Step 2. Randomly permute the site indices while holding fixed the site size. Calculate the test statistic for the permutation sample. By repeating this step 200 times, we can obtain 200 test statistics, (DD) QQ pp, pp = 1,, 200. Step 3. Calculate the p-value of this test as the proportion of the permutation samples with QQ pp (DD) QQoooooo (DD). 91

92 Omitted pretreatment and posttreatment confounders may cause bias. Sensitivity analysis may be conducted to assess the impact of omitted pretreatment and posttreatment confounders. Conceptualizing and investigating the relationships among multiple mediators may help with handling observed posttreatment covariates. 92

93 Average Effect Estimate Effect Size Between-Site Standard Deviation 95% Plausible Values of Site-Specific Effects Total Impact (SE=5.30, p=0.002) (p=0.03) (-32.22, 65.04) Indirect Effect 8.68 (SE=1.61, p<0.001) (p=0.06) (-5.28, 22.64) Direct Effect Correlation Between Direct and Indirect Effects 7.74 (SE=5.38, p=0.15) (p=0.055) (-38.83, 54.31) 93 93

94 Rely on correct mediator model specification Rely on correct outcome model specification Rely on correct distributional assumptions Applicable to discrete mediator or outcome Sampling distribution of indirect effect estimate is normal Hypothesis testing for the between-site variances Clarify identification assumptions to draw causal conclusions Path Analysis Yes Yes Yes No No No No Proposed Yes No No Yes Yes Yes Yes Between-site variance in mediation mechanism has not been discussed in causal inference literature. 94

95 May not be optimal when site sizes are extremely small. Robust to the misspecification of the outcome model, while less efficient than path analysis. Rely on correct specification of the mediator model. 95

96 Most Relevant Readings Hong, G. (2015). Causality in a social world: Moderation, mediation, and spill-over. West Sussex, UK: John Wiley & Sons. Hong, G., Deutsch, J., & Hill, H. D. (2015). Ratio-of-mediatorprobability weighting for causal mediation analysis in the presence of treatment-by-mediator interaction. Journal of Educational and Behavioral Statistics, 40(3), Qin, X., & Hong, G. (in press). A weighting method for assessing between-site heterogeneity in causal mediation mechanism. Journal of Educational and Behavioral Statistics. 96

97 Other Related Publications Bein, E., Deutsch, J., Porter, K., Qin, X., Yang, C., & Hong, G. (2015). Technical report on two-step estimation in RMPW analysis. MDRC. Hong, G. (2010). Ratio of mediator probability weighting for estimating natural direct and indirect effects. In JSM Proceedings, Biometrics Section. Alexandria, VA: American Statistical Association, pp Hong, G., Deutsch, J., & Hill, H. (2011). Parametric and non-parametric weighting methods for estimating mediation effects: An application to the National Evaluation of Welfare-to- Work Strategies. In JSM Proceedings, Social Statistics Section. Alexandria, VA: American Statistical Association, pp Hong, G., & Nomi, T. (2012). Weighting methods for assessing policy effects mediated by peer change. Journal of Research on Educational Effectiveness special issue on the statistical approaches to studying mediator effects in education research, 5(3), Huber, M. (2014). Identifying causal mechanisms (primarily) based on inverse probability weighting. Journal of Applied Econometrics, 29(6), Tchetgen Tchetgen, E. J., & Shpitser, I. (2012). Semiparametric theory for causal mediation analysis: Efficiency bounds, multiple robustness and sensitivity analysis. The Annals of Statistics, 40(3),

98 Acknowledgement Funding agencies University of Chicago Spencer Foundation William T. Grant Foundation IES Statistical and Research Methodology in Education Grant 98