Research Design: Causal inference and counterfactuals

Transcription

1 Research Design: Causal inference and counterfactuals University College Dublin 8 March 2013

2

3 Outline

4 Inference In regression analysis we look at the relationship between (a set of) independent variable(s) and a dependent variable. Statistical inference is concerned with the question how likely it is to observe this relationship given the null hypothesis of no relationship (frequentist) A different question is whether or not we can deduce that the independent variable is a cause of the dependent one. (Note: references in these slides can be found in the associated handout.)

5 Inference In regression analysis we look at the relationship between (a set of) independent variable(s) and a dependent variable. Statistical inference is concerned with the question how likely it is to observe this relationship given the null hypothesis of no relationship (frequentist) or how much we should update our beliefs concerning this relationship given our new evidence (Bayesian). A different question is whether or not we can deduce that the independent variable is a cause of the dependent one. (Note: references in these slides can be found in the associated handout.)

6 Association association causation

7 Association Given that, say, X and Y are correlation (associated), there are still many possible causal patterns at play.

8 Many possible patterns

9 Many possible patterns

10 Many possible patterns

11 Many possible patterns

12 Many possible patterns

13 Many possible patterns

14 Many possible patterns

15 Many possible patterns

16 Inference Generally, to make causal inferences from your analysis, additional assumptions need to be made in addition to the ones already made for associational or predictive inference.

17 Outline

18 Fundamental problem Imagine, there are two kinds of people, one group, T = 1, that has a college degree, and another group, T = 0, that does not. We want to measure where a college degree leads to a higher salary, Y.

19 Fundamental problem Imagine, there are two kinds of people, one group, T = 1, that has a college degree, and another group, T = 0, that does not. We want to measure where a college degree leads to a higher salary, Y. What we would like to know is the difference for any individual i whether they have a college degree or not: Y Ti=1 i Y Ti=0 i. However, for every individual i, we either observe Y Ti=1 i, or we observe Y Ti=0 i they either have the degree or they don t.

20 We wish... Respondent Degree Y T i=0 i Y T i=1 i effect 1 Yes Yes No No Yes Yes No Yes

21 We wish... we have... Respondent Degree Y T i=0 i Y T i=1 i 1 Yes Yes No 90 4 No 87 5 Yes Yes No 92 8 Yes 109 effect

22 Potential outcomes Potential outcome = { Y1i if T i = 1 Y 0i if T i = 0 E.g., Y 1i is the salary of individual i had (s)he a college degree, irrespective of whether (s)he actually does. (Angrist & Pischke 2009, 13-14)

23 Potential outcomes Potential outcome = { Y1i if T i = 1 Y 0i if T i = 0 E.g., Y 1i is the salary of individual i had (s)he a college degree, irrespective of whether (s)he actually does. Y i = Y 0i +(Y 1i Y 0i )T i = Y 0i +δt i, where δ = Y 1i Y 0i is the causal effect. (Angrist & Pischke 2009, 13-14)

24 Average treatment effect Because it is impossible to observe individual treatment effect, we usually turn to average treatment effect: E[δ] = E[Y 1i Y 0i ] = E[Y 1i ] E[Y 0i ], which we could naively estimate with ˆδ = E[Y 1i T i = 1] E[Y 0i T i = 0].

25 Average treatment effect Because it is impossible to observe individual treatment effect, we usually turn to average treatment effect: E[δ] = E[Y 1i Y 0i ] = E[Y 1i ] E[Y 0i ], which we could naively estimate with ˆδ = E[Y 1i T i = 1] E[Y 0i T i = 0]. This assumes that E[Y 1i ] reflects the salary for people with a college degree, irrespective of whether they got one or not, and that E[Y 0i ] reflects the salary without a college degree, irrespective of whether they got one or not.

26 Counterfactual causality By making such assumptions by looking at the ATE we are making a counterfactual argument. We are making assumptions of what Y 1i would have been, had i had a college degree.

27 Counterfactual causality By making such assumptions by looking at the ATE we are making a counterfactual argument. We are making assumptions of what Y 1i would have been, had i had a college degree. This implies that we cannot measure a causal effect, only estimate it.

28 Counterfactual causality By making such assumptions by looking at the ATE we are making a counterfactual argument. We are making assumptions of what Y 1i would have been, had i had a college degree. This implies that we cannot measure a causal effect, only estimate it. To understand when the ATE assumptions are reasonable, we need to look at the effect of covariates other variables that relate to Y, which we will denote by X.

29 Treatment effect: abbreviations ATE Average Treatment Effect E[δ] = E[Y 1i Y 0i ] ATT ATE for the Treated E[δ T ] = E[Y 1i Y 0i T i = 1] ATC ATE for the Control (untreated) E[δ C ] = E[Y 1i Y 0i T i = 0] PATE Population ATE E[Y 1i Y 0i ] SATE Sample ATE E n [Y 1i Y 0i ] LATE Local ATE E[Y 1i Y 0i X i = x] CATE Conditional ATE E[Y 1i Y 0i X i = x] and analogously we have LATT, PATT, SATC, etceta. Note that ATE, ATT, ATC implicitly refer to population values. E n [ ] is the sample mean, i.e. E n [x] = x = 1 n n i=1, where the E n allows for the formulation of conditional and counterfactual means.

30 Bias in causal inference Using shorthand E 01 = E[Y 0i T i = 1], etc., and taking π as the population proportion that received the treatment, E[δ] = πe[δ T i = 1]+(1 π)e[δ T i = 0] = π(e 11 E 01 )+(1 π)(e 10 E 00 )

31 Bias in causal inference Using shorthand E 01 = E[Y 0i T i = 1], etc., and taking π as the population proportion that received the treatment, can be decomposed into E[δ] = πe[δ T i = 1]+(1 π)e[δ T i = 0] = π(e 11 E 01 )+(1 π)(e 10 E 00 ) (E 11 E 00 ) = E[δ]+(E 01 E 00 )+(1 π){(e 11 E 01 ) (E 10 E 00 )}.

32 Bias in causal inference Using shorthand E 01 = E[Y 0i T i = 1], etc., and taking π as the population proportion that received the treatment, can be decomposed into E[δ] = πe[δ T i = 1]+(1 π)e[δ T i = 0] = π(e 11 E 01 )+(1 π)(e 10 E 00 ) (E 11 E 00 ) = E[δ]+(E 01 E 00 )+(1 π){(e 11 E 01 ) (E 10 E 00 )}. (E 11 E 00 ) observed difference in effect E[δ] average treatment effect (E 01 E 00 ) selection bias (1 π){(e 11 E 01 ) (E 10 E 00 )} differential treatment effect bias (Morgan & Winship 2007

33 SUTVA The stable unit treatment value assumption SUTVA is simply the a priori assumption that the value of Y for unit i when exposed to treatment t will be the same no matter what mechanism is used to assign treatment t to unit i and no matter what treatments the other units receive. (Rubin (1986: 961), as cited in Morgan & Winship (2007: 37))

34 Outline

35 When studying effect of, say, T on Y, by examining the statistical association between the two variables, we need to ascertain that the observed effect is not caused by a third variable, say, X. (Pearl 2000: )

36 When studying effect of, say, T on Y, by examining the statistical association between the two variables, we need to ascertain that the observed effect is not caused by a third variable, say, X. We can say that T and Y are confounded when there is a third variable X that influences both T and Y; such a variable is then called a confounder of T and Y. (Pearl 2000: )

37 Another way of saying this is that if E(Y T,X) E(Y T) and E(T X) E(T), X is a confounder of the effect of T on Y. (Lee 2005: 44)

38 If healthier patients take a drug and sicker patients do not, we can find an association between drug and recovery even when the drug does not work. If sicker patients take a drug and healthier patients do not, we might not find an association between drug and recovery even when the drug works. association causation The first example is also called a spurious effect (not to be confused with spurious regression).

39 Endogeneity The situation where cor(x,ε) 0 is called endogeneity. Endogeneity has three main causes: Measurement error in X Simultaneity or reverse causation Omitted variables

40 Note that confounding is a causal concept, not an associational one! (Pearl 2000)

41 Note that confounding is a causal concept, not an associational one! X has to have a causal effect on T and X has to have a causal effect on Y for there to be an issue. (Pearl 2000)

42 Outline

43 X affects both T and Y = control (Lee 2005: 43-48)

44 Do control This is the typical case of a confounding factor, and hence should be eliminated through controlling.

45 Do control

46 X affects both T and Y = control T affects Y, which in turn affects X = do not control (Lee 2005: 43-48)

47 Don t control

48 Don t control In this case, X is an effect of Y. By controlling for X, you can severily underestimate the effect of T on Y.

49 Don t control In this case, X is an effect of Y. By controlling for X, you can severily underestimate the effect of T on Y. Imagine that a college degree leads to a better income leads to a nicer car. Controlling for the price of the car in estimating the effect of having a college degree on income might cancel the effect.

50 X affects both T and Y = control T affects Y, which in turn affects X = do not control T affects X, which in turn affects Y = do not control... (Lee 2005: 43-48)

51 Don t control

52 Don t control To get the overall effect of T on Y, you want to include the effect through X.

53 Don t control To get the overall effect of T on Y, you want to include the effect through X. E.g. if you want to know the effect of changing the policy regarding smoking in pubs on the amount of smoking in general, you do not care through what mechanism this happened (through peer pressure, laziness, etc.), but only about the overall effect.

54 X affects both T and Y = control T affects Y, which in turn affects X = do not control T affects X, which in turn affects Y = do not control unless you explicitly want only the direct effect (Lee 2005: 43-48)

55 Maybe control

56 Maybe control Remember the following equation: β = β +φγ

57 Maybe control Remember the following equation: β = β +φγ Sometimes you are interested in β (so control), sometimes in β (so don t control).

58 Maybe control Example: A scholarship for poorer students might help them to get a college degree, which in turn might help them to earn more money later in life.

59 Maybe control Example: A scholarship for poorer students might help them to get a college degree, which in turn might help them to earn more money later in life. Having a scholarship on your CV, however, might also further your career, independent of the effect of having a college degree.

60 Maybe control Example: A scholarship for poorer students might help them to get a college degree, which in turn might help them to earn more money later in life. Having a scholarship on your CV, however, might also further your career, independent of the effect of having a college degree. To see the overall effect of the scholarship, don t control on having a college degree.

61 Maybe control Example: A scholarship for poorer students might help them to get a college degree, which in turn might help them to earn more money later in life. Having a scholarship on your CV, however, might also further your career, independent of the effect of having a college degree. To see the overall effect of the scholarship, don t control on having a college degree. To see the effect of having a scholarship, independent of the effect of getting a college degree, do control for college degree.

62 X affects both T and Y = control T affects Y, which in turn affects X = do not control T affects X, which in turn affects Y = do not control unless you explicitly want only the direct effect X affects Y, but not T, nor the effect of T on Y (Lee 2005: 43-48)

63 Maybe control

64 Maybe control When X affects Y, but not T, there is no confounding issue and the estimates for the effect of T on Y should not be affected by inclusion of X. However, including X in the model can still help for efficiency. (Gelman & Hill 2007: 177)

65 X affects both T and Y = control T affects Y, which in turn affects X = do not control T affects X, which in turn affects Y = do not control unless you explicitly want only the direct effect X affects Y, but not T, nor the effect of T on Y X affects Y, not T, but it does affect of effect of T on Y (interaction) (Lee 2005: 43-48)

66 Maybe control Here including the interaction in your model can highlight how the effect is different for different groups.

67 Maybe control Here including the interaction in your model can highlight how the effect is different for different groups. Note that it affects the interpretation, but that the estimation of the overall ATE is not affected by controlling for X.

68 Outline

69 The ideal experiment To avoid any effect of covariates the ideal is to randomly select participants for your research from the overal population (enables inference to the population) and to randomly assign the treatment to these participants (enables causal inference).

70 Experiment Field experiment Natural experiment Blocking Matching Regression etc.

71 Kitchen sink A typical approach in the quantitative social sciences is to collect a number of different theories / hypotheses, add them all as variables to a regression, and see who wins. This is the kitchen sink approach (or garbage can approach). If anything, the above discussion should have made clear that to draw causal inferences, a clear distinction of treatment from covariates is crucial. In other words: focus your research!

72 Kitchen sink A typical approach in the quantitative social sciences is to collect a number of different theories / hypotheses, add them all as variables to a regression, and see who wins. This is the kitchen sink approach (or garbage can approach). If anything, the above discussion should have made clear that to draw causal inferences, a clear distinction of treatment from covariates is crucial. In other words: focus your research! (Note that the garbage can phrase has also been used to argue against ignoring nonlinearities (Achen 2005), as opposed to careless specification of the causal effect.)

73 Kitchen sink Another way of putting the issue is that the above is all about trying to study the effect of a cause (treatment), rather than the cause of an effect. The latter is perhaps ill-defined and runs into the infinite regress of causation. (See Gerring (2001, 2012) for an extensive discussion of Y-centered and X-centered research.) (Gelman & Hill 2007: 187)

74 Causal diagrams The preceding examples underline how it is important to always draw out the causal diagram and consider carefully how you select cases and select controls when making causal inferences.

75 Equifinality Equifinality refers to the situation where a particular outcome might come about through different causal paths. No particular path might show a strong association between independent and dependent variable.

76 Equifinality Equifinality refers to the situation where a particular outcome might come about through different causal paths. No particular path might show a strong association between independent and dependent variable. Should this be seen as problematic for the counterfactual causal inference framework?

77 Outline

78 KKV s 5 rules 1 Construct falsifiable theories 2 Build theories that are internally consistent 3 Select dependent variables carefully No endogenous relationship Ensure variation in dependent variable 4 Maximize concreteness 5 State theories as encompassing as feasible (King, Keohane & Verba 1994: )

79 Gerring s criteria Clarity Manipulability Separation Independence (priority) Impact Mechanism

80 Discussion points How do causal inference and prediction relate? ( The proof of the pudding is in the eating? )

81 Discussion points How do causal inference and prediction relate? ( The proof of the pudding is in the eating? ) How does the theory thus far translate to qualitative research? Does it?

82 Discussion points How do causal inference and prediction relate? ( The proof of the pudding is in the eating? ) How does the theory thus far translate to qualitative research? Does it? Are all causal inferences counterfactual?

83 Discussion points How do causal inference and prediction relate? ( The proof of the pudding is in the eating? ) How does the theory thus far translate to qualitative research? Does it? Are all causal inferences counterfactual? How does this all relate to causal mechanisms in the sense of Hedstrom & Swedberg?