Potential Outcomes and Causal Inference

Transcription

1 Potential Outcomes and Causal Inference PUBL0050 Week 1 Jack Blumenau Department of Political Science UCL 1 / 47

2 Lecture outline Causality and Causal Inference Course Outline and Logistics The Potential Outcomes Framework (Intuition) The Potential Outcomes Framework (Detail) 2 / 47

3 Causality and Causal Inference

4 Knowledge and causality We do not have knowledge of a thing until we have grasped its why, that is to say, its cause. Aristotle, Physics 3 / 47

5 What are we doing in this course? We will be learning to make inferences about causal relationships: Causality The relationship between events where one set of events (the effects) is a direct consequence of another set of events (the causes). Causal inference The process by which one can use data to make claims about causal relationships. 4 / 47

6 What are we doing in this course? What does this mean in practice? Measuring the effect of some treatment on some outcome What is the effect of canvassing on vote intention? What is the effect of class size on test scores? What is the effect of peace-keeping missions on peace? What is the effect of austerity on support for Brexit? What is the effect of trade on support for Trump? What is the effect of the minimum wage on employment? What is the effect of the number of children on the risk of divorce? etc 5 / 47

7 Causal inference is hard I would rather discover one true cause than gain the Kingdom of Persia Democritus, BC 6 / 47

8 Causal inference and regression A reasonable question is: how is this different from regression, which also measures the effect of X on Y? Regression can play an important role in answering causal questions, but not always. We will see when regression serves as a useful descriptive tool, and when β k can be given a causal interpretation. 7 / 47

9 Course Outline and Logistics

10 Course outline 1: Potential Outcomes and Causal Inference 2: Randomized Experiments 3: Selection on Observables (I) 4: Selection on Observables (II) 5: Panel Data and Difference-in-Differences 6: Synthetic Control 7: Instrumental Variables (I) 8: Instrumental Variables (II) 9: Regression Discontinuity Designs 10: Overview and Review 8 / 47

11 Teaching staff Jack Blumenau Office: B.13, 29/30 Tavistock Square Office Hours: Friday 10:00-12:00 (by appointment) Philipp Broniecki Office: SPP Seminar room G.03, 29/30 Tavistock Square Office Hours: Wednesday 15:00-17:00 (by appointment) Book office hours via Moodle SPPBook 9 / 47

12 Prerequisites An introductory course in statistics/quantitative methods You should be familiar with (at least) t-tests and linear regression You should know what this is (and know how to interpret it): Y i = α + β 1 X 1 + β 2 X 2 + ɛ i If you are unsure, you could take a look at this website, and see how much of the material you recognise You can also speak to me 10 / 47

13 Assessment and key deadlines 100% of the assessment for this course will be via a 3000 word research paper in which you will apply the methods you have learnt to your own research question. I will provide guidance on the structure of these papers I will provide feedback on a one-page summary during the term This is very good preparation for your dissertations Key deadlines: December 7th: One page summary with a) clear research question; b) outline of methodological design; c) identification of data source(s) January 9th: Final project due 11 / 47

14 Learning objectives This course will provide you with: An understanding of the assumptions required to support causal claims made with quantitative data An overview of the most commonly used statistical methods which aim to estimate causal effects The practical skills required to implement these methods using R 12 / 47

15 Course Texts (I) 13 / 47

16 Course Texts (II) 14 / 47

17 Online course resources Course Website: Lecture summaries Seminar tasks Homeworks (set in class, solutions released on Monday) Moodle: (Key: regression) Mostly just links to the website Piazza: piazza.com/ucl.ac.uk/fall2018/publ0050/home All course-material related questions should be posted on Piazza is for administrative questions. 15 / 47

18 Seminars Seminar structure 1. Lecture review 2. Seminar tasks 3. Homework Seminar times: 1. Seminar 1: Friday, 10-11am, 26 Bedford Way, G11 2. Seminar 2: Friday, 11-12pm, 26 Bedford Way, G11 3. Seminar 3: Friday, 12-1pm, 26 Bedford Way, 316 If you ve not been assigned a seminar yet: 1. Surname A-H Seminar 1 2. Surname I-Q Seminar 2 3. Surname R-Z Seminar 3 16 / 47

19 The Potential Outcomes Framework (Intuition)

20 The Road Not Taken Two roads diverged in a yellow wood, And sorry I could not travel both And be one traveler, long I stood And looked down one as far as I could To where it bent in the undergrowth;... Two roads diverged in a wood, and I I took the one less traveled by, And that has made all the difference. Robert Frost ( ) 17 / 47

21 The Road Not Taken Frost s traveller highlights the central conundrum of causal analysis: An individual ( one traveler ) cannot take an action and the opposite action at the same time ( I could not travel both ) Once an action is taken, we therefore cannot tell what the world would have been like if an alternative action had been taken I.e. we cannot tell what is at the end of the road not taken. Frost also does what most of us do: We cannot observe the counterfactual outcomes that would have resulted from taking another path But we can imagine ( And that has made all the difference ) The Potential Outcomes Framework A language for describing the types of comparisons that Frost is making. A potential outcome is what happens at the end of each of the roads that the traveller might take. 18 / 47

22 Potential outcomes What does it mean for a causal factor to affect an outcome for an individual? e.g. time spent studying e.g. final PUBL0050 grade e.g. you In the counterfactual approach to causation: if the outcome for the individual would be different for different hypothetical values of the causal factor. These hypothetical outcomes (assosiated with hypothetical values of the causal factor) are known as potential outcomes. 19 / 47

23 Potential outcomes illustration Imagine we knew the grade a particular individual would receive for different amounts of study time: Each point on the line represents a potential outcome Each point tells us the hypothetical outcome associated with a hypothetical value of our causal factor Final Grade Hours of Study 20 / 47

24 Potential outcomes illustration Imagine we knew the grade a particular individual would receive for different amounts of study time: In this framework, causal effects are defined in terms of these potential outcomes If we could observe all of these outcomes for an individual, we could quantify the effect of spending 10 hours per week studying rather than 5 the average effect of spending an additional hour studying and so on Hours of Study Final Grade / 47

25 Fundamental Problem of Causal Inference For any given unit/individual we only observe one potential outcome: Final Grade Final Grade Hours of Study 12 Hours of Study To make causal inferences we have to impute (make educated guesses about) the other unrealised potential outcomes 22 / 47

26 Making comparisons across units Tempting: make comparisons of realised outcomes across units with different values of the causal factor This can lead to erroneous inferences (conclusions) Final Grade Person B Person A Hours of Study 23 / 47

27 What s the problem? 1. Heterogeneity: Different units have different potential outcomes 2. Confounding/endogeneity: when the values of the causal factor are systematically related to the potential outcomes (e.g. smarter students study more/less, students who benefit more from studying study more/less) These are the two central problems that the tools we study on the course aim to overcome. 24 / 47

28 Inference A central goal in statistics is inference: to draw conclusions about things we cannot see from things that we can see. Statistical inference The process of drawing conclusions about features/properties of the population on the basis of a sample. Causal inference The process of drawing conclusions about features/properties of the full set of potential outcomes on the basis of some observed outcomes. Generally we will be attempting to do both: using a sample to draw conclusions about the full set of potential outcomes in the population. 25 / 47

29 Break

30 The Potential Outcomes Framework (Detail)

31 Potential Outcomes History Counterfactual definitions of causality have a long intellectual history: If a person eats of a particular dish, and dies in consequence, that is, would not have died if he had not eaten of it, people would be apt to say that eating of that dish was the cause of his death John Stuart Mill, 1843 Neyman (1923)[1990] introduced the potential outcomes notation for experiments. Fisher (1925) proposed randomizing treatments to units. Rubin (1974) then extended the potential outcomes framework ( Rubin Causal Model, Holland, 1986) to observational studies 26 / 47

32 Potential outcomes notation (I) Warning All textbooks/articles use somewhat different notation for potential outcomes. We will be using the notation from Angrist and Pischke, Definition (Treatment) D i : Indicator of treatment intake for unit i { 1 if unit i received the treatment D i = 0 otherwise. D i denotes the treatment (causal variable) for unit i e.g. D i = 1: asprin; D i = 0: no asprin D i = 1: encouragement to vote; D i = 0: no encouragement to vote Defined for binary case, but will generalise to continuous treatments 27 / 47

33 Potential outcomes notation (II) Definition (Outcome) Y i : Observed outcome variable of interest for unit i Definition (Potential Outcome) Y 0i and Y 1i : Potential outcomes for unit i { Y1i Potential outcome for unit i with treatment Y di = Potential outcome for unit i without treatment Y 0i If D i = 1, Y 0i is what would have been the outcome if D i had been 0 If D i = 0, Y 1i is what would have been the outcome if D i had been 1 potential outcomes are fixed attributes for each i and represent the outcome that would be observed hypothetically if i were treated/untreated 28 / 47

34 Potential outcomes notation (III) Given Y 0i and Y 1i, it is straightfoward to define a causal effect. Definition (Causal Effect) For each unit i, the causal effect of the treatment on the outcome is defined as the difference between its two potential outcomes: τ i Y 1i Y 0i τ i is the difference between two hypothetical states of the world One where i receives the treatment One where i does not receive the treatment 29 / 47

35 Potential outcomes notation (IIII) Assumption The outcome (Y i ) is connected to the potential outcomes (Y 0i,Y 1i ) via: Y i = D i Y 1i + (1 D i ) Y 0i so { Y1i if D Y i = i = 1 Y 0i if D i = 0 A priori each potential outcome could be observed After treatment, one outcome is observed, the other is counterfactual Definition (Fundamental Problem of Causal Inference) Cannot observe both potential outcomes (Y 1i, Y 0i ) for the same unit i 30 / 47

36 No interference Assumption Assumption Recall that observed outcomes are realized as Y i = D i Y 1i + (1 D i ) Y 0i The non-interference assumption, implies that potential outcomes for unit i are unaffected by treatment assignment for unit j Also known as the Stable Unit Treatment Value Assumption (SUTVA) Rules out interference among units Examples: Effect of GOTV on spouse s turnout Effect of medication when patients share drugs 31 / 47

37 Illustration: Individual treatment effects (τ i ) Imagine a population with 4 units, where we observe both potential outcomes for each unit: i Y i D i Y 1i Y 0i τ i If we know both potential outcomes for each unit, we can easily estimate the Average Treatment Effect: τ ATE E[Y 1i Y 0i ] = 1 N (Y 1i Y 0i ) N i=1 = E[Y 1i ] E[Y 0i ] ATE = = = / 47

38 Illustration: Fundemental Problem of Causal Inference We never actually observe both potential outcomes. i Y i D i Y 1i Y 0i τ i ?? ?? 3 0 0? 0? 4 1 0? 1? ATE =?+?+?+? 4 =? Estimating individual effects or average effects requires observing unobservable potential outcomes and therefore cannot be accomplished without additional assumptions. 33 / 47

39 Illustration: Selection bias One intuitive approach is to make comparisons across units using Y i. i Y i D i Y 1i Y 0i τ i ?? ?? 3 0 0? 0? 4 1 0? 1? i.e. Compare the average observed outcome under treatment to the average observed outcome under control. However, recalling that ATE = 1.25 Difference in means = = 3 so, in this example at least Difference in means ATE 34 / 47

40 Statistics concepts and notation To proceed to a more general statement of selection bias, we will need some key building blocks of statistics Population all the units of interest Sample a subset of the units in the population Variable (e.g. Y 1i, X i ) a numerical measure Random Variable a variable whose outcome is the result of a random process number on a six-sided dice treatment status of unit i in a randomized experiment Y1i for a unit randomly selected from the population 35 / 47

41 Expectations and conditional expectations Expectations: The expectation of a random variable X is denoted E[X], where E[X] xpr[x = x] This is the average outcome of the random variable e.g. for a six-sided dice Conditional expectations: E[X] = = 3.5 The conditional expectation of a variable refers to the expectation of that variable amongst some subgroup e.g. E[Y 1i D i = 1] gives the conditional expectation of the random variable Y 1i when D i = 1 36 / 47

42 Statistical terms Parameter/Estimand a fixed feature of the population (e.g. E[Y 1i ]) Sample statistic a random variable whose value depends on the sample (e.g. the sample mean, Ȳ ) Estimator any function of sample data used to estimate a parameter (e.g. 1 n ni=1 Y i ) Unbiased estimator An statistic is an unbiased estimator of a parameter if the estimate it produces is on average (i.e. across an infinite number of samples) equal to the parameter. E.g. E[ 1 n Y i ] = E[Y i ] (1) n i=1 37 / 47

43 Common parameters/estimands/quantities of interest Definition (Average treatment effect) τ ATE = E[Y 1i Y 0i ] Definition (Average treatment effect on the treated) τ ATT = E[Y 1i Y 0i D = 1] Definition (Average treatment effect on the controls) τ ATC = E[Y 1i Y 0i D = 0] Definition (Average treatment effects for subgroups) τ ATEX = E[Y 1i Y 0i X i = x] 38 / 47

44 Average treatment effect (ATE) Earlier, we said that an intuitive quantity of interest (e.g. the parameter) is the Average Treatment Effect: τ ATE E[Y 1i Y 0i ] = E[Y 1i ] E[Y 0i ] (2) i.e. how outcomes would change, on average, if every unit were to go from untreated to treated (defined by potential outcomes). For a given sample, one obvious estimator of the ATE is the difference in group means (DIGM). Label units such that D i = 1 for i {1, 2,..., m} and D i = 0 for i {m + 1, m + 2,..., n} DIGM 1 m m i=1 Y i 1 n m n i=m+1 Y i (3) i.e. the difference in observed outcomes between treatment and control Key question Is the DIGM an unbiased estimator for the ATE? 39 / 47

45 Is the DIGM an unbiased estimator for the ATE? Remember our toy example: i Y i D i Y 1i Y 0i τ i τ ATE = = 1.25 DIGM = = 3 Is this true generally? 40 / 47

46 Is the DIGM an unbiased estimator for the ATE? Let s redefine the DIGM using τ i Y 1i Y 0i : DIGM = 1 m Y 1i 1 n m n m = 1 m = 1 m i=1 m i=1 m i=1 (Y 0i + τ i ) 1 n m τ i + 1 m m i=1 Y 0i i=m+1 n Y 0i i=m+1 n Y 0i i=m+1 Y 0i 1 n m E[DIGM] = E[τ i D i = 1] + {E[Y 0i D i = 1] E[Y 0i D i = 0]} = τ ATT + selection bias where τ ATT is the average treatment effect for the treated group 41 / 47

47 Is the DIGM an unbiased estimator for the ATE? Remember our toy example: i Y i D i Y 1i Y 0i τ i τ ATE = = 1.25 τ ATT = = 2 Selection bias = = 1 τ ATT + Bias = = 3 = DIGM Implication DIGM is an unbiased estimator of τ ATE if 1. τ ATT = τ ATE and 2. E[Y 0i D = 1] = E[Y 0i D = 0] i.e. there is no selection bias 42 / 47

48 Does E[Y 0i D i = 1] normally equal E[Y 0i D i = 0]? Normally? No. Does job training improve employment outcomes? Participants are self-selected from a population of individuals in difficult labor situations Post-training period earnings for participants would be higher than those for nonparticipants even in the absence of the program i.e. E[Y 0 D = 1] E[Y 0 D = 0] > 0 Does asprin releive headache symptoms? Those who take asprin self-select from the population of individuals who are suffering from headaches Drug-takers would have worse headaches than non-drug-takers in in the absence of pain-relief i.e., E[Y0 D = 1] E[Y 0 D = 0] < 0 Implications 1. Selection into treatment is often associated with potential outcomes. 2. Selection bias can be positive or negative! 43 / 47

49 Assignment Mechanism To work out the treatment effects we are interested in, we need to know something about the potential outcomes that we do not observe In order to infer these unobservable outcomes, we make assumptions about the assignment mechanism Definition (Assignment Mechanism) The assignment mechanism is the procedure that determines which units are selected for treatment. Examples include: Random assignment Selection on observables Selection on unobservables 44 / 47

50 Design Trumps Analysis Randomized experiments and comparative observational studies do not form a dichotomy, but rather are on a continuum. Randomized Experiments Observational Studies Rubin, 2008 Selection on observables Regression Matching Weighting Selection on unobservables Difference-in-Differences and synthetic control Instrumental Variables Regression Discontinuity Design 45 / 47

51 The value of potential outcomes The strength of [the potential outcomes framework] is that it allows us to make these assumptions more explicit than they usually are. When they are explicitly stated, the analyst can then begin to look for ways to evaluate or partially test them. Holland, / 47

52 Summing up Potential outcomes: Causality is defined by potential outcomes, not by observed outcomes FPOCI: We only ever observe one potential outcome per unit Inference: Statistical inference learning about the population from a sample Causal inference learning about the population of potential outcomes from the observed outcomes Selection bias: The difference in means is only an unbiased estimator for the ATE when there is no selection bias 47 / 47

53 Next week: Randomized Experiments