Potential Outcomes and Causal Inference I

Transcription

1 Potential Outcomes and Causal Inference I Jonathan Wand Polisci 350C Stanford University May 3, 2006

2 Example A: Get-out-the-Vote (GOTV) Question: Is it possible to increase the likelihood of an individuals turnout by making an appeal to vote? Non-experimental Research Design: Using survey data, regress reported voter turnout on reported contact with candidates (Rosenstone and Hansen, 1993) Experimental Research Design: Gerber and Green 1998 field experiment sample of households (with one or two individuals registered to vote) randomly assignmed to treatments

3 Example A: Get-out-the-Vote (GOTV) Experimental Research Design: Gerber and Green 1998 field experiment sample of households with one or two individuals registered to vote: random assignment to 0, 1, 2 or 3 types of treatments: in-person contact telephone call direct mail random assigment to appeal civic duty close election neighborhood solidarity Any concerns about design of experiment?

4 Example A: Get-out-the-Vote (GOTV) Design for in-person contact Group N Treatment 5,800 Control 23,500 Result: Group N Type Treatment assigned and received 1,600 compliers Treatment assigned not received 4,200 non-compliers Control group 23,500 compliers In-person Contact rate: 28 percent Is non-compliance a potential problem?

5 Example B: Ballot order effect Question: Is there an electoral (dis)advantage related to placement of a name on an election ballot? Non-experimental Research Design:? Experimental Research Design: Compare vote totals using randomly assigned placement of names (e.g., California gubernatorial recall).

6 Example B: Ballot order effect California law: randomly order the letters of the alphabet. use to order of the names in the first assembly district next district, top letter rotates to the bottom, and others move up one place, and so on Compare vote share on first page or not: District Potential Outcomes Actual Observed Y i (0) Y i (1) Treatment Outcome Y i ? ? ? ? ? ?

7 Examples How would measure the effects and test the hypotheses in these examples? what is the estimand of interest? is the effect statistically significant?

8 Rubin Causal Model What is meant by A causes B? Holland/Rubin the effect of A is B Glymor/ Lewis / Aristotle A is a cause of B What can be thought of as causes? Counterfactual / potential outcome mantra: NO CAUSATION WITHOUT MANIPULATION Holland s causality requirements: events, not attributes, are potential causes cause must directly affect individual cause always specified relative to a specific alternative

9 Examples of Treatments and Counterfactuals Outcome Treatment Counterfactual Death Received Drug Received Placebo Income Completed PhD Quit PhD Vote for Buchanan Butterfly Ballot format Absentee ballot format Citizen Voted Contacted for mobilization Not contacted Margin of votes Received X more dollars Did not receive more money Armed Conflict Adopted democratic rule Retained non-democratic rule

10 Rubin Causal Model Core features of Rubin s framework: 1. Causal effects are defined by potential outcomes for both randomized experiments and observational studies, and necessary assumptions to do so are stated formally, e.g., SUTVA (Stable Unit-Treatment-Value Assumption) Random and Non-Trivial Assignment Exclusion, causal effect only through treatment Monotonicity of compliance in assignment 2. The central role of a model of the assignment mechanism is made clear for all forms of inference. Mechanism creating missing data of potential outcomes is based on assignment (known or hypothesized) Enables a formal statement of benefits of randomized experiments for causal inference Enables the formal statement of possible dependence of assignment based on expected outcomes, i.e., confounded designs

11 Classical experiment Let (Y, X, W ) be observable random variables, 1. Y is the outcome of interest 2. X a pre-determined variable 3. W indicator for treatment status Define a stochastic assignment mechanism 1. randomly draw a unit from a population 2. assign treatment W = 1 to unit with probability p 0 3. measure all variables, including outcome of interest Potential outcomes are Y 0 and Y 1, but we observe only, Y = WY 1 + (1 W )Y 0

12 Classical experiment Condition 1 Let Y 1 = y 1 (X) and Y 0 = y 0 (X) for some functions y 1 () and y 0 (). Condition 2 Pr[W = 1 X = x] = p 0 for all w in the support of W Proposition If Conditions 1 and 2 hold, then 1. Exchangeability Pr[X x W = j] = Pr[X x], for j {1, 2}, x X 2. Conditional means identify average treatment effect E[Y W = 1] E[Y D = 0] = E[Y 1 Y 0 ] = ATE Show (1) is true by use of Bayes Rule and independence.

13 Classical experiment Conceptual difference from regression (or control function) methods: randomization, unlike control, does not attempt to remove the effect of a source of variation, but instead makes the source act like a random variable, equally likely to favor T or not T in an repetition. by randomization and replication the contribution of any source can be made small if N is large enough. Cochran (1965, 7 8),

14 Example B: Ballot order effect District Potential Outcomes Actual Observed Y i (0) Y i (1) Treatment Outcome Y i ? ? ? ? ? ? Null hypothesis, no difference between randomly assigned first placement (W = 1) and non-first placement (W = 0).

15 Example B: Ballot order effect District Potential Outcomes Actual Observed Y i (0) Y i (1) Treatment Outcome Y i (55.0) (72.0) (72.7) (70.0) (66.0) (78.9) Under null, could replace missing values with observed. With this assumption, possible to compare differences between average treated and control for other permutations of random assignment.

16 Example B: Ballot order effect Fisher Exact Tests: Example H 0 Y i (1) = Y i (0) for all i = 1,..., N H 1 Y i (1) Y i (0) for some i = 1,..., N Step 1. Use null hypothesis to obtain unobserved values. Step 2. Define a test statistic candidate, T = Ȳ (1) Ȳ (0) = 1 N 1 W i Y i (1) (1 W i )Y i (0) N 1 i=1 Under the null we expect T to be close to zero. Question: Is T = 5.07 in example too big for H 0 to be true? N 0 i=1

17 Example B: Ballot order effect Step 3. Calculate T for all possible permutations of treatment W1 W2 W3 W4 W5 W6 T

18 Example B: Ballot order effect Simply counting tells us that 12/20 assignment vectors W lead to a test statistic at least equal to 5.07 in absolute value. Thus, 5.07 is not rare at all and does not constitute strong evidence against the null hypothesis. Note: Assumption under the Null for the Fisher test is not equivalent to H 0 1 N 1 N 1 W i Y i (1) = (1 W i )Y i (0) i=1 N 0 i=1 Which null hypothesis is more interesting?

19 Randomization Inference Neyman s 1935 paper in the JRSS: Neyman: So long as the average yields of any treatments are identical, the question as to whether these treatments affect separate yields on single plots seems to be uninteresting and academic... Fisher:... It may be foolish, but that is what the z test was designed for, and the only purpose for which it has been used....

20 Randomization Inference General properties of test statistic T (W, Y obs, X) function of observables stochastic through dependence on W Y(0) and Y(1) are treated as fixed quantities. Most common test: T = Ȳ (1) Ȳ (0) = 1 N 1 W i Y i (1) 1 N 0 (1 W i )Y i (0) N 1 N 0 i=1 but could also do this with tranformations of data, likelihood, ranks, or model. Approach is very good for initial analysis to establish presence of effects. i=1

21 Example A: Get-out-the-Vote (GOTV) Even with random assignment to treatment, who receives treatment may not be random Observed Variable Z i {0, 1} D i {0, 1} Y i {0, 1} Description Assigned to treatment: to be contacted Was i actually treated / contacted Outcome : Voted or not If acceptance of treatment is correlated to expected outcome, then we have confounded assignment. Angrist, Imbens, and Rubin (1996, JASA) reinterpret IV estimator in temrs of the effect from a confounded experiment.

22 Example A: Get-out-the-Vote (GOTV) AIR notation Description D i (Z ) {0, 1} Y i (Z, D) {0, 1} Contact given assigment Z Voted given assigment and actuality Potential Treatment D i (Z i = 1) D i (Z i = 0) Potential Outcome Y i (Z i = 1, D i = 1) Y i (Z i = 1, D i = 0) Y i (Z i = 0, D i = 1) Y i (Z i = 0, D i = 0) Description Assigned to treatment Assigned to control Description Assigned to treatment, and treated Assigned to treatment, and not treated Assigned to control, and treated Assigned to control, and not treated Only going to see one outcome for an individual.

23 Confoundedness: Terminology Perfect compliance: All assigned to contact are contacted D i (Z i ) = Z i Non-compliance with Treatment: Subset unavailable for contact D i (Z ) Z Could D i (Z ) > Z be possible? Causal Effects: Intention-to-treat D i (1) D i (0) Y i (1, D i (1)) Y i (0, D i (0))

24 Causal Inference without Unconfounded Assigment 1. SUTVA a. If Z i = Z i then D i (Z ) = D i (Z ) b. If Z i = Z i and D i = D i then Y i (Z, D) = Y i (Z, D ) 2. Random and Non-Trivial Assignment P(Z = c) = P(Z = c ) for all c, c where Σc = Σc 3. Exclusion: Y (Z, D) = Y (Z, D) for all Z, Z, D 4. Non-zero causal effect of Z on D E[D i (1) D i (0)] 0 5. Monotonicity: D i (1) D i (0) > 0

25 Confoundedness: LATE AIR define the Local Average Treatment Effect as: ( [ ] E [Y i 1, Di (1) ) ( Y i 0, Di (0) )] E Y i (1) Y i (0) D i (1) D i (0) = 1 = [ ] E D i (1) D i (0) = ITT Vote ITT Contact

26 Example A: hypothetical selection An example of a model of voter mobilization from field experiment: Y i = β 0 + D i β 1 + u i D i = α 0 + Z i α 1 + v i { 1 D D i = i > 0 0 Di 0 cov(u i, v i ) 0 Structural approaches would posit (and estimate) such a model.

27 Confoundedness: LATE Decomposing ITT vote outcome (AIR eq 9): Y i ( 1, Di (1) ) Y i ( 0, Di (0) ) ( = Y i Di (1) ) ( Y i Di (0) ) ( ) ( ) = Y i (1)D i (1)+Y i (0)(1 D i (1)) Y i (1)D i (0)+Y i (0)(1 D i (0)) = Y i (1)D i (1) + Y i (0) Y i (0)D i (1) Y i (1)D i (0) Y i (0)) + Y i (0)D i (0) ( ) ( ) = Y i (1) D i (1) D i (0) Y i (0) D i (1) D i (0) ( )( ) = Y i (1) Y i (0) D i (1) D i (0)

28 Confoundedness: LATE AIR derivation continued (eqs 10 12) ( E [Y i 1, Di (1) ) ( Y i 0, Di (0) )] [( )( )] = E Y i (1) Y i (0) D i (1) D i (0) [ ] [ ] = (1)E Y i (1) Y i (0) D i (1) D i (0) = 1 P D i (1) D i (0) = 1 [ ] [ ] +( 1)E Y i (1) Y i (0) D i (1) D i (0) = 1 P D i (1) D i (0) = 1 [ ] [ ] +(0)E Y i (1) Y i (0) D i (1) D i (0) = 0 P D i (1) D i (0) = 0 [ ] [ ] = E Y i (1) Y i (0) D i (1) D i (0) = 1 P D i (1) D i (0) = 1 So, rearranging, [ ] E Y i (1) Y i (0) D i (1) D i (0) = 1 = E ( ( [Y i 1,D i (1) ) Y i 0,D i (0) E [ D i (1) D i (0)=1 ] ) ]

29 Example A: Get-out-the-Vote (GOTV) Vote Percent for Treatment: 47.2%; for Control: 44.8% Percent Contacted of Assigned Treatment: 27.9% Local Average Treatment Effect [ ] E Y i (1) Y i (0) D i (1) D i (0) = 1 = = = ( E [Y i 1, Di (1) ) ( Y i 0, Di (0) )] [ ] E D i (1) D i (0) ITT Vote ITT Contact =.087

30 Instrumental variables Problems Exclusion restriction assumptions are generally untestable and dubious (Moffit 1996; Heckman 1995, 1997) Precision of inference can be poor if instruments are weak or sample is small (Bound et al. 1995) Only generally valid for measuring ATE when treatment effect is constant for all individuals, and when there is no compliance problems.

31 I.V., a simple version If Z X is square, such as when X = [1, D] Z = [1, Z ] which we have in the case of the experiment, then Y i = β 0 + D i β 1 + u i Y = Xβ + u D i = α 0 + Z i α 1 + v i D = Z α + v β = (X P z X) 1 X P z Y = (X Z (Z Z ) 1 Z X) 1 X Z (Z Z ) 1 Z Y = ((Z X) 1 (Z Z )(X Z ) 1 X Z (Z Z ) 1 Z Y = ((Z X) 1 Z Y = cov(y, Z ) cov(x, Z )

32