Instrumental Variables

Transcription

1 Instrumental Variables

2 Caveat An Instrumental Variable is a somewhat complicated methodological idea. Can be technically challenging at first. Good applications often are embedded in the language and questions of specific fields of study. Academic politics about what methods work best and who should be cited play an outsized role in teaching. We don t have enough time to master the idea today.

3 Purpose Review the basics with some of the new lingo. Provide some advice about how to apply designbased thinking to IV settings. Give examples of IV studies. Inspire you to learn more on your own time.

4 Instrumental Variables As A More General Method

5 Learning About Causal Effects Randomized experiments have played a crucial role in establishing causality and in estimating the magnitude of effects in many fields. But randomized experiments (obviously) are not the only way that people generate evidence of causality. And it is not the only way that people use IV to estimate the magnitude of causal effects.

6 IV and RCT We have been talking about how to repair a broken RCT using a method called instrumental variables. Use the wald ratio or 2sls The idea here is that RCT is really an special case of the more general method of IV. A very important and convincing special case. But still: a special case

7 Instrumental Variables The logic of IV applies outside the domain of formally conducted RCTs. A variety of statistical tools, robustness checks, and analytical norms have grown up around the use of instrumental variables. Learning about IV methods has tremendous spill over effects. It helps you be a more critical reader of research papers. Even papers that don t use IV methods at all.

8 The Plan Instrumental Variables Simple Introduction Better Notation and Assumptions Wald Ratio and LATE IV as a generalized research design. Implementation from a design based perspective: Probing Assumptions Assessing potential threats to validity Interpreting results

9 Starting with an Easy Case

10 Constant Linear Effects Y = β 0 + β 1 D + ε In the past, this was the standard way to analyze causal relationships. (Perhaps with covariates.) Still serves as an off the shelf model for framing the discussion.

11 But Treatment is Not Randomly Assigned. Y = β 0 + β 1 D + ε Cov(D, ε) is non zero. Positive Cov(D, ε) means that treated people would have had higher D even in the absence of treatment. Selection bias.

12 If we estimate the model using standard regression then the coefficients will be biased. What to do?

13 Enter the IV Suppose we find a variable/situation/subpopulation that satisfies: E[ε Z = 1] = E[ε Z =0] The instrument is not associated with any other factors that affect Z. Combines Independence and Exclusion.

14 What Can You Do With Z? You want to know: Y = β 0 + β 1 D + ε But OVB means that the direct approach will provide a biased estimate of β 1.

15 What can you do with Z Condition on Z = 1 and take expectations. E[Y Z = 1] = E[β 0 + β 1 D + ε Z = 1] E[Y Z = 1] = β 0 + β1e[d Z=1] + E[ε Z=1] Condition on Z = 0 and take expectations E[Y Z = 1] = E[β 0 + β 1 D + ε Z = 0] E[Y Z = 1] = β 0 + β1e[d Z=0] + E[ε Z=0] Difference the two sides: E[Y Z = 1] - E[Y Z = 0] = {β 0 + β1e[d Z=1] + E[ε Z=1]} {β 0 + β1e[d Z=0] + E[ε Z=0]} Cancel and invoke exclusion/independence: E[Y Z = 1] - E[Y Z = 0] = β1 {E[D Z=1] E[D Z=0]}

16 What Can You Do With Z? E[Y Z = 1] - E[Y Z = 0] = β1 {E[D Z=1] E[D Z=0]} Divide both sides by the first stage: β1 = { E[Y Z = 1] - E[Y Z = 0] } / {E[D Z=1] E[D Z=0]} Success. For this to work, you need: Independence/Exclusion First Stage Constant Effects/Linearity

17 Moving To A More General Setting: Notation and Assumptions for Modern IV

18 Notation Z is the randomly assigned study arm. D = D 1 Z + D(0)(1 Z) is the treatment received expressed in terms of potential treatments under alternative values of Z. Y(d, z) is a potential outcome under hypothetical values of Z and D. Imagine a set of variables for every possible combination of Z and D.

19 Assumptions 1. First Stage: Z does affect D 2. Independence: Z is randomly assigned. 3. Exclusion: Z has no direct effect on outcomes. 4. Monotonicity: Z always affects D in the same direction. 5. SUTVA: Non-interference\No Spillovers

20 The First Stage Assumption 1 Treatment take up rates vary across subpopulations defined by Z. The treatment exposure rate is different in the treatment and control arms of the study. Z affects D. The instrument affects how many people are treated.

21 The First Stage Assumption 2 E[D Z = 1] E[D Z = 0] 0 Pr[D Z = 1] Pr[D Z = 0] 0 D i = D = α 0i + α 1i Z i E[α 1i ] 0 D = α 0 + α 1 Z + υ α 1 0 Is the first stage assumption testable?

22 Independence Assumption 1 The instrument is statistically independent of the potential outcome and potential treatment variables. People who receive Z = 1 are not more likely to respond to the instrument than people who receive Z = 0. (No assignment by people most likely to take advantage of the opportunity.) People who receive Z = 1 are not more likely to benefit from the treatment. People who receive Z = 0 are not more likely to do well on the outcome even in the absence of the treatment.

23 Independence Assumption 2 Pr(Y(d,z), D(z) Z = 1) = Pr(Y(d,z), D(z) Z = 0) Is the independence assumption testable? Can the independence assumption be investigator controlled?

24 Some Reinforcement The first two IV assumptions are: First Stage and Independence Both assumptions are at least partially testable: First Stage is easily tested: are treatment take up rates in the Z = 1 and Z = 0 groups? Independence is partially testable through balancing tests: are covariates similar in the Z = 1 and Z = 0 groups? Independence is very well justified by a known random assignment procedure. Then Z should be uncorrelated with covariates by construction

25 Exclusion Restriction 1 The instrumental variable has no direct effect on the outcome. The instrument only affects outcomes through its first stage effect on treatment exposure. The IV causal chain: Z affects D and (then) D affects Y.

26 Exclusion Restriction 2 Y(d,z) = Y(d) Before the exclusion restriction, there are four potential outcomes: Y(1,1) : outcome when D = 1, Z = 1 Y(1,0) : outcome when D = 1, Z = 0 Y(0,1) : outcome when D = 0, Z = 1 Y(0,0) : outcome when D = 0, Z = 0 Exclusion restriction collapses the four d,z pairs down to the standard pair of potential outcomes.

27 Exclusion Restriction 3 Typically not testable against data. RCTs are not an exception and many social/economic/educational RCTs likely violate the exclusion restriction. This point is difficult to see at first but here are some examples of threats to the exclusion restriction: Placebo effects, demoralization effects, Hawthorne effects, substitution effects, income effects. These things can all be created by commonly employed random assignment procedures.

28 Exclusion Restriction 4 Good papers justify the exclusion restriction using: Substantive theory and logic. Sensitivity analysis Smart partial tests the flow from the specific design and theory: the treatment should not affect group X, where it had no first stage. Bad papers justify the exclusion restriction by appealing to: Random assignment. Covariate balance. The idea that the results make sense. Older papers and bad new papers ignore the exclusion restriction entirely or bury it in a footnote.

29 Monotonicity Assumption 1 The instrument affects treatment exposure in the same direction for each person. Either the instrument nudges everyone towards the treatment or away from the treatment. Not both. Important in the newish heterogeneous treatment effects approach. Not testable. Appeal to logic. Perhaps try to investigate two-way flows.

30 Monotonicity Assumption 2 D(1) D(0) for all subjects; or D(1) D(0) for all subjects. Recall: D = D(0) + Z(D(1) D(0)) D i = D = α 0i + α 1i Z i Monotonicity requires that: α 1i 0 D(1) D(0) 0 for each i.

31 Spillover Effects (SUTVA) The instruments and treatments associated with person i have no influence on the instruments and treatments associated with person j. Y(di, dj) = Y(di) No herd immunity, peer effects, etc. Economists sometimes say that quasi-experiments are typically partial equilibrium studies.

32 Assumption Story Testable Against Data Strategies First Stage Z affects D. Yes. Test the first stage. Both precision and magnitude are important. Independence Z is as good as randomly assigned. Partial. Conduct balance tests on covariates. Exclusion Z has no direct effect on Y. Usually not. Examine logic and theory. Conduct study specific tests and sensitivity analysis if possible. Monotonicity Z weakly affects D in the same direction for everyone. Usually not. Sub group analysis to check sign of first stage. Two way flows may indicate a problem. No Spillovers Other people s treatments don t matter. Usually not. Tricky. But sometimes cluster level instruments can provide a solution. Conduct analysis at level that contains the spill.

33 Getting To LATE I will skip the derivation. But we can do it this afternoon if there is interest.

34 Combine Assumptions with Data Suppose we believe that assumptions 1 through 5 are reasonable in the context of some study. And suppose our study produces observations of (Y, D, Z) for each person. (Notice that the study only gives us data on realized outcomes and not potential outcomes.) What can we learn?

35 Intent To Treat ITT = E[Y Z = 1] E[Y Z = 0] Y = β 0 + β 1 Z + ε Also called the reduced form Effect of the instrument on the outcome. Mean differencing and regression give the same answer in the simple case.

36 What Does ITT Show? Fairly simple derivation shows that: ITT = E[Y(1) Y(0) D(1) D(0) = 1] x Pr(D(1) D(0) = 1] Average effect for compliers times the size of the complier population.

37 What Does The First Stage Show? F = E[D Z = 1] E[D Z = 0] Fairly simple derivation shows that: F = E[D(1) D(0)] = Pr(D(1) D(0) = 1] First stage reveals the fraction of the population who are compliers.

38 Wald Ratio Produces IV ITT/FS = E[Y(1) Y(0) D(1) D(0) = 1]

39 Who Are Compliers? Recall from earlier presentations: D(1) = 1, D(0) = 0 Complier D(1) = 1, D(0) = 1 Always Taker D(1) = 0, D(0) = 0 Never Taker D(1) = 0, D(0) = 1 Defier Monotonicity assumption rules out defiers.

40 Two Stage Least Squares

41 Wald Ratio Great for understanding the logic of IV and for clarifying identification of causal effects. But: Tedious and statistically costly to deal with covariates. Slightly annoying to compute standard errors. What if you have more than one Z? What if you have more than one D?

42 Practicality In practice, lots of people think in terms of Wald Ratios. Maybe they even start the analysis with Wald Ratios. Often the calculate simple Wald Ratios in their heads during seminars.

43 Practicality But when it comes down to actually doing their work and writing a paper They employ a method called Two Stage Least Squares.

44 Two Stage Least Squares We ll present things in regression notation because that is the easier way to understand this material. As in the previous example, you can always rewrite things in terms of potential outcomes to make causality and treatment effects clearer. For now, we ll start with constant treatment effects

45 Two Equations in TSLS 1. Causal Model (Structural Model) 2. First Stage

46 Causal Equation The causal or structural equation: Y = β 0 + β 1 D + ε Assume that D is not randomly assigned and omitted variable bias is a concern. Notice that Z does not appear in the model.

47 Terminology When D is assigned in a non-random way that we think will produce biased estimates, people say that: D is endogenous. D is determined simultaneously with Y D is subject to omitted variables bias (OVB) D is self-selected. Conditioning on D creates a selected sample. They all mean (basically) the same thing. We hate your paper. You need and instrument. (We ll hate that too.)

48 Stages of TSLS The first stage of TSLS involves estimating the first stage regression. The second stage involves estimating the causal model by replacing D with its predicted value.

49 First Stage D = α 0 + α 1 Z + υ Regression of treatment on instrument. Core idea is that we can decompose the variation in treatment exposure into two parts: The part generated by the design/instrument. The part generated by non-random choices. Compute predicted value from the equation: Dhat is the good variation or design based variation Uhat is the remaining (bad) variation.

50 Second Stage Y = β 0 + β 1 D + ε Y = β 0 + β 1 (α 0 + α 1 Z + υ) + ε Y = {β 0 + β 1 α 0 }+ {β 1 α 1 Z} + {β 1 υ + ε} Y = θ 0 + θ 1 Z + ξ θ 1 = β 1 α 1 is the reduced form or ITT effect. What should you do if you want to know β 1?

51 Insight: No Need To Divide If You Only Include The Good Variation From The First Stage Y = β 0 + β 1 (α 0 + α 1 Z) + ε Y = β 0 + β 1 (Dhat) + ε You obtain an unbiased estimate of β 1 because α 0 + α 1 Z consists of only the variation in D that is explained by Z.

52 Recap Intuitive TSLS Algorithm: Estimate the first stage model. Compute predicted values Regress Y on the predicted values. In practice, the whole thing is computed in one step.

53 Extensions of TSLS

54 Extensions of TSLS: Use The White Board. Adding covariates to the model. Multiple Instrumental Variables. Multiple Treatment Variables.

55 Implementation Strategies and Rules of Thumb

56 Implementation Test the first stage first. F > 10 is a common rule of thumb. Remember to investigate first stage by sub-population. Think about the logic of the first stage. Consider monotonicity assumption. Conduct balance tests. Develop theory needed to assess exclusion restriction. Devise sensitivity analysis for exclusion restriction. Try to find partial tests of exclusion.

57 Why Covariates? To make it more likely that the independence and exclusion restrictions actually hold. Remember that we may not have explicitly randomly assigned values of Z. Controlling for covariates increases power if the covariates explain variation in Y. Covariates can also be used to assess assumptions: balance tests. Covariates also facilitate sub-population analysis.

58 Balancing Tests? If Z is as good as randomly assigned then if you make a table of means for sub-groups with different values of Z, you should see balance. Unless you think Z is only as good as randomly assigned after controlling/stratifying on X. Then you need to adjust for X directly.

59 Why should you avoid IV methods? Very hard to come up with instruments that will credibly satisfy the assumptions. Why delude yourself? Even when it works, the standard errors from IV estimates are apt to be very large. Interpretation is weird: LATE or weighted average of LATEs may be a treatment effect for a special complier group that you do not care about. Weak instruments problem.

60 Examples With Investigator Control Oregon Health Insurance Experiment Moving To Opportunity Experiment

61 Examples with non-investigator randomization Vietnam Draft Lottery Charter School Lottery

62 Examples where quasiexperiments, theories, and models are used to justify IV

63 Effects of Obesity on Health Care Costs Cawley and Meyerhoefer. Problem is measurement error in weight and omitted variable bias in weight. Weight of relative is an instrument for the individual s weight. Claim is that shared household environment has no effect on weight. Controversial for lay people. Research from behavioral genetics finds no support for the claim that shared household environments affect weight. OLS finds that obesity raises costs by $650 per year. IV suggests that obesity raises costs by $2700 per year.

64 Effects of HIV Prevalence on Sexual Behavior Oster (2011) How come people in Africa don t practice safer sex now that HIV prevalence is so high? Problem: Local HIV Prevalence is not randomly assigned and is (obviously) partly determined by risky sexual practices. Instrument: Distance from the origin of the epidemic (center of Congo) Finds that after accounting for endogeneity, people do reduce risky sex when HIV prevalence is higher.

65 Other Examples Settler Mortality and Economic Institutions Family Size and Longer Run Well-being (Twins and Gender Hunting) Effects of Eminent Domain Laws on Investment Levels and Patterns (Heterogenous Judges) Effects of Foster Care on Child Outcomes (Heterogeneous Judges) Demand Curves for Fish: Fulton Fish Market (Stormy Days and Sunny Days)

66 Class Exercise: A Hypothetical Study Causal Effects of Flu Vaccination on Younger Adults

67 Side Track To Talk About Weak Instruments

68 Weak Instruments An instrument may be weak if the coefficient on the instrumental variable in the first stage is small. D i = X i γ + π 1 Z i + μ i Suppose that π 1 > 0 but not by very much?

69 Remember the Wald Ratio Wald ratio is the building block of more complicated IV strategies. W = E Y Z = 1 E[Y Z=0] E D Z = 1 E[D Z=0] W = ITT FS What if F 0?

70 Value of the Wald Ratio With Fixed ITT and Shrinking First Stage

71 Very Fast Background Health People 2020 report: About 25% of 18 to 64 year olds were vaccinated for influenza in The goal for 2020 is to push that number up to 80%. About 67% of adults over age 65 were vaccinated in The goal for 2020 is to get the number up to 90%.

72 Simple Question Does the general population of non-elderly adult actually benefit from increased vaccination?

73 How should we model this question?

74 Effects of Flu Vaccines How should we measure health? What do the coefficients mean? What would you hope to include in X? Why does the model include an interaction term? What are some possible threats to validity if we tried to estimate the model using OLS?

75 Think About Assumptions 1. Independence: Z is randomly assigned. 2. SUTVA: Non-interference\No Spillovers 3. Exclusion: Z has no direct effect on outcomes. 4. First Stage: Z does affect D 5. Monotonicity: Z always affects D in the same direction.

76 What should we use as an instrumental variable?

77 Candidate Instruments Dummy variable indicating whether the person works in a health care occupation. Price of flu vaccines by city and year in the US. State regulations that require insurance plans to cover flu vaccinations. Time series data on the total supply of vaccines by year. Other ideas?

78 Suppose we settle on one of the instruments. How do we proceed?

79 A Rough Plan Collect data on [Y, X, GT65, FluVac]. Collect data on Z. Often Z and [Y, X, GT65, FluVac] will come from different sources and you will need to merge two data sets together. Collecting data on Z may be a lot of work.

80 Implementation HospVisit i = X i β 0 + β 1 GT60 + β 2 FluVac i + β 3 (FluVac i GT60 i ) + ε i What is the First Stage? What is the Second Stage?

81 Stata Tips