Introduction to Causal Inference. Solutions to Problem Set 1. Required Problems

Transcription

1 Introduction to Causal Inference Solutions to Problem Set 1 Professor: Teppei Yamamoto Due Friday, July 15 (at beginning of class) Only the required problems are due on the above date. The optional problems will not directly count toward your grade, though you are encouraged to complete them as your time permits. If you choose not to work on the optional problems, use them for your self-study during the summer vacation. Required Problems Problem 1 Supposed that you have a random sample of size n from the population of interest. Answer the following questions that are designed to help you get familiar with potential outcomes. Try to keep your answers brief and your language precise. Throughout the problem, assume that the Stable Unit Treatment Value Assumption (SUTVA) holds. (a) Explain the notation Y i (0). The potential outcome for subject i if this subject were untreated. Another way to put it: the untreated potential outcome for subject i. (b) Contrast the meaning of Y i (0) with the meaning of Y i. The first is the potential outcome for subject i if this subject were untreated. The second is simply the observed outcome for subject i. (c) Contrast the meaning of Y i (0) with the meaning of Y i (1). Is it ever possible to observe both at the same time? Why? The first is the potential outcome for i under control. The second is the potential outcome for i under treatment. In any moment, only one of the two potential outcomes for i can be realized. A subject cannot be under control and treatment simultaneously, so observing both potential outcomes is not possible. This is known as the fundamental problem of causal inference. 1

2 (d) Explain the notation E[Y i (0) D i = 1], where D i is a binary variable that gives the treatment status for subject i, 1 if treated, 0 if control. The expected value of the potential outcome for subject i if the subject were untreated, given that this subject actually receives treatment. Another way to put it: the expected value of the untreated potential outcome for a subject in the treatment group. (e) Contrast the meaning of E[Y i (0)] with the meaning of E[Y i D i = 0]. The first is the expected value of the untreated potential outcome for subject i; the second is the expected value of observed outcomes for subject i, given that they were untreated. (f) Contrast the meaning of E[Y i (0) D i = 1] with the meaning of E[Y i (0) D i = 0]. The first is the expected value of the untreated potential outcome for a subject who actually receives treatment; the second is the expected value of the untreated potential outcome for a subject who is in fact untreated. (g) Which of the following quantities (that you explained in parts (d) through (f)) can be identified from observed information? Do not make any additional assumptions about the distributions of Y or D, except the SUTVA and also that there is at least one observation with D i = 1, and at least one with D i = 0 in the observed data. E[Y i (0) D i = 1] (1) E[Y i (0)] (2) E[Y i D i = 0] (3) E[Y i (0) D i = 0] (4) We can identify (3) E[Y i D i = 0] and (4) E[Y i (0) D i = 0]. Note also that under the Stable Unit Treatment Value Assumption (SUTVA), these two quantities are equal. (h) Now, assume that D i is randomly assigned to the units in this sample. Which of the above quantities can be identified from the observed data? Now we can also identify both (1) and (2) as E[Y i D i = 0]. Problem 2 This problem is for those of you who are unfamiliar with R, which we will use throughout the rest of the course in lectures and problem sets. If you are already comfortable using R, skip this problem and proceed to Problem If you haven t, download and install R and RStudio on your own computer. Make sure to choose the right versions for your operating system. 2

3 2. Complete Lesson 1 on this website. That is, you should watch each of the video tutorials (Parts 1.1 to 1.6) and complete all the swirl quizzes. (You will be able to find out how to install and use swirl by following the links on the above website.) As proof of your hard work, type the following directly in your console after you complete all the swirl quizzes: > savehistory(file = "pset1prob3.txt") and submit the file along with your problem set answers. Problem 3 This problem is designed for those of you who are already familiar with R and need a quick review of key concepts and operations. If you are new to R and just getting started with it, this problem is optional for you. 1. Using the data given below, answer the following questions in R. Major Foreign Holders of U.S. Treasury Securities Country Dec2007 Dec2006 Dec2005 Dec2004 Dec2003 Japan China, Mainland United Kingdom Oil Exporters Brazil Caribbean Banking Centers All Others Total Total Public Debt Outstanding Year Debt Held by the Public Intra-governmental Holdings Total Dec Dec Dec Dec Dec (a) How much did the debt held by the public increase from 2003 to 2007? What proportion of this new debt was bought up by foreigners? What proportion of the new debt was bought up by the Chinese? 3

4 increase < increase foreign.increase < foreign.increase / increase China.increase < China.increase / increase (b) What proportion of U.S. Treasury Securities held by foreigners in 2007 were held by the top six countries/groupings? top < top / (c) As a percentage, what is the nominal increase in holdings from 2003 to 2007 of these six countries/groupings as a group? What if we exclude Japan? top < (top top6.2003) / top * 100 top minus.japan < top minus.japan < (top minus.japan - top minus.japan) / top minus.japan * 100 (d) Create a vector of the percentage changes for each country from 2003 to Be sure to include names indicating which value corresponds to which country. top.2003 <- c(550.8, 159.0, 82.2, 42.6, 11.8, 47.3) top.2007 <- c(581.2, 477.6, 158.1, 137.9, 129.9, 116.4) percentages <- (top top.2003) / top.2003 * 100 names(percentages) <- c("japan", "China, Mainland", "United Kingdom", "Oil Exporters", "Brazil", "Caribbean Banking Centers") percentages (e) Repeat the exercises in parts (b) and (c) using the vectors you have just created. Use sum() and indices. sum(top.2007) / (sum(top.2007) - sum(top.2003)) / sum(top.2003) (sum(top.2007[-1]) - sum(top.2003[-1])) / sum(top.2003[-1]) (f) What was the ratio of debt held by the public relative to intra-governmental government holdings, for each year ? When did this ratio hit it s high point and low point? Make sure to include year labels. 4

5 public <- c(4044.2, , , , ) gov.holdings <- c(2953.7, , , , ) ratio.pub.gov <- public / gov.holdings names(ratio.pub.gov) <- 2003:2007 ratio.pub.gov 2. (a) Download the Middle East dataset available on the course website and save it to some convenient location. Change your working directory via setwd() or using the pulldown menu to the directory in which you saved the data. Read the data using the read.csv("filename.csv") command. How many observations (countries) are there? How many variables? [Don t just count off the screen. Imagine we were working with a much larger dataset.] ME <- read.csv("middleeast.csv") dim(me) Thus, the number of observations is 13 and the number of variables is 9. (b) Create a new variable in the data frame that equals GDP per capita in each country, and calculate the mean GDP per capita in the entire region. ME$GDP.pc <- ME$GDP / ME$Population mean(me$gdp.pc) # Create the GDP per capita variable. (c) What is the class of the religion variable? If it is not already a character string, coerce the plurality religion for each country to a character string. class(me$religion) ME$Religion <- as.character(me$religion) (d) You can use negative indices to remove rows or columns from objects. For example, X[-1,] will give you the matrix X less its first row. Using this trick, remove the capital and currency variables from the dataset. names(me) # Retrieve the variable names ME <- ME[,-c(5,7)] # Use negative indices # Can also do: # ME <- ME[,c(1:4,6,8:10)] # ME <- subset(me, select=-c(5,7)) ME (e) How many of these countries have an HDI value above 0.75? (Hint: The sum(x) function counts up the number of TRUEs in x if x is a logical vector.) 5

6 sum(me$hdi > 0.75) (f) How many of these countries have both a density equal to or above 35 and a population above 10 million? sum(me$density >= 35 & ME$Population > ) (g) Create a new variable within the Middle East dataset indicating whether each country has a high (above 0.75) or low HDI value. ME$hdi.hi <- ME$HDI > 0.75 ME$hdi.hi (h) Create a new dataset consisting of just the high HDI countries. ME.high.hdi <- ME[ME$hdi.hi==TRUE,] # or ME.high.hdi <- subset(me, hdi.hi==true) ME.high.hdi (i) The save.image function allows you to save your workspace into a file so you can retrieve your work later on. Use the following syntax to save your objects in your working directory: save.image(file = "nameyoulike.rdata"). (You can also do the same via RStudio s drop-down menu.) Now, close RStudio (don t forget to save your code as well!), reopen it, and load the saved workspace using the load function. save.image(file="middleeast.rdata") # close RStudio, reopen it, and type load(file="middleeast.rdata") Problem A Optional Problems You and some colleagues are conducting an intervention in Ghana s 2016 election 1. Your goal is to assess the effect of deploying a new biometric voting machine on the incidence of electoral fraud at polling stations. Though Ghana is made up of 275 constituencies, due to the political context you are only allowed to perform your experiment within one constituency. Unconcerned, you and your team randomly select eight polling stations in the constituency, and among the eight, randomly 1 This problem is loosely inspired by true events. If you are interested in the substantive or methodological issues, see Asunka et al

7 assign half to receive the new voting machines and the other to serve as a control group. D i gives the resulting treatment status for each polling station i, for i {1,..., N}, where D i {0, 1} and N = 8. The outcome of interest is the percentage of votes in a polling station attributable to fraud, Y i. a) Assume that both parts of SUTVA hold. Calculate, explaining your answer: i) For each polling station i, the number of potential outcomes that can be defined; ii) For each polling station i, the number of unit treatment effects that can be defined; and, iii) For the sample of polling stations, the number of (unconditional) average treatment effect estimands that can be defined. Throughout this problem we will see questions like those above. There is a general approach to them, which we introduce here. i) For each i we can write a potential outcome Y i (D i ). Given D i = {0, 1}, there are two potential outcomes for i, Y i (0) and Y i (1). ii) A unit (or individual) treatment effect is the difference between two potential outcomes for a given unit. With only two potential outcomes per unit, there is one unit treatment effect, Y i (1) Y i (0), for each unit. iii) Similarly, an average treatment effect is the difference between the mean outcome for units in one condition and the mean outcome for units under another condition. Because there is only one unit treatment effect defined per unit i, there is only one unconditional average treatment effect: E[Y (1) Y (0)]. (One unconditional average treatment effect, because as discussed in class, we can also define the average treatment effect among the treated units, E[Y (1) Y (0) D = 1], among the control units, E[Y (1) Y (0) D = 0], and among subsets of units conditional on values of some variable X, E[Y (1) Y (0) X = x].) b) A Ghanaian political insider gets wind of your study, and gives you some information. She says that because all of the polling stations in your study are within one small constituency, there will be interference between units. She explains how interference might occur in this case: In Ghana, the political operatives in each polling station who are responsible for committing fraud will move elsewhere if their efforts are frustrated. Their range of movement is predetermined by the geographic influence of the party bosses. Local conditions are such that the operatives in each polling station can only move to one, and only one, other polling station, as shown in Figure 1. 7

8 Figure 1: Operatives may move to only one other polling station, as indicated by the arrows (for example, from the second to third polling station, but not the reverse). Given this structure, answer again questions (i) - (iii). To fully understand the answer to this question, recall from class that an assumption often (almost always) invoked in causal inference is that observed outcomes for each unit are unaffected by the treatment status of other units. (This is one part of the the stable unit treatment value assumption, or SUTVA.) Intereference between units means that this is not true: assigning one unit to treatment or control may affect the outcome for another unit. Formally, interference can be ruled out if the following is true: Y (D1,D 2,...,D N )i = Y (D 1,D 2,...,D N )i if D i = D i, i. In the case of interference, intuitively this equation means the following. For any sample of N units, k of which are treated (and N k control), there are ( ) N k possible treatment vectors (different ways treatment could be assigned between all the units). The equality above implies that for any of these ( ) N k treatment vectors, the potential outcomes of unit i remain constant. We will use this insight to answer the question about potential outcomes. The actual solutions are as follows: i) The potential outcomes for each i now depend on the treatment status of its neighbors; we can write these potential outcomes as Y i (D i, D i+1 ). (Compare this to the previous answer.) D i is dichotomous, so the number of potential outcomes we can define for each i is now 2 2 = 4. ii) We noted earlier that an unit treatment effect is the difference between two potential outcomes for a given unit. We could define a single unit treatment effect given two 8

9 potential outcomes per unit. Now, with four potential outcomes per unit, the number of differences that we can take is an n-choose-k problem: we can define ( 4 2) = 6 unit treatment effects. iii) As before, the number of average and unit treatment effects that we can define is the same. If there are six unit treatment effects, there are also six average treatment effects. c) Your funder is keen to spend their budget, and demands that you field the intervention as described originally: N = 8, and for D i = {0, 1}, N i=1 D i = 4. You set aside your concerns, and observe the data in Table 1 on the percentage of votes in each polling station attributable to fraud. Table 1: Observed Data: Treatment and Ballot Stuffing Unit D i Y i 1 1 4% 2 1 2% 3 0 8% 4 1 9% % % 7 0 4% 8 1 1% Write out an appropriate estimator for the average treatment effect (ATE) given SUTVA. Drawing on your insights so far and your knowledge of causal inference, under what conditions will this estimator be unbiased? Apply it to the data and compute an estimate of the effect of the new biometric voting machines on the incidence of fraud. An appropriate estimator for the ATE is the difference in the expected values of observed outcomes in each condition: E[Y (1) D = 1] E[Y (0) D = 0]. (Notice how this differs from E[Y (1)] E[Y (0)], which defines the estimand.) The estimator will be unbiased so long as (1) treatment assignment is independent of potential outcomes, (2) there is no interference between units, and (3) the treatment itself (though not necessarily the effect) is stable across units. As seen in class, (1) will be satisfied under random assignment of treatment. The difference between (2) and (3) is important. (2) simply requires that, as outlined above, the potential outcomes of each unit are not affected by the treatment status of other units. (3) requires that treatment is uniform in substance and application across all observations. In a drug trial with treatment and control conditions, for instance, (3) could mean that the treated and control subjects all get exactly the same dosage or lack of dosage, respectively, for exactly the same drug. In social science applications like the one in this problem set, satisfying (3) can sometimes be challenged by implementing partners who tailor treatments to particular places as they see fit. 9

10 We compute our estimate as the difference in observed mean outcomes among treated and control units: sum(4, 2, 9, 1)/4 - sum(8, 12, 13, 4)/4 d) Given that you know the structure of the interference network, you believe you may be able to rescue something from the study. Using Figure 1, translate Table 1 into a table formatted like Table 2, where the latter columns give Y i (D i, D i 1 ) for the combinations of possible values for D i and D i 1. (And for i = 1, Y 1 (D 1, D 8 ), per Figure 1). Where you can, fill the cells of your table with values from Table 1; leave blank the cells representing unobserved potential outcomes. Table 2: Observed and Unobserved Potential Outcomes Unit D i D i 1 Y i (1, 1) Y i (1, 0) Y i (0, 1) Y i (0, 0) 1. 8 Table 3: Solution: Observed and Unobserved Potential Outcomes Unit D i D i 1 Y i (1, 1) Y i (1, 0) Y i (0, 1) Y i (0, 0) % % % % % % % % e) Formally express each of the estimands described below using potential outcomes; propose appropriate estimators for each; and finally estimate them with the help of your new table. i) The ATE, conditional on a neighbor taking treatment. ii) The ATE, conditional on a neighbor taking control. iii) The magnitude of effect modification due to assignment of treatment to a neighboring unit. i) Estimand: E[Y (1, 1) Y (0, 1)] Estimator: E[Y (1, 1) D i = 1, D i 1 = 1] E[Y (0, 1) D i = 0, D i 1 = 1] Estimate: 3% - 10% = -7% 10

11 ii) Estimand: E[Y (1, 0) Y (0, 0)] Estimator: E[Y (1, 0) D i = 1, D i 1 = 0] E[Y (0, 0) D i = 0, D i 1 = 0] Estimate: 5% - 8.5% = -3.5% iii) Estimand: E[Y (1, 1) Y (0, 1)] E[Y (1, 0) Y (0, 0)] [ Estimator: E[Y (1, 1) D i = 1, D i 1 = 1] E[Y (0, 1) D i = 0, D i 1 = 1] [E[Y (1, 0) D i = ] 1, D i 1 = 0] E[Y (0, 0) D i = 0, D i 1 = 0] Estimate: -7% + 3.5% = -3.5% Note: Key to the estimands being ATEs (and not, say ATTs or ATCs) is that our experimental design actually controls interference. That is, because we see the entire network of connections, and randomly allocate treatment/control to every unit in the network, we know that not just the treatment but also the presence of a treated neighbor is random. f) Having heard about your complicated experimental design, a colleague approaches you for advice about a potential field experiment. Their experiment, which randomizes the provision of information about the quality of schooling, will be conducted in a dense urban area. Treatment is to be assigned at the household level. After questioning your colleague extensively, you ascertain that the structure of the urban area can be described by a ring lattice network with the number of households given by N, and the interconnectedness of the households given by degree d: Figure 2: Households and connections between households in the village. This example has N = 6, where N represents the nodes or units, and d = 4, where d represents the degrees of the network, or how many connections each node/unit has to other nodes/units Figure 2 implies that any household in the village is connected to the four (d = 4) nearest households, so that interference can be ruled out between households that are three or more lots apart. We can rule out interference between any households not immediately connected by an edge. Given this setup, and given N > d, answer the following: i) For each household j, give the number of potential outcomes that can be defined. ii) For each household j, give the number of unit treatment effects that can be defined. 11

12 iii) Imagine now that the experimental intervention is no longer binary, but ternary. It comprises two distinct treatments and a control. In this scenario, how many potential outcomes are there for each j? And how many unit treatment effects? iv) Assume that as N increases the structure of the households continues to follow an expanding ring lattice as shown in figure 2, where the number of connections is still defined by d, for any d < N. In terms of k, the number of levels of treatment (e.g., k = 2 if treatment is binary), and d, the number of degrees in the ring lattice network, write down a general expression for M k,d, the number of unique unit treatment effects for any given unit j. An increase in which parameter, k or d, has worse implications for the tractability of a study? i) The potential outcomes for unit j are given by Y j (D j, D i 2, D i 1, D j+1, D j+2 ), which gives us 2 5 = 32 potential outcomes for each unit. ii) As before, we can calculate the number of UTEs as an n-choose-k problem: ( ) 32 2 = 496 iii) Potential outcomes for each unit j are now given by 3 5 = 243. There are now ( ) = unique UTEs for each unit j. iv) In this setting, a general expression for the number of unique UTEs for each unit j is M k,d = ( ) k d+1 2. Clearly d is the problematic parameter: increasing d increases the number of unique UTEs exponentially, while increasing k does not. Problem B This problem will introduce you to directed acyclic graphs (DAGs), and reasoning about them. Consider the following two DAGs, in which X, Y, and Z are observed, and U a and U b are hypothesized but unobserved. U a U b U a U b X D Z X Y Z X Y Figure 3: Causal Graph A Figure 4: Causal Graph B a) List all of the nodes in Graph A. b) List all of the edges in Graph A. c) List each path terminating at Y that exists in Graph A, and repeat for Graph B. Interpret the differences between graphs. 12

13 d) Explain the relationship between X D and X in Graph B. e) Is the relationship between X and Y identified in Graph B? Why? f) Assuming D i {0, 1}, write down the ATE of X on Y for Causal Graph B using the notation introduced in class. a) X, Y, Z, U A, U B. b) X Y, X Z, X U B, Y U B, Z U A. You could represent these pairs in any order, that is, switch the first and second nodes while also flipping the arrow. You simply need to preserve the directionality of the relationship. c) In Graph A, the paths that terminate at Y are: Z X Y, Z X U b Y, X Y, X U b Y, U b X Y, U a Z X Y, U b Y, and U a Z X U b Y. By contrast, in Graph B they are: X Y and U b Y. In Graph B, the edges connecting Z and X and U b and X have been severed by the do(x D ) operator, which exogenously sets X to particular values of D. d) X has been exogenously set to particular values by the do(x D ) operator. That is, there are values of X that can be exogenously set by assigning D to some value. In graph B this has been applied. e) In Graph B the relationship between X and Y is identified. This results from the use of the do(x D ) operator, which means that there is no unobserved variable that correlates with both X and Y. f) AT E = E[Y do(x 1 )] E[Y do(x 0 )] 13