Quantitative Analysis II PUBPOL 528C Spring 2017

Transcription

1 Quantitative Analysis II PUBPOL 528C Spring 201 Instructor: Brian Dillon Meeting time: W 5:30-8:20 bdillon2@uw.edu Class location: Parrington 108 Phone: TA: Austin Sell Office: Parrington 209G TA arsell@uw.edu Sections (Par 106): CA: Tuesday :30-5:20 CB: Friday 12:30-1:20 Instructor office hours: TA office hours (Par 12E): T 3:30-:30 and Th :00-5:00 (sign up on Doodle), and by appointment. Also 1 electronic OH I will explain in class. M :00-6:00 and W :30-5:20, and by appointment Textbook: A.H. Studenmund, Using Econometrics: A Practical Guide, 6 th Ed. (not the th ) Website: Course Objectives The goals of this course are to deepen your understanding of regression analysis and statistical modeling, and to develop your skills in applying these techniques to public policy and management issues. We will focus on choosing the right statistical framework for a particular question, estimating the relationship between multiple factors and an outcome of interest, and determining when and why statistical estimates can be interpreted as causal. Real world data will be used in most applications. Your aim should be to develop an understanding of both the underlying statistical theory and the practical applications of the course material. For better or worse, in recent years the public discourse in many policy arenas has become increasingly interested in evidence-based policy design and quantitative analysis. A mastery of basic econometrics and a firm understanding of how to apply these ideas to real problems are essential for your forward progress, both in the MPA program and in your careers to follow. Software We will use Stata for all problem sets and data assignments in this course. I will not presume that you have any prior experience with the program. If you would like to buy a copy of Stata for your computer, a 6-month license costs $5. Details are here (be sure to buy Stata IC, not small Stata ): Stata is available in the Parrington Hall computer lab. You can also access Stata remotely through the Evans School Terminal Server or through the UW Center for Studies in Demography and Ecology (CSDE). Instructions for remotely accessing Stata will be posted before the course begins. Excel might be useful for some of the assignments and for data manipulation. Prerequisites 1

2 This course is only open to students who have successfully completed PBAF 52. Substitute prerequisites from students outside of the Evans School will be considered on a case-by-case basis. Reading The schedule below gives an approximation of the reading schedule for the course. As the term progresses, I will give more specific guidance on exactly which parts of which chapters are relevant for each week. While the material in the lectures, quiz sections and problem sets is your best guide to what will be on the quizzes and exams, all of the material in the assigned chapters is fair game on any assessment. I will provide supplemental readings as the course progresses. These will be posted to the course website. Grading and Assignments Your grade will be based on 3 problem sets, 3 quizzes, a data analysis assignment, a written final exam during finals week, and completion of the pre-class questionnaire. Due dates are posted on the timeline below. Late work will receive a score of zero. Problem sets will be posted roughly one week before they are due. Group work is allowed and encouraged. However, working through the problem sets on your own is essential for doing well in this course. Even if you work with others, you must generate your own answers to submit. Each problem set is worth 10% of your final grade. Additional, non-graded problem sets may be provided. The 3 quizzes will be roughly similar to each other, although I cannot guarantee that they will all have the same number of questions or be of equal difficulty. Quizzes will be given in lecture. Each quiz is worth 10% of your final grade. Make-up quizzes will not be offered, but I will drop your lowest score from the 6 problems sets and quizzes, so missing one quiz does not have to affect your grade. Your final grade in this course will be based on the following: Pre-class Questionnaire (due by Tuesday, March 28, 11:59pm) 5% Problem sets and Quizzes (6 x 10%, drop the lowest) 50% Data Analysis Exam (take home, due May 30 at 11:59pm) 20% Final Exam (June 6, 5:30-:20, Par 108) 25% I will not curve individual quizzes, problem sets, or exams, but I will curve your final scores if necessary. My goal will be to ensure that the distribution of grades in the course is roughly similar to the recent historical distribution of grades in PBAF 528. Academic Integrity UW and the Evans School expect students to adhere to the highest standards of academic integrity and honesty. A student found to be cheating on a quiz or exam will receive a zero for that test. A second offense will lead to a zero for the course. Enrollment, Attendance, Absences Check the University Calendar for the policy on incompletes and withdrawals. We will adhere to the university dates and policies. If you are going to miss a class, talk to a classmate beforehand and arrange to get a copy of her/his notes. Office hours are not intended as a time to repeat material because of a 2

3 class absence. If you have a scheduling conflict for the final exam, you must contact me prior to the exam. Students who fail to do so will be given a zero for the exam and will forfeit the right to a makeup. If you need to leave class early, please tell me before class and choose a seat near the exit. Finally, when we have brief stretch breaks during class, please don t leave the room. Special Accommodations If you have an arrangement with UW DRS for exam or quiz accommodations, please me after the first class so that we can set up a meeting and discuss the best way to proceed. Communication I want you to succeed in this course so we will be as available as possible to answer your questions and support your progress. That said, here are a few rules to help us organize communication: i. The best ways to contact me are in office hours, before/after class, or over . ii. Use the discussion board for STATA or OFFICE HOUR questions. I will explain this in class. iii. If you me, I will get back to you within 8 hours. Except s sent on Friday, which might not be answered until Monday. iv. I only answer s that contain a greeting that includes my name and/or title, and a signature that includes your name. Course Schedule All dates other than the quizzes, final, and data analysis exams are subject to revision. Weekly reading assignments should be completed prior to the lecture, in case we move more quickly than expected. See below for more guidance on reading. For the first three weeks, quiz sections will be in the Language Learning Center, Denny Hall, room 15 (Tuesday) or 156 (Friday). Week # Class Dates Important Events Rough guide to reading 1 3/29 Sections in computer lab. Tuesday: Denny Hall 15; Friday: Ch. 1-2 Denny Hall /5 Sections in computer lab. Tuesday: Denny Hall 15; Friday: Ch. 3-5 Denny Hall 156; PS 1 due Sunday, April 9 at 11:59pm 3 /12 Sections in computer lab. Tuesday: Denny Hall 15; Friday: Ch. 6 Denny Hall 156; Quiz 1 in lecture /19 Ch. 5 /26 PS 2 due Sunday, April 30 at 11:59pm 6 5/3 Quiz 2 in lecture Ch /10 8 5/1 PS 3 due Sunday, May 21 at 11:59 pm Ch /2 Quiz 3 in lecture Ch /31 Data Analysis Exam due Tuesday, May 30 at 11:59 pm Ch. 11 Finals Final exam in Par 108 on Tuesday, June 6, 5:30-:20 3

4 Topic List In the matrix below I have listed most of the topics that we will cover this term. I will likely add to this, or choose not to cover some of these points. Before each lecture I will post an announcement listing the topic numbers that I expect to cover that week. You will notice that the location in the book of some content will not always line up with the reading schedule in the previous table (hence the above table is just a rough guide ). Also, a few concepts are given only brief or partial coverage in the book. If you look in the book for details on a topic and cannot find them, you can assume that I will provide the details in class or will give additional readings. I have also given an indication in the table of which content you will be expected to calculate or work out by hand ( do the math ) and which content you will need to work with in Stata (by writing code, interpreting Stata output, or both). All of that is subject to change.

5 # Concept Main analytical framework Location in book You are expected to Understand/ explain/ interpret 1 Central objective: associate the variation in some outcome of interest the dependent variable, Y with the 1 variation in some other variable or variables (the independent variables, X). 2 For example, X might be a variable indicating participation in a program, and Y is the outcome that the program 1 is supposed to impact. 3 The variance of a variable is a measure of its dispersion around the mean. The covariance and the correlation of 2.2, variables X and Y measure the extent to which they tend to move together around their respective means. 1.1 We never observe the real process that generates the data. 1 5 We can write down a statistical model that relates the dependent variable Y to the independent variables, the Xs. One example of such a model is: 1 (1) Y i = α + β 1 X 1i + β 2 X 2i + + β K X Ki + ε i In this model, α is the intercept coefficient, β 1 β K are the slope coefficients, and ε is a statistical error term that accounts for all of the variation in Y that is not explained by the Xs. The i subscript refers to a single observation. If we have N observations, then i=1,,n. Do the math Use in Stata 6 This is a statistical model, rather than a deterministic model, because we do not know the values of the coefficients with certainty. That is, we never observe reality, and we also never observe the exact coefficients of our model. Instead, we estimate two objects for each coefficient/parameter in the above model. In the case of X 1, we estimate the coefficient, β 1, which is our best guess at the value of the true coefficient, and the estimated standard error, SE(β 1), is a measure of how confident we are in that guess. We will define best later. 8 Because we generally only observe a sample, not the entire population of interest, slight differences in the sample composition can lead to differences in β 1 and SE(β 1) (even if we were to repeat the estimation on randomly generated samples). The smaller are the samples, the more likely it is that the estimated coefficients will be different. The distribution of coefficient estimates for different samples of the same size is called the sampling distribution. The estimated coefficient β 1 is the mean of the sampling distribution, and the estimated standard error, SE(β 1), is the standard deviation of the sampling distribution. 1 1,.2.2 5

6 Once we have estimated the coefficients, the predicted value of the outcome variable for each observation i is given by Y i = α + β 1X 1i + β 2X 2i + + β KX Ki 1.3, 1. 9 The residual, e i, is the difference between the predicted value and the actual value: e i = Y i Y i 1.3 Ordinary Least Squares (OLS) 10 There are infinite ways to (i) Model the relationships between Y and the Xs, and (ii) Choose values for the 1 coefficients, once we have decided on a specification for the model (by specification we mean equation ) 11 Of all possible ways to model the relationships between Y and the Xs, we are focused on those that are linear in.1 the coefficients Of all possible ways to choose the values of the coefficients, OLS turns out to be the best way to estimate the 2, model (i.e., estimate the coefficients) under certain circumstances (see the Gauss-Markov Theorem). 12 N 2 OLS chooses the values of α and β 1 β K that minimize the sum of squared residuals (RSS): RSS = i=1 e i By minimizing the sum of squared residuals, rather than the sum of residuals, OLS (1) penalizes larger residuals 2.1 more than smaller residuals, and (2) avoids having positive and negative residuals cancel each other out 1 In the bivariate case, with only one X variable, the OLS estimate of the slope coefficient is β 1 = cov(x,y) var(x) 2.1 cov(x, Y) = N i=1 (X i X)(Y i Y) and var(x) = i=1 (X i X) 2. Note that it would also be Ok to write the covariance N 1 N 1 and the variance with N in the denominator the difference depends on a minor point that is beyond our scope. The intuitive interpretation for an OLS coefficient is that it is a ratio of a covariance to a variance a measure of how much X and Y move together, normalized by how much X is just varying around on its own. 15 In the bivariate case, the OLS estimate of the intercept coefficient is α = Y β 1X. By choosing the intercept this 2.1 way, we ensure that the mean of the residuals is zero. 16 In the multivariate case, the formulas are more complicated, because they account for the relationships between 2.2 each X and Y, but also take into account the correlation between the Xs. 1 The interpretation of an OLS coefficient from a multivariate regression is A 1-unit increase in X k is associated 2.2 with a β K increase in Y, controlling for [list the other explanatory variables] 18 To estimate a model we need to find two objects the coefficients, and the standard errors. These should 1,.2 always be thought of together. In a statistical model, uncertainty is a key part of the modeling process. The coefficient estimate is essentially useless if it is not accompanied by a measure of how confident we are about the estimate (a standard error). 6

7 Testing 19 Under most circumstances, the coefficients estimated by OLS follow the Student s t distribution with N-k-1 degrees of freedom. For large N, the t distribution is essentially the normal distribution. 20 A two-tailed test of hypothesis H 0 : β k = S has the test statistic t = (β k S) SE(β k), which we compare to a table of critical values for some level of confidence α/2 with degrees of freedom N-k-1. We construct two-sided confidence intervals for β k as [β k ± SE(β k)tα 2,N k 1] 21 We might be interested in other tests based on the estimated coefficient and standard error. If the test of interest is 1-sided (e.g., we want to specifically test whether a program made people worse off), we run a 1-tailed test: a. The hypothesis can never be rejected (at conventional levels of significance) if the sign of the coefficient is the same as that under the null hypothesis. E.g., if β k is positive, we can never reject H 0 : β k 0 b. If the sign of the coefficient is the opposite of that under the null hypothesis, then the test statistic is the same as for a 2-tailed test, but the rejection region is larger (it is determined by α rather than α/2). 22 An F-test is a general approach to testing whether multiple hypotheses are true simultaneously. The standard form of the test: a. Ignore the null hypothesis and estimate the model. This is the unrestricted model. Retain the RSS. b. Impose the restrictions and re-estimated the model. This is the restricted model. Retain the RSS. c. Form the test statistic and compare to a table of F-distribution critical values with q degrees of freedom in the numerator and N-k-1 degrees of freedom in the denominator, where q is the number of constraints (restrictions). How good is the model? 23 The RSS is one part of the decomposition of the variance of Y. The other part is the explained sum of squares, or N N ESS, which is defined as ESS = i=1(y i Y). ESS + RSS = TSS, where TSS = i=1(y i Y). Note that TSS is like the variance of Y, except that it is not divided by (N-1). 2 Because the goal of this modeling exercise is to explain the variation in Y, the TSS is a measure of how much variation there actually is to explain. The more that the Y i are spread around the mean of Y i.e., the more that they vary the higher is the TSS. 25 R 2 = ESS/TSS gives the proportion of the variation in Y that is explained by the model. R 2 always lies between 0 and 1. That is, the model can never explain more of the variation than there is to explain. R 2 is fine as a rough measure of how much of the variation in Y we are explaining, for this specific sample. It is not a very useful tool

8 for determining how good the model is, because adding meaningless variables to the model can increase R 2, but can never decrease it. So a high R 2 is not necessarily evidence of a good model. 26 Adjusted R 2 corrects for that final problem by incorporating a penalty for every variable that is added to the 2. model. Adding an explanatory variable to the model can decrease adjusted R 2, if the explanatory power of the new variable is not sufficient to offset the penalty for adding a term. Adjusted R 2 can be negative. 2 An F-test for overall significance is a standard, theoretically-grounded way to evaluate the goodness-of-fit 5.6 Explanatory variables / Alternative specifications 28 A categorical variable is a variable that assigns each observation to one of a list of possible categories using numerical codes (e.g., 1=US citizen, 2=Permanent resident, 3=Visa holder, =Other) 29 A categorical variable cannot be entered directly in a model, because the numerical categories do not have any real meaning. If we used a different numbering scheme which would not change the category data in any meaningful way we would get different OLS results. Clearly not ideal. 30 Instead, to account for between-group differences, we construct separate dummy variables for each group, where the dummy variable takes a value of 1 if the observation is a member of the group, and 0 otherwise. When we include dummy variables, one must always be excluded. That is the reference group or the excluded group against which the others are compared 31 For example, if we want to model the outcome Y as a function of the residency status categorical variable from part a, we could build separate dummy variables for each category and estimate the following: (2) Y i = α + β 1 CITIZEN i + β 2 PERM i + β 3 VISA i + ε i Then the predicted values are: Y i = α + β 1(1) + β 2(0) + β 3(0) = α + β 1 for an i with CITIZEN i=1 Y i = α + β 1(0) + β 2(1) + β 3(0) = α + β 2 for an i with PERM i=1 Y i = α + β 1(0) + β 2(0) + β 3(1) = α + β 3 for an i with VISA i=1 Y i = α + β 1(0) + β 2(0) + β 3(0) = α for an i with OTHER i=1 32 In the above case, each subgroup has its own intercept. If there were additional continuous variables in the model, without any additional interactions, then the slope coefficients would be the same for all subgroups. Only the intercepts are different, in this case. 33 We can construct interactions between dummy variables to allow more specific subgroups to have their own intercepts. For example, we could add gender dummy variables to model (2), and then interact the gender dummy variables with the residency variables if we believe that the relationship between residency status and Y might differ across genders. 8

9 3 We can also add interaction terms between dummy variables and continuous variables, to allow each subgroup to have its own slope coefficient. In that case, we always include in the model the dummy variable, the continuous variable, and the interaction (never include the interaction without including each interacted variable on its own). 35 If some of the variables in our data are nested e.g., we have data on kids in schools, and every child in school A is in county B, and every child in county B is in State C, etc. then we can only include dummy variables for one level of subgroup effects (also called group effects, or [GROUP] fixed effects, or controls for [GROUP] ). The lower the level, the more we control for unobserved differences between groups. But the lower we go, the more variables we are including in the model, which reduces statistical power and tends to increase standard errors. 36 We can use OLS as long as we stick to models that are linear in the coefficients. A model can be linear in the coefficients but still allow for non-linear relationships between Y and the X variables. 3 Ways to model non-linear relationships between Y and X: i. Include higher-order X terms, such as X 2, X 3, etc., as explanatory variables. This is useful if we think that the marginal association between X and Y is different at different values of X. ii. Use log transformations, such as Semi-log: log Y i = α + β 1 X 1i + ε i Log-log: log Y i = α + β 1 log X 1i + ε i 38 Logged variables should be interpreted in percentage terms. The estimated coefficient from the semi-log specification gives the percentage increase in Y associated with a 1-unit increase in X. 39 The estimated coefficient from the log-log specification gives the percentage increase in Y associated with a 1 percent increase in X (i.e., the elasticity of Y with respect to X). The Gauss-Markov Theorem and the classical assumptions 0 The Gauss-Markov Theorem states that among all possible ways to estimate a model, OLS is the Best, Linear, Unbiased, Estimator (OLS is BLUE) when the classical assumptions are true. 1 Best = minimum variance, where variance refers to the variance of the regression. You can think of Best as Smallest standard errors, without introducing bias 2 Unbiased: refers to the estimated β coefficients. An estimator (or estimation method) is unbiased if it is correct on average. That is, if we could draw many different samples and estimate the coefficients for each sample, on average they would be equal to the true values of the coefficients. Note that you cannot know this for a specific empirical example, because you never observe the true model. Instead, statisticians have worked out through theory and simulations that if the classical assumptions hold, OLS will be unbiased. 9

10 3 Linear: linear in the coefficients. Other ways to modeling Y and X might have lower standard errors than OLS and be unbiased, but they would have to be non-linear in the coefficients, which goes beyond our scope. OLS is BLUE when the 6 classical assumptions hold. Because they are assumptions, they are never fully testable. But it is possible to run some tests that give an indication of whether the classical assumptions hold. 5 The assumptions that we will not spent a lot of time on: a. Linear, correctly specified, additive error. We only use linear models with additive errors. Whether the model is correctly specified is a slightly vague term, because it can refer to whether we have modeled the relationships in the right way, e.g. by using logs or higher order powers of X when appropriate, and it can also refer to whether there are important omitted variables. For us, the latter issue is more of a classical assumption 3 issue. But you might see people referring to the issue of possible omitted variables as a specification problem. b. Error is mean zero. Because of how we estimate the intercept, this is true by definition in OLS. c. No perfect multicollinearity. We will talk about this briefly. None of the X variables can be an exact linear function of the others. This is why we must always exclude one dummy variable. It can also be problematic to interpret coefficients if we include many, highly collinear variables in the model. Violations of classical assumption 3 6 Classical assumption 3 states that the X variables cannot be correlated with the error term. When this is violated, it is a case of omitted variable bias. A more specific type of omitted variable bias is selection bias, in which some units in the data are selecting into a situation that changes both X and Y. For example, if X is a dummy variable for participating in a program, and program eligibility requires attending sessions 3 weekdays in a row at 2pm, then only people who are unemployed or can take off of work to attend the sessions will enroll in the program. These people are selecting into participation, and they might have different outcomes from nonparticipants for reasons not caused by the program itself. Something unobserved about these people could be affecting both X and Y. But because we don t know what that is and it is not in the model, it introduces bias into the estimate coefficient on the X variable (and possibly on the other coefficient estimates, too). Selection bias and other forms of omitted variable bias are some of the main reasons that we cannot generally view our estimates as causal estimates. If X and Y are varying together because of other factors, we don t know what proportion of their co-movement is due those other factors, and what is due to X itself. 8 There is surely always a little bit of correlation between X and epsilon. But the more there is, the more likely it is that the coefficients are biased. 6.1, , 16.1, 1.2,

11 9 Recall that the problem is not that there are important omitted variables. There are always important variables that are not in the model. Bias is a problem when there are important omitted variables and those variables are correlated with explanatory variables in the model. 50 Technically, violations of c.a. 3 can affect both the coefficients and the standard errors. When discussing this issue, we usually focus on the fact that the coefficients are biased, but the standard errors are wrong too. 51 We can never fully test for violations of this assumption. However, if we have data on some additional variables that are not in the model but that could be inducing omitted variable bias, we can try including them and seeing what happens. This can happen via eyeball include those other variables and see how much the coefficients change or more formally via an F test (including the extra variables = unrestricted model; dropping them = restricted model). 52 One possible example of the above is subgroup effects including state, or county, or city dummy variables to pick up some of the unobserved variation between groups. When we do that, we are often not too concerned with the coefficients on the subgroup fixed effects. We include the subgroup effects to control for unobserved variation that would otherwise be in the error term, and this reduces the chance that the coefficients of interest, on other X variables, are biased 53 Sometimes there are omitted variables that are considered so critical, researchers will do follow-up studies to measure those variables and include them in the model (e.g. a follow-up phone call) 5 If we have panel data repeated observations at the level of analysis then we can estimated fixed effects regressions by modeling the changes in Y as a function of the changes in X, or by including dummy variables at the individual level. Using panel data without the individual fixed effects is called a Pooled model. In a pooled model we are effectively ignoring the panel structure. Child 1 in year 1 is treated as a different person from Child 1 in year 2, etc. 55 The main uses of panel data: a. We can include time period dummy variables in a pooled model, to control for average differences across periods. b. If we have panel data, it is usually best to include the fixed effects in the model. However, if the pooled and FE models give very similar estimates, or if an F-test shows that we cannot reject the possibility that the individual FE are jointly not different from zero, we might choose to leave the individual FE out of the model in order to improve the precision of the other estimates. 56 Another way to mitigate or eliminate omitted variable bias is to run an RCT or find a natural experiment. If individuals are assigned to different values of X completely randomly, then we know that any association between Y and X must come from X itself, not from some omitted factors. See below ,

12 Violations of classical assumptions and/or 5 5 The general term for the problem of a non-constant variance of ε is heteroskedasticity or heteroskedastic errors. 10 When there is no violation of this assumption, we say that the errors are homoskedastic. 58 OLS assumes homoskedasticity when it constructs the standard errors. So if the assumption is wrong, so are the 10 standard errors. Usually, but not always, the standard errors are biased downwards (too small). That leads to inflated t-statistics and an unjustifiably high probability of rejecting the null hypothesis in a t-test 59 Heteroskedasticity does not bias the estimated coefficients To detect heteroskedasticity: White s test To correct for general (unspecific) forms of heteroskedasticity: use robust standard errors n/a 62 If we have reason to believe that the variance of ε is different for members of certain subgroups, or that the errors n/a for members of a subgroup might be correlated, then we have a second possible violation of the classical assumptions. The fix for this: cluster the standard errors. This is only an option if we have a theory about the subgroups within which the errors might be correlated or within which the variance of ε might be constant. If we have multiple options for clustering, the higher level (e.g. state instead of county) is more cautious. But theory should be the guide clustering at a very high level just to be cautious is not advisable, because it can inflate the standard errors to correct for a problem that does not exist. However, if you are unsure of the appropriate level, cluster at a higher level, to be safe. Moving from associations to causation 63 The selection problem 16 6 Randomization solves the selection problem (often implementd via randomized, controlled trials, or RCTs) There are still challenges to interpretation of RCT results: How representative is the experimental sample for the population as a whole? 2. How successful was the experimenter at inducing compliance? 3. Could there be spillovers or interactions between the Treatment and Control groups?. Will outcomes change if a small program is implemented at larger scale? 5. Can we properly identify the causal mechanism? 66 Other approaches to causal modeling (a preview of PUBPOL 529): 1. Natural experiments 2. Instrumental variables 3. Matching 1.3, 16 12