Linear Regression & Correlation

Transcription

1 Linear Regression & Correlation Jamie Monogan University of Georgia Introduction to Data Analysis Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

2 Objectives By the end of these meetings, participants should be able to: Calculate and interpret the intercept and slope of a linear regression model with one independent variable. Draw statistical inferences from regression coefficients. Calculate and interpret Pearson s r and r 2. Explain the key assumptions behind a regression model. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

3 Simple Regression Imagine that we have a relationship between two interval or ratio level variables. For instance, imagine the relationship between a county s average educational level and its turnout in a presidential election. Ideally simple regression would only be done on truly continuous variables (i.e. variables that are extremely precise and can take any real value). In practice almost no social science variables meet that requirement. In practice, relationships are rarely perfect (board illustration). Even if imperfect, from a relationship like this, we could compute something referred to as a regression coefficient. A regression coefficient is the amount of change in the value of the dependent variable based on a one unit change in the independent variable. In effect what we are doing is computing the slope of the line representing the relationship between the values of two variables. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

4 Working Example: Education and Turnout in Texas Counties turnout<-c(33.9,35.3,36.2, 38.9,44,32.6,36.6,35.6,37.1, 35.4,39.9,35.9) highschool<-c(75,81.3,91.8, 86.7,75.5,89.4,77.8,76.9, 77.6,74.5,80.5,76.1) plot(y=turnout,x=highschool, xlab="percent High School Educated", ylab="percent Voter Turnout") Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

5 Simple Regression Since, in practice, data are never a straight linear relationship, regression lines are computed most commonly by method known as least squares. (Today s focus: Ordinary Least Squares, OLS.) This procedure in a two variable model with a small number of cases is relatively simple and easy to compute. This method seeks to minimize the mean squared error (not the error on any one specific case). y i = α + βx i + ɛ i is the basic formula for a population regression equation, where: α=population intercept (expected outcome for zero value on the independent variable, also commonly written as β 0 ) β=population slope (also commonly written as β 1 ) x i =value of the independent variable ɛ i =disturbance term y i =value of the dependent variable The part of that equation that we are most interested in is the slope. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

6 Sample Regression Equation The sample regression equation is commonly written in one of two ways. One way is to place hats over quantities we estimate: y i = ˆα + ˆβx i + u i, where: ˆα=sample intercept ˆβ=sample slope x i =value of the independent variable u i =error term y i =value of the dependent variable Another way is to use the Roman equivalents of the Greek letters: y i = a + bx i + e i, where: a=sample intercept b=sample slope x i =value of the independent variable e i =error term y i =value of the dependent variable Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

7 Simple Regression The sample regression coefficient, or slope (b), can be computed in a number of ways, ordinary least squares (OLS) being just one technique among many. In simple regression the formula to compute b is: b = (xi x)(y i ȳ) (xi x) 2 In other words, the slope is computed by taking the difference of each value of the dependent variable from the mean of the dependent variable multiplied by the difference of each value of independent variable from the mean of the independent variable. Then a sum of all those products, divided by the sum of squared deviations of the independent variable. Intuition: how much do the two variables go together (covariance) divided by how much does the independent variable vary (variance). Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

8 An Example: Calculating the Slope Coefficient County % Turnout (y) % High School (x) (y 36.8) (x 80.3) (x 80.3) 2 Dallas Tarran Collin Rockwell Delta Denton Ellis Hunt Johnson Kaufman Parker Wise Average Sum So we determine b = 36.6/384.5 =.095 Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

9 An Example: Calculating the Intercept From this point we can also compute the sample intercept as well. a = ȳ b x, where: a=sample intercept ȳ =mean of the dependent variable x=mean of the independent variable From the previous slide, we can see that 36.8-(-.095*80.3)=44.4 So the sample intercept is 44.4 In R, we can estimate a and b in one fell swoop: tx.mod<-lm(turnout~highschool) We then can view our results by typing: summary(tx.mod) Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

10 Mean Squared Error The formula for mean squared error is: se 2 = SSE n 2 = (yi ŷ i ) 2 n 2 Where: n=number of Cases (y i ŷ i ) is the difference between the real value of the dependent variable and the expected value based on the predictor. In effect the formula for the mean squared error tells us how much better we can accurately predict y by using the predicting variable than if we just guessed the mean every time. If we had just guessed the mean the formula would be: sy 2 (yi ȳ) 2 = n 1 Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

11 Statistical Inference We will use the statistics: t 0 = a α SE(a) and t 1 = b β SE(b) which are distributed Student s-t with n 2 degrees of freedom. Where: n n SSE = ei 2 = (y i (a + bx i )) 2, and s 2 = SSE/(n 2) so that: i=1 i=1 SE(a) = s e ( n i=1 x 2 i i (x i x) 2 ) 1 2 SE(b) = s e 1 ( i (x i x) 2 ) 1 2 Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

12 Statistical Inference What we re testing (software default): against the two-sided alternative: H 0 :α = 0, H 0 :β = 0. H A :α 0, H A :β 0. Is zero actually the null hypothesis you want? (Often it will be.) Do you really want a two-tailed test? (Often you won t.) Perhaps confidence intervals are a better choice! The basic form of confidence intervals: a ± t SE(a) b ± t SE(b) Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

13 Correlation Cofficient: Pearson s r With these data we could also create a measure referred to as the correlation coefficient. This value shows how well the independent variable predicts the dependent variable. This measure will range between -1 and 1. A correlation coefficient of 0 would suggest the absence of any relationship between the two variables. A value of 1 (such as in our example) would imply a perfect positive relationship. A value of -1 would suggest that additional years of education actually uniformly reduces turnout. The correlation coefficient for a linear relationship is known as a Pearson s r. The square of a Pearson s r (r 2 ) calculates the amount of variance explained by the predictor. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

14 A Side Note on r 2 Generally speaking it is considered best in the social sciences to have the largest r 2 possible (it is bounded at 1). However, as the number approaches 1, concerns about the data emerge. There are no perfect relationships in the social sciences and so r 2 over.90 or so are immediately suspect. (Unless we re in the time series world.) Generally, it is best to compare your r 2 against scholars doing similar work. For example, macro research regularly finds r 2 values of.75 or greater while political psychology research rarely reaches.25. Good research asks good questions. So finding a high r 2 on a question that has been decided in the literature is less important than research with a low r 2 on an interesting question or a brand new idea. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

15 Correlation How can we compute Pearson s r? r = N XY X Y [N X 2 ( X ) 2 ][N Y 2 ( Y ) 2 ] Obviously, this measure would be very complicated to compute for any model with multiple predictors or a large number of cases. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

16 Another plan for calculating r 2 Hint: This is going to become our favorite technique! With these two formulas now we can compute the proportional reduction in error (r 2 ). r 2 = s2 y s 2 e s 2 y This is the square of the linear coefficient or Pearson s r. You can test this in R by typing: cor(turnout,highschool) Take the square and you will see the r 2 value. r =.185 r 2 =.034 Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

17 Class Example Let s run our own linear regression model using data on 45 occupations in Does the percentage of males in an occupation who are high-school graduates influence the percentage of people who rate an occupation as prestigious? #Load library containing data: library(car) #Load data set: data(duncan) #Run linear model: model<-lm(prestige~education, data=duncan) #Summarize results: summary(model) #Correlation coefficient: cor(duncan$prestige, Duncan$education) #Alternate means of obtaining r^2 cor(duncan$prestige, Duncan$education)^2 Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

18 Interpret results from your example What do the results tell us about the relationship between education and occupational prestige? How well does education predict prestige? How confident are we of this result? Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

19 Residuals Residuals reflect the part of the dependent variable not predicted by the independent variable. Examination of the residuals can sometimes help a researcher identify predictors in the model that are missing. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

20 Regression Interpretation Recall that least squares regression fits relationships as a linear function. It is possible that the relationship you are seeking to explain is non-linear. It is generally worthwhile to graph the relationship between your chief predictor and your dependent variable to see if the relationship is linear. It is also wise to start by looking at how strongly your predictor and your dependent variable are correlated. With my initial example, it would have been clear that the relationship between high school and turnout is nowhere near significant in my sample of 12 (either directionally or in a two tailed test). In other words, I can not be sure if the relationship observed is due to a genuine relationship or measurement error. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

21 Regression Interpretation While statistical significance is important, it does not tell the whole story. First, larger samples will reach significance much more easily. Second, different types of analyses expect different kinds of results. Third, you must be careful about the regression assumptions. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

22 OLS Assumptions in Verbal and Scalar Form 1 Correct Specification (E{ɛ i } = 0 for i = 1,..., n) 2 Nonstochastic regressors ({ɛ 1,..., ɛ n } and {x 1,..., x n } are independent) 3 Error Properties a. Homoscedastic (V {ɛ i } = σ 2 for i = 1,..., n) b. Independent (Cov{ɛ i, ɛ j } = 0 for i, j = 1,..., n where i j) c. Normally Distributed (ɛ i N (0, σ 2 )) Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

23 Assumption Sets The weak set includes all but 3(c): sufficient for properties of β (and proof of Gauss-Markov). The strong set includes 3(c): sufficient for inference about β. In other words, four assumptions gets us to a good estimate of β, but we need all five to do confidence intervals and significance testing. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

24 Gauss-Markov Theorem Under the weak set assumptions OLS is BLUE (B)est (i.e., efficient) (L)inear (U)nbiased (E)stimator More on this in Fall Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25

25 For Next Time Read Agresti & Finlay, chapter 10, Introduction to Multivariate Relationships. Answer questions 9.4 and 9.11 Load the UN Human Development Report, 2005 data from Agresti s website. (Also listed in Table 9.13.) Choose an input and outcome variable in these data and create a research hypothesis. Run a linear regression model to test your hypothesis. Report and interpret the intercept, slope, r, and r 2. Draw a statistical inference about the slope coefficient. Code Hints un<-read.csv( header=true, na.strings=.. ) Basic syntax for correlation: cor(x, y, use = "everything") If you have missing observations in your variables, you may need to change the use option. Jamie Monogan (UGA) Linear Regression & Correlation POLS / 25