y n 1 ( x i x )( y y i n 1 i y 2

Transcription

1 STP3 Brief Class Notes Instructor: Ela Jackiewicz Chapter Regression and Correlation In this chapter we will explore the relationship between two quantitative variables, X an Y. We will consider n ordered pairs of observations (x,y). Both x and y can be observed (observational study) or y can be observed for specific values of x that are selected by the researcher (experiment). Data must display a linear trend on the scatterplot for us to consider fitting a linear equation that will expres in terms of x. We will use the following vocabulary with respect to x and y: y response variable or dependent variable x predictor variable/explanatory variable or independent variable If our scatterplot indicatexistence of a linear trend (visually we can see our data points clustering closely to some line), we would be interested in expressing quantitatively the strength of that linear relationship. The numerical measure we will use is called Linear Correlation Coefficient and we will denote it's value by r. Linear correlation Coefficient of X and Y, r : r= where s x = x n i x y i y s x = x i x = y s i y y n ( x i x s x )( y y i ) = z n xi z yi (standard deviations of x-s and y-s respectively) n n Linear Correlation coefficient gives information about strength and direction of a linear trend in our data. Values close to or - indicate extremely strong linear association between X and Y. Example. Consider people on the diet program. Let X=# of times per week person eats out, Y= change in weight (in pounds),(before-after) after three months period of dieting. Compute and interpret linear correlation r. (Make sure to plot the data first, scatterplot shows reasonably strong negative linear trend) First we compute means and standard deviations of our x and y values: x i y i x i x (x i x) y i y ( y i y) x=5/5=5 s x= =.35 y=0/5=8 s x= 0 =5.8

2 STP3 Brief Class Notes Instructor: Ela Jackiewicz Now we are ready to compute our z-scores and finally r. x i y i z x i = x i x s x z yi = y i y z xi z yi (3-5)/.35= (0-8)/5.8= r= 3. = 0.5 This value indicates very strong linear relationship between x and y. Properties of r: r is unit-free r, the closer is r to - or the stronger is linear relationship between x and y, r= or - indicates a perfect linear relationship r is symmetric, if X and Y are reversed, value of r remains the same. sign of r matches the sign of the slope of the regression line Note: r=0 does not necessarily mean that there is no relationship between x and y, just that there is no linear relationship. Warning about correlation: strong (positive or negative) correlation does not have to imply causal relationship between x and y. Example: Researchers found that there is a very strong positive correlation between number of storks present an number of babies born. Does it imply that storks cause babies to be born? Of course not, both can be influenced by some other factors. Observational studies can't prove causation. Experiments can prove causation. Inference concerning correlation. r=sample linear correlation coefficient can be regarded as an estimate of a population correlation ρ (Greek letter rho). In order to regard r as an estimate of ρ it must be reasonable to assume that both x and y were selected at random from the population. When x-s are specified by the researcher, r is not an appropriate estimate of ρ. Bivariate Random Sampling Model assumeach pair (x,y) is sampled randomly from the population of all such pairs.

3 STP3 Brief Class Notes Instructor: Ela Jackiewicz Tests of hypotheses involving population correlation coefficient. Our hypotheses are: H 0 : =0 (x and y are not linearly correlated in the population) vs H a :ρ 0 or H a :ρ> 0 or H a :ρ< 0 Test statistics: t s =r n r Assuming null hypothesis is true, t s has a t distribution with df=n- In our example we may ask the question: Assuming that a linear model applies, test the hypothesis that there is no linear relationship between x and y against an alternative that y decreases with increasing x. Use 5% significance level. Our test hypotheses are: H 0 :ρ=0 and H a :ρ< 0 t s =. (5 ) = 3.8 P-value=0.05<0.05 (.83) Null is rejected, we have evidence for alternative. Yes, y decreases with increasing x, there is statistically significant negative linear relationship between x and y. Notes about correlation inference: There is a difference between statistical significance and importance, weak correlation may be statistically significant, but not at all useful or informative For a fixed value of r, test statistics, t s, increases as a sample size increases, if n is large enough, ts will be large and will show correlation to be significant no matter how small r is. CI for r may help determine a practical significance. r ixtremely sensitive to extreme data points, be aware of outliers. We have plotted our data to determine if there appear to be a linear relationship, we found the correlation coefficient and tested to see if r was statistically significant. Now we will discuss how to find and interpret the equation of the line that best summarizes the linear relationship between two quantitative variables. Our line is called Least Squares Regression Line and will have a form: y=b 0 b x, where y is a predicted value of y for given x. b 0 =y-intercept, b =slope, a rate of change of y with respect to x. The name of the line reflects the fact that our line has smallest possible sum of squared errors (of all lines that can be possibly fitted to the given data). Errors or residuals are the vertical deviations of observations (data points) from the line : e= y ŷ, SS(resid)= e = (y i ŷ i ) is minimized by LS regression line.

4 STP3 Brief Class Notes Instructor: Ela Jackiewicz Formulas for the slope and y-intercept of Least-Squares Regression equation: y =b 0 b x, b =r s x, b 0 =y b x In our example: x=5/5=5 y=0/5=8 b = = b 0=8 ( )(5)=8, so our equation is : ŷ=8 x Equation tells us that for every time increase in x (times to eat out in a week), y (weight change) decreases by pounds. The more often you go out to eat the less pounds are lost. Units of the slope are : (units of y)/ (units of x) Notes about the regression line: ( x, y) lies on the fitted line we can use the regression line to predict values of y for given values of x. Interpreting the slope: If x increases by unit, y increases (or decreases) by b units (direction of change depends on the sign of b. Regression line is based on the assumption that data points are scattered about the line. If data are scattered about the curve, or do not show any trend at all, avoid the linear regression. Linear regression can be strongly affected by outliers. Always plot data first. Making predictions using our equation: Predict number of pounds lost if dieting person eats out 5 times per week: y=8-0=8 lb (this prediction is for x within data range) Warning about making predictions: It is safe to predict y for x within data range. Extrapolation - making predictions for values of the predictor variable outside the range of the observed values of the predictor variable (x) Grossly incorrect predictions can result from extrapolation. For ex. Prediction for x=5 i=- (weight gain of lb). One may think that value is reasonable, weight gain is to be expected if you eat out so often, but because we have no data in that range, we may not be sure that our particular trend will hold as far out of the range of x values. Problems we may encounter with our data: Outlier a data point that lies far from the regression line, relative to other data points Influential observation a data point whose removal causes the regression to change considerably. It is usually separated in the x-direction from the other data points. It pulls the regression line towards itself.

5 STP3 Brief Class Notes Instructor: Ela Jackiewicz Important Sums of Squares : There are three summations that are important in assessment of the fit of our regression line. Total Sum of squares SS total = y i y gives total variability in y values Residual Sum of Squares: SS resid = y i y i by the regression equation gives total variability in y values unexplained Regression Sum of Squares: the regression equation SS reg = y i y gives total variability in y valuexplained by Regression Identity: SS(total)=SS(reg)+SS(resid) In our example SS(total) was already computed. We can now obtain residual sum of squares, and regression sum of squares by obtaining predicted values of y for each x, by using our equation. x i y i ( y i y) ŷ i y i ŷ ( y i ŷ) ŷ i y ( ŷ i y) SS(total) 8 SS(resid) 88 SS(reg) We can verify that in our example : 0=88+8 We will use the three sums of squares in farther definitions. Assessing the fit of our regression line: = We can use SS(resid) to compute Residual Standard Deviation: s SS(resid) e n This measure tells us how close are the data points to our fitted regression line, the smaller is this value, the better is the fit of our regression line. If is smaller than (standard deviation of y-s), it means a good fit. The units of are the same as units of y.

6 STP3 Brief Class Notes Instructor: Ela Jackiewicz measures variability of y values around the mean ( y ) measures variability around the regression line. In the nice data, if it is not too small, we expect roughly : 8% of observed y-s to be within ± of the regression line 5% of observed y-s points to be within ± of the regression line and.% of observed y-s points to be within ±3 of the regression line In other words these percentages (roughly) of data points are expected to be within a vertical distances (parallel to the y axis) above and below the regression line. = In our example = (8/3)=.5lb<5.5= indicating a good fit. ( s 0 y 5 =5.5 ) Another value that tells us about the fit of the regression line is: Coefficient of Determination: r SS reg SS resid = = SS total SS total (square of linear corr. coef. r) This measure gives us proportion of total variability in y values that ixplained by our regression line. If this is close to (00%), then that indicates a good fit. 0 r In our example r = 8 0 = 88 0 =0.83, it indicates a good fit, 83% of total variability in y-values ixplained by the regression line. There is an approximate relationship of r to and. This approximation is good for large data r If this ratio is small, it indicates a good fit Above formula can also be expressed as r s e and used to approximate r for large data. In our example 0.83 =0. and approximation, since our data is small. = =0.8, and r 0. not a great

7 STP3 Brief Class Notes Instructor: Ela Jackiewicz Hypothesis test and CI for the slope: Assuming that the following linear model applies to X and Y: Y = 0 X Term ϵ in our model represents random error, we include it in the model to reflect that Y varieven if X is fixed. We can test hypothesis related to the true slope β The terms we compute for Least Squares Regression are estimating parameters in our model: b 0 estimates 0, b estimates If we want to test if slope is zero (indicating no relationship between x and y), our hypotheses are: H 0 : =0 vs H a :β 0 or H a :β > 0 or H a :β < 0 Choice of alternative depends on the question. Test statistics: t s = b SE b df=n-, Standard Error of the slope b : SE b = (x i x) = s x n, the second formula iasier to compute, The test statistics we are using here iquivalent to the one we used before in a test for population correlation coefficient, so we do not need to repeat it. We can also compute Confidence interval for a true slope: 5% CI for : b ±t.05 SE b In our example SE b =.5.35 =.53 and 5% CI for is : ±3.8 (0.53), ±3.8 (0.53), (-3., -0.3) CI indicates that the slope is significantly different than 0 and negative.

8 STP3 Brief Class Notes Instructor: Ela Jackiewicz CALCULATOR ) To use above computational formulas and/or find basic statistics for x and y use STAT EDIT place x-s and y-s on L and L respectively, then STAT CALC, select -Var Stats <enter> -Var Stats L, L <enter> )To find the regression line: STAT EDIT place x-s and y-s on L and L respectively STAT CALC, select LinReg(ax+b) <enter> LinReg(ax+b) L, L <enter> Output should display slope, y intercept, r and r.if you do not see r and r, then go to the CATALOG, select DIAGNOSTICS ON <enter><enter>, next time you run the regression it will show both values. 3) To perform the test: STAT TESTS use LinRegT-Test, place Xlist L Ylist L, then select appropriate alternative hypothesis (They use β for β, test is for both β and ρ ) In the output value of s=, you also get values of s x and there - ) To compute CI for β use STAT TESTS use LinRegT-interval, place Xlist L Ylist L, then select appropriate C-level If you do not have the CI on your calculator, use s from the LinRegT-Test output to compute SE b = s x n and compute CI by hand.

9 STP3 Brief Class Notes Instructor: Ela Jackiewicz Least squares Linear Regression Example Is number of beers consumed a good predictor of BAC (blood alcohol content)? Can you drive legally after beers (legal limit is.08)? Random sample of students from Ohio State University participated in a study. Each student was assigned randomly number of beers to drink and after 30 minutes a police officer measured their BAC in grams of alcohol per deciliter of blood. The results are given in the table below. # of beers=x BAC=y Answer following questions: Graph these data to confirm existence of a linear trend. Obtain LS regression line for these data. What is the slope of your line telling you about change in BAC after you drink a next beer? Do you think y-intercept value is sensible? Compute linear correlation coefficient and coefficient of determination. What % of total variability in BAC ixplained by the regression line? Are r and r indicating a good fit? Clearly answer both questions posed at the beginning of the problem. Can we use our equation to safely predict BAC for one that drinks 5 beers? Given SS(resid)=.005, compute, compare it to and decide if it indicates a good fit. Test the hypothesis of no linear relationship between x and y against directional alternative hypothesis that y increases with increasing x. Obtain 5% CI for the true slope β