Topic - 12 Linear Regression and Correlation

Transcription

1 Topic 1 Linear Regression and Correlation

2 Correlation & Regression Univariate & Bivariate tatistics U: frequenc distribution, mean, mode, range, standard deviation B: correlation two variables Correlation linear pattern of relationship between one variable () and another variable () an association between two variables graphical representation of the relationship between two variables Warning: No proof of causalit Cannot assume causes

3 1. Correlation Analsis Correlation coefficient measures the strength of the relationship between and ample Pearson s correlation coefficient r å ( ) å i å i i n i å ( å )( å ) i i n ( ) å i n i

4 Pearson s Correlation Coefficient r indicates strength of relationship (strong, weak, or none) direction of relationship positive (direct) variables move in same direction negative (inverse) variables move in opposite directions r ranges in value from 1. to trong Negative No Rel. trong Positive

5 Limitations of Correlation linearit: can t describe nonlinear relationships e.g., relation between aniet & performance no proof of causation Cannot assume causes

6 ome Correlation Patterns Linear relationships Curvilinear relationships Y Y X X Y Y X X

7 ome Correlation Patterns trong relationships Weak relationships Y Y X X Y Y X X

8 Eample The table shows the heights and weights of n 1 randoml selected college football plaers. Plaer Height, Weight, r 38 (6.4)(61).861

9 Eample scatter plot 1 catterplot of Weight vs Height Weight r.861 trong positive correlation Height As the plaer s height increases, so does his weight.

10 Inference using r The population coefficient of correlation is called ( rho ). We can test for a significant correlation between and using a test: H H 1 : r : r ¹ r Test tatistic : t Reject H if t > t n r 1 r or t < t a / a / with n df.

11 Eample r.861 Is there a significant positive correlation between weight and height in the population of all college football plaers? H H 1 : r : r > Test tatistic : t r n 1 r Use the ttable with n 8 df to bound the pvalue as pvalue <.5. There is a significant positive correlation between weight and height in the population of all college football plaers.

12 . Linear Regression Regression: Correlation + Prediction Regression analsis is used to predict the value of one variable (the dependent variable) on the basis of other variables (the independent variables). Dependent variable: denoted Y Independent variables: denoted X 1, X,, X k

13 Eample Let be the monthl sales revenue for a compan. This might be a function of several variables: 1 advertising ependiture time of ear 3 state of econom 4 size of inventor We want to predict using knowledge of 1,, 3 and 4.

14 ome Questions Which of the independent variables are useful and which are not? How could we create a prediction equation to allow us to predict using knowledge of 1,, 3 etc? How good is this prediction? We start with the simplest case, in which the response is a function of a single independent variable,.

15 Model Building A statistical model separates the sstematic component of a relationship from the random component. Data tatistical model stematic component + Random errors In regression, the sstematic component is the overall linear relationship, and the random component is the variation around the line.

16 A imple Linear Regression Model Eplanator and Response Variables are Numeric Relationship between the mean of the response variable and the level of the eplanator variable assumed to be approimatel linear (straight line) Model: Y a + b1 + e e ~ N(, s ) b 1 > Þ Positive Association b 1 < Þ Negative Association b 1 Þ No Association

17 Picturing the imple Linear Regression Model Y Regression Plot Error: e b lope 1 a Intercept X

18 imple Linear Regression Analsis a + b + e ˆ a + b + e actual value of a score ŷ predicted value Variables: Independent Variable Dependent Variable Parameters: a Intercept β lope ε ~ normal distribution with mean and variance s

19 imple Linear Regression Model ˆ a + b bsloped/d intercept a

20 The Method of Least quares The equation of the bestfitting line is calculated using a set of n pairs ( i, i ). We choose our estimates a and b to estimate a and b so that the vertical distances of the points from the line, are minimized. Bestfittingline : ˆ a + b Choosea and b tominimize E å( ˆ) å( a b)

21 Least quares Estimators b a b b a n n n + å å å å å å å and where Bestfittingline : squares: Calculatethesumsof ˆ ) )( ( ) ( ) ( b a b b a n n n + å å å å å å å and where Bestfittingline : squares: Calculatethesumsof ˆ ) )( ( ) ( ) (

22 Eample The table shows the IQ scores for a random sample of n 1 college freshmen, along with their final calculus grades. tudent IQ cores, Calculus grade, Use our calculator to find the sums and sums of squares. å å å 46 å 3634 å

23 Bestfittingline : and a b ˆ (46) (46)(76) (76) (46) Bestfittingline : and a b ˆ (46) (46)(76) (76) (46) Eample

24 The Analsis of Variance The total variation in the eperiment is measured b the total sum of squares: Total å ( ) The Total is divided into two parts: ür (sum of squares for regression): measures the variation eplained b using in the model. üe (sum of squares for error): measures the leftover variation not eplained b.

25 The Analsis of Variance We calculate R ( ) E Total R ( )

26 The ANOVA Table Total df n 1 Regression df Error df 1 n 1 1 n Mean quares MR R/(1) ME E/(n) ource df M F Regression 1 R R/(1) MR/ME Error n E E/(n) Total n 1 Total

27 The Calculus Problem ( ) 1894 R ( ) E Total R ource df M F Regression Error Total 9 56.

28 Testing the Usefulness of the Model (The F Test) You can test the overall usefulness of the model using an F test. If the model is useful, MR will be large compared to the uneplained variation, ME. Hpothesis H : model is not usefulin predicting Test tatistic : MR F ME Reject H if F > F with 1 and n df a. This test is eactl equivalent to the ttest, with t F.

29 Minitab Output To test H : b Least squares regression line Regression Analsis: versus The regression equation is Predictor Coef E Coef T P Constant Rq 7.5% Rq(adj) 66.8% Analsis of Variance ource DF M F P Regression Residual Error Total ME Regression coefficients, a and b t F

30 Testing the Usefulness of the Model The first question to ask is whether the independent variable is of an use in predicting. If it is not, then the value of does not change, regardless of the value of. This implies that the slope of the line, b, is zero. H : b versus H : b a ¹

31 Testing the Usefulness of the Model The test statistic is function of b, our best estimate of b. Using ME as the best estimate of the random variation s, we obtain a t statistic. H : b versus H : b a ¹ Teststatistic: t b ME whichhasa t distributi on with df n or a confidenceinterval: b ± t a / ME

32 The Calculus Problem Is there a significant relationship between the calculus grades and the IQ scores at the 5% level of significance? H : b versush : b ¹ a t b M E/ / Reject H when t >.36. ince t 4.38 falls into the rejection region, H is rejected. There is a significant linear relationship between the calculus grades and the IQ scores for the population of college freshmen.

33 Measuring the trength of the Relationship If the independent variable is of useful in predicting, ou will want to know how well the model fits. The strength of the relationship between and can be measured using: Correlation coefficient : r Coefficient of determination : r R Total

34 Measuring the trength of the Relationship ince Total R + E, r measures ü ü the proportion of the total variation in the responses that can be eplained b using the independent variable in the model. the percent reduction the total variation b using the regression equation rather than just using the sample mean bar to estimate. r R Total For the calculus problem, r.75 or 7.5%. Meaning that 7.5% of the variabilit of Calculus cores can be elain b the model.

35 Estimation and Prediction ø ö ç ç è æ + + ± ø ö ç ç è æ + ± n ME t n ME t / / ) ( 1 1 ˆ ) ( 1 ˆ a a : when Topredicta particularvalueof : when To estimatethe averagevalueof ø ö ç ç è æ + + ± ø ö ç ç è æ + ± n ME t n ME t / / ) ( 1 1 ˆ ) ( 1 ˆ a a : when Topredicta particularvalueof : when estimatethe averagevalueof To Confidence interval Prediction interval

36 The Calculus Problem Estimate the average calculus grade for students whose IQ score is 5 with a 95% confidence interval. Calculate ˆ (5) 79.6 ˆ ± æ 1 (5 46) ç + è ± 6.55 or 7.51to ö ø

37 The Calculus Problem Estimate the calculus grade for a particular student whose IQ score is 5 with a 95% confidence interval. Calculate ˆ (5) 79.6 ˆ ±.36 æ 1 (5 46) ç è ± 1.11 or 57.95to1.17. ö ø Notice how much wider this interval is!

38 Minitab Output Confidence and prediction intervals when 5 Predicted Values for New Observations New Obs Fit E Fit 95.% CI 95.% PI (7.51, 85.61) (57.95,1.17) Values of Predictors for New Observations New Obs 1 5. Fitted Line Plot ügreen prediction bands are alwas wider than red confidence bands Regression 95% CI 95% PI Rq 7.5% Rq(adj) 66.8% üboth intervals are narrowest when bar

39 Estimation and Prediction Once ou have ü ü determined that the regression line is useful used the diagnostic plots to check for violation of the regression assumptions. You are read to use the regression line to ü Estimate the average value of for a given value of ü Predict a particular value of for a given value of.

40 Estimation and Prediction The best estimate of either E() or for a given value is ˆ a + b Particular values of are more difficult to predict, requiring a wider range of values in the prediction interval.

41 Regression Assumptions Remember that the results of a regression analsis are onl valid when the necessar assumptions have been satisfied. Assumptions: 1. The relationship between and is linear, given b a + b + e.. The random error terms e are independent and, for an value of, have a normal distribution with mean and constant variance, s.

42 Diagnostic Tools 1. Normal probabilit plot or histogram of residuals. Plot of residuals versus fit or residuals versus variables 3. Plot of residual versus order

43 Residuals The residual error is the leftover variation in each data point after the variation eplained b the regression model has been removed. ˆ Residual ˆ i i or i a b i If all assumptions have been met, these residuals should be normal, with mean and variance s.

44 Normal Probabilit Plot ü If the normalit assumption is valid, the plot should resemble a straight line, sloping upward to the right. ü If not, ou will often see the pattern fail in the tails of the graph. 99 Normal Probabilit Plot of the Residuals (response is ) 95 9 Percent Residual 1

45 Residuals versus Fits ü If the equal variance assumption is valid, the plot should appear as a random scatter around the zero center line. ü If not, ou will see a pattern in the residuals. 15 Residuals Versus the Fitted Values (response is ) 1 Residual Fitted Value 9 1