Correlation and Regression

Transcription

1 Correlation and Regression Marc H. Mehlman University of New Haven All models are wrong. Some models are useful. George Box the statistician knows that in nature there never was a normal distribution, there never was a straight line, yet with normal and linear assumptions, known to be false, he can often derive results which match, to a useful approximation, those found in the real world. George Box (University of New Haven) Correlation and Regression 1 / 64

2 Table of Contents 1 Bivariate Data 2 Correlation 3 Simple Regression 4 Variation 5 Logistic Regression 6 Logistic Regression 7 Chapter #9 R Assignment (University of New Haven) Correlation and Regression 2 / 64

3 Bivariate Data Bivariate Data and Scatterplots Bivariate Data and Scatterplots (University of New Haven) Correlation and Regression 3 / 64

4 Bivariate Data Bivariate data comes from measuring two aspects of the same item/individual. For instance, (70, 178), (72, 192), (74, 184), (68, 181) is a random sample of size four obtained from four male college students. The bivariate data gives the height in inches and the weight in pounds of each of the for students. The third student sampled is 74 inches high and weighs 184 pounds. Can one variable be used to predict the other? Do tall people tend to weigh more? Definition A response (or dependent) variable measures the outcome of a study. The explanatory (or independent) variable is the one that predicts the response variable. (University of New Haven) Correlation and Regression 4 / 64

5 Bivariate Data Bivariate data For each individual studied, we record data on two variables. We then examine whether there is a relationship between these two variables: Do changes in one variable tend to be associated with specific changes in the other variables? Here we have two quantitative variables recorded for each of 16 students: 1. how many beers they drank 2. their resulting blood alcohol content (BAC) Student ID Number of Beers Blood Alcohol Content (University of New Haven) Correlation and Regression 5 / 64

6 Bivariate Data Scatterplots A scatterplot is used to display quantitative bivariate data. Each variable makes up one axis. Each individual is a point on the graph. Student Beers BAC (University of New Haven) Correlation and Regression 6 / 64

7 Bivariate Data > plot(trees$girth~trees$height,main="girth vs height") girth vs height trees$girth trees$height (University of New Haven) Correlation and Regression 7 / 64

8 Bivariate Data How to scale a scatterplot Same data in all four plots Both variables should be given a similar amount of space: Plot is roughly square Points should occupy all the plot space (no blank space) (University of New Haven) Correlation and Regression 8 / 64

9 Bivariate Data Interpreting scatterplots After plotting two variables on a scatterplot, we describe the overall pattern of the relationship. Specifically, we look for Form: linear, curved, clusters, no pattern Direction: positive, negative, no direction Strength: how closely the points fit the form and clear deviations from that pattern Outliers of the relationship (University of New Haven) Correlation and Regression 9 / 64

10 Bivariate Data Form Linear No relationship Nonlinear (University of New Haven) Correlation and Regression 10 / 64

11 Bivariate Data Direction Positive association: High values of one variable tend to occur together with high values of the other variable. Negative association: High values of one variable tend to occur together with low values of the other variable. (University of New Haven) Correlation and Regression 11 / 64

12 Bivariate Data Strength The strength of the relationship between the two variables can be seen by how much variation, or scatter, there is around the main form. (University of New Haven) Correlation and Regression 12 / 64

13 Bivariate Data Outliers An outlier is a data value that has a very low probability of occurrence (i.e., it is unusual or unexpected). In a scatterplot, outliers are points that fall outside of the overall pattern of the relationship. (University of New Haven) Correlation and Regression 13 / 64

14 Bivariate Data Adding categorical variables to scatterplots Two or more relationships can be compared on a single scatterplot when we use different symbols for groups of points on the graph. The graph compares the association between thorax length and longevity of male fruit flies that are allowed to reproduce (green) or not (purple). The pattern is similar in both groups (linear, positive association), but male fruit flies not allowed to reproduce tend to live longer than reproducing male fruit flies of the same size. (University of New Haven) Correlation and Regression 14 / 64

15 Correlation Correlation Correlation (University of New Haven) Correlation and Regression 15 / 64

16 Correlation Definition Given the bivariate data, (x 1, y 1 ),, (x n, y n ), the sample correlation coefficent (sample Pearson product-moment correlation coefficient) is r def = 1 n 1 n ( ) ( xj x yj ȳ j=1 The population correlation coefficient is denoted as ρ def = 1 N s x s y ). N ( ) ( ) xj µ X yj µ Y j=1 where the above sum is summed over the entire population of size N. One thinks of r as an estimator of ρ. σ X σ Y (University of New Haven) Correlation and Regression 16 / 64

17 Correlation One can also use the formula r = n( n j=1 x jy j ) ( n j=1 x j)( n j=1 y j) [ n ( n j=1 x n ) ] [ 2 j 2 j=1 x j n ( n j=1 y n ) ] 2 j 2 j=1 y j R command: > cor(trees$girth,trees$height) [1] (University of New Haven) Correlation and Regression 17 / 64

18 Correlation One can also use the formula r = n( n j=1 x jy j ) ( n j=1 x j)( n j=1 y j) [ n ( n j=1 x n ) ] [ 2 j 2 j=1 x j n ( n j=1 y n ) ] 2 j 2 j=1 y j R command: > cor(trees$girth,trees$height) [1] (University of New Haven) Correlation and Regression 17 / 64

19 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

20 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

21 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

22 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

23 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

24 Correlation The correlation coefficient measures the strength of any linear relationship between X and Y. Properties of Correlation: cor(x, Y ) = cor(y, X ). 1 r 1, and scale invariant. if r is positive there is a positive linear relationship between the two variables. if r is negative there is a negative linear relationship between the two variables. the closer r is to one, the stronger the linear relationship between the two variables. if r = 1 (ie, r = 1 or 1), all the data points lie on a straight line. (University of New Haven) Correlation and Regression 18 / 64

25 Correlation r has no unit r = standardized value of x (unitless) standardized value of y (unitless) r = (University of New Haven) Correlation and Regression 19 / 64

26 Correlation r is not resistant to outliers Correlations are calculated using means and standard deviations, and thus are NOT resistant to outliers. Just moving one point away from the linear pattern here weakens the correlation from 0.91 to 0.75 (closer to zero). (University of New Haven) Correlation and Regression 20 / 64

27 Correlation Correlation 14 (University of New Haven) Correlation and Regression 21 / 64

28 Correlation Caution: Correlation is not Causation Definition When calculating correlation, a lurking variable is a third factor that explains the relationship between the two correlated variables Example (Lurking Variables) There is a strong correlation between shoe size and reading skills among elementary school children. The lurking variable is There is a strong correlation between the number of firefighters at a fire site and the amount of damage. The lurking variable is Caution: Beware correlations based on averaged data. While there is a strong correlation average age and average height among children, the correlation between age and height for individual children is much, much lower. (University of New Haven) Correlation and Regression 22 / 64

29 Correlation Definition Two variables are confounded when their effects on the response variable can not be distinguished from each other. The confounded variables can be either explanatory or lurking variables (or only work in the presence of each other). The only way to distinguish between two confounded variables is to redesign the experiment. Example When I m stressed, I get muscle cramps. However, when I m stressed, I also drink lots of coffee and lose sleep. Are the cramps caused by stress, or coffee, or lack of sleep, or some combination of the above? Example A classic example of confounding: A study suggests that people who carry matches are more likely to develop lung cancer. Is it the matches or is there confounding here with a lurking variable? (University of New Haven) Correlation and Regression 23 / 64

30 Correlation Definition Two variables are confounded when their effects on the response variable can not be distinguished from each other. The confounded variables can be either explanatory or lurking variables (or only work in the presence of each other). The only way to distinguish between two confounded variables is to redesign the experiment. Example When I m stressed, I get muscle cramps. However, when I m stressed, I also drink lots of coffee and lose sleep. Are the cramps caused by stress, or coffee, or lack of sleep, or some combination of the above? Example A classic example of confounding: A study suggests that people who carry matches are more likely to develop lung cancer. Is it the matches or is there confounding here with a lurking variable? (University of New Haven) Correlation and Regression 23 / 64

31 Correlation Definition Two variables are confounded when their effects on the response variable can not be distinguished from each other. The confounded variables can be either explanatory or lurking variables (or only work in the presence of each other). The only way to distinguish between two confounded variables is to redesign the experiment. Example When I m stressed, I get muscle cramps. However, when I m stressed, I also drink lots of coffee and lose sleep. Are the cramps caused by stress, or coffee, or lack of sleep, or some combination of the above? Example A classic example of confounding: A study suggests that people who carry matches are more likely to develop lung cancer. Is it the matches or is there confounding here with a lurking variable? (University of New Haven) Correlation and Regression 23 / 64

32 Correlation Establishing causation Establishing causation from an observed association can be done if: 1) The association is strong. 2) The association is consistent. 3) Higher doses are associated with stronger responses. 4) The alleged cause precedes the effect. 5) The alleged cause is plausible. Lung cancer is clearly associated with smoking. What if a genetic mutation (lurking variable) caused people to both get lung cancer and become addicted to smoking? It took years of research and accumulated indirect evidence to reach the conclusion that smoking causes lung cancer. (University of New Haven) Correlation and Regression 24 / 64

33 Correlation Theorem (Test for Correlation) Let The test statistic is for H 0. H 0 : ρ = 0 vs H 1 : ρ 0 t = r 1 r 2 n 2 t(n 2) One can also use Table A-6 with test statistic r. R command: cor.test(x, Y) (one can also do one sided tests with R). (University of New Haven) Correlation and Regression 25 / 64

34 Using R Correlation Example > cor.test(trees$girth,trees$height) Pearson s product-moment correlation data: trees$girth and trees$height t = , df = 29, p-value = alternative hypothesis: true correlation is not equal to 0 95 percent confidence interval: sample estimates: cor Note that one is assuming that the (trees$height, trees$girth) are sampled from a bivariate normal distribution. (University of New Haven) Correlation and Regression 26 / 64

35 Correlation Example Each day, for the last 63 days, measurements of the time Joe spends sleeping and the time he spends watching tv are taken. Assume time spent sleeping and time spent watching tv form a bivariate normal random variable. A sample correlation of r = 0.12 is calculated. Find the p value of H 0 : ρ = 0 versus H A : ρ 0. Solution: > tstat=0.12*sqrt((63-2)/(1-0.12^2)) > tstat [1] > 2*(1-pt(tstat,61)) [1] There is little evidence that the time Joe spends sleeping and the time Joe spends watching tv is correlated. (University of New Haven) Correlation and Regression 27 / 64

36 Correlation Example Each day, for the last 63 days, measurements of the time Joe spends sleeping and the time he spends watching tv are taken. Assume time spent sleeping and time spent watching tv form a bivariate normal random variable. A sample correlation of r = 0.12 is calculated. Find the p value of H 0 : ρ = 0 versus H A : ρ 0. Solution: > tstat=0.12*sqrt((63-2)/(1-0.12^2)) > tstat [1] > 2*(1-pt(tstat,61)) [1] There is little evidence that the time Joe spends sleeping and the time Joe spends watching tv is correlated. (University of New Haven) Correlation and Regression 27 / 64

37 Simple Regression Simple Regression Simple Regression (University of New Haven) Correlation and Regression 28 / 64

38 Simple Regression Let X = the predictor or independent variable Y = the response or dependent variable. Given a bivariate random variable, (X, Y ), is there a linear (straight line) association between X and Y (plus some randomness)? And if so, what is it and how much randomness? Definition (Statistical Model of Simple Linear Regression) Given a predictor, x, the response, y is y = β 0 + β 1 x + ɛ x where β 0 + β 1 x is the mean response for x. The noise terms, the ɛ x s, are assumed to be independent of each other and to be randomly sampled from N(0, σ). The parameters of the model are β 0, β 1 and σ. (University of New Haven) Correlation and Regression 29 / 64

39 Simple Regression Conditions for Regression Inference The figure below shows the regression model when the conditions are met. The line in the figure is the population regression line µy= β0 + β1x. For each possible value of the explanatory variable x, the mean of the responses µ(y x) moves along this line. The Normal curves show how y will vary when x is held fixed at different values. All the curves have the same standard deviation σ, so the variability of y is the same for all values of x. The value of σ determines whether the points fall close to the population regression line (small σ) or are widely scattered (large σ). 8 (University of New Haven) Correlation and Regression 30 / 64

40 Simple Regression Moderate linear association; regression OK. Obvious nonlinear relationship; regression inappropriate. y = x y = x One extreme outlier, requiring further examination. Only two values for x; a redesign is due here y = x y = x (University of New Haven) Correlation and Regression 31 / 64

41 Simple Regression Given bivariate random sample from the simple linear regression model, (x 1, y 1 ), (x 2, y 2 ),, (x n, y n ) one wishes to estimate the parameters of the model, (β 0, β 1, σ). Given an arbitrary line, y = mx + b define the sum of the squares of errors to be n i=1 [y i (mx i + b)] 2. Using Calculus, one can find the least squares regression line, y = b 0 + b 1 x, that minimizes the sum of squares of errors. (University of New Haven) Correlation and Regression 32 / 64

42 Simple Regression Theorem (Estimating β 0 and β 1 ) Given the bivariate random sample, (x 1, y 1 ), (x n, y n ), the least squares regression line, y = b 0 + b 1 x is obtained by letting ( ) sy b 1 = r and b 0 = ȳ b 1 x. s x where b 0 is an unbiased estimator of β 0 and b 1 is an unbiased estimator of β 1. Note: The point ( x, ȳ) will lie on the regression line, though there is no reason to believe that ( x, ȳ) is one of the data points. One can also calculate b 1 using b 1 = n( n j=1 x jy j ) ( n j=1 x j)( n j=1 y j) n n j=1 x j 2 ( n j=1 x j) 2. (University of New Haven) Correlation and Regression 33 / 64

43 Simple Regression Example > plot(trees$girth~trees$height,main="girth vs height") > abline(lm(trees$girth ~ trees$height), col="red") girth vs height trees$girth trees$height Since both variables come from trees, in order for the R command lm (linear model) to work, trees has to be in the R format, data.frame. > class(trees) # "trees" is in data.frame format - lm will work. [1] "data.frame" > g.lm=lm(girth~height,data=trees) > coef(g.lm) (Intercept) trees$height (University of New Haven) Correlation and Regression 34 / 64

44 Simple Regression Example > plot(trees$girth~trees$height,main="girth vs height") > abline(lm(trees$girth ~ trees$height), col="red") girth vs height trees$girth trees$height Since both variables come from trees, in order for the R command lm (linear model) to work, trees has to be in the R format, data.frame. > class(trees) # "trees" is in data.frame format - lm will work. [1] "data.frame" > g.lm=lm(girth~height,data=trees) > coef(g.lm) (Intercept) trees$height (University of New Haven) Correlation and Regression 34 / 64

45 Simple Regression Definition The predicted value of y at x j is ŷ j def = b 0 + b 1 x j. The predicted value, ŷ, is a unbiased estimator of the mean response, µ y. Example Using the R dataset trees, one wants the predicted girth of three trees, of heights 74, 83 and 91 respectively. One uses the regression model girth height for our predictions. The work below is done in R. > g.lm=lm(girth~height,data=trees) > predict(g.lm,newdata=data.frame(height=c(74,83,91))) (University of New Haven) Correlation and Regression 35 / 64

46 Simple Regression Never make forecasts, especially about the future. Samuel Goldwyn The regression line only has predictive value for y at x if 1 ρ 0 (if no significant linear correlation, don t use the regression line for predictions.) If ρ 0, then ȳ is best predictor of y at x. 2 only predict y for x s within the range of the x j s one does not predict the girth of a tree with a height of 1000 feet. Interpolate, don t extrapolate. r (or r 2 ) is a measure of how well the regression equation fits data. bigger r better data fits regression line better prediction. (University of New Haven) Correlation and Regression 36 / 64

47 Simple Regression Outliers and influential points Outlier: An observation that lies outside the overall pattern. Influential individual : An observation that markedly changes the regression if removed. This is often an isolated point. Child 19 = outlier (large residual) Child 19 is an outlier of the relationship (it is unusually far from the regression line, vertically). Child 18 = potential influential individual Child 18 is isolated from the rest of the points, and might be an influential point. (University of New Haven) Correlation and Regression 37 / 64

48 Simple Regression Outlier All data Without child 18 Without child 19 Influential Child 18 changes the regression line substantially when it is removed. So, Child 18 is indeed an influential point. Child 19 is an outlier of the relationship, but it is not influential (regression line changed very little by its removal). (University of New Haven) Correlation and Regression 38 / 64

49 Simple Regression Definition Given a data point, (x j, y j ), the residual of that point is y i ŷ i. Note: 1 Outliers are data points with large residuals. 2 The residuals should be approximately N(0, σ). 3 The regression equation gives the smallest residuals 2 = (y ŷ) 2 possible. (University of New Haven) Correlation and Regression 39 / 64

50 Simple Regression R command for finding residuals: Example > g.lm=lm(girth~height,data=trees) > residuals(g.lm) (University of New Haven) Correlation and Regression 40 / 64

51 Simple Regression Definition Given bivariate data, (x 1, y 1 ),, (x n, y n ), the residual plot is a plot of the residuals against the x j s. If (X, Y ) is bivariate normal, the residuals satisfy the Homoscedasticity Assumption: Definition (Homoscedasticity Assumption) The assumption that the variance around the regression line is the same for all values of the predictor variable X. In other words the pattern of the spread of the residual points around the x axis does not change as one travels left to right on the x axis. There should not be discernible patterns in the residual plot. (University of New Haven) Correlation and Regression 41 / 64

52 Simple Regression R command for testing if Linear Model applies (residuals approximately N(0, σ)). Example > g.lm=lm(girth~height,data=trees) > par(mfrow=c(2,2)) # visualize four graphs at once > plot(g.lm) > par(mfrow=c(1,1)) # reset the graphics defaults Residuals vs Fitted Normal Q Q Residuals Standardized residuals Fitted values Theoretical Quantiles Standardized residuals Scale Location Standardized residuals Residuals vs Leverage Cook's distance Fitted values Leverage (University of New Haven) Correlation and Regression 42 / 64

53 Variation Variation Variation (University of New Haven) Correlation and Regression 43 / 64

54 Variation: Variation y j ȳ }{{} = ŷ j ȳ }{{} + y j ŷ j }{{}. total deviation explained deviation unexplained deviation From here, using some math, one gets the following sum of squares, (SS), n (y j ȳ) 2 j=1 }{{} SS TOT =total variation = n (ŷ j ȳ) 2 j=1 }{{} SS A =explained variation (University of New Haven) Correlation and Regression 44 / 64 + n (y j ŷ j ) 2 j=1 }{{} SS E =unexplained variation.

55 Variation Definition The coefficient of determination is the portion of the variation in y explained by the regression equation r 2 def = SS n A j=1 = (ŷ j ȳ) 2 SS n TOT j=1 (y j ȳ) 2. Properties of the Coefficient of Determination: 1 r 2 = (r) 2 = (correlation coefficient) 2. 2 r 2 = proportion of variation of Y that is explained by the linear relationship between X and Y. Example Using R, since > (cor(trees$girth,trees$height))^2 [1] one concludes that approximately 27% of variation in tree Girth is explained by tree Height and 73% by other factors. (University of New Haven) Correlation and Regression 45 / 64

56 Variation r = 0.3, r 2 = 0.09, or 9% The regression model explains not even 10% of the variations in y. r = 0.7, r 2 = 0.49, or 49% The regression model explains nearly half of the variations in y. r = 0.99, r 2 = , or ~98% The regression model explains almost all of the variations in y. (University of New Haven) Correlation and Regression 46 / 64

57 Variation Definition The variance of the observed y i s about the predicted ŷ i s is s 2 def = SS E n 2 = (yj ŷ j ) 2 n 2 = y 2 j b 0 yj b 1 xj y j, n 2 which is an unbiased estimator of σ 2. The standard error of estimate (also called the residual standard error) is s, an estimator of σ. Note: (b 0, b 1, s) is an estimator of the parameters of the simple linear regression model, (β 0, β 1, σ). Furthermore, b 0, b 1 and s 2 are unbiased estimators of β 0, β 1 and σ 2. (University of New Haven) Correlation and Regression 47 / 64

58 Variation Definition Let y be a future observation corresponding to x. A (1 α)100% Prediction Interval for y is a confidence interval where y will be in the confidence interval (1 α)100% of the time. A prediction interval a confidence interval that not only has to contend with the variability of the response variable, but also the fact that β 0 and β 1 can only be approximated. Theorem ((1 α)100% Prediction Interval for y given x = x ) A (1 α)100% Prediction Interval for y given x = x is ŷ ± m where ŷ = b 0 + b 1 x and the margin of error is m = t α/2 (n 2) s n + (x x) 2 n j=1 (x j x) 2. }{{} SEŷ (University of New Haven) Correlation and Regression 48 / 64

59 A confidence interval for y: Variation (University of New Haven) Correlation and Regression 49 / 64

60 Variation R commands: Example > g.lm=lm(girth~height,data=trees) > predict(g.lm,newdata=data.frame(height=c(74,83,91)),interval="prediction",level=.90) fit lwr upr > summary(g.lm) Call: lm(formula = Girth ~ Height, data = trees) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Height ** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 29 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 29 DF, p-value: (University of New Haven) Correlation and Regression 50 / 64

61 Logistic Regression Multivariate Regression Multivariate Regression (University of New Haven) Correlation and Regression 51 / 64

62 Logistic Regression Given multivariate variate data, (x (1) 1, x (1) 2,, x (1) k, y 1 ), (x (2) 1, x (2) 2,, x (2) k, y 2 ),, (x (n) 1, x (n) 2,, x (n) k, y n ) where x (i) 1, x (i) 2,, x (i) k is a predictor of the response y i, one explores the following possible model. Definition (Statistical Model of Multivariate Linear Regression) Given a k dimensional multivariate predictor, (x (i) 1, x (i) 2,, x (i) k ), the response, y i, is y i = β 0 + β 1 x (i) β k x (i) k + ɛ i where β 0 + β 1 x (i) β k x (i) k is the mean response. The noise terms, the ɛ i s are assumed to be independent of each other and to be randomly sampled from N(0, σ). The parameters of the model are β 0, β 1,, β k and σ. (University of New Haven) Correlation and Regression 52 / 64

63 Logistic Regression Definition Given a multivariate normal sample, ( ) ( ) x (1) 1,, x (1) k, y 1,, x (n) 1,, x (n) k, y n, the least squares multiple regression equation, is the linear equation that minimizes ŷ = b 0 + b 1 x b k x k, n (ŷ j y j ) 2, j=1 where ŷ j def = b 0 + b 1 x (j) b k x (j) k. (University of New Haven) Correlation and Regression 53 / 64

64 Logistic Regression There must be at least k + 2 data points to do obtain the estimators n b 0, b j s and s 2 def j=1 = (y i ŷ i ) 2 n k 1 of β 0, β j s and σ 2, where b 0, the y intercept, is the unbiased, least square estimator of β 0. b j, the coefficient of x j, is the unbiased, least square estimator of β j. s 2 is an unbiased estimator of σ 2 and s is an estimator of σ. Due to computational intensity, computers are used to obtain b 0, b j s and s 2. (University of New Haven) Correlation and Regression 54 / 64

65 Logistic Regression Example > g.lm=lm(mpg~disp+hp+wt+qsec, data=mtcars) > par(mfrow=c(2,2)) > plot(g.lm) > par(mfrow=c(1,1)) Does the linear model fit? Residuals vs Fitted Normal Q Q Residuals Standardized residuals Fitted values Theoretical Quantiles Standardized residuals Scale Location Standardized residuals Residuals vs Leverage Cook's distance Fitted values Leverage (University of New Haven) Correlation and Regression 55 / 64

66 Logistic Regression Example (cont.) > summary(g.lm) Call: lm(formula = mpg ~ disp + hp + wt + qsec, data = mtcars) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) ** disp hp wt ** qsec Signif. codes: 0 *** ** 0.01 * Residual standard error: on 27 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 4 and 27 DF, p-value: 3.311e-10 (University of New Haven) Correlation and Regression 56 / 64

67 Logistic Regression Inflation Problem: As k increases r 2 increases, but the increase in predictability is illusionary. Solution: Best to use Definition The adjusted coefficient of determination is The p value of a test of is associated with a F -statistic. The p value of a test of is associated with a t-statistic. R 2 adj = 1 n 1 n k 1 (1 R2 ). H 0 : β 1 = = β k = 0 versus H A : not H 0 H 0 : β j = 0 versus H A : β j 0 (University of New Haven) Correlation and Regression 57 / 64

68 Logistic Regression Factor Analysis: One strives for the best fit (largest R 2 and smallest p value associated with the F statistic) with the fewest number of independent variables. Independent variables that are mostly independent of the dependent variable or highly correlated with another dependent variable can be discarded. It is an art. Doing this mechanically (on a machine) is called stepwise regression. (University of New Haven) Correlation and Regression 58 / 64

69 Logistic Regression Logistic Regression Logistic Regression (University of New Haven) Correlation and Regression 59 / 64

70 Logistic Regression A variable that takes on only the values 0 and 1 is a dummy variable. ex, gender, infected, etc If dummy variable is an independent variable, use methods of this chapter. If dummy variable is the dependent variable, use logistic regression. Let Y, the dependent variable is the dummy variable and use Ỹ = ln ( p 1 p ) in place of Y. Here p is the probability of Y = 1. (University of New Haven) Correlation and Regression 60 / 64

71 Chapter #9 R Assignment Chapter #9 R Assignment Chapter #9 R Assignment (University of New Haven) Correlation and Regression 61 / 64

72 Chapter #9 R Assignment (from the book Mathematical Statistics with Applications by Mendenhall, Wackerly and Scheaffer (Fourth Edition Duxbury 1990)) Fifteen alligators were captured and two measurements were made on each of the alligators. The weight (in pounds) was recorded with the snout vent length (in inches this is the distance between the back of the head to the end of the nose). The purpose of using this data is to determine whether there is a relationship, described by a simple linear regression model, between the weight and snout vent length. lnlength ~ lnweight. The authors analyzed the data on the log scale (natural logarithms) and we will follow their approach for consistency. > lnlength = c(3.87, 3.61, 4.33, 3.43, 3.81, 3.83, 3.46, 3.76, , 3.58, 4.19, 3.78, 3.71, 3.73, 3.78) > lnweight = c(4.87, 3.93, 6.46, 3.33, 4.38, 4.70, 3.50, 4.50, , 3.64, 5.90, 4.43, 4.38, 4.42, 4.25) (University of New Haven) Correlation and Regression 62 / 64

73 Chapter #9 R Assignment 1 Create a scatterplot of lnlength lnweight, complete with the regression line. 2 What is the slope and y intercept of the regression line? 3 Predict lnlength when lnweight is five. 4 Use graphs to decide if lnlength lnweight satisfies the requirements for being a linear model. 5 Find a 95% prediction interval for lnlength when lnweight is five. 6 What is the p value of a test of H 0 : β 1 = 0 versus H A : β 1 0? 7 What is the standard error of estimate? 8 What is the coefficient of determination, R 2. 9 What is the explained variation, the unexplained variation and the total variation? 10 What is the F statistic of H 0 : β 1 = 0 versus H A : β 1 0 and what is its degrees of freedom? 11 Using the correlation test, what is the p value of a test that H 0 : ρ = 0 versus H A : ρ 0? (University of New Haven) Correlation and Regression 63 / 64

74 Chapter #9 R Assignment First enter into R: > class(state.x77) # "lm" needs a data.frame not a matrix [1] "matrix" > st = as.data.frame(state.x77) # make state.x77 a data.frame > class(st) # "st" is a data.frame [1] "data.frame" > colnames(st)[4] = "Life.Exp" # no spaces in variable names > colnames(st)[6] = "HS.Grad" # no spaces in variable names 1 Do a multivariate regression with Life.Exp as the response variable and Population, Income, Illiteracy, Murder, HS.Grad, Frost and Area as explanatory variables. (a) Show that the multivariate regression linear model fits this data. (b) What is R 2 and adjusted R 2? (c) Which explanatory variables are relevant at the 0.05 significance level? 2 Do another multivariate regression, but only with explanatory variables Murder and HS.Grad. What is R 2 and adjusted R 2? 3 Comparing the adjusted R 2 in the above two problems, what do you conclude? (University of New Haven) Correlation and Regression 64 / 64