Dependence and scatter-plots. MVE-495: Lecture 4 Correlation and Regression

Transcription

1 Dependence and scatter-plots MVE-495: Lecture 4 Correlation and Regression It is common for two or more quantitative variables to be measured on the same individuals. Then it is useful to consider what relationship there ma be between the variables. Scatter-plot represents one variable plotted against another. Etension of a rubber spring as a function of applied force Fuel econom (miles travelled on one gallon) as a function of speed. Data on ear 1997 for various US car models There seems to be a clear linear relationship between the variables, known as Hooke s law, although there is some non-linear dependence for small values of the force. The most economic speeds are about 50 km/h and km/h for man models.

2 Linear relations and correlation A particularl common and important relationship which ma arise between variables is when the points lie approimatel upon a straight line. A statistical tool which measures the strength of such a linear relationship is the correlation coefficient between the variables. Assuming that the variables being measured are X and Y, and that the observations are ( 1, 1 ), ( 2, 2 ),..., ( n, n ), then the sample correlation coefficient is Properties of the correlation coefficient Sample correlation coefficient r is an estimate of its theoretical counterpart, correlation coefficient ρ between random variables defined as ρ = cov(x, Y ), var X var Y r = S XY SXX S YY, where where cov(x, Y ) = E [ (X E X)(Y E Y ) ] is the covariance between X and Y. S XY = Σ i ( i )( i ȳ) = Σ i i i n ȳ S XX = Σ i ( i ) 2 = Σ i 2 i n 2 S YY = Σ i ( i ȳ) 2 = Σ i 2 i nȳ 2. Linear vs. non-linear relations Value of both r and ρ alwas lies between 1 and +1. When it is equal (close) to 1 or to +1, X and Y are (almost perfectl) linearl dependent. Hence the points in the scatter plot lie eactl on (close to) a straight line with negative or positive slope respectivel. Positive (negative) value of r means that the larger values of X tend to correspond to larger (smaller) values of Y. If the correlation coefficient is near zero, then there is no evidence of a linear relation between X and Y. There ma however still be a strong non-linear relation. The value of the correlation coefficient for the points in this scatter-plot which lie eactl on the curve = 2 is zero!

3 Linear relation and causation If r is far from zero, so there is strong evidence of a linear association, this does not impl that a change in the value of X will cause a change in the value of Y. It is etremel eas to fall into the error of believing that this does prove something of the kind, especiall if the value of Y measures some biological value (e.g., blood pressure) which treatment aims to change. There are at least two other possibilities: It is changes in the value of Y which cause changes in the value of X. Both X and Y are being controlled b some third unobserved variable (spurious correlation). Onl additional eperimental or surve work can establish such a causal link but this does not rest on correlation alone. Fitting a straight line Assuming that there is good reason to believe that X values could be used to predict Y values, the scatter-plot shows no clear evidence of a non-linear relation between X and Y, it is sensible to fit Y values to X values b means of a straight line which equation will be ŷ = a + b, Note that we are here treating X and Y asmmetricall regarding X as a predictor variable and Y as a response or outcome variable. We are in essence assuming that changes in X can be epected to cause changes in Y. The precautions noted earlier in making such assumptions must of course be observed in this case. where ŷ denotes the fitted value.

4 Residuals Having constructed our line, we shall then have two distinct points in our scatter diagram corresponding to each observation ( i, i ). The first is the observation point itself, and the second is the point ( i, ŷ i ), the fitted point on the straight line. The difference e i = i ŷ i is called the i-th residual. The usual method of fitting the straight line is the method of least squares in which the constants a and b are chosen so that S 2 = e e e2 n min It can be proved that the values i e i S 2 = = 0 i e 2 i = i e i = 6.38 S 2 = i e 2 i = b = S XY /S XX a = ȳ b minimise the sum of squares of the residuals. Regression model It can be shown that S YY = Σ i ( i ȳ) 2 = Σ i ( i ŷ i ) 2 + Σ i (ŷ i ȳ) 2 or, in other terms, SS total = SS residual + SS regression. The second term can also be written as SS regression = r 2 SS total, Therefore theoretical model of regression is i = α + β i + ε i, where the intercept α and the slope β are unknown parameters to be estimated and ε i, in the simplest case, are assumed to be independent and commonl N(0, σ 2 )-distributed with unknown σ. where r is the correlation coefficient. Thus r 2 represents the proportion of the total variation in the response values which is accounted for as being caused b variation in the X values. The first term represents the part of the variation in the responses which is presumed to be random.

5 Testing significance of linear regression Assuming the regression model above is correct, we can use it to estimate man things. Each of the estimates has an approimate normal distribution, with its own standard error, and from this we can construct as usual t-confidence intervals for the true value of the quantit being estimated. The ke question is whether the regression is significant, i.e. is the slope β is non-zero. For β we have: 1 Estimate: b = S XY /S XX ; Associated standard error s/ S XX ; Corresponding (1 α) confidence interval for β: b ± t n 2,α/2 s/ S XX. where s 2 = S 2 /(n 2) the mean sum of squares of the residuals. Thus the true value of the slope β is covered b the random interval (b t n 2,α/2 s/ S XX, b + t n 2,α/2 s/ S XX ) with probabilit 1 α. Therefore, if this interval does not contain 0, β is significantl non-zero at 100α% error level. This means that it is still possible that β = 0, but the probabilit to observe such etreme values of b is less than α. (1) Confidence intervals for linear regression Similarl, one can check various values of the other characteristics: 2 α the intercept of the regression line on the Y -ais: Estimate: a = ȳ b ; Associated standard error: s 1/n + 2 /S XX ; Corresponding (1 α) confidence interval for α: a ± t n 2,α/2 s 1/n + 2 /S XX. 3 The MEAN value of Y when X = 0 : Estimate: ŷ( 0 ) = a + b 0 ; Associated standard error: s 1/n + ( 0 ) 2 /S XX ; Corresponding (1 α) confidence interval for the mean response when X = 0 : a + b 0 ± t n 2,α/2 s 1/n + ( 0 ) 2 /S XX. 4 An INDIVIDUAL value of Y when X = 0 : Estimate: ŷ( 0 ) = a + b 0 ; Associated st. error: s 1 + 1/n + ( 0 ) 2 /S XX ; Corresponding (1 α) prediction interval: a + b 0 ± t n 2,α/2 s 1 + 1/n + ( 0 ) 2 /S XX.

6 Regression in Matlab Matlab s fitlm function fits a regression line (linear model) to the data. For the Hooke s law eample in the beginning, lm=fitlm(force,etension) plot(lm) Tping lm shows further details: Solid red is the regression line, but an straight line within the dotted curves would not contradict data at 95% confidence. Significance of coefficients The line reads that the response variable is modelled as a linear combination of a constant and one eplanator variable. Estimate gives the corresponding coefficients, so that etension is modelled as a linear function force. SE is the standard error of these coefficient estimates, i.e. s/ S XX for the gradient estimate as in Eq. (1). Residuals etension ( force) are stored in vector lm.residuals.raw and can be used to check the regression assumptions. pvalue is another machiner to check the regression significance. If this value is less than the error level ou have chosen to work with, sa 5% (equivalentl, 95% confidence), then the corresponding coefficient is significantl non-zero (with this confidence). Since < 0.05, the regression is significant at 95% confidence level. It is natural to epect that zero force implies zero etension, but substituting force=0 into the regression equation gives etension=intercept= However, the corresponding pvalue is > 0.05, thus the intercept is not significant and could be set to 0 with 5% error associated with such a decision.