Regression and correlation

Transcription

1 Cotets 43 Regressio ad correlatio 1. Regressio. Correlatio Learig outcomes You will lear how to explore relatioships betwee variables ad how to measure the stregth of such relatioships. You should ote from the outset that simply establishig a relatioship is ot eough. You may establish, for example, a relatioship betwee the umber of hours a perso works i a week ad their hat size. Should you coclude that workig hard causes your head to elarge? Clearly ot, ay relatioship existig here is ot causal! Time allocatio You are expected to sped approximately te hours of idepedet study o the material preseted i this workbook. However, depedig upo your ability to cocetrate ad o your previous experiece with certai mathematical topics this time may vary cosiderably. 1

2 Regressio 43.1 Itroductio Problems i egieerig ofte ivolve the exploratio of the relatioship(s) betwee two or more variables. The techique of regressio aalysis is very useful ad well-used i this situatio. This Sectio will look at the basics of regressio aalysis ad should eable you to apply regressio techiques to the study of relatioships betwee variables. Just because a relatioship exists betwee two variables does ot ecessarily imply that the relatioship is causal. You might fid, for example that there is a relatioship betwee the hours a perso speds watchig TV ad the icidece of lug cacer. This does ot ecessarily imply that watchig TV causes lug cacer. Assumig that a causal relatioship does exist, we ca measure the stregth of the relatioship by meas of a correlatio coefficiet discussed i the ext Sectio. As you might expect, tests of sigificace exist which allow us to iterpret the meaig of a calculated correlatio coefficiet. Prerequisites Before startig this Sectio you should... Learig Outcomes After completig this Sectio you should be able to... 1 study Descriptive Statistics usig Workbook 36 make sure you ca fid the expectatio ad variace of sums of variables usig Workbook uderstad the terms idepedet ad depedet variables. 4 uderstad the terms biased ad ubiased estimators. uderstad what is meat by the terms regressio aalysis ad regressio lie. uderstad the method of least squares for fidig a lie of best fit.

3 1. Regressio As we have already oted, relatioship(s) betwee variables are of iterest to egieers who may wish to determie the degree of associatio existig betwee idepedet ad depedat variables. Kowig this ofte helps egieers to make predictios ad, o this basis, to forecast ad pla. Essetially, regressio aalysis provides a soud kowledge base for which accurate estimates of the values of a depedet variable may be made oce the values of related idepedet variables are kow. It is worth otig that i practice the choice of idepedet variable(s) may be made by the egieer o the basis of experiece ad/or prior kowledge sice this may idicate to the egieer which idepedet variables are likely to have a substatial ifluece o the depedet variable. I summary, we may state that the priciple objectives of regressio aalysis are: (a) to eable accurate estimates of the values of a depedet variable to be made from kow values of a set of idepedet variables; (b) to eable estimates of errors resultig from the use of a regressio lie as a basis of predictio. Note that if a regressio lie is represeted as y = f(x) where x is the idepedet variable, the the actual fuctio used (liear, quadratic, higher degree polyomial etc.) may be obtaied via the use of a theoretical aalysis or perhaps a scatter diagram (see below) of some real data. Note that a regressio lie represeted as y = f(x) iscalled a regressio lie of y o x. Scatter Diagrams A useful first step i establishig the degree of associatio betwee two variables is the plottig of a scatter diagram. Examples of pairs of measuremets which a egieer might plot are: (a) volume ad pressure; (b) acceleratio ad tyre wear; (c) curret ad magetic field; (d) torsio stregth of a alloy ad purity. If there exists a relatioship betwee measured variables, it ca take may forms. I this work, eve though a outlie itroductio to o-liear regressio is give at the ed of the Workbook, we shall focus o the liear relatioship oly. I order to produce a good scatter diagram you should follow the steps give below: 1. Give the diagram a clear title ad idicate exactly what iformatio is beig displayed;. Choose ad clearly mark the axes; 3. Choose carefully ad clearly mark the scales o the axes; 4. Idicate the source of the data. 3 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

4 Examples of scatter diagrams are show below. Figure 1 Figure Figure 3 Figure 1 shows a associatio which follows a curve, possibly expoetial, quadratic or cubic; Figure shows a reasoable degree of liear associatio where the poits of the scatter diagram lie i a area surroudig a straight lie; Figure 3 represets a radomly placed set of poits ad o liear associatio is preset betwee the variables. Note that i figure, the word reasoable is ot defied ad that while poits close to the idicated straight lie may be explaied by radom variatio, those far away may be due to assigable variatio. The rest of this uit will deal with liear associatio oly although it is worth otig that techiques do exist for trasformig may o-liear relatioships ito liear oes. We shall ivestigate liear associatio i two ways, firstly by usig educated guess work to obtai a regressio lie by eye ad secodly by usig the well-kow techique called the Method of Least Squares. Regressio Lies by Eye Note that at a very simple level, we may look at the data ad, usig a educated guess, draw a lie of regressio by eye through a set of poits. However, fidig a regressio lie by eye is usatisfactory as a geeral statistical method sice it ivolves guess-work i drawig the lie with the associated errors i ay results obtaied. The guess-work ca be removed by the method of least squares i which the equatio of a regressio lie is calculated usig data. Essetially, we calculate the equatio of the regressio lie by miimisig the sum of the squared vertical distaces betwee the data poits ad the lie. The Method of Least Squares (i) A Elemetary View We assume that a experimet has bee performed which has resulted i pairs of values, say (x 1,y 1 ), (x,y ),, (x,y ) ad that these results have bee checked for approximate liearity HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 4

5 o the scatter diagram give below. y P (x,y ) P 1 (x 1,y 1 ) Q y = mx c Q Q 1 P (x,y ) O x The vertical distaces of each poit from the lie y = mx c are easily calculated as y 1 mx 1 c, y mx c, y 3 mx 3 c y mx c These distaces are squared to guaratee that they are positive ad calculus is used to miimise the sum of the squared distaces. Effectively we are miimizig the sum of a two-variable expressio ad eed to use partial differetiatio. If you wish to follow this up ad look i more detail at the techique, ay good book (egieerig or mathematics) cotaiig sectios o multi-variable calculus should suffice. We will ot look at the details of the calculatios here but simply ote that the process results i two equatios i the two ukows m ad c beig formed. These equatios are: xy m x c x =0 (i) ad y m x c =0 (ii) The secod of these equatios (ii) immediately gives a useful result. Rearragig the equatio we get y m x c =0 or, put more simply ȳ = m x c where ( x, ȳ) isthe mea of the array of data poits (x 1,y 1 ), (x,y ),, (x,y ). This shows that the mea of the array always lies o the regressio lie. Sice the mea is easily calculated, the result forms a useful check for a plotted regressio lie. Esure that ay regressio lie you draw passes through the mea of the array of data poits. Elimiatig c from the equatios gives a formula for the gradiet m of the regressio lie, this is: 5 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

6 xy x y m = ( ) x x This is ofte writte as m = S xy S x The quatity S x is, of course, the variace of the x-values. The quatity S xy is kow as the covariace (of x ad y) ad will appear agai later i this Workbook whe we measure the degree of liear associatio betwee two variables. Kowig the value of m eables us to obtai the value of c from the equatio ȳ = m x c Summary The least squares regressio lie of y o x has the equatio ȳ = m x c. Remember that: m = xy x y ( ) x ad that c is give by the equatio c =ȳ m x x It should be oted that the coefficiets m ad c obtaied here will give us the regressio lie of y o x. This lie is used to predict y values give x values. If we eed to predict the values of x from give values of y we eed the regressio lie of x o y. The two lies are ot the same except i the (very) special case where all of the poits lie exactly o a straight lie. It is worth otig however, that the two lies cross at the poit ( x, ȳ). It ca be show that the regressio lie of x o y is give by x = m y c where m = xy x y ( ) y ad c = x m ȳ y HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 6

7 Example A warehouse maager of a compay dealig i large quatities of steel cable eeds to be able to estimate how much cable is left o of his partially used drums. A radom sample of twelve partially used drums is take ad each drum is weighed ad the correspodig legth of cable measured. The results are give i the table below: Weight of drum ad cable (x) kg. Measured legth of cable (y) m Fid the least squares regressio lie i the form y = mx c ad use it to predict the legths of cable left o drums whose weights are: (i) 35 kg (ii) 85 kg (iii) 100 kg I the latter case state ay assumptios which you make i order to fid the legth of cable left o the drum. Solutio Excel calculatios give x = 700, x = 4400, the formulae xy x y m = ( ) x ad c =ȳ m x x give m =3ad c = 0. Our regressio lie is y =3x 0. Hece, the required predicted values are: y = 1860 xy = so that y 35 = =85 y 85 = = 35 y 100 = = 80 all results beig i metres. To obtai the last result we have assumed that the liearity of the relatioship cotiues beyod the rage of values actually take. 7 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

8 A article i the Joural of Soud ad Vibratio 1991(151) explored a possible relatioship betwee hypertesio (defied as blood pressure rise i mm of mercury) ad exposure to oise levels (measured i decibels). Some data give is as follows: Noise Level (x) Blood pressure rise (y) Noise Level (x) Blood pressure rise (y) (a) Draw a scatter diagram of the data. (b) Commet o whether a liear model is appropriate for the data. (c) Calculate a lie of best fit of y o x for the data give. (d) Use your regressio lie predict the expected rise i blood pressure for a exposure to a oise level of 97 decibels. Your solutio HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 8

9 (a) Eterig the data ito Excel ad plottig gives Blood Pressure icrease versus recorded soud level Soud Level (Db) (b) A liear model is appropriate. (c) Excel calculatios give x = 1656, x = , y =86, xy = 7654 so that m = ad c = Our regressio lie is y =0.1743x (d) The predicted value is: y 97 = = 6.78 mm mercury. Blood Pressure rise (mm Mercury) The Method of Least Squares (ii) A Modellig View We take the depedet variable Y to be radom variable whose value, for a fixed value of x depeds o the value of x ad a radom error compoet say e ad we write Y = mx c e Adoptig the otatio of coditioal probability, we are lookig for the expected value of Y for a give value of x. The expected value of Y for a give value of x is deoted by E(Y x) =E(mx c e) =E(mx c)e(e) The variace of Y for a give value of x is give by the relatioship V (Y x) =V (mx c e) =V (mx c)v (e), assumig idepedece. If µ Y x represets the true mea value of Y for a give value of x the µ Y x = mx c, assumig a liear relatioship holds, is a straight lie of mea values. If we ow assume that the errors e are distributed with mea 0 ad variace σ we may write E(Y x) = E(mx c)e(e) = mx c sice E(e) = 0. 9 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

10 ad V (Y x) =V (mx c)v (e) =σ sice V (c) =0. This implies that for each value of x, Y is distributed with mea mx c ad variace σ. Hece whe the variace is small the observed values of Y will be close to the regressio lie ad whe the variace is large, at least some of the observed values of Y may ot be close to the lie. Note that the assumptio that the errors e are distributed with mea 0 ad variace σ may be made without loss of geerality. If the errors had ay other mea, we could subtract it ad the add the mea to the value of c. The ideas are illustrated i the followig diagram. y E(y x) =mx c e p y p O x 1 x x 3 x p x The regressio lie is show passig through the meas of the distributios for the idividual values of x. Ay radomly selected value of y may be represeted by the equatio y p = mx p c e p where e p is the error of the observed value of y its expected value, amely E(Y x p )=µ y xp = mx p c Note that e p = y p mx p c so that the sum of the squares of the errors is give by S = e p = (y p mx p c) ad we may miimize the quatity S by usig the method of least squares as before. The mathematical details are omitted as before ad the equatios obtaied for m ad c are as before, amely xy x y m = ( ) x ad c =ȳ m x. x Note that sice the error e p i the pth observatio essetially describes the error i the fit of the model to the pth observatio, the sum of the squares of the errors e p will ow be used to allow us to commet o the adequacy of fit of a liear model to a give data set. HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 10

11 Adequacy of Fit We ow kow that the variace V (Y x) =σ is the key to describig the adequacy of fit of our simple liear model. I geeral, the smaller the variace, the better the fit although you should ote that it is wise to distiguish betwee poor fit ad a large error variace. Poor fit may suggest, for example, that the relatioship is ot i fact liear ad that a fudametal assumptio made has bee violated. A large value of σ does ot ecessarily mea that a liear model is a poor fit. It ca be show that sum of the squares of the errors say SS E ca be used to give a ubiased estimator ˆσ of σ via the formula ˆσ = SS E p where p is the umber of idepedet variables used i the regressio equatio. I the case of simple liear regressio p =sice we are usig just x ad c ad the estimator becomes: ˆσ = SS E The quatity SS E is usually used explicitly i formulae whose purpose is to determie the adequacy of a liear model to explai the variability foud i data. Two ways i which the adequacy of a regressio model may be judged are give by the so-called Coefficiet of Determiatio ad the Adjusted Coefficiet of Determiatio. The Coefficiet of Determiatio Deoted by R, the Coefficiet of Determiatio is defied by the formula R =1 SS E SS T where SS E is the sum of the squares of the errors ad SS T is the sum of the squares of the totals give by (y r ȳ) = yi ȳ. The value of R is sometimes referred loosely as represetig the amout of variability explaied or accouted for by a regressio model. For example, if after a particular calculatio it was foud that R =0.884, we could say that the model accouts for about 88% of the variability foud i the data. However, deductios made o the basis of the value of R should be treated cautiously, the reasos for this are embedded i the followig properties of the statistic. It ca be show that: (a) 0 R 1 (b) a large value of R does ot ecessarily imply that a model is a good fit; (c) addig a regressor variable (simple regressio becomes multiple regressio) always icreases the value of R. This is oe reaso why a large value of R does ot ecessarily imply a good model; (d) models givig large values of R ca be poor predictors of ew values if the fitted model does ot apply at the appropriate x-value. Fially, it is worth otig that to check the fit of a liear model properly, oe should look at plots of residual values. I some cases, tests of goodess-of-fit are available although this topic is ot covered i this workbook. 11 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

12 The Adjusted Coefficiet of Determiatio Deoted (ofte) by Radj, the Adjusted Coefficiet of Determiatio is defied as Radj =1 SS E/( p) SS T /( 1) where p is the umber of variables i the regressio equatio. For the simple liear model, p = sice we have two regressor variables x ad 1. It ca be show that: (a) R adj is a better idicator of the adequacy of predictive power tha R sice it takes ito accout the umber of regressor variables used i the model; (b) R adj does ot ecessarily icrease whe a ew regressor variable is added. Both coefficiets claim to measure the adequacy of the predictive power of a regressio model ad their values idicate the proportio of variability explaied by the model. For example a value of R or R adj = may be iterpreted as idicatig that a model explais 97.51% of the variability it describes. For example, the drum ad cable example cosidered previously gives the results outlied below with R =96. ad R adj =0.958 I geeral, Radj is (perhaps) more useful tha R for comparig alterative models. I the cotext of a simple liear model, R is easier to iterpret. I the drum ad cable example we would claim that the liear model explais some 96.% of the variatio it describes. Drum & Cable x Cable Legth y xy Predicted Error (x) (y) Values Squares Sum of x Sum of x Sum of y Sum of y Sum of xy SSE = = 700 = 4400 = 1860 = = m =3 c = 0 SST = R = Radj = HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 1

13 Use the drum ad cable data give origially ad set up a spreadsheet to verify the values of the Coefficiet of Variatio ad the Adjusted Coefficiet of Variatio. Your solutio As per the table above givig R =0.96 ad R adj = Sigificace Testig for Regressio Note that the results i this sectio apply to the simple liear model oly. Some additios are ecessary before the results ca be geeralized. The discussios so far pre-suppose that a liear model adequately describes the relatioship betwee the variables. We ca use a sigificace test ivolvig the distributio to decide whether or ot y is liearly depedet o x. We set up the followig hypotheses: H 0 : m =0 ad H 1 : m 0 It may be show that the test statistic is F test = SS R SS E /( ) where SS R = SS T SS E ad rejectio at the 5% level of sigificace occurs if F test >F 0.05,1, Note that we have oe degree of freedom sice we are testig oly oe parameter (m) ad that deotes the umber of pairs of (x, y) values. A set of tables givig the 5% values of the F -distributio is give at the ed of this Workbook. Example Test to determie whether a simple liear model is appropriate for the data previously give i the drum ad cable example above. 13 HELM (VERSION 1: April 9, 004): Workbook Level : Regressio

14 Solutio We kow that SS T = SS R SS E where SS T = y ( y) is the total sum of squares (of y) sothat (from the spreadsheet above) we have: SS R = = Hece SS R F test = SS E /( ) = /(1 ) = 5.5 From tables, the critical value is F 0.05,1,10 = Hece, sice F test >F 0.05,1,10,wereject the ull hypothesis ad coclude that m 0. HELM (VERSION 1: April 9, 004): Workbook Level : Regressio 14