11 Correlation and Regression

Size: px

Start display at page:

Download "11 Correlation and Regression"

Sophie May
6 years ago
Views:

1 11 Correlatio Regressio 11.1 Multivariate Data Ofte we look at data where several variables are recorded for the same idividuals or samplig uits. For example, at a coastal weather statio, we might record the midday temperature, pressure, wid speed directio visibility the daily precipitatio each day for a umber of days. We might, i fact, record these variables at several differet weather statios o the same days. Such data are called multivariate. If there are oly two variables, the data are called bivariate. With bivariate data we ca plot oe variable agaist the other i a scatter plot (see Figure 1). With more tha two variables we ca plot each possible pair ( with some computer programs eve look at three dimesioal plots) but remember that importat features of the structure of the data might remai hidde. Figure 1 shows visibility measuremets i km take at two weather statios o each of thirty days. The data are give i table Liear Associatio Correlatio Liear associatio betwee two variables X Y ca be measured by the covariace of X Y. The covariace of X Y is covar(x, Y ) = E{[X E(X)][Y E(Y )]} = E(XY ) E(X)E(Y ) A chage i the uits of X Y would chage the value of the covariace. To overcome this problem we defie the correlatio betwee X Y to be the covariace divided by each of the stard deviatios. This does ot deped o uits is always betwee 1 1. Positive values deote that large X teds to go with large Y small X with small Y. Negative values deote that large X teds to go with small Y small X with large Y. Note: Zero correlatio betwee two variables does ot, by itself, mea that they are statistically idepedet. Simple estimators of covariace correlatio are available. To estimate the covariace, covar(x, Y ), the estimator S xy /( 1) is ofte used, where S xy = = (x i x)(y i ȳ) x i y i 1 is the sum of products of x y. To estimate the correlatio, ρ(x, Y ), the estimator is ofte used where r xy = S xy Sxx S yy x i y i S xx = = (x x) 2 ( x 2 i 1 ) 2 x i 1

2 is the sum of squares of x S yy = = (y ȳ) 2 ( yi 2 1 ) 2 y i is the sum of squares of y The value of r xy is always such that 1 r xy 1. We oly obtai r xy = ±1 if the poits o a scatter plot lie exactly alog a straight lie. Note. 1. Noe of this implies causality. 2. We ca calculate S xx as or x 2 i ( x i ) 2 / x 2 i x We ca calculate S xy as or x i y i ( x i )( y i )/ x i y i xȳ. 4. If the distributios of both tests are ormal, we ca test the ull hypothesis that the true correlatio is zero. If we ca ot assume ormality the we ca use rak correlatio (see below) Example For the data i table 1 we calculate the followig sums. Let x i be the visibility at A y i be the visibility at B o day i. x i = 328 y i = 342 x 2 i = 6886 y 2 i = 6934 x i y i =

3 Statio Statio Statio A B A B A B Table 1: Visibility measuremets at two statios o thirty days Thus So the sample correlatio coefficiet is S xx = = S yy = = S xy = = r = = Rak Correlatio Because we ofte do ot kow the appropriate probability distributio for sets of data, if we wish to test the sigificace of associatios we ofte use a oparametric test based o raks. We simply calculate the correlatio of the raks this ca be compared with stard tables to test the ull hypothesis that there is o associatio. For example, for the data i Table 1 the rak correlatio betwee visibilities at the two statios is this is sigificat i a two-tailed test at the 0.1% level. Note that, as well as ot requirig ormality, the rak correlatio test does ot require that the relatioship is liear, simply that it is mootoic, that is that it is either icreasig or decreasig Regressio Itroductio to regressio Usig the data i Table 1, ca we fid a way to predict visibility at Statio B from the visibility at Statio A? If we ca, this does ot mea that the visibility at A causes the visibility at B. It simply meas that the iformatio from A is useful if we wat to kow what is happeig at B. No doubt the visibilities at both statios are affected by the geeral prevailig weather coditios. I other cases, particularly whe data are collected from deliberate experimets, as i below, we might be able to ifer that oe variable causes aother. I correlatio we cosider the relatioship betwee two variables o a equal footig. I regressio we build a model where the value of oe variable, called the depedet variable, depeds o the values take by oe or more other variables called idepedet variables or regressors or explaatory variables or covariates or cocomitat variables. 3

4 Visibility (km) at Statio B Visibility (km) at Statio A Figure 1: Scatter plot: Visibility measuremets (km) at two statios o thirty days where There are may differet kids of regressio models. Oe of the simplest is y i = α + βx i + ε i, y i is observatio umber i o the depedet variable, x i is observatio umber i o the explaatory variable, ε i is a rom error α β are parameters whose values usually have to be estimated. Let us suppose that we have observatios, where a observatio cosists of a pair of values x i, y i. Without loss of geerality we assume that ε 1,..., ε are take from a distributio with zero mea. It is ofte also assumed that they are idepedet, ormally distributed have the same variace. We will make these assumptios uless otherwise stated. This model is called a ordiary liear regressio o a sigle covariate with ormal errors. See Figure 2. 4

5 y y 1 y 2 * (x 1, y 1 ) y = α + βx ε 1 ε 2 * (x 2, y 2 ) x 1 x 2 x Figure 2: Ordiary least squares regressio o a sigle covariate Example I order to ivestigate how the breakig load of rope of a certai type chages with wear, samples are subjected to simulated workig coditios for differet legths of time, up to 1600 hours. the breakig loads of the samples are determied. The data are as follows Aalysis Workig time Breakig load Workig time Breakig load x hours y toes x hours y toes Is the model appropriate? A scatter plot of y agaist x should be used. Are the poits clustered aroud a straight lie? Figure 3 shows the data from the example. A straight lie, actually the regressio lie of breakig stregth o workig time, has bee superimposed. 5

6 y x Figure 3: Poits clustered roud a straight lie: Breakig Stregth y toes agaist workig time x hours for rope. The usual method of estimatio is called least squares. Excel will calculate the estimates. It will also provide a test of the ull hypothesis that β = 0. This ull hypothesis represets the idea that the depedet variable does ot, i fact, deped o the explaatory variable. The least squares estimates are the values of α β which miimise the sum of squared residuals where, if our least squares estimates of α β are ˆα ˆβ. The residual for observatio i is ˆε i = y i ˆα ˆβx i. The sum of squared residuals (or residual sum of squares) is W = [y i ˆα ˆβx i ] 2. Least squares chooses the values of ˆα ˆβ so that W is miimised. It is easy to show that ˆα = ȳ ˆβ x where the sum of products of x y is ˆβ = S xy /S xx, the sum of squares of x is S xy = (x i x)(y i ȳ), S xx = (x i x) 2, 6

7 the sample mea of y is the sample mea of x is It is easily show that ȳ = y i / x = x i /. S xy = S xx = x i y i 1 x i y i ( x 2 i 1 ) 2 x i. After the parameters of the model have bee determied we ca check some of the model assumptios. We ca: 1. Plot the regressio lie y = ˆα + ˆβx o our graph of the data to check that it is a reasoable fit. (See Figure 3). 2. Examie the residuals, ˆε i = y i ˆα ˆβx i. If the model is reasoable the the residuals will be approximately idepedet idetically distributed with zero mea. It is ofte a good idea to plot the residuals agaist: (a) origial data order i. Is there ay chage i mea or variace with time, positio or ay other aspect of the experimet? (b) the idepedet variable x. Is there ay curvature or ay depedece of the variace o x? We ca also check the idepedece of the residuals. Does there seem to be ay relatioship betwee ˆε i ˆε i 1, ˆε i 2,...? Tests are available but a visual ispectio of a plot may suffice Example cotiued We ca do the calculatios as follows. The umber of observatios is = 27. We calculate totals as follows. 27 x i = y i = the 27 x 2 i = x i y i = S xy = = S xx = = The meas are 7

8 x = = Thus ȳ = = ˆβ = S xy = 5690 S xx = ˆα = ȳ ˆβ x = = So, our estimate of the mea breakig stregth for rope samples after x hours is ŷ = x this is the equatio of the lie plotted o figure 3. Now we ca calculate the residuals. For example, the fourth residual is ˆε 4 = y 4 ŷ 4 = y 4 (ˆα + ˆβx 4 ) = 2.43 ( ) = 0.33 Figure 4 shows the residuals plotted agaist workig time (x). There does ot appear to be cause for cocer Problems The data below are from a series of tests of a h-held istrumet for measurig the depth of water. All of the depths are i metres. The table gives accurate depths x measured by a survey vessel the readigs y give by the istrumet at the same locatios. Depth Readig Depth Readig Depth Readig x y x y x y Plot the data o a graph (with x o the horizotal axis). Does there appear to be a liear relatioship? 2. Calculate the sample correlatio coefficiet betwee x y. 3. Estimate the parameters α, β of a regressio of y o x. 4. Plot your fitted regressio lie o your graph of the data commet o the fit. 5. Calculate the residuals, plot them agaist x commet o the resultig graph. 8

9 residual x Figure 4: Residuals agaist workig time (x) hours, rope example 9

1 Inferential Methods for Correlation and Regression Analysis

1 Inferential Methods for Correlation and Regression Analysis 1 Iferetial Methods for Correlatio ad Regressio Aalysis I the chapter o Correlatio ad Regressio Aalysis tools for describig bivariate cotiuous data were itroduced. The sample Pearso Correlatio Coefficiet