Regression and Correlation

Size: px

Start display at page:

Download "Regression and Correlation"

Annice Gilmore
6 years ago
Views:

1 1 Topic 6 Regression and Correlation Contents 6.1 Introduction Correlation Definition Calculation of r Least Squares Regression Line General Equation of Regression Line Theoretical Derivation Calculation of the Regression Line Confidence Intervals Hypothesis Tests Hypothesis Test for b Hypothesis Test for r Multiple Linear Regression Summary and Assessment Learning Objectives draw scatterplots to investigate for a relationship between x and y define the Product-Moment Correlation Co-efficient, r interpret values of r as giving a good or bad linear relationship between variables calculate r using a methodical manual method distinguish between an explanatory variable and a response variable calculate the equation of a Least Squares Regression Line through a set of points calculate confidence intervals for the slope of the regression equation carry out hypothesis tests for b and r solve a multiple linear regression problem with the aid of a computer package

2 2 TOPIC 6. REGRESSION AND CORRELATION 6.1 Introduction One of the most important things that should have been learnt in the last chapter is how to investigate whether there is a significant difference between two samples. It is often also interesting to know what type of relationship there is between two sets of variables. This can be a very useful discovery as the statistics obtained can then be used in decision making. The goal is to use statistical data to derive a procedure that will allow a prediction, with a determined level of confidence, of further observations or measurements in the future. A typical example to be discussed below is the following. Data is collected on the housing market. It is a good guess that there will be a dependency between the size of house and the price. The latter is simply a number and the former can be measured through number of floors, number of rooms etc. There are also categorical parameters that determine the price, like "good location" or "close to grocery facilities". The collected data can be plotted on a scatterplot (x-y Cartesian diagram) and some relationship can be looked for. Mathematical tools can then be developed that will allow a function to be calculated (expressing price in terms of number of rooms etc.) that will represent this dependency. Moreover, it will be desirable to have numerical values representing the confidence that should be placed in accepting this function. This topic will mainly be concerned with linear relationships although others will be briefly mentioned. The starting point in looking a relationship between two sets of results is by making a visual inspection of given sets of data. Scatterplots are graphs that show the relationship between two quantitative variables. Examples are Fuel consumption per speed; Fuel consumption per weight; Spending for leisure per income; Prices for hard disks per memory size; Performance of PC s per memory (RAM or ROM) size; Example Problem: Consider the following average high and low temperatures in Montreal (in degrees Celsius) and the measured gas consumption for heating (in litres) for a house. Jan Feb March April May June high low gas

3 6.2. CORRELATION 3 July Aug Sept Oct Nov Dec high low gas Solution: The following is a data plot of gas consumption against low and high temperatures. In both cases the data plot suggests a strong linear relationship. In general when interpreting a scatterplot, first look for an overall pattern, and describe form, direction, and strength. Two variables are positively associated if above average values of one variable tend to result in above average values of the other. Two variables are negatively associated if above average values of one variable tend to result in below average values of the other. In the example above the data are negatively associated. 6.2 Correlation Definition Correlation is a number that measures strength and direction of the linear relationship between two data sets. If random variables x and y are given, each consisting of n individual data, then the correlation, r, between x and y is defined as

4 4 TOPIC 6. REGRESSION AND CORRELATION where and are sample means and s x and s y are the standard deviations of the samples. It can be shown that correlation has the following properties: r does not have a dimension; correlation is invariant under change of units of measurements; r is independent from interchanging the role of x and y; r is always a number between -1 and 1. r is defined as the Product Moment Correlation Coefficient. Values close to -1 or 1 indicate strong linear relationships, whilst values near 0 indicate a lack of relationship. Negative values mean that as one variable increases, the other decreases. Some examples of scatterplots are given below. Note The equation given above is not usually the most efficient way to calculate r. Even although it may look more complicated, a better one for computational purposes is: $ %! "!# "&'#)( *+&-, %.&'#)( *+&-, /

5 6.3. LEAST SQUARES REGRESSION LINE Calculation of r Example Problem: Find the correlation coefficient for the data set: x y Solution: For the equation 021 3!46587:9;4<5=467 >?3!4<5"@A9)BC4D5E+@-F?3!4<7.@A9)BC467E+@GF to be used, it is best to set up a table. x y x 2 y 2 xy JILK <7 1MIN 4<5 1POQK 4<7 1JILKO 4<587 1SRUT 9 > 9 V 9 \ ] \ \ c Also, n = 4. Now, 021 RWV RUT ILKWVXIN?RYVZOQK So, 0[1 KWV_^.I 1`K.abKc This is close to 0 so it shows that there is little relationship between x and y. 6.3 Least Squares Regression Line It is now desired to draw the "best straight line" through the points on the scatterplot. Given two random variables (data sets), when a relationship is being investigated, one of the variables is usually considered to be an explanatory variable (often denoted x), and the other a response variable (often denoted y). A regression line is a line that describes how the response variable y changes when the explanatory variable x changes. If plotted on a graph, this line will indeed be the best one that can be drawn through the points. d

6 t i q m q e 6 TOPIC 6. REGRESSION AND CORRELATION General Equation of Regression Line The general equation of a straight line is usually given as, y = mx + c where m is the gradient (slope) of the line and y the intercept with the y axis. In statistics, the constants are usually referred to instead by the letters a and b. The equation y = a + bx is calculated as follows: e - = f!g<h8i f!g<hkj'l)mcg<hon+j g6h g<i prq g isl g h f Note that n is the number of points and that b has to be calculated first as a depends on it. Regression lines are used, among others, for prediction. Of course, a regression line is only useful if a linear relationship between the data sets is expected Theoretical Derivation q`pvu e This section aims to show how the regression equations are formed. The notation t is used to denote a regression line for the data set x and y. Here h stands for the predicted value of the response variable (as opposed to the observed i value). The difference of the predicted data and observed data of any point, y, is given by the valuew q ialwi t pxu e, is called the error (or residual) of the observed data honylwi y, and is the vertical distance between y and the regression line. The least squares regression line is the regression line that minimises the sum of (the squares of) errors. Thus, it is desired to minimise z q g w j g {}m p u e hn~liu j The problem is then to find values a and b that will make S as small as possible. Partial differentiation of S with respect to both a and b and equating the results to zero will achieve this. Two simultaneous equations are formed and they can then be solved to give the values a and b quoted earlier.

7 ƒ ƒ 6.3. LEAST SQUARES REGRESSION LINE 7 = ƒ! < "! ˆ < = < ƒ! 6 " A )ŠC 6 o + Œr : Notes Ž In least squares regression analysis the roles of x and y are distinct. Changing them round gives different regression lines (what is happening is that rather than minimising errors vertically, they are being minimised horizontally). Ž A change of one standard deviation in x corresponds to a change of r standard deviations in y. Ž It is always the case that 6 ` Calculation of the Regression Line As in the correlation section, there is a convenient way of calculating the regression line through a set of points by simply setting up a table. Examples 1. Problem: Find the regression line for the data set: x y It is assumed that x is the explanatory variable and y the response. Solution: x y x 2 xy Sum = 10 Sum = 48 Sum = 30 Sum = 142 Note that n = 5. = ƒ "! ƒ! 6 " A )ŠC 6 o + Œr :

8 8 TOPIC 6. REGRESSION AND CORRELATION Now,! "! ˆ = r P s š Uœ šlžw Ÿ U[ Pœ Qž k A ) o + P r Z Qž )šlž P Qž ª œ Qž Qž ` «and r U = X «W XšLž `ž.«the equation is therefore y = x To draw this line on a scatterplot, simply choose two values of x (usually one high and one low) and calculate y. x = 0 ± = 0.4 x = 5 ± y = = 23.4 The line is then plotted on the scattergraph as seen below. Notice that here, since the example was purely theoretical, it did not matter which values should be used for x and which for y - they were simply used "as they came". However sometimes the order is important. If, for example, you were examining the effect of how cholesterol levels of humans depend on age, it would be necessary to make cholesterol the response variable (y) and age the explanatory variable (x). It would be nonsensical to have an equation that calculates a person s age given their cholesterol level. 2. Montreal heating data Problem: Similar calculations to those in the abstract example above are carried out on the data relating to the use of gas for heating in section 6.1 and the following parameters for the least squares regression lines are found to be: ²

9 6.3. LEAST SQUARES REGRESSION LINE 9 r b a High Low Solution: The two equations for the regression lines are thus Gas Used = x High Temperature Gas Used = x Low Temperature Again, it is much more sensible to find equations for "gas used" in terms of "temperature" rather than the other way round, so "gas used" is taken as the response variable and "temperature" the explanatory one. The lines can now be drawn on the earlier graphs as follows: 3. Problem: The following four data sets are from Frank J. Anscombe, Graphs in statistical analysis. x y x y x y x y Solution: The correlation and least squares lines for the four data sets are shown in the following list: ³

10 10 TOPIC 6. REGRESSION AND CORRELATION Set 1: r = , y 1 = x Set 2: r = , y 2 = x Set 3: r = , y 3 = x Set 4: r = , y 4 = x The scatterplots can then be plotted as usual. On inspection of the plots it is realised that only the third and fourth represent strong linear relationships (both with one influential outlier). The first data set represents a moderate linear relationship, while the second represents a curved relationship. This shows that correlation and the least squares regression line can suggest a linear relationship even if there is none so visual inspection of the data is vital. EC Data The following data represent the number of members in the EC Council of Ministers of (1) current EC members and of (2) potential EC members, and the populations of the member states: (1) Current members

11 6.4. CONFIDENCE INTERVALS 11 Country Current Members Population (millions) Germany Great Britain France Italy Spain Netherlands Greece Belgium Portugal Sweden Austria Denmark Finland Ireland Luxembourg (2) The agreed number of seats of potential members according to the EC meeting in Nice end of 2000: Country Potential Members Population(millions) Poland Romania Hungary Bulgaria Slovakia Czech Republic Lithuania Latvia Slovenia Estonia Cyprus Malta Calculate the correlation coefficient and the equations of the regression lines for both sets of results, draw the scatterplots including the least squares regression lines and comment on the suitability of the equation. Make sure you choose the response and explanatory variables sensibly. 6.4 Confidence Intervals The last section showed that regression lines can always be found even if the graph looks anything but linear! It is therefore important to develop a procedure in order to µ

12 12 TOPIC 6. REGRESSION AND CORRELATION gain confidence in the least squares regression line. Some assumptions are made about the about the error term For the random variable U ¹6 ò» The standard deviation of is a constant¼ The distribution of is Normal Errors associated with different observations are independent Under the assumptions above let ½ ¾ º À;ÁÃÂÄvÅ É Ä ¾ ÂÆÈÇ. Then ÁÉ ÄÌ ¾ Ç ÆËÊ+Ç distribution with (n-2) degrees of freedom, and is an unbiased estimator for ¼ ¾ has a chi-square It is desirable to find confidence intervals for the slope of the line, b. It can be shown that b follows the t distribution with (n-2) degrees of freedom, with parameters mean = b and standard error = ÍÎ Î Ï ¹ÑÐÒ:ÓJÔ ÒÕ+¾ËÖØ ¹ÑÐÛÜÓÞÝ ÛÕ ÓÚÙ ¾ Note that the standard error can also be calculated from the formula ß~àbáYà ºãâ ävå ÐØæçÓ ä ¾ Õ ÓÚÙ Thus the end-points of a 95% confidence interval for the slope b are bè té-êéëé ¾ëì-íÉ Ä ¾ äâ å ÐØæçÓ ä ¾ Õ ÓÚÙ The value t (n-2),0.025 can be obtained from tables in the same way as it was when the difference between two sample means was being examined. In general, the end points of a (1-2î )100% confidence interval are given by: bè tï íé Ä ¾ äâ å ÐØæçÓ ä ¾ Õ ÓÚÙ Example Problem: Consider the data below: x y Find a 95% confidence interval for the slope of the regression line. ð

13 ñ û ü ö ö ö ö 6.4. CONFIDENCE INTERVALS 13 Solution: It is easily confirmed that a = , b = and r = ò ùlú òôó õøöø ü ýÿþ þ The standard error is equal to ó õøöø The "cut-off" points for the t distribution with 4 degrees of freedom and a 95% confidence interval are þ ù ú ý þ Therefore, the 95% confidence interval This gives the interval [0.5679, ]. The two extreme regression lines are y = x and y = x All three equations are drawn on the graph below. Thus, with 95% confidence it can be said that the true linear regression line is within the cone displayed in the data plot, the boundaries of the cone being the two regression lines with slope and respectively. The cone seems to be very large, but keep in mind that there is only a small number of data. Response time to 999 calls. A study is made into the response time to 999 calls. It measured the time, y, (in minutes) it takes the ambulance to arrive at the scene against the distance, x, (in kilometres) between the station and the scene. The following data are collected: x y Calculate the least squares regression line and draw it on a scatterplot together with two more lines that determine a cone formed by the 95% confidence interval values for b.

14 14 TOPIC 6. REGRESSION AND CORRELATION 6.5 Hypothesis Tests Hypothesis tests can be used to check whether certain parameters produced in the methods of analysis in this chapter are significantly different from zero Hypothesis Test for b It would be useful to know if a calculated regression line really does have an upward (or downward) slope, or if instead that the points actually lie on a horizontal line. If it can be shown that the value of b is zero then it can be concluded that the line is horizontal (or at least that it doesn t have a significant slope). A hypothesis test can be used to do this. It has already been mentioned that b follows a t distribution with (n-2) degrees of freedom and the formula to calculate the standard error of b (the slope of the regression line) has been given as,. This can be used in the test statistic that can be calculated for b in much the same way as it was for small samples in Topic 4. As an example, take the data used in the last section. x y b is calculated as and the standard error as Set up the hypotheses: H 0 : b = 0 H 1 : b 0 The test statistic is! #%$&$ " " $')( ' " This calculates as " $+*,- /. 10%230 Drawing a t distribution curve with 4 degrees of freedom and a significance level of 5% results in the following diagram. 4

15 6.5. HYPOTHESIS TESTS 15 Since the test statistic is in the shaded area the alternative hypothesis (H 1 ) is accepted. There is evidence at the 5% level that the slope of the regression line is not equal to zero. Note that for convenience b is used as the notation in the hypotheses even although in the hypothesis test it actually refers to the population as opposed to the sample Hypothesis Test for r As has been mentioned so far, the correlation coefficient, r, measures the closeness of the linear relationship between any two variables. The values range from -1 to +1 with the extreme values meaning a very good relationship and values close to 0 implying no relationship. A t statistic can be used for r, in much the same way as it was with b, to determine whether r is significantly different from zero. The test statistic is given by the equation ::<;= 8?>@<A 8 freedom. with (n - 2) degrees of Examples 1. Problem: x y The scatterplot is drawn below Solution: The plot does not look very linear and this is borne out by calculation of r, which is A hypothesis test is now carried out. H 0 : r = 0 H 1 : r 6F 0 G

16 K K 16 TOPIC 6. REGRESSION AND CORRELATION The test statistic is HJI L M<M<NO This calculates as K?P?QR MTSUOV Q<Q Compare with the t distribution with 4 degrees of freedom. Since the test statistic is not in the shaded region the null hypothesis is accepted. There is no evidence, at the 5% level, of a significant relationship between the two variables. One-tailed tests can also be used. 2. Problem: Recall the Montreal heating data in section 6.1produced the following results: r b a High Low The two equations for the regression lines were Gas Used = W High Temperature Gas Used = W Low Temperature There were 12 data points considered. Solution: Consider the first value of r, namely A hypothesis test can be used to confirm that this is significantly less than 0. H 0 : r X 0 H 1 : r Y 0 The test statistic, HZI L MM<NO K?P?Q<R MCSDOV QQ calculates as The t curve is drawn below (10 degrees of freedom) with a significance level of 0.1% shaded. [

17 6.6. MULTIPLE LINEAR REGRESSION 17 Since the test statistic is in the shaded area, the null hypothesis is rejected. There is evidence using a significance level of that the value of r is less than 0. It can be said, then, that there is a very highly significant negative correlation between the two variables. This can similarly be repeated for the other value of r in the example. Once again in this section the "r" in the hypothesis test actually refers to the population rather than the sample. 6.6 Multiple Linear Regression This topic ends with a brief look at multiple linear regression. The calculations are carried out from first principles but would more usually be analysed by a statistical computer package such as Minitab. Consider the following table showing information about apartment blocks sold in a big city: Number Apartments (x) Number Floors (y) Price (in $million) (z) Using the method of least squares, it is desired to find the equation of a plane that will predict the price of an apartment block (z) in terms of the numbers of apartments (x) and number of floors (y). The equation of the plane is z = a + bx + cy where a, b and c are parameters to be estimated. \

18 18 TOPIC 6. REGRESSION AND CORRELATION For an observed data point z, the error is given by: ]_^a`cbed `f^a`gbihfbkjmlnbio-p For least squares, it is required to minimise qr]ts. Therefore the problem becomes: Minamise u_^ qwv `gbihxbkjml7byomp{z s. This equation can be partially differentiated with respect to a, b and c to obtain: S ^~} v `gbihfbijml7bio-p z v bc tz a S j ^~} v `gbihfbijml7bio-p z v b lƒz S o ^ } v `gbihfbijml7bio-p z v b p{z To minimise, equate all 3 to 0 (the "2 s" cancel): b } ` + h } j } l ˆo } p^a b } lš` ˆh } l Œj } l s ˆo } l pž^/ b } p ` ˆh } pg j } lpg ˆo } p s ^a Now for the data above a series of values such as can be calculated and substituted into these equations. It can be shown that: } l s ^/ 3 } l ^/ %, } l ^ D 3 } p s ^ D } l ` ^ % % U } y = 59 } ` s ^/š3 3 D œ } p `f^ U 3 1 } `x^ š 1š Also note that ž 1 = n, the number of points (in this case 8). The equations become: 8a + 480b + 59c = a b c = a b + 499c = The method of solution of simultaneous equations can be used to solve for a, b and c (matrix methods can also be employed). The solution is a = , b = and c = The equation of the least squares plane through the data is z = x y Ÿ

19 6.7. SUMMARY AND ASSESSMENT Summary and Assessment At this stage you should be able to: draw scatterplots to investigate for a relationship between x and y define the Product-Moment Correlation Co-efficient, r interpret values of r as giving a good or bad linear relationship between variables calculate r using a methodical manual method distinguish between an explanatory variable and a response variable calculate the equation of a Least Squares Regression Line through a set of points calculate confidence intervals for the slope of the regression equation carry out hypothesis tests for b and r solve a multiple linear regression problem with the aid of a computer package

20 20 GLOSSARY Glossary Correlation is a number that measures strength and direction of the linear relationship between two data sets.

21 ANSWERS: TOPIC 6 21 Answers to questions and activities 6 Regression and Correlation EC Data (page 10) The correlation co-efficient, r, calculates as , which suggests a good linear correlation. However the graph does appear rather curved. The regression equation is y = x For the second table, r calculates as and the regression equation is y = x The graph is shown below. The first scatter plot does not necessarily suggest a linear relationship between number of inhabitants and seats, but the second does look linear. The combined data set does, in fact, suggest a logarithmic relationship instead of a linear one. Response time to 999 calls. (page 13) If you have Microsoft Excel, the regression equation and correlation coefficient can be obtained very quickly. The command "Correl" (from the function menu) will produce the Product Moment Correlation Co-efficient, r, whilst the commands "slope" and "intercept" will give you b and a respectively.

22 22 ANSWERS: TOPIC 6 This package is used on the response times for 999 calls data and the values obtained are: r = b = a = The regression equation is therefore y = x Now, confidence intervals for b can be calculated. The standard error of b is given by 1ª «««T D D«± ²)³µ T D D«3 ³ The value of t 0.025, 13 is found from tables to be (If you are using Excel, you might like to try using the "Tinv" function to also obtain this number, but note that it automatically does a two-tailed test so 0.05 has to be entered as opposed to 0.025). So the 95% confidence interval for b is ¹ x This gives an interval of [0.1392, ]. Therefore, the two lines to produce the cone are given by y = x y = x The lines are drawn on the scatterplot as follows º

AD HOC DRAFTING GROUP ON TRANSNATIONAL ORGANISED CRIME (PC-GR-COT) STATUS OF RATIFICATIONS BY COUNCIL OF EUROPE MEMBER STATES

Strasbourg, 29 May 2015 PC-GR-COT (2013) 2 EN_Rev AD HOC DRAFTING GROUP ON TRANSNATIONAL ORGANISED CRIME (PC-GR-COT) STATUS OF RATIFICATIONS BY COUNCIL OF EUROPE MEMBER STATES TO THE UNITED NATIONS CONVENTION