Mathematics for Economics MA course

Transcription

1 Mathematics for Economics MA course Simple Linear Regression Dr. Seetha Bandara

2 Simple Regression Simple linear regression is a statistical method that allows us to summarize and study relationships between two continuous (quantitative) variables: One variable, denoted x, is regarded as the predictor, explanatory, or independent variable. The other variable, denoted y, is regarded as the response, outcome, or dependent variable.

3 Introduction The primary goal of quantitative analysis is to use current information about a phenomenon to predict its future behavior. Current information is usually in the form of a set of data. In a simple case, when the data form a set of pairs of numbers, we may interpret them as representing the observed values of an independent (or predictor ) variable X and a dependent ( or response) variable Y. lot size Man-hours

4 Introduction Statistical relation between Lot size and Man-Hour The goal of the analyst who studies the data is to find a functional relation y f (x) between the response variable y and the predictor variable x. Man-Hour Lot size

5 Regression Function The statement that the relation between X and Y is statistical should be interpreted as providing the following guidelines: 1. Regard Y as a random variable. 2. For each X, take f (x) to be the expected value (i.e., mean value) of y. 3. Given that E (Y) denotes the expected value of Y, call the equation E( Y) f ( x) the regression function.

6 Pictorial Presentation of Linear Regression Model

7 Historical Origin of Regression Regression Analysis was first developed by Sir Francis Galton, who studied the relation between heights of sons and fathers. Heights of sons of both tall and short fathers appeared to revert or regress to the mean of the group.

8 Construction of Regression Models Selection of independent variables Since reality must be reduced to manageable proportions whenever we construct models, only a limited number of independent or predictor variables can or should be included in a regression model. Therefore a central problem is that of choosing the most important predictor variables. Functional form of regression relation Sometimes, relevant theory may indicate the appropriate functional form. More frequently, however, the functional form is not known in advance and must be decided once the data have been collected and analyzed. Scope of model In formulating a regression model, we usually need to restrict the coverage of model to some interval or region of values of the independent variables.

9 Uses of Regression Analysis Regression analysis serves Three major purposes. 1. Description 2. Control 3. Prediction The several purposes of regression analysis frequently overlap in practice

10 Formal Statement of the Model General regression model Y 0 1 X 1. 0, and 1 are parameters 2. X is a known constant 3. Deviations are independent N(o, 2 )

11 Meaning of Regression Coefficients The values of the regression parameters 0, and 1 are not known.we estimate them from data. 1 indicates the change in the mean response per unit increase in X.

12 Regression Line If the scatter plot of our sample data suggests a linear relationship between two variables i.e. y x 1 0 we can summarize the relationship by drawing a straight line on the plot. Least squares method give us the best estimated line for our set of sample data.

13 Regression Line We will write an estimated regression line based on sample data as yˆ b0 b1 x The method of least squares chooses the values for b 0, and b 1 to minimize the sum of squared errors SSE n n 2 ( y ˆ i yi ) i1 i1 2 y b b x 0 1

14 Regression Line Using calculus, we obtain estimating formulas: or n i n i i i n i n i n i i i i i n i i n i i i x x n y x y x n x x y y x x b ) ( ) ( ) )( ( b x y b 1 0 x y S S b r 1

15 Estimation of Mean Response Fitted regression line can be used to estimate the mean value of y for a given value of x. Example The weekly advertising expenditure (x) and weekly sales (y) are presented in the following table. y x

16 Point Estimation of Mean Response From previous table we have: n 10 y x The least squares estimates of the regression coefficients are: b n xy x x) ( x 2 xy n x y 10(818755) (564)(14365) 2 10(32604) (564) b (56.4)

17 Point Estimation of Mean Response The estimated regression function is: ŷ x Sales Expenditure This means that if the weekly advertising expenditure is increased by $1 we would expect the weekly sales to increase by $10.8.

18 Point Estimation of Mean Response Fitted values for the sample data are obtained by substituting the x value into the estimated regression function. For example if the advertising expenditure is $50, then the estimated Sales is: Sales (50) 1368 This is called the point estimate (forecast) of the mean response (sales).

19 Example:Retail sales and floor space It is customary in retail operations to asses the performance of stores partly in terms of their annual sales relative to their floor area (square feet). We might expect sales to increase linearly as stores get larger, with of course individual variation among stores of the same size. The regression model for a population of stores says that SALES = AREA +

20 Example:Retail sales and floor space The slope 1 is as usual a rate of change: it is the expected increase in annual sales associated with each additional square foot of floor space. The intercept 0 is needed to describe the line but has no statistical importance because no stores have area close to zero. Floor space does not completely determine sales. The term in the model accounts for difference among individual stores with the same floor space. A store s location, for example, is important.

21 Residual The difference between the observed value y i and the corresponding fitted value. ŷ i e i y i yˆ i Residuals are highly useful for studying whether a given regression model is appropriate for the data at hand.

22 Example: weekly advertising expenditure y x y-hat Residual (e)

23 Estimation of the variance of the error terms, 2 The variance 2 of the error terms i in the regression model needs to be estimated for a variety of purposes. It gives an indication of the variability of the probability distributions of y. It is needed for making inference concerning regression function and the prediction of y.

24 Regression Standard Error To estimate we work with the variance and take the square root to obtain the standard deviation. For simple linear regression the estimate of 2 is the average squared residual. s y To estimate, use ˆ x ei ( yi yi ) n 2 n 2 s 2 y. x sy. x s estimates the standard deviation of the error term in the statistical model for simple linear regression. 2

25 Regression Standard Error y x y-hat Residual (e) square(e) y-hat = X total S y.x

26 Basic Assumptions of a Regression Model A regression model is based on the following assumptions: 1. There is a probability distribution of Y for each level of X. 2. Given that µ y is the mean value of Y, the standard form of the model is f ( x ) y where is a random variable with a normal distribution with mean 0 and standard deviation.

27 Conditions for Regression Inference You can fit a least-squares line to any set of explanatory-response data when both variables are quantitative. If the scatter plot doesn t show an approximately linear pattern, the fitted line may be almost useless.

28 Conditions for Regression Inference The simple linear regression model, which is the basis for inference, imposes several conditions. We should verify these conditions before proceeding with inference. The conditions concern the population, but we can observe only our sample.

29 Conditions for Regression Inference In doing Inference, we assume: 1. The sample is an SRS from the population. 2. There is a linear relationship in the population. 1. We can not observe the population, so we check the scatter plot of the sample data. 3. The standard deviation of the responses about the population line is the same for all values of the explanatory variable. 1. The spread of observations above and below the least-squares line should be roughly uniform as x varies.

30 Conditions for Regression Inference Plotting the residuals against the explanatory variable is helpful in checking these conditions because a residual plot magnifies patterns.

31 Analysis of Residual To examine whether the regression model is appropriate for the data being analyzed, we can check the residual plots. Residual plots are: Plot a histogram of the residuals Plot residuals against the fitted values. Plot residuals against the independent variable. Plot residuals over time if the data are chronological.

32 Analysis of Residual A histogram of the residuals provides a check on the normality assumption. A Normal quantile plot of the residuals can also be used to check the Normality assumptions. Regression Inference is robust against moderate lack of Normality. On the other hand, outliers and influential observations can invalidate the results of inference for regression Plot of residuals against fitted values or the independent variable can be used to check the assumption of constant variance and the aptness of the model.

33 Analysis of Residual Plot of residuals against time provides a check on the independence of the error terms assumption. Assumption of independence is the most critical one.

34 Residual plots The residuals should have no systematic pattern. The residual plot to right shows a scatter of the points with no individual observations or systematic change as x increases. Residuals Degree Days Residual Plot Degree Days

35 Residual plots The points in this residual plot have a curve pattern, so a straight line fits poorly

36 Residual plots The points in this plot show more spread for larger values of the explanatory variable x, so prediction will be less accurate when x is large.

37 Variable transformations If the residual plot suggests that the variance is not constant, a transformation can be used to stabilize the variance. If the residual plot suggests a non linear relationship between x and y, a transformation may reduce it to one that is approximately linear. Common linearizing transformations are: 1 x, log( x) Variance stabilizing transformations are: 1, y log( y), y, y 2

38 Inference about the Regression Model When a scatter plot shows a linear relationship between a quantitative explanatory variable x and a quantitative response variable y, we can use the least square line fitted to the data to predict y for a give value of x. Now we want to do tests and confidence intervals in this setting.

39 Inference about the Regression Model We think of the least square line we calculated from a sample as an estimate of a regression line for the population. Just as the sample mean population mean µ. x is an estimate of the

40 Inference about the Regression Model We will write the population regression line as 0 1 x The numbers and are parameters that describe the 0 1 population. We will write the least-squares line fitted to sample data as b0 b1 x This notation reminds us that the intercept b 0 of the fitted line estimates the intercept 0 of the population line, and the slope b 1 estimates the slope 1.

41 Confidence Intervals and Significance Tests In our previous lectures we presented confidence intervals and significance tests for means and differences in means.in each case, inference rested on the standard error s of the estimates and on t or z distributions. Inference for the slope and intercept in linear regression is similar in principal, although the recipes are more complicated. All confidence intervals, for example, have the form estimate t* Se estimate t* is a critical value of a t distribution.

42 Confidence Intervals and Significance Tests Confidence intervals and tests for the slope and intercept are based on the sampling distributions of the estimates b 1 and b 0. Here are the facts: If the simple linear regression model is true, each of b 0 and b 1 has a Normal distribution. The mean of b 0 is 0 and the mean of b 1 is 1. That is, the intercept and slope of the fitted line are unbiased estimators of the intercept and slope of the population regression line.

43 Confidence Intervals and Significance Tests The standard deviations of b 0 and b 1 are multiples of the model standard deviation. 2 1 ) ( ) ( 1 x x s b S SE b n i i b x x x n s b S SE ) ( 1 ) ( 0

44 Confidence Intervals and Significance Tests

45 Example:Weekly Advertising Expenditure Let us return to the Weekly advertising expenditure and weekly sales example. Management is interested in testing whether or not there is a linear association between advertising expenditure and weekly sales, using regression model. Use =.05

46 Example:Weekly Advertising Expenditure Hypothesis: H Decision Rule: Reject H 0 if or 0 H a t : : t t t ;8 t. 025;8 t

47 Example:Weekly Advertising Expenditure Test statistic: t b1 S( b 1 ) S y. x S( b ) ( x x) b t

48 Example:Weekly Advertising Expenditure Conclusion: Since t =4.5 > then we reject H 0. There is a linear association between advertising expenditure and weekly sales.

49 Confidence interval for 1 b 1 t b ( ; n2) 2 ( S( 1)) Now that our test showed that there is a linear association between advertising expenditure and weekly sales, the management wishes an estimate of 1 with a 95% confidence coefficient.

50 Confidence interval for 1 For a 95 percent confidence coefficient, we require t (.025; 8). From table B in appendix III, we find t(.025; 8) = The 95% confidence interval is: b 1 t 10.8 ( ; n2) 2 ( S( b1 )) 2.306(2.38) (5.31, 16.3)

51 Example: Do wages rise with experience? Many factors affect the wages of workers: the industry they work in, their type of job, their education and their experience, and changes in general levels of wages. We will look at a sample of 59 married women who hold customer service jobs in Indiana banks. The following table gives their weekly wages at a specific point in time also their length of service with their employer, in month. The size of the place of work is recorded simply as large (100 or more workers) or small. Because industry, job type, and the time of measurement are the same for all 59 subjects, we expect to see a clear relationship between wages and length of service.

52 Example: Do wages rise with experience?

53 Example: Do wages rise with experience?

54 Example: Do wages rise with experience? From previous table we have: n 59 y x 4159 y x xy The least squares estimates of the regression coefficients are: b 1 r s s y x b 0 y bx

55 Example: Do wages rise with experience? What is the least-squares regression line for predicting Wages from Length of service (Los)? Suppose a woman has been with her bank for 125 months. What do you predict she will earn? If her actual wages are $433, then what is her residual? The sum of squared residuals for the entire sample is 59 ( y ˆ ) 2 i y i i1

56 Example: Do wages rise with experience? Do wages rise with experience? The hypotheses are: H 0 : 1 = 0, H a : 1 > 0 The test statistics The P- value is: Conclusion: b1 t SE b 1

57 Example: Do wages rise with experience? A 95% confidence interval for the average increase in wages per month of stay for the regression line in the population of all married female customer service workers in Indiana bank is b t * SE 1 1 b The t distribution for this problem has n-2 = 57 degrees of freedom

58 Example: Do wages rise with experience? Regression calculations in Practice are always done by software. The computer out put for the case study is given in the following slide.

59 Example: Do wages rise with experience?

60 Using the regression Line One of the most common reasons to fit a line to data is to predict the response to a particular value of the explanatory variable. In our example, the least square line for predicting the weekly earnings for female bank customer service workers from their length of service is yˆ x

61 Using the regression Line For a length of service of 125 months, our leastsquares regression equation gives yˆ (.5905)(125) $423 per week There are two different uses of this prediction. We can estimate the mean earnings of all workers in the subpopulation of workers with 125 months on the job. We can predict the earnings of one individual worker with 125 months of service.

62 Using the regression Line For each use, the actual prediction is the same,.but the margin of error is different for the two cases. To estimate the mean response, we use a confidence * interval. y 0 1x To estimate an individual response y, we use prediction interval. yˆ $423 A prediction interval estimates a single random response y rather than a parameter like µ y

63 Using the regression Line The main distinction is that it is harder to predict for an individual than for the mean of a population of individuals. Each interval has the usual form y ˆ t * SE The margin of error for the prediction interval is wider than the margin of error for the confidence interval.

64 Using the regression Line The standard error for estimating the mean response when the explanatory variable x takes the value x* is:

65 Using the regression Line The standard error for predicting an individual response when the explanatory variable x takes the value x* is:

66 Prediction of a new response ( ) We now consider the prediction of a new observation y corresponding to a given level x of the independent variable. In our advertising expenditure and weekly sales, the management wishes to predict the weekly sales corresponding to the advertising expenditure of x = $50. ŷ

67 Interval Estimation of a new response ( ) The following formula gives us the point estimator (forecast) for y. yˆ b0 b1 x 1- % prediction interval for a new observation ŷ is: yˆ t ( S ) Where S f ( ; n2) ( x x) S y. x 1 2 n ( x x) f ŷ

68 Example In our advertising expenditure and weekly sales, the management wishes to predict the weekly sales if the advertising expenditure is $50 with a 90 % prediction interval. S S yˆ f f (50) 1368 S y. x n ( x x) ( x x) We require t(.05; 8) = ( )

69 Example The 90% prediction interval is: ˆ t(.05;8) y ( S (72.11) (1233.9, ) f )

70 Analysis of variance approach to Regression analysis Analysis of Variance is the term for statistical analyses that break down the variation in data into separate pieces that correspond to different sources of variation. It is based on the partitioning of sums of squares and degrees of freedom associated with the response variable. In the regression setting, the observed variation in the responses (y i ) comes from two sources.

71 Analysis of variance approach to Regression analysis Consider the weekly advertising expenditure and the weekly sales example. There is variation in the amount ($) of weekly sales, as in all statistical data. The variation of the y i is conventionally measured in terms of the deviations: y i y

72 Analysis of variance approach to Regression analysis The measure of total variation, denoted by SST, is the sum of the squared deviations: SST ( y y) If SST = 0, all observations are the same(no variability). The greater is SST, the greater is the variation among the y values. When we use the regression model, the measure of variation is that of the y observations variability around the fitted line: y i 2 ˆ i y i

73 Analysis of variance approach to Regression analysis The measure of variation in the data around the fitted regression line is the sum of squared deviations (error), denoted SSE: SSE y i yˆ ( i For our Weekly expenditure example SSE = SST = What accounts for the substantial difference between these two sums of squares? ) 2

74 Analysis of variance approach to Regression analysis The difference is another sum of squares: SSR ( yˆ y) i SSR stands for regression sum of squares. SSR is the variation among the predicted responses ŷ i. The predicted responses lie on the least-square line. They show how y moves in response to x. The larger is SSR relative to SST, the greater is the role of regression line in explaining the total variability in y observations. 2

75 Analysis of variance approach to Regression analysis In our example: SSR SST SSE This indicates that most of variability in weekly sales can be explained by the relation between the weekly advertising expenditure and the weekly sales.

76 Formal Development of the Partitioning We can decompose the total variability in the observations y i as follows: y y yˆ y y yˆ i i The total deviation y i y can be viewed as the sum of two components: The deviation of the fitted value around the mean y. The deviation of y i around the fitted regression line. i i ŷ i

77 Formal Development of the Partitioning Skipping quite a bit of messy algebra, we just state that this analysis of variance equation always holds: ( y ) ( ˆ ) ( ˆ i y yi y yi yi ) Breakdown of degree of freedom: n 11 ( n 2)

78 Mean squares A sum of squares divided by its degrees of freedom is called a mean square (MS) Regression mean square (MSR) MSR SSR 1 Error mean square (MSE) SSE MSE n 2 Note: mean squares are not additive.

79 Mean squares In our example: MSR SSR SSE MSE n

80 Analysis of Variance Table The breakdowns of the total sum of squares and associated degrees of freedom are displayed in a table called analysis of variance table (ANOVA table) Source of Variation SS df MS F-Test Regression SSR 1 MSR =SSR/1 MSR/MSE Error SSE n-2 MSE Total SST n-1 =SSE/(n-2)

81 Analysis of Variance Table In our weekly advertising expenditure and weekly sales example the ANOVA table is: Source of variation SS df MS Regression Error Total

82 Analysis of Variance Table The Analysis of Variance table reports in a different way quantities such as r2 and s that are needed in regression analysis. It also reports in a different way the test for the overall significance of the regression. If regression on x has no value for predicting y, we expect the slope of the population regression line to be close to 0.

83 Analysis of Variance Table That is the null hypothesis of no linear relationship is: H 0 0 : 1 We standardize the slope of the leastsquares line to get a t statistic.

84 F-Test for 1 = 0 versus 1 0 The analysis of variance approach starts with sums of squares. If regression on x has no value for predicting y, we expect the SSR to be only a small part of the SST, most of which will be made of the SSE. The proper way to standardize this comparison is to use the ratio F MSR MSE

85 F-Test for 1 = 0 versus 1 0 In order to be able to construct a statistical decision rule, we need to know the distribution of our test statistic F. When H 0 is true, our test statistic, F, follows the F- distribution with 1, and n-2 degrees of freedom. Table C-5 on page 513 of your text gives the critical values of the F-distribution at = 0.05 and.01.

86 F-Test for 1 = 0 versus 1 0 Construction of decision rule: At = 5% level Reject H 0 if F F( ;1, n 2) Large values of F support H a and Values of F near 1 support H 0.

87 F-Test for 1 = 0 versus 1 0 Using our example again, let us repeat the earlier test on 1. This time we will use the F-test. The null and alternative hypothesis are: H : 0 Let =.05. Since n=10, we require F(.05; 1, 8). From table 5-3 we find that F(.05; 1, 8) = Therefore the decision rule is: Reject H 0 if: 0 H a : F 5.32

88 F-Test for 1 = 0 versus 1 0 From ANOVA table we have MSR = MSE = Our test statistic F is: Decision: F Since 20.47> 5.32, we reject H 0, that is there is a linear association between weekly advertising expenditure and weekly sales.

89 F-Test for 1 = 0 versus 1 0 Equivalence of F Test and t Test: For given level, the F test of 1 = 0 versus 1 0 is equivalent algebraically to the two sided t- test. Thus, at a given level, we can use either the t-test or the F-test for testing 1 = 0 versus 1 0. The t-test is more flexible since it can be used for one sided test as well.

90 Analysis of Variance Table The complete ANOVA table for our example is: Source of Variation SS df MS F-Test Regression Error Total

91 Computer Output The EXCEL out put for our example is: SUMMARY OUTPUT Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 10 ANOVA df SS MS F Significance F Regression Residual Total Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Intercept AD-Expen (X)

92 Coefficient of Determination Recall that SST measures the total variations in y i when no account of the independent variable x is taken. SSE measures the variation in the y i when a regression model with the independent variable x is used. A natural measure of the effect of x in reducing the variation in y can be defined as: R 2 SST SSE SSR 1 SST SST SSE SST

93 Coefficient of Determination R 2 is called the coefficient of determination. 0 SSE SST, it follows that: 0 R 2 1 We may interpret R 2 as the proportionate reduction of total variability in y associated with the use of the independent variable x. The larger is R 2, the more is the total variation of y reduced by including the variable x in the model.

94 Coefficient of Determination If all the observations fall on the fitted regression line, SSE = 0 and R 2 = 1. If the slope of the fitted regression line b 1 = 0 so that yˆ, SSE=SST and R 2 i y = 0. The closer R 2 is to 1, the greater is said to be the degree of linear association between x and y. The square root of R 2 is called the coefficient of correlation. r 2 R

95 Correlation Coefficient Recall that the algebraic expression for the correlation coefficient is ) ( ) ( ) ( ) ( ) )( ( y y n x x n y x xy n r y y x x y y x x r

96

97

98

99