1 / 27 Lecture 13 Simple Linear Regression October 28, 2010
2 / 27 Lesson Plan 1. Ordinary Least Squares 2. Interpretation
3 / 27 Motivation Suppose we want to approximate the value of Y with a linear function of X. y i α + βx i i = 1, 2,..., n However the data will not fit perfectly. Assume y i = α + βx i + ε i i = 1, 2,..., n What values of α and β are the best? The main idea is to minimize the following quantity ε 2 i = (y i α βx i ) 2
4 / 27 OLS Derivation First take the partial derivatives with respect to α and β. n (y i α βx i ) 2 α = na = (2α 2Y i + 2βx i ) = 0 y i β α = ȳ β x x i n (y i α βx i ) 2 β β = x 2 i = ( 2βx 2 i 2x i y i + 2ax i ) = 0 x i y i α x i
5 / 27 OLS Derivation If you substitute the value for α in the expression for β, it turns out that, after some algebra manipulations, [ ] β x x i = x i y i ȳ β [ x 2 i x 2 i n x 2 ] = x i x i y i n mȳ x [ nβ x 2 x 2] = n [yx ȳ x] β = [yx ȳ x] [x 2 x 2 ]
6 / 27 OLS Derivation If you look at the numerator of β, yx ȳ x looks like E(YX) E(Y )E(X) Therefore you should not be surprised that yx ȳ x [y i ȳ] [x i x] n 1 yx ȳ x is the sample covariance of Y and X.
7 / 27 OLS Derivation If you look at the denominator of β, x 2 x 2 looks like E(X 2 ) E(X) 2 Therefore you should not be surprised that x 2 x 2 [x i x] 2 n 1 x 2 x 2 is the sample variance of X.
8 / 27 OLS Estimates or α = ȳ β x β = Cov(x, y) Var(x) = (y i ȳ) (x i x) (x i x) 2 β = x i y i x 2 i x i y i n ( ) 2 x i n
9 / 27 OLS Estimates Notice that sometimes, for example in the book, a different formula is considered. If you define (centering) x = x x ỹ = y ȳ Then, because x and ỹ have mean zero, α = ȳ β x ỹ i x i β = x 2 i
10 / 27 Example: Leaning Tower of Pisa Lean (m) = distance between where a point on the tower would be if the tower were straight and where it actually was. Suppose we want to fit the following line Lean = α + βyear + ε Year Lean 1975 2.9642 1976 2.9644 1977 2.9656 1978 2.9667 1979 2.9673 1980 2.9688 1981 2.9696 1982 2.9698 1983 2.9713 1984 2.9717 1985 2.9725 1986 2.9742 1987 2.9757
Example: Leaning Tower of Pisa Y = Lean; X = Year 13 13 y i = 38.6018 x i y i = 76470.34 13 13 x i = 25753 x 2 i = 51016875 β = α = 13 x i y i 13 xi 2 13 y i 13 β 13 13 x i y i 13 ( ) 2 13 x i 13 13 x i = 0.0009318681 13 = 1.123338 11 / 27
Example: Leaning Tower of Pisa Lean 2.964 2.968 2.972 2.976 76 78 80 82 84 86 Year 12 / 27
13 / 27 Example: Leaning Tower of Pisa The intercept: α = 1.1233. The value of y when x = 0. Is this number useful? The slope: β = 0.00093. The amount by which the variable y changes when x increases by one unit. Does it matter how we record year(e.g. 87 versus 1987)? The fitted line represents the predicted values of y.
14 / 27 Interpolation and Extrapolation Interpolation is the process to determine the response y for a specific value of the explanatory variable x inside the range of x. What is the lean value for the Pisa tower at time 1977.5? y = 1.123338 + 0.0009318681 1977.5 = 2.966107 Extrapolation is the process to determine the response y for a specific value of the explanatory variable x outside the range of x. What is the lean value for the Pisa tower at time 2000? y = 1.123338 + 0.0009318681 2000 = 2.987074
15 / 27 OLS The difference between the true values and the theoretical values represent the error: ε i = y i (α + βx i ) An important property of OLS is that the sum of squared errors is minimum (ordinary least squares) To emphasize that the unknown intercept and slope are estimated by the data, we often write ŷ i = ˆα + ˆβx i ŷ i represents the predicted (fitted) value. It is also an estimate!
Example: Leaning Tower of Pisa 16 / 27
17 / 27 Y = 1.123338 + 0.0009318681X + ε If we substitute the data for X, Y into the OLS estimate: X Y 10000 ε (10000 ε) 2 1975 2.9642 4.2197 17.8065 1976 2.9644-3.0989 9.6031 1977 2.9656-0.4175 0.1743 1978 2.9667 1.2637 1.5970 1979 2.9673-2.0549 4.2227 1980 2.9688 3.6263 13.1505 1981 2.9696 2.3073 5.3254 1982 2.9698-5.0109 25.1100 1983 2.9713 0.6703 0.4493 1984 2.9717-4.6483 21.6071 1985 2.9725-5.9670 35.6054 1986 2.9742 1.7142 2.9387 1987 2.9757 7.3956 54.6949 1.332268e-11 192.2857
Y = 1.12 + 0.00093X + ε If we substitute the data for X, Y into an arbitrary estimate of the regression line we can see that X Y 10000 ε (10000 ε) 2 1975 2.9642 74.5 5550.25 1976 2.9644 67.2 4515.84 1977 2.9656 69.9 4886.01 1978 2.9667 71.6 5126.56 1979 2.9673 68.3 4664.89 1980 2.9688 74.0 5476.00 1981 2.9696 72.7 5285.29 1982 2.9698 65.4 4277.16 1983 2.9713 71.1 5055.21 1984 2.9717 65.8 4329.64 1985 2.9725 64.5 4160.25 1986 2.9742 72.2 5212.84 1987 2.9757 77.9 6068.41 915.1 64608.35 18 / 27
19 / 27 Example: α and β 2009-10-30 to 2010-10-2 Google Amazon S&P 500 Return 5 0 5 10 Return 5 0 5 10 Return 6 4 2 0 2 4 6 0 10 20 30 40 50 Week 0 10 20 30 40 50 Week 0 10 20 30 40 50 Week
20 / 27 Example: α and β The beta coefficients represents how the expected return of a stock is related to the performance of a financial market. The alpha is the extra return awarded to the investor for taking a risk, instead of accepting the market return. It turns out that: R (stock) t = α + βr (market) t Google versus S&P: α = 0.0993, β = 1.0807 Amazon versus S&P: α = 0.7295, β = 1.2235
Example: α and β 6 4 2 0 2 4 6 5 0 5 10 S&P 500 Google 6 4 2 0 2 4 6 5 0 5 10 S&P 500 Amazon 21 / 27
22 / 27 Example: α and β If you were to invest in stocks, you would like to have high returns and little risk (for example variance). If you were evaluating the linear regression between a single stock and a general portfolio, β would measure the amount of variance (risk) associated with a single stock which cannot be diversified by investing in the portfolio. Because β represents the relationship between the stock and the portfolio return.
23 / 27 Demand Elasticity Demand Elasticity is an important concept in economics. Suppose we want to study the relationship between the price and the demand of a specific good. The demand for a drug which cures a severe disease is typically inelastic. No matter how expensive, people prefer to get cured (inelastic demand). If the price for pasta increases people may start buying rice instead (elastic demand).
24 / 27 Demand Elasticity Elasticity is defined as Elasticity = q p p q From a practical point of view, the easiest way to evaluate elasticity is to define a model and then derive the corresponding value. For example, given the following data describing how much are you willing to buy (q) at the level price (p) p 40 41 42 43 44 45 46 47 q 15.81 15.95 14.94 15.02 14.63 14.28 13.95 13.42
25 / 27 Demand Elasticity A simple approach consists in evaluating a linear regression and deriving the corresponding elasticity. 8 (p i p)(q i q) = 8 (p i p) 2 = 8 p i q i = 14.52305 8 q i 2 = 42 β = 14.52305 = 0.3457868 42 α = q β p = 29.79191 Therefore ˆq = 29.79191 0.3457868p
26 / 27 Demand Elasticity The next step is to evaluate the theoretical elasticity Elasticity = q p p q = β pˆq Therefore p 40 41 42 43 44 45 46 47 q 15.81 15.95 14.94 15.02 14.63 14.28 13.95 13.42 ˆq 15.96 15.61 15.27 14.92 14.58 14.23 13.89 13.54 Ê 0.87 0.91 0.95 1.00 1.04 1.09 1.15 1.20 If p < 43 the demand is inelastic If p > 43 the demand is elastic
27 / 27 Demand Elasticity Elasticity is defined as Elasticity = q p p q = q/q p/p The beta coefficient represents the change in y associated with a unit change in x. The elasticity is measured in percentage. The elasticity is unit-less