Simple Linear Regression

Size: px

Start display at page:

Download "Simple Linear Regression"

Tabitha Sanders
5 years ago
Views:

1 Simple Linear Regression Christopher Ting Christopher Ting : christophert@smu.edu.sg : : LKCSB 5036 January 7, 017 Web Site: Christopher Ting QF 30 Week 4 January 7, 017 1/9

2 Learning Objectives Gain deep insights into simple or univariate OLS classical conditions FOC first-order condition solutions of two FOC s OLS estimators weights of simple OLS distribution of OLS estimators properties of residuals hypothesis testing significance test of OLS estimates Define BLUE best linear unbiased estimator Gain deeper insights into asymptotic properties, consistent properties, and coefficient of determination of simple OLS Describe how OLS estimates can be applied to forecasting Develop a working knowledge of OLS regression by applying the theory to hedging an equity portfolio with stock index futures Christopher Ting QF 30 Week 4 January 7, 017 /9

3 Innards of Simple OLS Classical Conditions Given n pairs of observations on explanatory variable X i and dependent variable Y i, Model 1 of QF30 is Y i = a + bx i + e i, i = 1,,..., n, 1 where e i is the noise. Model 0 is Y i = a + e i. Assumptions: A1 Ee i = 0 for every i A Ee i = σ e A3 Ee i e j = 0 for every i, j A4 X i, e j are independent for each i, j d A5 e i N0, σ e Christopher Ting QF 30 Week 4 January 7, 017 3/9

4 Innards of Simple OLS First-Order Conditions of Least Squares Least Squares: Minimizing the sum of squared errors: min â, b e i = Yi â bx i e i â e i b = = Yi â bx i = 0 X i Yi â bx i = 0 These least squares minimization conditions are ordinary. Christopher Ting QF 30 Week 4 January 7, 017 4/9

5 Innards of Simple OLS Ordinary Least Squares Solutions Solution of first FOC Y i = â + bxi = ny = nâ + n bx = Y = â + bx = â = Y bx Solution of second FOC X i Y i = X i â + bx i = = = X i Y i = X i Y i = X i â + b X i X i Y bx + b X i Yi Y = b X i Xi X X i Yi Y = b = X i Xi X X i Christopher Ting QF 30 Week 4 January 7, 017 5/9

6 Innards of Simple OLS OLS with Centered Regressor More convenient to start with the centralized linear model Y i = a + b X i X + e i, a = a + bx OLS FOC min â, b e i = Y i â b X i X Y i â b X i X = 0 Xi X Y i â b X i X = 0 Christopher Ting QF 30 Week 4 January 7, 017 6/9

7 Innards of Simple OLS Linear Estimators Solution of FOC s â = Y b = Xi X Y i Y Xi X X i X Define the weights v i := 1 n Xi X X Xi X ; w Xi X i := Xi X Hence, linear combinations: â = v i Y i ; b n = w i Y i Remark: Xi X = n 1 σ X Christopher Ting QF 30 Week 4 January 7, 017 7/9

8 Innards of Simple OLS Properties of Weights and OLS Estimators Properties of v i v i = 1, vi = 1 n + X Xi X, v i X i = 0 Properties of w i w i = 0, wi = 1 Xi X, w i X i = 1 Finite sample properties of OLS estimators: Unbiasedness â = v i a + bxi + e i = a + v i e i = E â = a b = w i a + bxi + e i = b + w i e i = E b = b Christopher Ting QF 30 Week 4 January 7, 017 8/9

9 Innards of Simple OLS Variance and Covariance of OLS Estimators V â n = E â a = E v i e i = = σ e 1 n + X Xi X E vi E e i V b C â, b n = E b b = E w i e i = = σe 1 Xi X E wi E e i â = E a b n n b = E v i e i w j e j = σe = σe X Xi X j=1 v i w i Christopher Ting QF 30 Week 4 January 7, 017 9/9

10 Innards of Simple OLS Distribution of OLS Estimators Slope estimator b = Xi X Y i Y Xi X d ; b N b, σe 1 Xi X Intercept estimator â = Y b X ; â d N a, σ e 1 n + X Xi X 3 Christopher Ting QF 30 Week 4 January 7, /9

11 Innards of Simple OLS Distribution of OLS Estimators in Matrix Form To incorporate C â, b = σe X Xi X. Normal distribution â a d N b, b σ e σ e 1 n + X X i X X X i X σ e σ e X X i X 1 X i X Christopher Ting QF 30 Week 4 January 7, /9

12 Innards of Simple OLS Properties of Residuals Once the estimates â and b are obtained, we have ê i = Y i â b X i. 4 The variance of ê i is estimated as σ e = 1 n ê i 5 Christopher Ting QF 30 Week 4 January 7, 017 1/9

13 Series of residuals Innards of Simple OLS Hypothesis Testing ê i = Y i â b X i, Unbiased estimator of residual variance σ e = 1 n i = 1,,..., n Testing null hypothesis H 0 : b = β e.g. β = 0 t n = σ e b β ê i 1 n X i X Testing null hypothesis H 0 : a = α e.g. α = 0 6 â α t n = 7 1 σ e n + X n X i X Christopher Ting QF 30 Week 4 January 7, /9

14 Innards of Simple OLS Illustrative Example: Estimates Let X i := X i X, and Y i := Y i Y. Observation X i Y i Xi Yi Xi Yi X ixi X iyi X i Y i Total b = = 0.75, â = = Y i = X i Christopher Ting QF 30 Week 4 January 7, /9

15 Innards of Simple OLS Regression Result σ e = For a estimate, the standard error is For b estimate, the standard error is ê i = Christopher Ting QF 30 Week 4 January 7, /9

16 Innards of Simple OLS Gauss-Markov Theorem Gauss-Markov Theorem states that among all linear and unbiased estimators, the OLS estimators â and b have the minimum variances, i.e., V â and V b are the smallest possible and thus the OLS estimators are efficient estimation efficiency. OLS estimators under the classical conditions are called BLUE, viz. Best Linear Unbiased Estimators for the linear regression model. Y i = a + bx i + e i, i = 1,,..., n Christopher Ting QF 30 Week 4 January 7, /9

17 Asymptotic Limits Estimation with Asymptotically Large Sample When X i and Y i are stationary, by the Law of Large Numbers, lim n X n = µ X, lim n Y n = µ Y When n is asymptotically large, the biased second-order estimators approach the population variances σ X, σ Y, and covariance σ XY. S X := 1 n Xi X, S Y = 1 n Yi Y S XY := 1 n Xi X Y i Y When n is asymptotically large, OLS slope estimator is expressed as b = Xi X Y i Y Xi X = S XY SX 8 Christopher Ting QF 30 Week 4 January 7, /9

18 Asymptotic Limits Consistent Properties of OLS Covariance between X i and Y i when Y i = a + bx i + e i is CX i, Y i = CX i, a + bx i + e i = b VX i + Ce i, X i = b VX i = b = CX i, Y i VX i Hence from 8, lim b = b. n Implications: OLS b estimator is consistent: lim b = b n OLS â estimator is consistent: Since â = Y b X, lim â = µ Y b µ X = a n Christopher Ting QF 30 Week 4 January 7, /9

19 Asymptotic Limits Decomposition Consider Ŷ i ê i = â + b X i = Y i â b X i = Y i Ŷi TSS = ESS + RSS Yi Y = Ŷi Y + }{{} }{{} ê i }{{} Total Sum of Squares Explained Sum of Squares Residual Sum of Squares ESS is expressed as ESS = â + b Xi â b X = b Xi X. Christopher Ting QF 30 Week 4 January 7, /9

20 Asymptotic Limits Coefficient of Determination The population correlation coefficient is ρ XY = σ XY σ X σ Y. The sample estimate r XY is Xi X X i X r XY = n Xi X Xi X = S XY 9 S X S Y The OLS slope estimator is then b = S XY S X = r XY S X S Y S X = r XY S Y S X Consequently, ESS = r XY SY SX Coefficient of determination R ns X = r XY ns Y R = ESS TSS = r XY ns Y ns Y = r XY 10 Christopher Ting QF 30 Week 4 January 7, 017 0/9

21 Asymptotic Limits Illustrative Example: Goodness of Fit From Slides 14 and 6, we can compute the following quantities Sample correlation coefficient: TSS: ESS: RSS: R : Christopher Ting QF 30 Week 4 January 7, 017 1/9

22 Forecasting Forecasting The OLS forecast of Yn+1 given X n+1 is Ŷ n+1 = â + b X n+1 = Y b X + b X n+1 = Y + b X n+1 X. Now, by summing up and divide by n, we obtain Y = a + b X + 1 n e i. The point forecast is thus given by Ŷ n+1 = a + b X + 1 n e i + b X n+1 X Christopher Ting QF 30 Week 4 January 7, 017 /9

23 Forecasting Forecasting Error The true Yn+1 is a + bx n+1 + e n+1, so the forecast error is Y n+1 Ŷn+1 = b X n+1 X b X n+1 X + e n+1 1 n = b b Xn+1 X + e n+1 1 n e i e i Christopher Ting QF 30 Week 4 January 7, 017 3/9

24 Forecasting Properties of the OLS Forecast The forecast error conditional on X n+1 is normally distributed. The forecast is unbiased E Y n+1 Ŷn+1 X n+1 = 0 Variance of the OLS Forecast V Y n+1 Ŷn+1 Xn+1 = Xn+1 X V b + σ e + 1 n σ e = σe n + Xn+1 X Xi X The t-statistic of the forecast t n = Y n+1 Ŷn+1 σ e n + X n+1 X X i X Christopher Ting QF 30 Week 4 January 7, 017 4/9

25 Forecasting Point Forecast and Confidence Interval The point forecast is Ŷ n+1 = â + b X n+1 11 Since the forecast is a random variable, it has a confidence Interval associated with it. With 95% probability, the forecasted value falls within the confidence interval bounded by Ŷ n+1 ± t n, 97.5% σ e n + Xn+1 X Xi X Christopher Ting QF 30 Week 4 January 7, 017 5/9

26 Forecasting Illustrative Example: Forecast Continuing from Slides 14 and 6, suppose X 11 =. The forecast Ŷ11: The forecast standard error: Ŷ11: At the 95% confidence level, the forecast lower bound: At the 95% confidence level, the forecast upper bound: Christopher Ting QF 30 Week 4 January 7, 017 6/9

27 Hedging Applications Application: Hedging with Futures An institutional investor holds a portfolio of Japanese stocks that has returns following closely that of the Nikkei 5 stock index returns S t+1 /S t. Multiplier of Nikkei 5 futures traded on SGX is 500. To hedge against a potential bear market going forward, the investor forms a hedged portfolio such that the change of hedged portfolio s value P t+1 = P t+1 P t is P t+1 = f S t+1 h 500 F t+1 where f is a constant proportional factor that equates the unhedged value of the portfolio to S t, h is the number of contracts, and F t is the futures price. In effect, the investor wants to minimize the risk or variance of P t+1 : V P t+1 = f V S t+1 + h 500 V F t+1 f h 500 C S t+1, F t+1 Christopher Ting QF 30 Week 4 January 7, 017 7/9

28 Hedging Applications Solution to Hedging How many contracts h should the investor short? The FOC for minimizing V P t+1 with respect to decision variable h yields h 500 V F t+1 500f C St+1, F t+1 = 0 The risk-minimizing optimal hedge is to short h = f C S t+1, F t V F t+1 Estimation: Run the following regression Since b = C S t+1, F t+1 V S t+1 = a + b F t+1 + e t+1 F t+1, the number of contracts to short is h = b f 500 Christopher Ting QF 30 Week 4 January 7, 017 8/9

29 Hedging Applications Sample Exam Short Question On January 0, 017, the value of the portfolio is 38 billion yen, the Nikkei 5 index is 19,137.91, and the OLS estimate for b is How many contracts should the fund manager short? Christopher Ting QF 30 Week 4 January 7, 017 9/9

Quantitative Analysis of Financial Markets. Summary of Part II. Key Concepts & Formulas. Christopher Ting. November 11, 2017

Quantitative Analysis of Financial Markets. Summary of Part II. Key Concepts & Formulas. Christopher Ting. November 11, 2017 Summary of Part II Key Concepts & Formulas Christopher Ting November 11, 2017 christopherting@smu.edu.sg http://www.mysmu.edu/faculty/christophert/ Christopher Ting 1 of 16 Why Regression Analysis? Understand