CHAPTER 6: USING MULTIPLE REGRESSION

Transcription

1 CHAPTER 6: USING MULTIPLE REGRESSION There re my situtios i which oe wts to predict the vlue the depedet vrible from the vlue of oe or more idepedet vribles. Typiclly: idepedet vribles re esily mesurble Collectio of idepedet vribles explis the depedet vrible Exmple: The retur of most stocks is closely tied to the retur o the mrket. I fice it is importt to try d predict the retur o stock (depedet vrible) from the retur o the mrket (e.g. S&P500 Idex). Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 06

2 Note tht for idividul stock: Aul Retur Ed of Yer' s Price + Divided - Lst Yer' s Lst Yer' s Price Price Yer Closig Stock Price Divideds Aul Retur o Lilly Yer Mrket Retur Aul Retur o Lilly Regressio Fit Aul Retur o Lilly 994 $ 6.50 $ % 995 $ 7.88 $ % % 73.8% 48.68% 996 $ 37.3 $ % % 36.4% 6.68% 997 $ $ % % 7.73% 5.40% 998 $ 4.75 $.5.56% %.56% 4.0% 999 $ $ % % 63.39% 47.60% 000 $ 73.5 $ % % 9.33% -8.0% 00 $ $ % % 6.7% 46.09% 00 $ $.0-4.6% % -4.6% 9.3% 003 $ $.4.73% %.73% 3.05% 004 $ $ % 004.3% 4.74% -0.90% % 9.4% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 07

3 Plottig Aul Retur gist Mrket Retur i Sctter Plot d icludig the best lier fit we get: Lilly Retur vs. Mrket 80.00% 60.00% Lilly Retur 40.00% 0.00% 0.00% -5.00% 0.00% 5.00% 0.00% 5.00% 0.00% 5.00% 30.00% 35.00% -0.00% % Mrket Retur Aul Retur o Lilly Lier (Aul Retur o Lilly) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 08

4 CONCLUSION: There ppers to be lier reltioship, but the wide dispersio bout the best fittig lie idictes we do ot hve perfect lier reltioship. Possibly other fctors besides mrket retur determie the ul retur o the Lilly stock. INTERMEZZO: ANALYSIS OF VARIANCE PROBLEM DESCRIPTION Suppose you hve lier fuctio f ( x,, ) xk,, k, b x + + k xk + b k i xi + b i were the prmeters,, k,b ukow. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 09

5 You hve N dt poits (,, y ) y geerted by this lier fuctio but the dt poits coti some mesuremet error. Together with ech dt-poit y of this lier fuctio you hve the iput- vribles (,, xk ) x. If the mesuremet error were to be zero d you kew the vlues of the prmeters,, k,b the y k i i x i + b Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 0

6 OBJECTIVE:. Obti best-guess for the prmeters:,, k, b. Estimte the ucertity i the mesuremet error. MEETING THE FIRST OBJECTIVE: Clculte,, k, b such tht the sum of squres error s (SSE) is miimized. N k N k ( y i xi b) Mi ( y i xi b),, k b i i Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge

7 MEETING THE SECOND OBJECTIVE: Itroduce the rdom vrible Y represetig mesuremet of the fuctio f ( x,, xk,, k, b) k i i x i + b d ssume tht:.. where ε does ot deped o the iput-vribles (or idepedet vribles ( k x,, x ) : k k i i i E[ Y,,, b] x + b Y E[ Y,,, b] + ε k Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge

8 Assume tht error term ε is orml distributed with me 0 d stdrd devitio σ or vrice σ PROBLEM THAT REMAINS TO BE SOLVED: Estimtio of the vrice σ THE ANALYSIS OF VARIANCE Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 3

9 ANALYSIS OF VARIANCE: Simplest Approch: Suppose f ( x,, xk b) b Hece, we hve dt ( y,, y ) d we eed to represet this etire set of dt by oe best-guess b. As it turs the lest-squres fit for b equls: b y y We clculte -observed error terms: ε y b y y,,, Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 4

10 d estimte the vrice for the error-term usig ( ε ) ( y b) ( y y) BUT! We kow estimtor of vrice bove is BIASED. A ubised estimtor of the vrice is: ( y y) Coclusio: A ubised estimtor for the vrice of the error term is ( ε Iformlly, we divide by (-) isted of dividig by s we used prmeter b to represet the etire dt set: Sttistici refers to this s loosig degree of freedom. ) ( y,, y ) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 5

11 Simple Lier Regressio: Suppose f ( x,, xk b x + ) b Hece, we hve dt ( y,, y ) d we eed to represet this etire set of dt by oe lier lie specified by two best-guesses, b d the idepedet vrible x. Simple lier regressio gives us the these lestsqures prmeter estimtes â d b. We clculte -observed error terms: ε y x b,,, d estimte the vrice for the error-term usig ( ε ) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 6

12 BUT! As it turs out the estimte for the vrice of the error-term bove is BIASED. A ubised estimtor of the vrice i of the error term i this cse is: Iformlly, we divide by (-) isted of dividig by s we used prmeters, b to represet the etire dt set: ( ε ) ( y,, y ) We lost gi degree of freedom. HOW MUCH BETTER DOES THE SIMPLE LINEAR REGRESSION EXPLAIN THE VARIANCE IN THE DATA COMPARED TO THE SIMPLEST APPROACH OF ONE POINT ESTIMATE? Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 7

13 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 8 ANSWER IS GIVEN BY R-SQUARED! ) ( y y SS ) ( b x y SSE SS SSE SS SSE SS R ) ( ) ( ) ( ) ( y y b x y y y b x y

14 y x b ( ) Vrice Residuls Smple Vrice Smple Vrice Oe sys tht R-Squred metric fvors the more complicted pproch s (-)/(-) is less th d does ot ccout for the fct tht simple lier regressio uses oe more prmeter to represet the dt. Multiple Lier Regressio: Suppose f ( x, ), xk,, k, b x + + k xk + b Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 9

15 Hece, we hve dt ( y,, y ) d we eed to represet this etire set of dt by oe hyper-ple lie specified by (k+) best-guesses Multiple lier regressio gives us the these lest-squres fits We clculte -observed error terms:,,, b k.,,, b k. ε k i i i y x b,,, d estimte the vrice for the error-term usig ( ε BUT! As it turs out the estimte for the vrice of the error-term bove is BIASED. ) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 0

16 A ubised estimtor of the vrice i of the error term is: ( ) k + ( ε Iformlly, we divide by -(k+) isted of dividig by s we used (k+) prmeters to represet the etire dt set: ( y,, y ) ) We lost (k+) degrees of freedom. HOW MUCH BETTER DOES THE MULTIPLE LINEAR REGRESSION APPROACH EXPLAIN THE VARIANCE IN THE DATA COMPARED TO THE SIMPLEST APPROACH OF ONE POINT ESTIMATE? Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge

17 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge ANSWER IS GIVEN BY R-SQUARED! ) ( y y SS ) ( i k i b x y SSE i SS SSE SS SSE SS R ) ( ) ( ) ( ) ( i k i i k i y y b x y y y b x y i i

18 ( k + ) Vrice Residuls Smple Vrice Oe sys tht R-Squred fvors the more complicted pproch cosiderbly s (-k-)/(-) is less th d does ot ccout for the fct tht multiple lier regressio uses k more prmeters to represet the dt. Oe sys if you wt to hve mesure tht compres the vrice i the multiple regressio fit compred to the simplest pproch oe should be comprig vrices d ot ust the SS d SSE. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 3

19 ADJUSTED R-SQUARED: Adusted R Vrice Residuls Smple Vrice SSE - (k + ) SS - Prcticl Implictios: Oe c lwys improve the R-squred vlue by icludig more idepedet vribles. It ppers tht we re obtiig better fit. However, with the sme umber of dt poits, the umber of dt poits per fitted prmeter decreses d we ituitively feel loss of ccurcy i ech fitted prmeter. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 4

20 I the extreme cse if we tke s my idepedet vribles (d prmeters) s the origil dt we obti perfect fit (R-Squred). We bsiclly get out dt bck. Is this perfect fit? No, you eed to ccout for the complexity of the model. You c icrese the umber of idepedet vribles s log s the ADJUSTED R-SQUARED icreses. STRIVE FOR THE SIMPLEST MODEL THAT BEST EXPLAINS THE VARIANCE IN THE DATA Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 5

21 MULTIPLE LINEAR REGRESSION: FORECASTING AUTO SALES Suppose you wt to Forecst Auto-Sles (i thousds of crs). Wht re some vribles tht might ifluece uto sles i prticulr qurter of the yer? Previous Qurter s GNP (Gross Ntiol Product) Previous Qurter s Iterest Rte File Auto Multiple Lier Regressio.xls cotis Auto-Sles Dt for STEP : Let GNP(x) be Gross Ntiol Product i Qurter x. LgGNP(x,k) GNP(x-k) : i.e. GNP k qurters before Crete lg vribles from the origil dt. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 6

22 Historicl dt Yer Qurter Sles GNP Iterest Rte LgGNP() LgIt() % Yer Qurter Sles GNP Iterest Rte LgGNP() LgIt() % % % % % % % 70.90% % % % % % % % % % % % % % % STEP. Use Dt Alysis Toolpck Regressio Alysis to perform Multiple Lier Regressio Alysis. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 7

23 STRATEGY: Use Yers (5 Yers) for the regressio lysis d use Yers ) for vlidtio of the regressio lysis. Note: Hve to use Dt Alysis Toolpck becuse of multiple idepedet vribles. INTERCEPT d SLOPE oly work i the cse of SINGLE LINEAR REGRESSION. Coefficiets Itercept LgGNP() LgIt() S(q,x) Sles i Qurter q d Yer x. S(q,x)0.473*LgGNP(x,)-50.3*LgINT(x,)+56.9 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 8

24 INTEPRETATION: Whe GNP icreses AUTO SALES INCREASE Billio Dollrs icrese i GNP will result i pproximtely 47 more crs beig sold oe-percetge poit icrese i iterest (sy from 6% to 7%) will result i reductio of cr sles of 50.3*000* PREDICTION: Iterest Rte i Curret Qurter 8.8%, GNP 399 billio Predicted Auto Sles Next Qurter 0.473* * Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 9

25 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 30 UNCERTAINTY OF PREDICTION MAKE NORMAL ASSUMPTION OF RESIDUALS: STEP 5A: Model the Error Term (Residuls) s orml distributed rdom vrible with me µ0 d stdrd devitio s ε, such tht: (LOST THREE DEGREES OF FREEDOM) i i s 3 ε ε ε + ε b X X X X b Y E Y ],,,, [

26 INTERMEZZO: THE NORMAL DISTRIBUTION My biologicl pheome (height, weight, legth) follow bell-shped curve tht c be represeted by orml distributio. Cosider the productio of me shoes. You wt to offer these shoes i my differet sizes. However, you eed to decide the percetge of shoes to produce i ech size. Let Y be the legth of me s feet. Y N(µ, σ): E[Y] µ ( xu) σ fy ( y µσ, ) e σ π Vr(Y) σ Some hdy rules of thumb: Pr( µ σ < Y < µ + σ ) 0.68 Pr( µ σ < Y < µ + σ ) 0.95 Pr( µ 3σ < Y < µ + 3σ ) 0.99 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 3

27 Probbility Desity Fuctio - N(,0.5) % 95% 99% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 3

28 RETURNING TO UNCERTAINTY OF PREDICTION: Iterest Rte i Curret Qurter 8.8%, GNP 399 billio Predicted Auto Sles Next Qurter 0.473* * % Credibility Itervl 668.3± % Credibility Itervl 668.3± * ± % Credibility Itervl 668.3± 3* ± Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 33

29 WHAT ABOUT THE NORMALITY ASSUMPTION OF THE RESIDUALS?. Norml Distributio Assumptio of Residuls: Residuls re orml distributed rdom vrible with me µ0 d stdrd devitio s ε, s ε 3 i ε i. Rescle Residuls to Stdrdized Residuls by dividig by s ε. Hece, with the ssumptio bove, Stdrdized Residuls re Stdrd Norml Distributed with me 0 d stdrd devitio. (i.e. N(0,)) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 34

30 3. Order the Stdrdized Residuls such tht 4. clculte: ε s () ε < ε () s ε < < ε s ε < ε ( ) ( ) ε() ε() ε( ) ζ() F, ζ() F,, ζ( ) F sε sε sε where F is the stdrd orml cumultive distributio fuctio. s ε 5. Plot the poits. ( ) (, ζ () ),(, ζ() ),(, ζ( ) ),(, ζ( ) ) If Stdrdized Residuls re i fct N(0,) distributed these poits should ll be oe the lie yx. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 35

31 00% NORMAL PROBABILITY PLOT Stdrd Norml CDF Vlues 80% 60% 40% 0% 0% 0% 0% 40% 60% 80% 00% Empiricl CDF Vlues ARE RESIDUALS NORMALLY DISTRIBUTED? Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 36

32 VALIDATION OF REGRESSION: STRATEGY: Use Yers (5 Yers) for the regressio lysis d use Yers ) for vlidtio of the regressio lysis. Regressio Coefficiet Regressio Coefficiet Itercept Yer Qurter Sles GNP Iterest Rte LgGNP() LgIt() PREDICTION RESIDUALS % % % % % % % % % % % % % % % % % Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 37

33 PLOT OF ADDITIONAL RESIDUALS IN NORMAL PDF WITH MEAN 0 d STANDARD DEVIATION NORMAL DISTRIBUTION RESIDUALS.00E-03.50E-03.00E E E RESIDUALS NORMAL PDF.5% BOUND 97.5% BOUND VALIDATION SET Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 38

34 CONCLUSION: All dditiol observtios fll well withi 95% credibility itervl. Oly oe observtio does flls o the boudry of this 95% credibility itervl (out of 8 observtios). Norml Probbility Plot + Grph with dditiol observtios support the lier fit d the ucertity model for predictios. CONCERN: R-SQUARED 30.94% Smple Vrice Vrice Residuls ADJUSTED R-SQUARED 4.04% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 39

35 NEXT ATTEMPT: Avilble iformtio cotis: QUARTER OF THE YEAR d UNEMPLOYMENT RATE. Suggestio: Use QUARTER OF THE YEAR d UNEMPLOYMENT RATE i previous qurter s dditiol idepedet vrible i multiple lier regressio. Existig Idepedet Vribles: Previous Qurter s GNP (Gross Ntiol Product) Previous Qurter s Iterest Rte New Idepedet Vribles: Previous Qurter s Uemploymet rte Qurter of the Yer Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 40

36 MODELING QUARTER OF THE YEAR AS A INDEPENDENT VARIABLE GNP, Iterest Rte d Uemploymet Rte re mesurble vribles with turl ttribute scle: GNP is mesured i the ttribute: Dollrs ($) Iterest Rte is mesured i: Percetge Poits (%) Uemploymet Rte is mesured i: Percetge Poits (%) A higher GNP Vlue i terms of dollrs is better A lower Iterest Rte i terms of % is better A Lower Uemploymet Rte i terms of % is better Origil Dt ssigs to first qurter, to secod qurter, 3 to third qurter, 4 to fourth qurter. Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 4

37 Yer Qurter Sles GNP It Uemp LgGNP() LgIt() LgUEMp() % 5.70% % 5.90% % 5.90% % 5.70% % 6.00% % 5.90% % 6.0% 70.90% 6.00% % 7.30% % 6.0% % 7.70% % 7.30% % 7.40% % 7.70% % 7.40% % 7.40% % 7.40% % 7.40% % 7.40% % 7.40% % 8.30% % 7.40% % 8.80% % 8.30% % 9.40% 58.80% 8.80% % 0.00% 63.40% 9.40% % 0.70% % 0.00% % 0.40% % 0.70% % 0.0% % 0.40% % 9.40% % 0.0% % 8.50% % 9.40% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 4

38 HOWEVER: We c ot ssig iterprettio of Better or Worse to higher qurter umber. Modelig this vrible s Q,,3,4 would do tht! Better Approch: Itroduce Three Biry Dummy Vribles Q, Q, Q3: Q,Q0,Q30: First Qurter (Jury - Mrch) Q0,Q,Q30: Secod Qurter (April - Jue) Q0,Q0,Q3: Third Qurter (July September) Q0,Q0,Q30: Fourth Qurter (October December) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 43

39 Coefficiets Itercept LgGNP() LgIt() LgUEMp() Q Q Q S(q,x) Sles i Qurter q d Yer x. S(q,x) 0.44*LgGNP(x,)-857.7*LgINT(x,) *LgUEMP()+7.08*Q +3.64*Q+75.97*Q Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 44

40 INTEPRETATION: Billio Dollrs icrese i GNP will result i pproximtely 4 more crs beig sold A oe-percetge poit icrese i iterest (sy from 6% to 7%) will result i reductio of cr sles of 857.7*000* (Approx. 8000) oe-percetge poit icrese i uemploymet rte (sy from 6% to 7%) will result i reductio of cr sles of *000* (Approx ) Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 45

41 Durig Jury Mrch cr sles ru 7000 higher th October December Durig April Jue cr sles ru 3000 higher th October December Durig July September cr sles ru higher th October December PREDICTION: I Fourth Qurter of the yer: Iterest Rte 8.8%, GNP 369 billio, Uemploymet Rte 7.% Predicted Auto Sles First Qurter 0.44* * * * +3.64* * Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 46

42 Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 47 UNCERTAINTY OF PREDICTION MAKE NORMAL ASSUMPTION OF RESIDUALS: STEP 5A: Model the Error Term (Residuls) s orml distributed rdom vrible with me µ0 d stdrd devitio s ε, such tht: (WE LOST SEVEN DEGREES OF FREEDOM) i i s 7 ε ε ],,,,,,,,,,,, [ Q Q Q X X X b Y E Y + ε b Q Q Q X X X

43 SSE VARIANCE RESIDUALS STANDARD ERROR 76. *STANDARD ERROR *STANDARD ERROR SS SAMPLE VARIANCE R-Squred 73.54% Adusted R-Squred 63.6% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 48

44 RETURNING TO UNCERTAINTY OF PREDICTION: Predicted Auto Sles First Qurter 0.44* * * * +3.64* * % Credibility Itervl ± % Credibility Itervl ± * ± % Credibility Itervl ± 3* ± Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 49

45 WHAT ASSUMPTION WAS USED ABOVE? NORMAL PROBABILITY PLOT.000 Stdrd Norml CDF Vlues Empiricl CDF Vlues Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 50

46 VALIDATION OF REGRESSION: STRATEGY: Use Yers (5 Yers) for the regressio lysis d use Yers ) for vlidtio of the regressio lysis. Yer Qurter Sles GNP It Uemp PREDICTED SALES RESIDUALS % 7.40% % 7.30% % 7.0% % 7.00% % 7.0% % 7.0% % 6.90% % 6.80% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 5

47 PLOT OF ADDITIONAL RESIDUALS IN NORMAL PDF WITH MEAN 0 d STANDARD DEVIATION 76. NORMAL DISTRIBUTION RESIDUALS.50E-03.00E-03.50E-03.00E E E RESIDUALS NORMAL PDF.5% BOUND 97.5% BOUND VALIDATION SET Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 5

48 SUMMARY: We icluded more iformtio to predict Auto Sles (i.e. Uemploymet Rte d Qurter of the Yer) d were ble to chieve higher R-Squred Vlue d cosequetly smller Stdrd Error. The ucertity i the predictio is less s result of the smller stdrd error The model is LESS VALID s more residuls of the vlidtio set fll outside the.5% or 97.5% credibility bouds SO, DID WE IMPROVE OUR MODEL? Aswer: Check the Adusted R-Squred Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 53

49 THREE PARAMETER MODEL R-SQUARED 30.94% ADJUSTED R-SQUARED 4.04% SEVEN PARAMETER MODEL R-SQUARED 73.54% ADJUSTED R-SQUARED 63.6% SIX PARAMETER MODEL R-SQUARED 7.63% ADJUSTED R-SQUARED 64.57% FIVE PARAMETER MODEL R-SQUARED 69.37% ADJUSTED R-SQUARED 6.57% Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 54

50 VALIDATION SIX PARAMETER MODEL LgGNP() LgIt() LgUEMp() Q Q INTERCEPT NORMAL DISTRIBUTION RESIDUALS.50E-03.00E-03.50E-03.00E E E RESIDUALS NORMAL PDF.5% BOUND 97.5% BOUND VALIDATION SET Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 55

51 VALIDATION FIVE PARAMETER MODEL LgGNP() LgIt() LgUEMp() Q INTERCEPT NORMAL DISTRIBUTION RESIDUALS.50E-03.00E-03.50E-03.00E E E RESIDUALS NORMAL PDF.5% BOUND 97.5% BOUND VALIDATION SET Lecture Notes by: Dr. J. Ree v Dorp Chpter 6 - Pge 56