ACE 562 Fall Lecture 8: The Simple Linear Regression Model: R 2, Reporting the Results and Prediction. by Professor Scott H.

Transcription

1 ACE 56 Fall 5 Lecure 8: The Simple Linear Regression Model: R, Reporing he Resuls and Predicion by Professor Sco H. Irwin Required Readings: Griffihs, Hill and Judge. "Explaining Variaion in he Dependen Variable," Secion 8.1; "Reporing-Summarizing Resuls," Secion 8.; and "Predicing Expendiure," Secion 7.3 in Learning and Pracicing Economerics ACE 56, Universiy of Illinois a Urbana-Champaign 8-1

2 Overview There are wo major reasons for analyzing he linear saisical model, Explain how y changes as x changes Predic y given x Technique of LS guaranees ha esimaed line will be he bes-fiing, in he sense of having he smalles sum of squared errors Despie being bes-fiing he degree of fi can vary considerably for LS lines If he scaerplo of daa are close o he line, we would say he LS line fis well If he scaerplo of daa are no close o he line, we would say he LS line fis poorly Suggess he need o quanify how well a LS line fis he daa ACE 56, Universiy of Illinois a Urbana-Champaign 8-

3 y y x x ACE 56, Universiy of Illinois a Urbana-Champaign 8-3

4 Sandard Error of Regression as a Measure of Fi To begin, noe ha he esimaed LS line yields a se of fied values y = b + b x ˆ 1 The associaed errors of fi are given by eˆ = y yˆ I is naural o firs consider he sandard error as a measure of fi Provides a measure of he ypical regression error The formula o esimae he sandard error of he regression is eˆ ˆ ˆ ˆ ˆ σ T ˆ σ e + e + + e = = = = T T T Noe ha unis of measuremen for ˆ σ are always he same as for y ACE 56, Universiy of Illinois a Urbana-Champaign 8-4

5 For he food expendiure problem, we found ha ˆ ˆ σ = σ = = We inerpreed his as saying ha he ypical, or expeced, error for he food expendiure LS regression line is $6.84 per week Some quesions quickly arise Is $6.84 per week big or small? How does $6.84 per week compare o oher LS lines? Suggess he need for a measure of fi ha does no depend on he unis of measuremen of he dependen variable y R, or he coefficien of deerminaion, provides a pure, uni-less measure of fi ACE 56, Universiy of Illinois a Urbana-Champaign 8-5

6 R as a Measure of Fi In he previous secion we noed ha he esimaed regression errors are given by, eˆ = y yˆ Re-arranging his expression, we can show ha he value of y can be decomposed ino wo componens, y = yˆ + eˆ To begin he derivaion of R i is helpful o subrac he mean of y from boh sides of he equaion ( y y) = ( yˆ y) + eˆ In words, his says, Toal deviaion in y = componen explained by x + unexplained componen ACE 56, Universiy of Illinois a Urbana-Champaign 8-6

7 ACE 56, Universiy of Illinois a Urbana-Champaign 8-7

8 Since we are ineresed in variaion and no deviaion, le s square boh sides of he previous equaion, ( y y) = [( yˆ y) + eˆ ] Which can be expanded as follows, y y = yˆ y + eˆ + yˆ y eˆ ( ) ( ) ( ) Now, sum boh sides of he previous equaion, T T T T ( y ) ( ˆ ) ˆ ( ˆ ) ˆ y = y y + e + y y e = 1 = 1 = 1 = 1 Since, T ( yˆ y) eˆ = ( ye ˆ ˆ yeˆ ) = 1 = 1 T T = ye ˆˆ y eˆ = 1 = 1 T T = ye ˆˆ = 1 ACE 56, Universiy of Illinois a Urbana-Champaign 8-8

9 T = ( b + b x )ˆ e = 1 1 T = b eˆ + b xeˆ 1 = 1 = 1 T = b xeˆ = T = 1 Hence, he earlier relaionship reduces o, T T T ( y ) ( ˆ ) ˆ y = y y + e = 1 = 1 = 1 This is an imporan relaionship, which shows he decomposiion of oal sample variaion in y ino explained and unexplained componens ACE 56, Universiy of Illinois a Urbana-Champaign 8-9

10 Now, define he following erms, T = 1 T = 1 ( y y) ( yˆ y) = Sum of Squares Toal (SST) = Sum of Squares Regression (SSR) T eˆ = Sum of Squares Error (SSE) = 1 Hence, SST = SSR + SSE This decomposiion is provided in he regression oupu of virually all economeric packages Usually labeled as he analysis of variance able ACE 56, Universiy of Illinois a Urbana-Champaign 8-1

11 ACE 56, Universiy of Illinois a Urbana-Champaign 8-11

12 A widespread use of he informaion in he analysis of variance able is o define a measure of he proporion of variaion in y explained by x wihin he regression model To obain his measure, firs divide he previous equaion by SST o obain he relaionship in proporionae form, SST SSR SSE = + SST SST SST or, SSR 1 = + SST SSE SST Now, we can define, SSR R = SST Shows ha R measures he oal sample variaion in y explained by he variaion in x The formal erm for deerminaion R is coefficien of ACE 56, Universiy of Illinois a Urbana-Champaign 8-1

13 R ofen is saed in percenage erms as follows, R = SSR SST X 1 Noe ha by subsiuing ino he SST equaion in proporionae form, we obain, or, 1 R = R + SSE SST = 1 SSE SST Two imporan limis can be placed on R, R 1 Why is R non-negaive? Wha does i mean if R is? Wha does i mean if R is 1? ACE 56, Universiy of Illinois a Urbana-Champaign 8-13

14 Correlaion and R The sample correlaion coefficien for wo random variables x and y is, r xy, = cov( ˆ xy, ) var( ˆ x) var( ˆ y) There are wo ineresing relaionships beween r, and R in he case of he simple linear regression model, r = R xy, x y r = R yy ˆ, ACE 56, Universiy of Illinois a Urbana-Champaign 8-14

15 R for he Food Expendiure Example For he household food expendiure and income example, he relevan calculaion is, 86.6 R = = Indicaes we are able o explain 31.7% of he oal variaion in food expendiure by he variaion in income This leaves 68.3% of he variaion unexplained, suggesing he explanaory power of he model is low Typical of cross-secional daa Regressions based on economic ime-series daa end o have much higher Rs due o shared ime-rends of he variables ACE 56, Universiy of Illinois a Urbana-Champaign 8-15

16 ACE 56, Universiy of Illinois a Urbana-Champaign 8-16

17 Sample Regression Oupu from Excel SUMMARY OUTPUT Regression Saisics Muliple R R Square Adjused R Square Sandard Error Observaions 4 ANOVA df SS MS F Significance F Regression Residual Toal Coefficiens Sandard Error Sa P-value Lower 95% Upper 95% Inercep X Variable ACE 56, Universiy of Illinois a Urbana-Champaign 8-17

18 Words of Wisdom from Peer Kennedy (p. 7) In general, economericians are ineresed in obaining good parameer esimaes where good is no defined in erms of R. Consequenly, he measure R is no of much imporance in economerics. Unforunaely, however, many praciioners ac as hough i is imporan, for reasons ha are no enirely clear, as noed by Cramer (1987, p.53), These measures of goodness of fi have a faal aracion. Alhough i is generally conceded among insiders ha hey do no mean a hing, high values are sill a source of pride and saisfacion o heir auhors, however hard hey may ry o conceal hese feelings. Implicaions I is a misake o focus oo closely on of economeric success R as a measure A low R does no necessarily indicae he esimaed parameers do no provide useful informaion ACE 56, Universiy of Illinois a Urbana-Champaign 8-18

19 ACE 56, Universiy of Illinois a Urbana-Champaign 8-19

20 SUMMARY OUTPUT Regression Saisics Muliple R.56 R Square.3 Adjused R Square.3 Sandard Error 6.84 Observaions 4. ANOVA df SS Regression Residual Toal Coefficiens Sandard Error Inercep X Variable SUMMARY OUTPUT Regression Saisics Muliple R.6 R Square.7 Adjused R Square.1 Sandard Error 5.96 Observaions. ANOVA df SS Regression Residual Toal Coefficiens Sandard Error Inercep X Variable ACE 56, Universiy of Illinois a Urbana-Champaign 8-

21 Reporing he Resuls of Regression Analysis There are several, sandard mehods of reporing regression resuls One common form is, yˆ = x R =.317 (4.8) (.553) ( se..) where s.e. sands for esimaed sandard error Anoher is o replace he sandard errors by - saisics for a zero null, yˆ = x R =.317 (1.84) (4.) ( sa.) ACE 56, Universiy of Illinois a Urbana-Champaign 8-1

22 Finally, i has become commonplace in recen years o also repor p-values in eiher reporing forma, yˆ = x R =.317 (4.8) (.553) ( se..) [.733] [.16] [ p value] yˆ = x R =.317 (1.84) (4.) ( sa.) [.733] [.16] [ p value] If resuls for a large number of regressions mus be repored, a abular forma should be employed The same basic informaion should be repored in he able ACE 56, Universiy of Illinois a Urbana-Champaign 8-

23 Predicion wih Regression Models Predicion is a subjec of grea pracical imporance Ofen given lile reamen in exbooks We will cover he subjec in deail The erms predicion and forecas can be used inerchangeably In a regression seing, we wan o predic he value of he dependen variable y for a given value of he independen variable x Example: Wha would be he level of food expendiure for a family ha has a weekly income of $6? An example of cross-secional predicion We will consider wo approaches o answering his specific quesion ACE 56, Universiy of Illinois a Urbana-Champaign 8-3

24 Case 1: β 1, β, x and σ Known Assume a linear saisical model is he daa generaing process for food expendiure, y = β + β x + e 1 where, as before, y is food expendiure, x is income, and e and y are assumed o be iid wih he following disribuions, e ~ N(, σ ) and y ~ N( β + β x, σ ) 1 I is imporan o noe ha we are assuming ha β1and β are known Since he saisical model holds for any observaion, we can wrie he following version, y = β + β x + e 1 where y is he prediced value of food expendiure for a given value of income x ACE 56, Universiy of Illinois a Urbana-Champaign 8-4

25 A his poin, y is no a predicion because he value of e is unknown The bes we can do is o use he expeced value of y as our predicion, [ ] E( y ) = yˆ = E β + β x + eo = β + β x 1 1 ŷ is called he leas squares predicor Noe ha ŷ can differ from y because he fuure disurbance e may differ from is implici predicor, which is is mean value of Hence, ŷ is a random variable and we are ineresed in is properies in a repeaed sampling conex I is convenional o examine he sampling properies of he forecas error, raher han he sampling properies of ŷ direcly ACE 56, Universiy of Illinois a Urbana-Champaign 8-5

26 Forecas error The forecas error is defined as he difference beween he acual y and he predicion ŷ f = y yˆ = ( β + β x + e ) ( β + β x ) = e 1 1 Noe ha he forecas error in his case is exacly equal o he error erm in he saisical model Based on he above relaionship, we can examine some imporan sampling properies of he forecas error Mean forecas error We can examine he expeced value of he forecas error ha we should expec in repeaed sampling, E( f) = E( y yˆ ) = E( e ) = This shows ha he leas square predicor is an unbiased linear predicor On average, in he repeaed sampling sense, he prediced food expendiure will equal he acual value ACE 56, Universiy of Illinois a Urbana-Champaign 8-6

27 Variance of he forecas error While he leas squares predicion is unbiased, i may sill be wide of he mark for any paricular predicion The reliabiliy of he predicion in repeaed sampling is measured by he variance of he predicion var( f) = E( y yˆ ) = E( e ) = σ Shows ha he variance of he forecas error is exacly equal o he variance of he regression error erm (also assumed o be known) Sandard error of he forecas ( ) = var( ) = = se f f σ σ ACE 56, Universiy of Illinois a Urbana-Champaign 8-7

28 95% confidence inerval for forecas We can consruc a sandard normal random variable as follows, Z f y yˆ f ~ N(,1) var( f ) σ = = Since Z f is a sandard, normal random variable, we can wrie, Subsiuing for Z f, P[ 1.96 Z f 1.96] =.95 P y yˆ = σ [ ].95 Muliply he inequaliy in he brackes by σ, P[ 1.96σ y yˆ 1.96 σ ] =.95 Now, add ŷ o each erm, P[ yˆ 1.96σ y yˆ σ ] =.95 ACE 56, Universiy of Illinois a Urbana-Champaign 8-8

29 Hence, he 95 percen confidence inerval for y is, yˆ ± 1.96σ Inerpreaion: In repeaed sampling, we expec 95% of inerval predicions o conain he realized y We can generalize o any predicion confidence level, 1 α, as follows, Py [ ˆ Z σ y yˆ + Z σ] = 1 α α α and, ŷ ± Z α σ ACE 56, Universiy of Illinois a Urbana-Champaign 8-9

30 ACE 56, Universiy of Illinois a Urbana-Champaign 8-3

31 Case : β 1, βand σ Esimaed We sar wih he same assumpion ha a linear saisical model is he daa generaing process for food expendiure, y = β + β x + e 1 Again, since he saisical model holds for any observaion, we can wrie he following version, y = β + β x + e 1 where y is he prediced value of food expendiure for a given value of income x However, we now make he more realisic assumpion ha β1and β mus be esimaed In his case, he bes we can do is: 1) replace β 1 and β wih he esimaors b 1 and b, and ) replace he unknown error wih is expeced value of zero ACE 56, Universiy of Illinois a Urbana-Champaign 8-31

32 The leas squares predicor is hen, ŷ = b1+ bx Noe ha ŷ can now differ from y for wo reasons 1. The fuure disurbance e may differ from is implici predicor, which is is mean value of. The esimaors b 1 and b are likely o produce esimaes ha differ from he rue populaion parameers β 1 and β Hence, ŷ is a random variable and we are ineresed in is properies in a repeaed sampling conex Again, i is convenional o examine he sampling properies of he forecas error, raher han he sampling properies of ŷ direcly ACE 56, Universiy of Illinois a Urbana-Champaign 8-3

33 Forecas error The forecas error is defined as he difference beween he acual y and he predicor ŷ f = y yˆ = ( β + β x + e ) ( b + b x ) 1 1 or, f = y yˆ = ( β b) + ( β b ) x + e 1 1 Noe ha he forecas error in his case does no simply equal he regression error erm The forecas error is now a funcion of hree random variables, b 1, b, and e Based on he above relaionship, we can again examine he sampling properies of he forecas error ACE 56, Universiy of Illinois a Urbana-Champaign 8-33

34 Mean forecas error The expeced value of he forecas error ha we should expec in repeaed sampling is, E( f) = E( y yˆ ) = E[( β1 b1) + ( β b) x + e] = [ β1 Eb ( 1)] + [ β Eb ( )] x + Ee ( ) = [ β1 β1] + [ β β] x + E( e) = Ee ( ) = This shows ha even when he parameers have o be esimaed he leas squares predicor is unbiased On average, in he repeaed sampling sense, he prediced food expendiure will equal he acual value ACE 56, Universiy of Illinois a Urbana-Champaign 8-34

35 Variance of he forecas error While he leas squares predicor is unbiased, i may sill be wide of he mark for any paricular predicion The reliabiliy of he predicor is measured by he variance of he forecas error var( f ) = E( y yˆ ) = E[( β b) + ( β b ) x + e )] 1 1 Expanding he square, var( f) = E[( β b) + (( β b ) x ) + e ( β b)( β b ) x + ( β b ) x e + ( β b) e ] Take he expecaions hrough o each erm, var( f) = E[( β b) ] + E[(( β b ) x ) ] + E[ e ] E[( β b)( β b ) x ] + E[( β b ) x e ] + E[( β b) e ] Which reduces o, var( f ) = E[( β b) ] + E[(( β b ) x ) ] + E[ e ] + E[( β b)( β b ) x ] Now change he noaion, ACE 56, Universiy of Illinois a Urbana-Champaign 8-35

36 var( f) = var( b) + var( b ) x + cov( b, b ) x + σ 1 1 The nex sep is o subsiue he definiions of var( b 1), var( b ), and cov( b1, b ) ha we derived earlier, var( f ) T x 1 1 = σ x T T T ( x x) ( x x) = 1 = 1 = σ + x + σ x + σ T ( x x) = 1 Afer some fairly edious algebra, his can be reduced o, 1 ( x x) var( f ) = σ T T ( x x) = 1 ACE 56, Universiy of Illinois a Urbana-Champaign 8-36

37 1 var( f ) = σ T T ( x x) = 1 ( x x) Key poins: Since erm in brackes mus be posiive, forecas error variance is larger han variance of he regression Reflecs fac ha forecas error is influenced no only by he regression error, bu also ha parameers mus now be esimaed. The greaer he disance beween he mean of x and x, he greaer he variance of he forecas error In oher words, he more disan is he observaion for he independen variable from is mean, he more uncerain is he predicion All else consan, he larger he sample, he smaller he variance of he forecas error. ACE 56, Universiy of Illinois a Urbana-Champaign 8-37

38 Sandard error of he forecas In he definiion of he variance of he forecas error, he variance of he regression, σ, is assumed o be known This is rarely, if ever, likely o be rue in pracice Replace σ by is esimaor ˆ σ and derive he esimaed forecas error variance 1 var( ˆ f ) = ˆ σ T T ( x x) = 1 ( x x) The esimaed sandard error of he forecas is hen, se ˆ ( f) = var( ˆ f) ACE 56, Universiy of Illinois a Urbana-Champaign 8-38

39 95% confidence inerval for predicion Previously, we consruced a sandard normal random variable as follows, Z f = y yˆ var( f ) ~ N(,1) Bu we now mus replace var( f ) wih is esimae var( ˆ f ), which resuls in a -disribued random variable f y yˆ y yˆ = = var( ˆ f ) se ˆ ( f ) ~ α, T From a -able, we know ha when T = 4 and α =.5, he associaed criical values are +.4 and -.4 Since f is a -disribued random variable, we can wrie, P[.4 f.4] =.95 ACE 56, Universiy of Illinois a Urbana-Champaign 8-39

40 Subsiuing for f, P y yˆ = se ˆ ( f ) [.4.4].95 Muliply he inequaliy in he brackes by se ˆ ( f ), P[.4 se ˆ( f) y yˆ.4 se ˆ( f)] =.95 Now, add ŷ o each erm, Py [ ˆ.4 se ˆ( f) y yˆ +.4 se ˆ( f)] =.95 Hence, he 95 percen confidence inerval for y is, yˆ ±.4 se ˆ( f) We can generalize o any predicion confidence level, 1 α, as follows, P[ yˆ se ˆ( f) y yˆ se ˆ( f)] 1 α, T + α, T = α and, yˆ ± se ˆ( f ) α, T ACE 56, Universiy of Illinois a Urbana-Champaign 8-4

41 ACE 56, Universiy of Illinois a Urbana-Champaign 8-41

42 Predicion Inervals in he Food Expendiure Example Earlier, we generaed he following esimaes of he food expendiure-income relaionship for a sample of 4 households, b 1 = b =.33 ˆ σ = Based on hese esimaes he LS predicor is, yˆ = x If we se x o $6, hen he predicion of household expendiure is, y ˆ = (6) = 1.3 ACE 56, Universiy of Illinois a Urbana-Champaign 8-4

43 Predicion in he Food Expendiure Example 4 35 Food Expendiure ($/week) Income ($/week) ACE 56, Universiy of Illinois a Urbana-Champaign 8-43

44 The corresponding esimae of he variance of he forecas error is, 1 (6 69.8) var( ˆ f ) = = ,34.6 The esimaed sandard error of he forecas is, se ˆ ( f ) = = The criical value for a -disribuion wih α =.5 and 38 degrees of freedom is.4, and hence, he 95% CI for he predicion is, 1.3 ± or, ( 7.5 y 35.39) ˆ ACE 56, Universiy of Illinois a Urbana-Champaign 8-44

45 Inerpreaion Guidelines: In repeaed sampling, we expec 95% of inerval predicions o conain he realized y If we use he inerval predicor o compue a large number of inerval predicions like 1.3 ± , 95% of hese inervals will conain he realized y I is incorrec o sae, Given ha income is $6 per week, here is a.95 probabiliy ha he realized y will be beween $7.5 and $ Remember ha our confidence is in he predicor no he paricular predicion ACE 56, Universiy of Illinois a Urbana-Champaign 8-45

46 Compromise language, Given ha income is $6 per week, we are 95% confiden ha he inerval beween $7.5 and $35.39 will conain he realized y. Given ha income is $6 per week, we are 95% confiden ha he inerval beween $7.5 and $35.39 will conain he realized food expendiure per week. where confiden is undersood o apply o he predicion inerval esimaor in repeaed sampling no he $7.5 o $35.39 inerval esimae Summary Our predicion inerval suggess ha a household wih $6 in weekly income will spend somewhere beween $7.5 o $35.39 on food Such a wide inerval means ha our poin predicion, $1.3, is no reliable We migh be able o improve our predicion by measuring he effec of facors oher han income ACE 56, Universiy of Illinois a Urbana-Champaign 8-46

47 Forecas CI's in he Food Expendiure Example Food Expendiure ($/week) Income ($/week) ACE 56, Universiy of Illinois a Urbana-Champaign 8-47