ACE 56 Fall 005 Lecure 5: he Simple Linear Regression Model: Sampling Properies of he Leas Squares Esimaors by Professor Sco H. Irwin Required Reading: Griffihs, Hill and Judge. "Inference in he Simple Regression Model: Esimaor Sampling Characerisics and Properies," Ch. 6 in Learning and Pracicing Economerics Opional Reading: Kennedy. Appendix A: Sampling Disribuions: he Foundaion of Saisics, in A Guide o Economerics (Ag Library reserve) ACE 56, Universiy of Illinois a Urbana-Champaign 5-
Overview In he previous secion, we specified a linear economic model ha led o he following saisical model, known as he simple linear regression model y = β + β x + e We assumed ha e and y are independen and idenically disribued normal random variables In compac form, he assumpions of he simple linear regression model are, SR. y = β+ β x + e, =,..., SR. Ee ( ) = 0 Ey ( ) = β+ βx SR3. var( e ) = var( y ) = σ SR4. cov[ e, e ] = cov[ y, y ] = 0 s s s SR5. he variable x is no random and mus ake on a leas wo differen values SR6. e ~ N(0, σ ) y ~ N( β + β x, σ ) ACE 56, Universiy of Illinois a Urbana-Champaign 5-
One sample of daa was obained consising of observaions on food expendiure and income for fory households We assumed ha he sample daa was generaed by he previous saisical model Given his sample of daa, we developed he following rules (esimaors) for esimaing he inercep and slope parameers of he (rue, bu unknown) linear saisical model b y x x y = = = = x x = = b = y bx Using he sample daa and leas squares esimaors, we compued he following leas squares esimaes of he unknown inercep and slope for he saisical model b = 7.383 b = 0.33 A his poin, he esimaes are simply compued numbers ha have no saisical properies! ACE 56, Universiy of Illinois a Urbana-Champaign 5-3
We can never know how close hese paricular numbers are o he rue values we are rying o esimae While we canno know he accuracy of he leas squares esimaes, we can examine he properies of he esimaor under repeaed sampling As before, we imagine "hiing" he esimaor wih many hypoheical samples and examining is performance across he samples Repeaed sampling in he food expendiure example can be hough of as seing income levels o be he same across samples, so ha we jus randomly selec new households for he given levels of income In effec, assume ha we can perform a conrolled experimen where he se of values for x are fixed across repeaed samples, bu he values for y vary randomly Applying he leas squares esimaion rules o each new (hypoheical) sample leads o differen b and b esimaes Consequenly, he leas squares esimaion rules b and b are random variables ACE 56, Universiy of Illinois a Urbana-Champaign 5-4
Viewing b and b as random variables leads o he following imporan quesions Wha are he means, variances, covariances, and forms of he sampling disribuions for he random variables b and b? Since he leas squares esimaors are only one way of using he sample daa o obain esimaes of he unknown parameers β and β, how well do he leas squares esimaors compare o oher esimaors in repeaed sampling? If someone proposes an esimaion rule for a paricular saisical model, your nex quesion should be: Wha are is sampling characerisics and how good is i? ---Griffihs, Hill and Judge, LPE, p.09 ACE 56, Universiy of Illinois a Urbana-Champaign 5-5
ACE 56, Universiy of Illinois a Urbana-Champaign 5-6
ACE 56, Universiy of Illinois a Urbana-Champaign 5-7
Means, Variances and Covariances of b and b he sampling disribuions shown on he previous page can be developed heoreically We will concenrae on he derivaion of wo imporan properies of he sampling disribuion of b : mean and variance he derivaions for he mean and variance of b and he covariance of b and b are similar, so here is no need o repea he process o develop he desired formulas for he mean and variance of b, i is helpful o firs derive a new version of he leas squares formula for b We sar by re-saing he original version, b y x x y = = = = x x = = ACE 56, Universiy of Illinois a Urbana-Champaign 5-8
he previous formula can be re-wrien as, b = = ( x x)( y y) = ( x x) Simply muliplying ou he righ hand erm of he numeraor, we ge, b ( x x)( y y) ( x x) y ( x x) y = = = = = ( x x) ( x x) = = or, b = ( x x) y y ( x x) = = ( x x) = Since, = ( x x) = 0, his can be simplified o ACE 56, Universiy of Illinois a Urbana-Champaign 5-9
b = = = ( x x) y ( x x) Nex, b = wy = where, w = = ( x x) ( x x) Finally, subsiue he original formula for he saisical model, y = β+ βx + e, for y, b = w y = w ( β + β x + e ) = = ACE 56, Universiy of Illinois a Urbana-Champaign 5-0
or, and, since b = β w + β wx + we = = = b = = = β + we = w = 0 and wx = his new formula for he leas squares esimaor is quie valuable in deriving properies of b ACE 56, Universiy of Illinois a Urbana-Champaign 5-
Mean he mean is derived by aking he expecaion of b, Eb ( ) = E β + we = Eb ( ) E( β ) wee ( ) = + Eb ( ) = = β his shows ha he mean of he sampling disribuion of b is β, he populaion slope parameer Wha happens if our saisical model is no correcly specified? In his case, omiing an imporan variable will make and Ee ( ) 0 Eb ( ) β wee ( ) = + = ACE 56, Universiy of Illinois a Urbana-Champaign 5-
If β is posiive and he omied variable ends o increase he size of posiive errors, hen Eb ( ) > β If β is posiive and he omied variable ends o increase he size of negaive errors, hen Eb ( ) < β Discussion highlighs he imporance of using economic heory o correcly specify he saisical model Specificaion quesions dominae applied economeric work We will discuss his issue exensively nex semeser ACE 56, Universiy of Illinois a Urbana-Champaign 5-3
Variance he variance of b in repeaed samples is var( b ) = E[ b E( b )] = E[ b β ] Variance measures he precision of b in he sense ha i ells us how much he esimaes produced by b vary from sample-o-sample he lower he variance of an esimaor, he greaer he sampling precision he greaer he variance of an esimaor, he lower he sampling precision Key poin: An esimaor is considered more precise han anoher esimaor if is sampling variance is less han ha of anoher esimaor Economericians place a high prioriy on developing precise esimaors ACE 56, Universiy of Illinois a Urbana-Champaign 5-4
ACE 56, Universiy of Illinois a Urbana-Champaign 5-5
We can derive he variance of b as follows, var( b) = var β + we = var we = = var( ) var( ) scov(, s) = = s= b = w e + ww e e s var( b ) = w var( e) = σ σ = = var( b ) = w = w and, var( b ) σ = ( x x) = where = w = = ( x x) ACE 56, Universiy of Illinois a Urbana-Champaign 5-6
he sandard deviaion of b is found in he usual manner se( b) = var( b) = σ ( x x) = Noice ha he erm "sandard error" (se) is normally used in place of sandard deviaion of he sampling disribuion o see why, define he error ha arises in esimaing he rue slope parameer as f = b β Applying our rules for he mean and variance of he ransformaion of a random variable we find ha E( f) = 0 and var( f) = var( b ) Since he expeced esimaion error is zero, we can say ha he ypical error (wihou regard o sign) is given by he sandard deviaion of f, which equals he sandard deviaion of b If we replace "ypical" wih "sandard" we can say ha se( b ) measures he sandard esimaion error for b, or in abbreviaed form, sandard error ACE 56, Universiy of Illinois a Urbana-Champaign 5-7
Form Now ha we have derived he mean and variance of he sampling disribuion of b, we can urn our aenion o he form of he sampling disribuion Earlier, we noed ha b = wy = where, w = = ( x x) ( x x) Wriing he formula in he above forma shows ha he leas squares rule b is a linear funcion of he y he y are normally disribued (by assumpion) Any linear funcion of normally disribued random variables is iself normally disribued ACE 56, Universiy of Illinois a Urbana-Champaign 5-8
ACE 56, Universiy of Illinois a Urbana-Champaign 5-9 hus, b is normally disribued in repeaed sampling ~, ( ) b N x x β σ = We will simply sae he mean and variance resuls for b, ( ) Eb β = var( ) ( ) x b x x σ = = = Using a similar argumen as we did for b, i can be shown ha ~, ( ) x b N x x β σ = =
Covariance Finally, he covariance beween random variables b and b in repeaed sampling is, cov( b, b ) x = σ ( x x) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-0
Probabiliy Calculaions Using Sampling Disribuions While i is fun (!) o simply derive sampling disribuions, heir real usefulness is found in making probabiliy saemens abou our esimaes Le's suppose ha he rue regression model is y = 8+ 0.5x + e and all of he sandard assumpions hold for he error erm and σ = 50 Now assume ha we are "given" a se of = 5 values for he independen variable x and ( x x) = 5,000 = We now have all he informaion o derive he sampling disribuion of b he expeced value of b is given in his example as Eb ( ) = β = 0.5 ACE 56, Universiy of Illinois a Urbana-Champaign 5-
he sampling variance of b is b = = = var( ) σ 50 0.0033 5,000 ( x x) = Puing he wo ogeher wih our normaliy assumpion b ~ N (0.5,0.0033) Suppose we wan o deermine he probabiliy ha our esimae from a single sample is beween 0.0 and 0.30 Pr(0.0 b 0.30) =? We can ransform his ino a sandard normal probabiliy saemen o obain he answer 0.0 0.5 b 0.5 0.30 0.5 = 0.0033 0.0033 0.0033 Pr? Pr( 0.8660 Z 0.8660) 0.658 ACE 56, Universiy of Illinois a Urbana-Champaign 5-
Facors ha Affec he Variances and Covariance of he Sampling Disribuions of b and b var( b ) x = = σ ( x x) = var( b ) = σ ( x x) = cov( b, b ) x = σ ( x x) = Imporan observaions:. he variance of he error erm ( σ ) appears in each formula he larger he variance, σ, he greaer he uncerainy abou where he values of y will fall relaive o he populaion mean, E( y ) Sample informaion available o esimae βand β is less precise he larger is σ ACE 56, Universiy of Illinois a Urbana-Champaign 5-3
ACE 56, Universiy of Illinois a Urbana-Champaign 5-4
ACE 56, Universiy of Illinois a Urbana-Champaign 5-5
. he sum of squares for x denominaor of each formula = ( x x) appears in he Sum of squares for x measures he spread, or variaion, of x he larger he variaion in x, he smaller are he variances and covariance of b and b he more informaion we have abou x, he more precisely can we esimae βand β 3. he larger he sample size, he smaller he variances and covariance of b and b As increases, he sum of squares for increases unambiguously Effec is clear for var( b ) and cov( b, b ) Same impac on var( b ) because and variaion in x appear in denominaor he more informaion we have abou x, he more precisely can we esimae βand β ACE 56, Universiy of Illinois a Urbana-Champaign 5-6
ACE 56, Universiy of Illinois a Urbana-Champaign 5-7
y y y y x x x x x Which daa sample would you raher use o esimae he linear relaionship beween x and y? ACE 56, Universiy of Illinois a Urbana-Champaign 5-8
4. he erm for var( b ) = x appears in he numeraor of he formula x measures he disance of he daa on x from he = origin he more disan from zero are he daa on x, he independen variable, he more difficul i is o accuraely esimae he inercep β 5. x appears in he numeraor of cov( b, b ) Covariance has he opposie sign as x Covariance increases in magniude he larger is x An undersanding of he previous five poins is of grea value when inerpreing regression resuls in applied research ACE 56, Universiy of Illinois a Urbana-Champaign 5-9
y y y y x x x x Which daa sample would you raher use o esimae he inercep in he linear relaionship beween x and y? ACE 56, Universiy of Illinois a Urbana-Champaign 5-30
y y y y x x x x x Demonsraion of negaive covariance beween slope and inercep esimaes when mean of x is posiive ACE 56, Universiy of Illinois a Urbana-Champaign 5-3
Summary of Key Facors Affecing Precision of Leas Squares Esimaors In he following able, characerisics of he sample daa are caegorized in erms of impac on precision Sample Characerisic High Variance of y Low Variance of y High Variaion in x Low Variaion in x Large Small Small Disance from Origin and x Large Disance from Origin and x More Precision Less Precision ACE 56, Universiy of Illinois a Urbana-Champaign 5-3
Sampling Properies of he Leas Squares Esimaors Since here are a number of differen rules for obaining esimaes of β and β, how can we be assured ha b and b are he "bes" rules? Previously, we developed four main crieria for "good" esimaors Compuaional cos: Esimaor is a linear funcion of sample daa Unbiasedness: In repeaed sampling, he esimaor generaes esimaes ha on average equal he populaion parameer Efficiency: Of all possible unbiased esimaors, here is no oher esimaor ha has a smaller variance Consisency: As he sample size increases he probabiliy mass of he of esimaor "collapses" on he populaion parameer We will examine he leas squares esimaor b o see if i mees hese four crieria Similar resuls hold for b ACE 56, Universiy of Illinois a Urbana-Champaign 5-33
Compuaional Cos Earlier, we noed ha, where, w b = = wy = = ( x x) ( x x) Wriing he formula in he above forma shows ha he leas squares rule b is a linear funcion of he y Unbiasedness We wan o know wheher he expeced value of b is in fac equal o β Earlier, we showed ha he mean is derived by aking he expecaion of he following version of he formula for b, Eb ( ) = E β + we = ACE 56, Universiy of Illinois a Urbana-Champaign 5-34
Eb ( ) E( β ) wee ( ) = + Eb ( ) = = β Shows ha he leas squares esimaor b is unbiased Efficiency For a given sample size, we wan o know wheher he sampling variance of b is smaller han any oher unbiased, linear esimaor Desire an esimaor ha gives us he highes probabiliy of obaining an esimae close o he rue parameer value In oher words, is here a differen esimaor ha produces a sampling variance smaller han he following formula, var( b ) = σ ( x x) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-35
For all unbiased linear esimaors, var(b ) is he smalles sampling variance possible Proof is found on pp. 78-79 of Hill e. al, Undergraduae Economerics and many oher exs Consisency We wan o show ha as he sample size increases, he probabiliy mass of he of esimaor "collapses" on he populaion parameer his can be demonsraed informally by noing he formula for he sampling variance of b, var( b ) = σ ( x x) = and ha b = σ = x x lim var( ) 0 ( ) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-36
Summary of Sampling Properies Discussion he leas squares esimaors b and b of he populaion parameers β and β are, Linear Unbiased Efficien Consisen he firs hree properies are sufficien o prove ha b and b are he bes linear unbiased esimaors (BLUE) of β and β In his conex "bes" implies minimum variance sampling disribuion Known as he Gauss-Markov heorem ACE 56, Universiy of Illinois a Urbana-Champaign 5-37
Key Poins: he esimaors b and b are bes when compared o similar esimaors, hose ha are linear and unbiased. he Gauss-Markov heorem does no say ha b and b are he bes of all possible esimaors. he esimaors b and b are bes wihin heir class because hey have he minimum variance. In order for he Gauss-Markov heorem o hold, he assumpions (SR-SR5) mus be rue. If any of he assumpions -5 are no rue, hen b and b are no he bes linear unbiased esimaors of β and β. he Gauss-Markov heorem does no depend on he assumpion of normaliy. In he simple linear regression model, if we wan o use a linear and unbiased esimaor, hen we have o do no more searching. he Gauss-Markov heorem applies o he leas squares esimaors. I does no apply o he leas squares esimaes from a single sample. ACE 56, Universiy of Illinois a Urbana-Champaign 5-38
Esimaing he Variance of he Error erm Recall ha e and y were assumed o be iid wih he following disribuions, e ~ N(0, σ ) and y ~ N( β + β x, σ ) Unless σ is known, which is highly unlikely, i will have o be esimaed as well Again, we canno use he leas squares principle, as does no appear in he sum of squares funcion, σ = = = = S( β, β ) e ( y β β x ) Insead, we apply a "heurisic" procedure based on he definiion of σ ACE 56, Universiy of Illinois a Urbana-Champaign 5-39
he original definiion of σ in he saisical model is, σ var( y) = var( e) = = E[ e ] In oher words, he variance is he expeced value of he squared errors Given his definiion, i would be naural o esimae as he average of he squared errors σ In order o do his, we mus firs obain esimaes of he populaion errors using our sample daa as e = y b b x ˆ We can hen develop our sample esimaor of σ as, eˆ eˆ + eˆ +... + eˆ ˆ = = Noice he squared sample errors are averaged by dividing by - no (or -) Accouns for he fac ha regression parameers ( β, β ) have o be esimaed ACE 56, Universiy of Illinois a Urbana-Champaign 5-40
Many regression packages repor somehing called he "sandard error of he regression" his is simply he square roo of he esimaed variance of he error erm, eˆ ˆ ˆ ˆ... ˆ σ ˆ σ e + e + + e = = = = Warning: Do no confuse he sandard error of he regression wih he sandard error of he sampling disribuion of he leas squares esimaors b and b ACE 56, Universiy of Illinois a Urbana-Champaign 5-4
Sampling Properies of Variance Esimaor he rule derived for esimaing he populaion variance of he error erm ( σ ) is, ˆ = σ = eˆ Jus as was he case wih b and b, we are ineresed in he sampling properies of ˆ σ he same four crieria are applied when asking wheher ˆ σ is a "good" esimaion rule Compuaional cos, unbiasedness, efficiency, consisency I is obvious ha ˆ σ is no a linear esimaor I can be shown ha ˆ σ is unbiased, efficien, and consisen Bes unbiased esimaor (BUE) Proof can be found in advanced economerics books ACE 56, Universiy of Illinois a Urbana-Champaign 5-4
he nex issue is he form of he sampling disribuion of he variance esimaor ˆ σ o derive he sampling disribuion, firs noe ha he populaion random errors are disribued as, e ~ N(0, σ ) =,..., Consequenly, e ~ N(0,) =,..., σ and, e σ ~ χ =,..., If we sum over he ransformed random errors, i= e σ ~ χ ACE 56, Universiy of Illinois a Urbana-Champaign 5-43
Based on he previous resul, we can generae he sampling disribuion of ˆ σ, ˆ ~ σ σ χ Noe ha he sampling disribuion of ˆ σ is proporional o a chi-square wih - degrees of freedom ACE 56, Universiy of Illinois a Urbana-Champaign 5-44
Esimaors of he Variances and Covariance of b and b Recall ha he variances and covariance of he sampling disribuions of b and b were funcions of he unknown parameer σ We can generae esimaors of he variances and covariances of b and b by simply replacing σ wih ˆ σ in he earlier formulas, var( ˆ b ) x = = ˆ σ ( x x) = var( ˆ b ) ˆ = σ ( x x) = cov( ˆ b, b ) ˆ x = σ ( x x) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-45
Likewise, he esimaors for he sandard errors of b and b are, x = se ˆ( b ˆ ˆ ) = var( b) = σ ( x x) = se ˆ( b ) var( ˆ ) ˆ σ = b = ( x x) = his maerial is ricky: We developed esimaors of he variances, covariance and sandard errors of he leas squares esimaors of b and b!! While we will no consider he exra complexiy, he esimaors of he variances, covariance and sandard errors are hemselves random variables whose values vary in repeaed sampling ACE 56, Universiy of Illinois a Urbana-Champaign 5-46
Sample Esimaes of he Variances and Covariance of b and b We can now use our sample daa on food expendiure and income o generae esimaes of he variances and covariance of b and b Firs, esimae he variance of he error erm, eˆ = 780.4 ˆ σ = = = 46.853 38 and he sandard error of he regression, ˆ ˆ 46.853 6.845 σ = σ = = Nex, esimae he variances, sandard errors, and covariance of b and b, x = var( ˆ b ˆ ) = σ = 46.853(0.349) = 6.0669 ( x x) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-47
se ˆ ( b ) = var( ˆ b ) = 6.0669 = 4.0083 ˆ ˆ var( b ) = σ = 46.853(0.0000653) = 0.003 ( x x) = se ˆ ( b ) = var( ˆ b ) = 0.003 = 0.0557 ˆ b b ˆ x cov(, ) = σ = 46.853( 0.00455) = 0.34 ( x x) = ACE 56, Universiy of Illinois a Urbana-Champaign 5-48
Sample Regression Oupu from Excel SUMMARY OUPU Regression Saisics Muliple R 0.56309607 R Square 0.370775 Adjused R Square 0.990547 Sandard Error 6.8449384 Observaions 40 ANOVA df SS MS F Significance F Regression 86.6357 86.635 7.6438 0.0005536 Residual 38 780.4573 46.8596 oal 39 607.04779 Coefficiens Sandard Error Sa P-value Lower 95% Upper 95% Inercep 7.3837543 4.008356335.84956 0.07396-0.73759 5.4977 X Variable 0.35333 0.0559349 4.00378 0.00055 0.03763 0.3448903 ACE 56, Universiy of Illinois a Urbana-Champaign 5-49
Summary of Esimaes for Food Expendiure Daa b = 7.383 b = 0.33 ˆ σ = 46.853 var( ˆ ) 6.0669 b = cov( ˆ b, b ) = 0.34 var( ˆ b ) = 0.003 Based on his informaion and he assumpion ha he saisical model is correcly specified, we can esimae he disribuions of e and y as, e ~ N(0, 46.853) y ~ N(7.38 + 0.33 x, 46.853) We also can esimae he sampling disribuions of b and b as, b ~ N(7.38,6.0669) b ~ N (0.33, 0.003) ACE 56, Universiy of Illinois a Urbana-Champaign 5-50
Inerpreaion Guidelines Regression sandard error ˆ ˆ 46.853 6.845 σ = σ = = We say, he ypical error of he regression model, wihou regard o sign, is esimaed o be $6.845/week. Sandard error of parameer esimaes se ˆ ( b ) = var( ˆ b ) = 6.0669 = 4.0083 We say, he ypical error in esimaing β, wihou regard o sign, is esimaed o be 4.0083. se ˆ ( b ) = var( ˆ b ) = 0.003 = 0.0557 We say, he ypical error in esimaing β, wihou regard o sign, is esimaed o be 0.0557. ACE 56, Universiy of Illinois a Urbana-Champaign 5-5
ACE 56, Universiy of Illinois a Urbana-Champaign 5-5
ACE 56, Universiy of Illinois a Urbana-Champaign 5-53
ACE 56, Universiy of Illinois a Urbana-Champaign 5-54