Geometry of LS. LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT

Transcription

1 OCTOBER 7, 2016 LECTURE 3 GEOMETRY OF LS, PROPERTIES OF σ 2, PARTITIONED REGRESSION, GOODNESS OF FIT Geometry of LS We ca thik of y ad the colums of X as members of the -dimesioal Euclidea space R Oe ca defie a subspace of R called the colum space of a k matrix X, that is a collectio of all vectors i R that ca be writte as liear combiatios of the colums of X: SX { z R : z Xb, b b 1, b 2,, b k R k} For two vectors a, b i R, the distace betwee a ad b is give by the Euclidea orm 1 of their differece a b a b a b Thus, the LS problem, miimizatio of the sum-of-squared errors y Xb y Xb, is to fid, out of all elemets of SX, the oe closest to y: mi y ỹ SX ỹ 2 The closest poit is foud by "droppig a perpedicular" That is, a solutio to the LS problem, ŷ X β must be chose so that the residual vector û y ŷ is orthogoal perpedicular to each colum of X: û X 0 As a result, û is orthogoal to every elemet of SX Ideed, if z SX, the there exists b R k such that z Xb, ad û z û Xb 0 The collectio of the elemets of R orthogoal to SX is called the orthogoal complemet of SX: S X {z R : z X 0} Every elemet of S X is orthogoal to every elemet i SX As we have see i Lecture 2, the solutio to the LS problem is give by ŷ X β X X X 1 X y P X y, P X X X X 1 X is called the orthogoal projectio matrix For ay vector y R, Furthermore, the residual vector will be i S X: P X y SX y P X y S X 1 1 For a vector x x 1, x 2,, x, its Euclidea orm is defied as x x x x2 i 1

2 To show 1, first ote, that, sice the colums of X are i SX, P X X X X X 1 X X X, ad, sice P X is a symmetric matrix, X P X X Now, X y P X y X y X P X y X y X y 0 Thus, by the defiitio, the residuals y P X y S X The residuals ca be writte as û y P X y I P X y M X y, M X I P X I X X X 1 X, is a projectio matrix oto S X The projectio matrices P X ad M X have the followig properties: P X + M X I This implies, that for ay y R, Symmetric: Idempotet: P X P X P X, ad M X M X M X Orthogoal: y P X y + M X y P X P X, M X M X P X P X X X X 1 X X X X 1 X X X X 1 X P X M X M X I P X I P X I 2P X + P X P X I P X M X P X M X P X I P X P X P X P X P X P X 0 2

3 This property implies that M X X 0 Ideed, M X X I P X X X P X X X X 0 Note that, i the above discussio, oe of the regressio assumptios have bee used Give data, y ad X, oe ca always perform least squares, regardless of what data geeratig process stads behid the data However, oe eeds a model to discuss the statistical properties of a estimator such as ubiasedess ad etc Properties of σ 2 The followig estimator for σ 2 was suggested i Lecture 2: σ Û Û Y i X i β 2 It turs out that, uder the usual regressio assumptios A1-A4, σ 2 is a biased estimator First, write Û M X Y The last equality follows because M X X 0 Next, Now, sice U M X U is a scalar, tra deotes the trace of a matrix A E U M X U X E tr U M X U X M X Xβ + U M X U σ 2 Û Û U M X M X U U M X U U M X U tr U M X U, E tr M X UU X because trabc trbca tr M X E UU X because tr ad expectatio are liear operators σ 2 tr M X The last equality follows, because by Assumptio A3, E UU X σ 2 I Next, tr M X tr I X X X 1 X tr I tr X X X 1 X tr I tr X X 1 X X tr I tr I k k 3

4 Thus, E σ 2 k σ2 2 The estimator σ 2 is biased, but it is easy to modify σ 2 to obtai ubiasedess Defie s 2 σ 2 k k 1 Y i X i β 2 It follows from 2 that Es 2 σ 2 Partitioed regressio We ca partitio the matrix of regressors X as follows: ad write the model as X X 1 X 2, Y X 1 β 1 + X 2 β 2 + U, X 1 is a k 1 matrix, X 2 is k 2, k 1 + k 2 k, ad β1 β, β 1 ad β 2 are k 1 ad k 2 -vectors respectively Such a decompositio allows oe to focus o a group of variables ad their correspodig parameters, say X 1 ad β 1 If β1 β, β 2 the oe ca write the followig versio of the ormal equatios: β 2 X X β X Y as X 2X 1 X 2X 2 β 2 X 1 X 1 X 1X 2 β1 X 1 Y X 2Y Oe ca obtai the expressios for β 1 ad β 2 by ivertig the partitioed matrix o the left-had side of the equatio above Alteratively, let s defie M 2 to be the projectio matrix o the space orthogoal to the space S X 2 : M 2 I X 2 X 2X 2 1 X 2 The, I order to show that, first write β 1 X 1M 2 X 1 1 X 1M 2 Y 3 Y X 1 β1 + X 2 β2 + Û 4 4

5 Note that by the costructio, M 2 Û Û Û is orthogoal to X 2, M 2 X 2 0, X 1Û 0, X 2Û 0 Substitute equatio 4 ito the right-had side of equatio 3: X 1M 2 X 1 1 X 1M 2 Y X 1M 2 X 1 1 X 1M 2 X 1 β1 + X 2 β2 + Û X 1M 2 X 1 1 X 1M 2 X 1 β1 + X 1M 2 X 1 1 X 1Û M 2 X 2 0 ad M 2 Û Û β 1 Sice M 2 is symmetric ad idempotet, oe ca write X 1 M 2 X 1 β 1 M 2 X 1 M 2 X 1 1 M2 X 1 M 2 Y X 1 X1 1 X 1 Ỹ, X 1 X 2 X 2X 2 1 X 2X 1 residuals from the regressio of X 1 o X 2, Ỹ M 2 Y Y X 2 X 2X 2 1 X 2Y residuals from the regressio of Y o X 2 Thus, to obtai coefficiets for the first k 1 regressors, istead of ruig the full regressio with k 1 + k 2 regressors, oe ca regress Y o X 2 to obtai the residuals Ỹ, regress X 1 o X 2 to obtai the residuals X 1, ad the regress Ỹ o X 1 to obtai β 1 I other words, β 1 shows the effect of X 1 after cotrollig for X 2 Similarly to β 1, oe ca write: For example, cosider a simple regressio for i 1,, Let s defie a -vector of oes: β 2 X 2M 1 X 2 1 X 2M 1 Y, M 1 I X 1 X 1X 1 1 X 1 Y i β 1 + β 2 X i + U i, l I this case, the matrix of regressors is give by 1 X 1 1 X 2 l X 1 X 5

6 Cosider ad Now, l l Therefore, M 1 I l l l 1 l, β 2 X M 1 Y X M 1 X M 1 I 1 ll, ad M 1 X X l l X X Xl X 1 X X 2 X X X X l X 1 X i Thus, the matrix M 1 trasforms the vector X ito the vector of deviatios from the average We ca write β 2 Xi X Y i Xi X 2, Xi X Y i Y Xi X 2 Goodess of fit Write, by the cotructio, Y P X Y + M X Y Ŷ + Û, Ŷ Û P X Y M X Y Y P X M X Y 0 Suppose that the model cotais a itercept, ie the first colum of X is the vector of oes l The total variatio i Y is Y M 1 Y Ŷ + Û M1 Ŷ + Û Ŷ M 1 Ŷ + Û M 1 Û + 2Ŷ M 1 Û 6

7 Sice the model cotais a itercept, l Û 0, ad M 1 Û Û However, Ŷ Û 0, ad, therefore, Note that Y M 1 Y Ŷ M 1 Ŷ + Û Û, or 2 Ŷi Ŷ + Ûi 2 Y l Y l Ŷ l Ŷ Ŷ + l Û Hece, the averages of Y ad its predicted values Ŷ are equal, ad we ca write: 2 Ŷi Y + Ûi 2, or 5 T SS ESS RSS T SS ESS + RSS, total sum-of-squares, 2 Ŷi Y explaied sum-of-squares, Û 2 i residual sum-of-squares The ratio of the ESS to the T SS is called the coefficiet of determiatio or R 2 : R 2 Ŷi Y 2 Properties of R 2 : 1 Û 2 i 1 Û Û Y M 1 Y Bouded betwee 0 ad 1 as implied by decompositio 5 This property does ot hold if the model does ot have a itercept, ad oe should ot use the above defiitio of R 2 i this case If R 2 1 the Û Û 0, which ca happe oly if Y SX, ie Y is exactly a liear combiatio of the colums of X 7

8 Icreases by addig more regressors Proof Cosider a partitioed matrix X Z W Let s study the effect of addig W o R 2 Let P X X X X 1 X projectio matrix correspodig to the full regressio, P Z Z Z Z 1 Z projectio matrix correspodig to the regressio without W Defie also M X I P X, M Z I P Z Note that sice Z is a part of X, P X Z Z, ad P X P Z P X Z Z Z 1 Z Z Z Z 1 Z P Z Cosequetly, M X M Z I P X I P Z I P X P Z + P X P Z I P X P Z + P Z M X Assume that Z cotais a colum of oes, so both short ad log regressios have itercepts Defie Û X M X Y, Û Z M Z Y Write: 0 ÛX ÛZ ÛX ÛZ Û XÛX + Û ZÛZ 2Û XÛZ Next, Û XÛZ Y M X M Z Y Y M X Y Û XÛX Hece, Û ZÛZ Û XÛX R 2 shows how much of the sample variatio i y was explaied by X However, our objective is to estimate populatio relatioships ad ot to explai the sample variatio High R 2 is ot ecessary a idicator of the good regressio model, ad a low R 2 is ot a evidece agaist it Oe ca always fid a X that makes R 2 1, just take ay liearly idepedet vectors Because such a set spas the whole R space, ay y R ca be writte as a exact liear combiatio of the colums of that X 8

9 Sice R 2 icreases with iclusio of additioal regressors, istead researchers ofte report the adjusted coefficiet of determiatio R 2 : R R 2 k 1 Û Û/ k Y M 1 Y/ 1 The adjusted coefficiet of determiatio discouts the fit whe the umber of the regressors k is large relative to the umber of observatios R 2 may decrease with k However, there is o strog argumet for usig such a adjustmet 9