Lecure 6: Learnng for Conrol (Generalsed Lnear Regresson) Conens: Lnear Mehods for Regresson Leas Squares, Gauss Markov heorem Recursve Leas Squares Lecure 6: RLSC - Prof. Sehu Vjayakumar
Lnear Regresson Model M y f ( x) x x : Lnear Model 0 j j j here x ( x, x,..., x, ) Inpu vecor, m (,,...,, ) regresson parameers m 0 he lnear model eher assumes ha he regresson funcon f(x) s lnear, or ha he lnear model s a reasonable approxmaon. he npus x can be : Quanave npus ransformaons of quanave npus such as log, square roo ec. 3 Bass expansons (e.g. polynomal represenaon) : x x, x3 x,... Ineracon beeen varables : x3 x x Dummy codng of levels of qualave npu In all hese cases, he model s lnear n he parameers, even hough he fnal funcon self may no be lnear. Lecure 6: RLSC - Prof. Sehu Vjayakumar
Poer of Lnear Models 0 y g() yx f x, g x 0 x x x 3 x 4 x d f g() s lnear: only lnear funcons can be modeled hoever, f x s acually preprocessed, complcaed funcons can be realzed z z z z x z example : x z z z d d Lecure 6: RLSC - Prof. Sehu Vjayakumar 3
Lecure 6: RLSC - Prof. Sehu Vjayakumar 4 Leas Squares Opmzaon Leas Squares Cos Funcon Mnmze Cos n n N N daa ranng of N here x f J x x x x, here, ) ( # )) ˆ( ( J J 0 Soluon :
Wha are e really dong? Leas Squares Soluon : y pred x pred We seek he lnear funcon of ha mnmzes he sum of he squared resduals from Y Lnear leas squares fng Lecure 6: RLSC - Prof. Sehu Vjayakumar 5
More nsghs no he LS soluon he Pseudo-Inverse pseudo nverses are a specal soluon o an nfne se of soluons of a non-unque nverse problem (e alked abou n he prevous lecure) he marx nverson above may sll be ll-defned f s close o sngular and so-called Rdge Regresson needs o be appled Rdge Regresson I here Mulple Oupus: jus lke mulple sngle oupu regressons W Y Lecure 6: RLSC - Prof. Sehu Vjayakumar 6
Geomercal Inerpreaon of LS Resdual vecor : y s orhogonal o he space spanned by columns of snce J 0 And hence, y s he opmal reconsrucon of n he range of Lecure 6: RLSC - Prof. Sehu Vjayakumar 7
Physcal Inerpreaon of LS all sprngs have he same sprng consan pons far aay generae more force (danger of oulers) sprngs are vercal soluon s he mnmum energy soluon acheved by he sprngs Lecure 6: RLSC - Prof. Sehu Vjayakumar 8
Mnmum varance unbased esmaor Gauss-Markov heorem Leas Squares esmae of he parameers has he smalles varance among all lnear unbased esmaes. Leas Squares are also called BLUE esmaes Bes Lnear Unbased Esmaors ˆ ( H ) : Leas Squares here H ( ) Esmae In oher ords, Gauss-Markov heorem says ha here s no oher marx C such ha he esmaor formed by ~ C ll be boh unbased and have a smaller varance han ˆ. ˆ ( Leas Squares Esmae ) s an Unbased Esmae snce E( ˆ ) (Homeork!!) Lecure 6: RLSC - Prof. Sehu Vjayakumar 9
Gauss-Markov heorem (Proof) E( ~ ) E( C) E( C( ε)) E( C Cε) C CE( ε) C For Unbased Esmae : E( ~ ) C C I Var ( ~ ) E[( ~ E E E[( ~ ( ~ )( ~ ))( ~ ( ~ )) ] ) ] E[( C )( C ) ] E[( C Cε )( C Cε ) E[( Cε)( Cε) ]... snce C I CE[ εε ] C CC ] Lecure 6: RLSC - Prof. Sehu Vjayakumar 0
Gauss-Markov heorem (Proof) We an o sho ha Var ( ˆ ) Var ( ~ ) Le C D ( ) ( D ( ) ) I snce C I D I I D 0 Var ( ~ ) Var ( ~ ) CC ( DD DD DD ( D ( ) )( D ( ) ) ( ) ( )( ) D( ) ( )... snce D 0 Var ( ˆ ) ) I s hs suffcen o sho ha dagonal elemens s rue by defnon. Hence, proved. of DD are non negave. Lecure 6: RLSC - Prof. Sehu Vjayakumar
Based vs unbased Bas-Varance decomposon of error E fˆ( x ) Eyˆ f ( x ) E yˆ Eyˆ var( nose) bas var( esmae ) Gauss-Markov heorem says ha Leas Squares acheves he esmae h he mnmum varance (and hence, he mnmum Mean Squared Error) among all he unbased esmaes (bas=0). Does ha mean ha e should alays ork h unbased esmaors?? No!! snce here may exss some based esmaors h a smaller ne mean squared error hey rade a lle bas for a larger reducon n varance. Varable Subse Selecon and Shrnkage are mehods (hch e ll explore soon) ha nroduce bas and ry o reduce he varance of he esmae. Lecure 6: RLSC - Prof. Sehu Vjayakumar
Recursve Leas Squares he Sherman-Morrson-Woodbury heorem A zz A A zz A z A z More General: he Marx Inverson heorem A BC A A BI CA B CA Recursve Leas Squares Updae n Inalze : P I here (noe P ) For every ne daa pon x, (noe ha x ncludes he bas erm) : n n n n P xx P f no forgeng P P n x P x here f forgeng n W n P n x n x Lecure 6: RLSC - Prof. Sehu Vjayakumar 3
Recursve Leas Squares (con d) Some amazng facs abou recursve leas squares Resuls for W are EACLY he same as for normal leas squares updae (bach updae) afer every daa pon as added once! (no eraons) NO marx nverson necessary anymore NO learnng rae necessary Guaraneed convergence o opmal W (lnear regresson s an opmal esmaor under many condons) Forgeng facor allos o forge daa n case of changng arge funcons Compuaonal load s larger han bach verson of lnear regresson Bu don ge fooled: f daa s sngular, you sll ll have problems! Lecure 6: RLSC - Prof. Sehu Vjayakumar 4