Lecure VI Regresson (Lnear Mehods for Regresson) Conens: Lnear Mehods for Regresson Leas Squares, Gauss Markov heorem Recursve Leas Squares Lecure VI: MLSC - Dr. Sehu Vjayakumar
Lnear Regresson Model M y f ( ) + + ε : Lnear Model 0 j j j here (,,...,, ) Inpu vecor, m (,,...,, ) regresson parameers m 0 he lnear model eher assumes ha he regresson funcon f() s lnear, or ha he lnear model s a reasonable appromaon. he npus can be : Quanave npus ransformaons of quanave npus such as log, square roo ec. 3 Bass epansons (e.g. polynomal represenaon) :, 3,... Ineracon beeen varables : 3 Dummy codng of levels of qualave npu In all hese cases, he model s lnear n he parameers, even hough he fnal funcon self may no be lnear. Lecure VI: MLSC - Dr. Sehu Vjayakumar
Poer of Lnear Models 0 y g() y() f (, ) g( + 0 ) 3 4 d f g() s lnear: only lnear funcons can be modeled hoever, f s acually preprocessed, complcaed funcons can be realzed φ ( z) z () z z Φz () φ φ d () z eample : Φz () z d Lecure VI: MLSC - Dr. Sehu Vjayakumar 3
Lecure VI: MLSC - Dr. Sehu Vjayakumar 4 Leas Squares Opmzaon Leas Squares Cos Funcon Mnmze Cos ( ) ( ) n n N N daa ranng of N here f J, here, ) ( # )) ˆ ( ( ( ) ( ) ( ) ( ) J J + + 0 ( ) Soluon :
Wha are e really dong? Leas Squares Soluon : y ( ) pred pred We seek he lnear funcon of ha mnmzes he sum of he squared resduals from Y Lnear leas squares fng Lecure VI: MLSC - Dr. Sehu Vjayakumar 5
More nsghs no he LS soluon he Pseudo-Inverse + ( ) pseudo nverses are a specal soluon o an nfne se of soluons of a non-unque nverse problem (e alked abou n he prevous lecure) he mar nverson above may sll be ll-defned f s close o sngular and so-called Rdge Regresson needs o be appled Rdge Regresson + ( + γi) here γ << Mulple Oupus: jus lke mulple sngle oupu regressons W ( ) Y Lecure VI: MLSC - Dr. Sehu Vjayakumar 6
Geomercal Inerpreaon of LS Subspace S spanned by he columns of Vecor of resdual errors (ohorgonal o y) [ ] Resdual vecor: y [ ] [ ] 0 y s orhogonal o he space spanned by columns of snce J 0 ( ) y s an orhogonal Projecon of on S And hence, y s he opmal reconsrucon of n he range of Lecure VI: MLSC - Dr. Sehu Vjayakumar 7
Physcal Inerpreaon of LS y all sprngs have he same sprng consan pons far aay generae more force (danger of oulers) sprngs are vercal soluon s he mnmum energy soluon acheved by he sprngs Lecure VI: MLSC - Dr. Sehu Vjayakumar 8
Mnmum varance unbased esmaor Gauss-Markov heorem Leas Squares esmae of he parameers has he smalles varance among all lnear unbased esmaes. Leas Squares are also called BLUE esmaes Bes Lnear Unbased Esmaors ˆ ( H ) : Leas Squares here H ( ) Esmae In oher ords, Gauss-Markov heorem says ha here s no oher mar C such ha he esmaor formed by ~ C ll be boh unbased and have a smaller varance han ˆ. ˆ ( Leas Squares Esmae) s an Unbased Esmae snce E( ˆ ) (Homeork!!) Lecure VI: MLSC - Dr. Sehu Vjayakumar 9
Gauss-Markov heorem (Proof) E( ~ ) E( C) E( C( + ε)) E( C + Cε) C + CE( ε) C For Unbased Esmae : E( ~ ) C C I Var( ~ ) E[( ~ E E E[( ~ ( ~ )( ~ ))( ~ ( ~ )) ] ) ] E[( C )( C ) ] E[( C + Cε )( C + Cε ) E[( Cε)( Cε) ]... snce C I CE[ εε ] C σ CC ] Lecure VI: MLSC - Dr. Sehu Vjayakumar 0
Gauss-Markov heorem (Proof) We an o sho ha Var( ˆ ) Var( ~ ) Le C D + ( ) ( D + ( ) ) I snce C I D + I I D 0 Var( ~ ) Var( ~ ) σ CC σ ( DD σ DD σ DD σ ( D + ( ) )( D + ( ) ) + ( ) ( )( ) + D( ) + σ ( )... snce D 0 + Var( ˆ ) ) I s hs suffcen o sho ha dagonal elemens s rue by defnon. Hence, proved. of σ DD are non negave. Lecure VI: MLSC - Dr. Sehu Vjayakumar
Based vs unbased Bas-Varance decomposon of error E { } fˆ( ) σ + E{ yˆ } { } ( f ( )) + E ( yˆ E{ yˆ }) ε var( nose) + bas + var( esmae) Gauss-Markov heorem says ha Leas Squares acheves he esmae h he mnmum varance (and hence, he mnmum Mean Squared Error) among all he unbased esmaes (bas0). Does ha mean ha e should alays ork h unbased esmaors?? No!! snce here may ess some based esmaors h a smaller ne mean squared error hey rade a lle bas for a larger reducon n varance. Varable Subse Selecon and Shrnkage are mehods (hch e ll eplore soon) ha nroduce bas and ry o reduce he varance of he esmae. Lecure VI: MLSC - Dr. Sehu Vjayakumar
Recursve Leas Squares he Sherman-Morrson-Woodbury heorem ( A zz ) A + A zz A z A z More General: he Mar Inverson heorem ( A BC ) A + A B( I + CA B) CA Recursve Leas Squares Updae n+ W n P n+ ( ) n ( ) n Inalze : P I here γ << (noe P ) γ For every ne daa pon (, ) (noe ha ncludes he bas erm) n n n+ n P P f no forgeng P P n P here λ λ λ + < f forgeng + : Lecure VI: MLSC - Dr. Sehu Vjayakumar 3
Recursve Leas Squares (con d) Some amazng facs abou recursve leas squares Resuls for W are EACLY he same as for normal leas squares updae (bach updae) afer every daa pon as added once! (no eraons) NO mar nverson necessary anymore NO learnng rae necessary Guaraneed convergence o opmal W (lnear regresson s an opmal esmaor under many condons) Forgeng facor λ allos o forge daa n case of changng arge funcons Compuaonal load s larger han bach verson of lnear regresson Bu don ge fooled: f daa s sngular, you sll ll have problems! Lecure VI: MLSC - Dr. Sehu Vjayakumar 4