Linear Regression Linear Regression with Shrinkage

Transcription

1 Lear Regresso Lear Regresso h Shrkage

2 Iroduco Regresso meas predcg a couous (usuall scalar oupu from a vecor of couous pus (feaures x. Example: Predcg vehcle fuel effcec (mpg from 8 arbues:

3 Lear Regresso Isaces: <x, > Lear: appg from x o (x Gve, bass fucos, h( x { h0 ( x,..., h ( x}, (defe h 0 ( x Fd coeffces,..., } ( x daa f ( x; { h ( x h( x assumes he fucoal mappg s lear s parameers

4 Bass Fucos There are ma bass fucos e ca use eg Polomal Radal bass fucos Sgmodal Sples, Fourer, Waveles, ec ( x x h ( exp ( s x x h µ s x x h µ σ (

5 Lear Regresso Esmao mze he resdual error predco loss erms of mea squared error o rag samples. ( J f ( x; ( ( J h x H H H h... h ( emprcal squared loss N samples bass fucos eghs bass fucos ( ( N samples measuremes

6 Lear Regresso Soluo B seg he dervaves of H H o zero, e ge he soluo (as e dd for SE: ( ( ˆ ( H H H A b The soluo s a lear fuco of he oupus. here A H H b x marx of bass fucos H x vecor

7 Sascal ve of lear regresso I a sascal regresso model e model boh he fuco ad ose Observed oupu fuco + ose ( x f ( x; +ε here, e.g., ε ~ N(0, σ Whaever e cao capure h our chose faml of fucos ll be erpreed as ose

8 Sascal ve of lear regresso f(x; s rg o capure he mea of he observaos gve he pu x: E [ x] E[ f ( x; + ε x] f ( x; here E[ x] s he codoal expecao of gve x, evaluaed accordg o he model (o accordg o he uderlg dsrbuo P

9 Sascal ve of lear regresso Accordg o our sascal model ( x f ( x; +ε, ε ~ N(0, σ he oupus gve x are ormall dsrbued h mea f (x; ad varace σ : p( x,, σ exp ( f ( x; πσ σ (e model he ucera he predcos, o us he mea

10 axmum lkelhood esmao Gve observaos e fd he parameers ha maxmze he lkelhood of he oupus: axmze log-lkelhood { },,...,(, ( x x D ( ( k k x f x p L ; exp,, (, ( σ πσ σ σ ( ( k k x f L ; log, ( log σ πσ σ mmze

11 axmum lkelhood esmao Thus LE arg m ( ( f x; Bu he emprcal squared loss s J ( ( f ( x; Leas-squares Lear Regresso s LE for Gaussa ose!!!

12 Pseudo Iverse ( H H s called he pseudo-verse of H (sce H ll o usuall be square. The predcos o he rag daa are ˆ H ˆ ˆ ( H H H A b H ( H H here S s called he ha marx. Ths compues a orhogoal proeco of o he space spaed b he colums of H H S

13 Geomerc erpreao of lear regresso h o pu pos

14 Numercal ssues compug Recall ha H s a x marx. If < or f some of he colums (feaures are colear, he A s o full rak (so de(a 0 Eve f A s sgular, ˆ H ˆ s sll he proeco of oo he colum space of H here s us more ha oe a o express ha proeco erms of he colums of H (he model s udefable. - Ho do e compue A f s o of full rak? Use SVD Use regularzao - A

15

16 Lear regresso h regularzao If here are correlaed feaures, her coeffces mgh become poorl deerme ad exhb hgh varace. A large posve coeffce o oe varable ca be caceled b a smlarl large egave coeffce o s correlaed cous. Soluos: Selec a subse of srog pus subse seleco Add a regularzao erm o corol eghs. ehods usg Derved Ipu Drecos

17 Rdge Regresso Rdge regresso shrks he regresso coeffces b mposg a peal o her sze ( also called egh deca I rdge regresso, e add a quadrac peal o he eghs: N J ( 0 x + λ here λ 0 s a ug parameer ha corols he amou of shrkage. Ths s equvale o N rdge ˆ arg m here s s relaed o λ x 0 subec o s

18 Sadardzg I rdge regresso, e add a quadrac peal o all he eghs excep he offse 0 N J ( 0 x + λ We do o pealze he bas erm 0, sce e a a shf pu o shf he oupu b he same amou. N We ca esmae he offse b ( The remag coeffces are esmaed usg rdge regresso hou, usg he ceered daa 0 x No he pu marx X (ceered has (o + colums. Sce rdge s o vara o scalg of pus, e usuall also sadardze he pus,.e., e use z 0 x σ x x /

19 Rdge Regresso Soluo Rdge regresso marx form: J ( ( Z( Z +λ here Z ( X X s he ceered marx The soluo s ˆ ( Z Z+ λi rdge The problem s o sgular eve f Z Z s o full rak. Sll lear. For orhogoal pus he rdge esmaes are he scaled verso of leas squares esmaes: rdge LS ˆ γ ˆ 0 γ Z ˆ LS ( X X X

20 SVD ad LS Assume X s ceered. Le he SVD be X UDV, here U s x orhogoal marx h s colums spag he colum space of X V s x orhogoal marx h s colums spag he ro space of X D s x dagoal marx h dagoal eres d d... d 0 called he sgular values of X. I s eas o sho (do! ha he predcos o he rag se are ˆ Xˆ LS X ( X X X UU

21 Rdge ad SVD The rdge soluos are here u are he colums of U. Noe λ X ˆ rdge X ( X UD( D+ λi 0, d /( d X DU + λ. + λi d u d + λ Lke lear regresso, rdge compues he coordaes of h respec o he orhoormal bass U. I he shrks hese coordaes b he facor of /( +λ d d Thus he greaer shrkage s appled o bass vecors h smaller. Wha does small mea? d X u d

22 PCA ad Rdge If X UDV,, he he Ege decomposo of he sample covarace marx s X X VD V v The egevecors are he prcple compoes drecos of X. The frs prcple compoe z Xv ud has he larges varace Var ( z Var( Xv d / Hece small sgular values d correspod o drecos he colum spaces of X havg small varace, ad rdge shrks hese drecos he mos.

23 PCA ad Rdge I s easer o deerme he grade of he plae he log dreco ha he shor. Rdge proecs agas poeall hgh varace of grade esmaes he shor dreco.

24 Rdge regresso s AP h Gaussa pror J ( log log σ ( P( D Ν( X P( x, σ Ν( 0, τ ( X + + cos τ Ths s he same obecve fuco ha rdge solves, usg λσ /τ Rdge: J ( ( X( X +λ

25 The Lasso (L-Peal Lasso (leas absolue shrkage ad seleco operaor uses a L peal o he eghs N J ( 0 x + λ Ths s equvale o N rdge ˆ arg m 0 x subec o here s relaed o λ Ths ecourages sparc,.e., some eghs go exacl o 0. I s lke sof feaure seleco. Hoever, e mus o use quadrac programmg (or erave mehods.

26 L vs L peales Coours of he LS error fuco L L β + β β 0 β +β I Lasso he cosra rego has corers; he he soluo hs a corer he correspodg coeffces becomes 0 (he > more ha oe.

27 Lasso s AP h Laplace pror Cosder a double-sded expoeal pror The he AP esmae mmzes Ths s he same obecve fuco ha lasso solves, usg cos X X x Ν J + + ( ( Laplace(, ( log ( α σ α σ exp( exp( Laplace( ( P α α α α α α σ λ

28 Examples See examples of regresso a hp://e.kpeda.org/k/lear_regresso