Dimension Reduction. Curse of dimensionality

Transcription

1 Deso Reuco

2 Deso Reuco Curse of esoaly h 5 feaures esos, each quaze o levels, creae 5 possble feaure cobaos, age ho ay saples you ee o esae p? ho o you vsualze he srucure a 5 esoal space?

3 Oher probles ze of he local regos eee for esy esao geg larger a larger o capure r% of he aa, ege legh s r / =, r=., =.63, =, r=., =.8 Daa e o bouary, creag bouary se Coser ufor srbuo, p% eror Eeror probably s -p =, p=.8,.89 eeror N=, p=.8, eeror 3

4 oluos - Reuco Fsher s lear scra Preserve class separao specal case of prcple copoe aalyss Mul-esoal scalg Preserve sace easures Prcpal copoe aalyss Bes aa represeao o ecessarly bes class separao 4

5 Fsher s lear scra -class Gve -esoal saples X, X X, X a lear rasfor aps -D saples oo a le bes preserves class separao X {,,..., hch Iuvely, goo feaures are hose h large separao of eas relave o varaces y } 5

6 , y, 6

7 Caveas he aure of he proble s ha abguy gh arse he you reuce proble eso a goo reuco algorh ay ze he proble, bu ay o copleely elae he proble 7

8 Caveas co he fgures also sugges ha, soees, o ge beer perforace, s ecessary o crease he eso ore feaures, o o ecrease 8

9 9 I he orgal -esoal space Beee class scaer Wh class scaer Ieally, fuco shoul be large s s s X s s X

10 I he rasfore -esoal space Beee class scaer Wh class scaer Ieally, fuco shoul be large y ˆ ˆ ˆ s s s y ˆ ˆ ˆ ˆ s s F

11 Or b ˆ ˆ X X s s y s ˆ ˆ ˆ ˆ B ˆ ˆ ˆ ˆ s s F

12 he Aalyss F: geeralze Raylegh quoe o aze F, s he geeralze egevecor assocae h he larges geeralze egevalue B B B or

13 3 Proof: ˆ ˆ ˆ ˆ * * * * * * * * * * * c F s s F B B B B B B B B B

14 Eaple Larger he h-class sprea reco ges eephasze 4

15 Fsher s lear scra c-class Wh c- scra fucos Proec fro -space o c--space Aga, ry o aze beee-class scaer o h-class scaer rao for bes separably 5

16 6 I he orgal feaure space Wh class scaer Easy geeralzao o c classes c X X

17 Beee Class caerg More rcy oal ea & oal scaer oal scaer s ae of caer h a class caer beee classes c B oal ea c 7

18 8 c B c c c c X X oal ea & oal scaer ar c c X X X

19 Meag oal scaer = beee class scaer + h class scaer I hypohess esg Beee class scaer s sgfca Wh class scaer s sgfca error E.g., hree ffere reae opo surgery, rug, placebo Large beee class scaer eas oe reae s ore effecve ha he ohers Large h class scaer eas ha saples eas varao aog subecs of he sae reae 9

20 I he rasfore c--esoal space y y W ~ ~ ~ ~ B y c c c y y y y ~ ~,..., y c ~ ~ ~ ~ W c ~ ~ B W W W W B ~ J W ~ B B W BW W W rasfore easure s a ar Use eera for sprea volue

21 Mul-Desoal calg Gve obecs a a cofuso slary or s-slary ar Dsace slary ca be ubers Euclea sace or rag F a ebeg a -esoal space here he sace slary s preserve

22 Algorhs

23 Algorhs B s slar o covarace ar a ca be recosruce by ege vecors a egevalues 3

24 4

25 Mul-Desoal calg Orgal space eso Reuce-esoal space eso {,,..., } { y, y,..., y } y y elec such a ay o preserve he -/ sace easurees hrough eso reuco 5

26 MD oluo F as close o orgal as possble Merc MD f s a oooc, erc - preservg fuco f f a b NoMerc MD ra orers are he sae boh f ca be ay oooc fuco 6

27 7 Possble Cos ress Fucos c c c f c,, " '

28 Grae Desce A search echas ar a a arbrarly chose sarg po Move a reco egave grae o ze he cos fuco 8

29 A erave algorh f f ' f,, f f, ' f f 9

30 her grae recos y c y y y c' y y c" y y y 3

31 Ho ay esos? Aga, for vsualzao purpose, s usually or 3 3

32 Eaple 3

33 A Eaple oves, h rag fro - ba o eural o goo R ca be cosere a rao varable h he uerlyg uverse beg all veers ER = epece average rags fro all veers varr = ER -ER varace sprea rags fro all veers cov, = E[R -ER R -ER ] covarace correlao of rags of o oves 33

34 A Eaple co. Covarace ar: a ar h ery beg cov, cov, s syercal Has QLQ ege ecoposo Wha are he physcal eag of Q a L? 34

35 35 PCA Prcpal Copoe Aalyss here X X X X X X X X X X 3 3

36 36 PCA Prcpal Copoe Aalyss N e he covarace ar assue he ea s zero for o X X

37 Prcpal Copoe Aalyss Erac a se of copac bass hch bes escrbe he aa se u u. u u u. u.. u. u u. u. {,,..., } v. v... v. v X U Σ V 37

38 u Ca be sho ha s a orgal vecor s a bass vecor v s he sgfcace of he bass vecor s he egh of he parcular bass vecor If aa se s hghly correlae, usually oly a fe bases are sgfca Use v v u sea of Reuce esoaly fro o or less 38

39 Ipora VD properes Orhogoal bases Iporace rae as reco Boy-fe coorae syse Ucorrelae copoes 3 u u v v 39

40 4 Furherore U Σ U U Σ U V Σ U X X V Σ U X VD of he saples ca be use o erve he PCA rasfor of he class he sae bass fucos relae egevalues VD PCA VD PCA u u

41 C C Ho o Use PCA X U X Σ U U Σ U coy-bea-e-up: Re: proeco ecoposo o pora aa esos Gree: assage accorg o porace Blue: recosruco oo pora bass Represe boy-fe coorae syse, e.g., for slary search 4

42 4 Mah Deal u u u u u u u u u u u u u u u u M <<, oly a fe esos are ep Ebeg are he ros of u C C C C

43 u s represe Boy-fe Ucorrelae Iuo Iporace-rae esos Isea of usg orgal vecors proece o saar bass, use proece o u Use as ay or as fe as you a recall eso reuco 43

44 Cavea PCA gves he eso for bes represeao of aa, hch oes o ecessarly ples bes eso for scrao of aa Bes represeao Bes scrao 44

45 Cavea PCA s sesve o aa preprocessg Ceerg Noralzao Dffere oralzao eghg gves ffere preferece o feaures NBA player salary = f hegh, ppg he uber of pora esos e.g., hegh a ppg are correlae shoul be preserve 45

46 Cavea XX s a very frequely see ah cosruc rea as a vecor PCA rea as a vecor of rao varables KL rea as a vecor of paral ervaves Hessa XX s yerc, posve seefe Ege values are real a >= 46

47 Kerel PCA A geeralzao of PCA he feaure space Iea s hs: Lear srucures gh o es orgal feaure space Bu gh es afer a olear appg o a hgher-esoal space Lear algebra ca be use for aa aalyss hgher esoal space Wh erel rcs, appg ee o be acually oe 47

48 Kerel CPA Requrees: oly er proucs are use ecoposg covarace ar Do, ca be oe I he orgal space I Kerel space hou eplc appg 48

49 Mah Deals Copue covarace ar F ege vecors a values Represe erel ah represeable copoes R N Rq q q 49

50 5 Deals q K, q α Kα Kα α K q q q Rq N N K N K K N K N K N N,,,,, N R q

51 olve Ho o Use? K: erel ar, : ege vecors Represeao Kα Nα oly, are eee : solve he prevous sep Possble o f represeao bass a ap uo vecors usg Kerel fuco hou eplc appg q K, 5

52 PCA a MD PCA proves a lear soluo o a verso of he erc MD Dsace easurees are real a syercal Use a parcular efo of sace: er prouc Cavea: er prouc requres a coorae syse org hle parse sace oes o Ier prouc efes par-se sace bu o vce versa Pu all he par-se saces o a ar, =sace beee feaures a hs s he Gra ar Recall ha M s ae of ra-oe arces Oly a sall uber -3 for vsualzao purpose of hose are ep f sgular values rop off qucly appg o a loer eso space 5

53 Fal Noes Oher echques, such as elf-orgazao Map OM are avalable OM s scusse laer o-supervse echques 53