Kernels. Nuno Vasconcelos ECE Department, UCSD

Transcription

1 Kerels Nu Vasels ECE Departmet UCSD

2 Prpal mpet aalyss Dmesalty reut: Last tme we saw that whe the ata lves a subspae t s best t esg ur learg algrthms ths subspae D subspae 3D y φ φ λ λ y ths a be e by mputg the prpal mpets f the ata prpal mpets φ are the egevetrs f Σ prpal legths λ are the egevalues f Σ

3 Prpal mpet aalyss learg 3

4 Prpal mpet aalyss 4

5 PCA by SVD we et saw that PCA a be mpute by the SVD f the ata matr retly gve wth e eample per lum reate the etere ata-matr mpute ts SVD I ΜΠΝ 3 prpal mpets are lums f N egevalues are λ π 5

6 Etess ay we wll tal abut erels turs ut that ay algrthm whh epes the ata thrugh t-pruts ly.e. the matr f elemets a be erele ths s usually beefal we wll see why later fr w we l at the quest f whether PCA a be wrtte the frm abve reall the ata matr s K 6

7 Etess we saw that the etere-ata matr a the varae a be wrtte as I Σ the egevetr φ f egevalue λ s φ α α φ φ hee the egevetr matr s φ α α λ φ λ φ Γ Γ λ α λ α K 7 Γ Γ λ λ K

8 Etess we et te that frm the egevetr empst a.e. Λ Σ Λ Γ Γ Λ Σ Γ Γ ΓΛΓ 8

9 Etess summary we have Σ Λ Γ ΓΛΓ ths meas that we a bta PCA by assemblg - mputg ts ege-empst ΛΓ PCA the prpal mpets are the gve by Γ the egevalues are gve by Λ 9

10 Etess what s terestg here s that we ly ee the matr K M K K K M ths s the matr f t-pruts f the etere ata- M K K ths s the matr f t-pruts f the etere atapts te that yu t ee the pts themselves ly 0 ther t-pruts smlartes

11 Etess t mpute PCA we use the fat that K K but f K has egeempst ΛΓ K K ΓΛΓ ΓΛΓ ΓΛ Γ the - h has egeempst Λ Γ

12 Etess summary t get PCA mpute the t-prut matr K mpute ts ege-empst ΛΓ PCA the prpal mpets are the gve by Γ the egevalues are gve by Λ the pret f the ata-pts the prpal mpets s gve by Γ K Γ ths allws the mputat f the egevalues a PCA effets whe we ly have aess t the t-prut matr K

13 he t prut frm hs turs ut t be the ase fr may learg algrthms If yu mapulate a lttle bt yu a wrte them t prut frm Deft: a learg algrthm s t prut frm f gve a trag set D { y... y } t ly epes the pts thrugh ther t pruts. fr eample let s l at -meas 3

14 Clusterg We saw that t terates bewtee lassfat: re-estmat: estmat: µ * arg m µ ew te that µ µ µ µ µ µ 4

15 Clusterg a b th t t th t t µ mbg the tw we a wrte the tp equat as a fut f the t pruts l l µ 5

16 he erel tr why s ths terestg? ser trasfrmat f the feature spae: true a mappg : Z suh that mz > m f the algrthm ly epes the ata thrugh t-pruts the the trasfrme spae t ly epes 3 φ φ 6

17 he t prut mplemetat the trasfrme spae the learg algrthms ly requres t-pruts te that we -lger ee t stre the ly the t-prut matr terestgly ths hls eve whe s fte mesal we get a reut frm fty t! there s hwever stll e prblem: whe m[ ] s fte the mputat f the t pruts ls mpssble 7

18 he erel tr stea f efg efg mputg fr eah a fr eah par smply pyefe the fut a wr wth t retly. K K s alle a t-prut erel fat se we ly use the erel why efe? ust efe the erel K retly! ths way we ever have t eal wth the mplety f... ths s usually alle the erel tr 8

19 Quests I am fuse! hw I w that f I p a fut K t s equvalet t? geeral t s t. We wll tal abut ths later. f t s hw I w what s? yu may ever w. E.g. the Gaussa erel K s very ppular. It s t bvus what s... e the pstve se we t w hw t hse. Chsg stea K maes fferee. why s t that usg K s easer/better? mplety. let s l at a eample. σ 9

20 Plymal erels stll R ser the square f the t prut betwee tw vetrs K M K 0 K

21 Plymal erels a be wrtte as M [ ] M K K K K hee we have M K K R R : wth L L L M

22 Plymal erels the pt s that whle has mplety O ret mputat f K has mplety O ret evaluat s mre effet by a fatr f as ges t fty ths maes the ea feasble BW yu ust met ather erel famly ths mplemets plymals f se rer geeral the famly f plymal erels s efe as { L} K I t eve wat t th abut wrtg w!

23 Kerel summary. D t easy t eal wth apply feature trasfrmat : Z suh that mz >> m. mputg t epesve: wrte yur learg algrthm t-prut frm stea f we ly ee 3. stea f mputg efe the t-prut erel K a mpute K retly te: the matr M K L K L M s alle the erel r Gram matr 4. frget abut a use K frm the start! 3

24 Quest what s a g t-prut erel? ths s a ffult quest see Prf. Leret s wr prate the usual repe s: p a erel frm a lbrary f w erels we have alreay met the lear erel K the Gaussa famly K e the plymal famly σ { L } K 4

25 Dt-prut erels ths may t be a ba ea we rp the beefts f a hgh-mesal spae wthut a pre mplety the erel smply as a few parameters σ learg t wul mply trug may parameters up t what f I ee t he whether K s a erel? Deft: a mappg : R y y s a t-prut erel f a ly f y <y> where : H H s a vetr spae a <..> a tprut H 3 H 5

26 Pstve efte matres reall that e.g. Lear Algebra a Applats Strag Deft: eah f the fllwg s a eessary a suffet t fr a real symmetr matr A t be sem pstve efte: A 0 0 all egevalues f A satsfy λ 0 all upper-left submatres A have -egatve etermat v there s a matr R wth epeet rws suh that A R R upper left submatres: A a a a 3 a A A a a a 3 3 a a a 3 a3 a33 a a L 6

27 Pstve efte matres prperty v s partularly terestg R <> > A s a t-prut erel f a ly f A s pstve efte frm v ths hls f a ly f there s R suh that A R R hee wth <y> Ay R Ry y : R R R.e. the t-prut erel A A pstve efte s the staar t-prut the rage spae f the mappg R 7

28 Pstve efte erels hw I ete ths t f pstve efteess t futs? Deft: a fut y s a pstve efte erel f l a {... l } the Gram matr s pstve efte. M K L L M le R ths allws us t he that we have a pstve efte erel 8

29 Dt prut erels herem: y y s a t-prut erel f a ly f t s a pstve efte erel summary t he whether a erel s a t prut: he f the Gram matr s pstve efte fr all pssble sequees {... l } es the erel have t be a t-prut erel? t eessarly. Fr eample eural etwrs a be see as mplemetg erels that are t f ths type hwever: yu lse the parallelsm. what yu w abut the learg mahe may lger hl after yu erele t-prut erels usually lea t ve learg prblems. Usually yu lse ths guaratee fr t-prut 9

30 Clusterg s far ths s mstly theretal hw es t affet my algrthms? ser fr eample the -meas algrthm lassfat: re-estmat: µ * arg m µ ew a we erele the lassfat step? 30

31 Clusterg well we saw that th th b l t l l µ ths a the be erele t µ l µ l l 3

32 Clusterg furthermre ths a be e wth relatve effey µ l l th agal etry f Gram matr the assgmet f the pt ly requres mputg fr eah luster ths s a sum f etres f Gram matr mpute e per luster whe all pts are assge 3

33 Clusterg te hwever that we at epltly mpute µ ths s prbably bl fte t mesal... ay ase f we efe a Gram matr K fr eah luster t pruts betwee pts luster a S the sale sum f the etres ths matr S l l 33

34 Clusterg we bta the erel -meas algrthm lassfat: arg m K S l l * l re-estmat: upate l S l l but we lger have aess t the prttype fr eah luster 34

35 Clusterg wth the rght erel ths a wr sgfatly better tha regular -meas -meas erel -meas 35

36 Clusterg but fr ther applats where the prttypes are mprtat ths may be useless e.g. mpress we a try replag the prttype by the lsest vetr but ths s t eessarly ptmal 36

37 PCA we saw that t get PCA mpute the t-prut matr K mpute ts ege-empst ΛΓ PCA the prpal mpets are the gve by Γ the egevalues are gve by Λ the pret f the ata-pts the prpal mpets s gve by K te that mst f ths hls whe we erele we ly have t hage the matr K frm t φ φ the ly thg we a lger aess are the PCs Γ Γ 37

38 Kerel meths mst learg algrthms a be erele erel PCA erel LDA erel ICA et. as -meas smetmes we lse sme f the features f the rgal algrthm but the perfrmae s frequetly better et wee we wll l at the aal applat the supprt vetr mahe 38

39 39