Kernel-based Methods and Support Vector Machines

Transcription

1 Kerel-based Methods ad Support Vector Maches Larr Holder CptS 570 Mache Learg School of Electrcal Egeerg ad Computer Scece Washgto State Uverst

2 Refereces Muller et al. A Itroducto to Kerel-Based Learg Algorthms IEEE Trasactos o Neural Networks :

3 Learg Problem Estmate fucto f : R N {-+} usg trag data sampled from P Wat f mmzg epected error rsk R[f] R[ f ] loss f dp P ukow so compute emprcal rsk R emp [f] R emp [ f ] loss f

4 Overft Usg R emp [f] to estmate R[f] for small ma lead to overft

5 Overft Ca restrct the class F of f I.e. restrct the VC dmeso h of F Model selecto Fd F such that leared f F mmzes R emp [f] s overestmate of R[f] Wth probablt - δ ad >h: hl + l δ / 4 R[ f ] R f h emp[ ] +

6 Overft Tradeoff betwee emprcal rsk R emp [f] ad ucertat estmate of R[f] Epected Rsk Ucertat Emprcal Rsk Complet of F

7 Margs Cosder a trag sample separable b the hperplae f w + b Marg s the mmal dstace of a sample to the decso surface We ca boud the VC dmeso of the set of hperplaes b boudg the marg w marg

8 Nolear Algorthms Lkel to uderft usg ol hperplaes But we ca map the data to a olear space ad use hperplaes there Φ: R N F Φ Φ

9 Curse of Dmesoalt Dffcult of learg creases wth the dmesoalt of the problem I.e. Harder to lear wth more features But dffcult based o complet of learg algorthm ad VC of hpothess class Hperplaes are eas to lear Stll mappg to etremel hgh dmesoal spaces makes eve hperplae learg dffcult

10 Kerel Fuctos For some feature spaces F ad mappgs Φ there s a trck for effcetl computg scalar products Kerel fuctos compute scalar products F wthout mappg data to F or eve kowg Φ

11 Kerel Fuctos Eample: kerel k : 3 3 z z z R R Φ k Φ Φ Τ Τ

12 Kerel Fuctos Iverse multquadratc : tah : Sgmodal Polomal: ep Gaussa RBF : c c k d θ κ θ

13 Support Vector Maches Supervsed learg w + b K Mappg to olear space w Φ + b K Mmze subject to Eq. 8 m w w b Eq.8

14 Support Vector Maches Problem: w resdes F where computato s dffcult Soluto: remove depedec o w Itroduce Lagrage multplers 0 Oe for each costrat Eq. 8 Ad use kerel fucto

15 Support Vector Maches Φ + Φ L b L b b L w w w w w Substtutg last two equatos to frst ad replacg Φ Φ j wth kerel fucto k j

16 Support Vector Maches Subject to : ma j j j j k Ths s a quadratc optmzato fucto.

17 Support Vector Maches Oce we have we have w ad ca perform classfcato + + Φ Φ j j j j k b b k b f where sg sg

18 SVMs wth Nose Utl ow assumg problem s learl separable some space But f ose s preset ths ma be a bad assumpto Soluto: Itroduce ose terms slack varables ξ to the classfcato w + b ξ ξ 0 K

19 SVMs wth Nose Now we wat to mmze m w b ξ w + C ξ Where C > 0 determes tradeoff betwee emprcal error ad hpothess complet

20 SVMs wth Nose Subject to : ma j j j j C k where C s lmtg the sze of the Lagrage multplers

21 Sparst Note that ma trag eamples wll be outsde the marg Therefore ther optmal 0 Ths reduces the optmzato problem from varables dow to the umber of eamples o or sde the marg 0 ad 0 ad 0 0 ad 0 < < f C f C f ξ ξ ξ w marg

22 Kerel Methods Fsher s lear dscrmat Fd a lear projecto of the feature space such that classes are well separated Well separated defed as a large dfferece the meas ad a small varace alog the dscrmat Ca be solved usg kerel methods to fd olear dscrmats

23 Applcatos Optcal patter ad object recogto Ivarat SVM acheved best error rate 0.6% o USPS hadwrtte dgt recogto problem Better tha humas.5% Tet categorzato Tme-seres predcto

24 Applcatos Gee epresso profle aalss DNA ad prote aalss SVM method 3% of classfg DNA traslato tato stes outperforms best eural etwork 5% Vrtual SVMs corporatg pror bologcal kowledge reached -% error rate

25 Kerel Methods for Usupervsed Learg Prcpal Compoets Aalss PCA used usupervsed learg PCA s a lear method Kerel-based PCA ca acheve o-lear compoets usg stadard kerel techques Applcato to USPS data to reduce ose dcated a factor of 8 performace mprovemet over lear PCA method

26 Summar + Kerel-based methods allow lear-speed learg o-lear spaces + Support vector maches gore all but the most dfferetatg trag data those o or sde the marg + Kerel-based methods ad SVMs partcular are amog the best performg classfers o ma learg problems - Choosg a approprate kerel ca be dffcult - Hgh dmesoalt of orgal learg problem ca stll be a computatoal bottleeck