The Support Vector Machine

Transcription

1 he Supprt Vectr Machne Nun Vascncels (Ken Kreutz-Delgad) UC San Deg

2 Gemetrc Interpretatn Summarzng, the lnear dscrmnant decsn rule 0 f g> ( ) > 0 h*( ) = 1 f g ( ) < 0 has the fllng prpertes th It dvdes X nt t half-spaces he bundary s the hyperplane th: nrmal dstance t the rgn b/ g()/ gves the sgned dstance frm pnt t the bundary g() = 0 fr pnts n the plane g( ) = + b g() > 0 fr pnts n the sde pnts t ( pstve sde ) g() < 0 fr pnts n the negatve sde b g ( ) 2

3 Lnear Dscrmnants Fr n, ur gal s t eplre the p y smplcty f the lnear dscrmnant y =1 let s assume lnear separablty f the tranng data One handy trck s t use class labels y {-1,1} nstead f y {0,1}, here y = 1 fr pnts n the pstve sde y = -1 fr pnts n the negatve sde he decsn functn then becmes 1 f g ( ) > 0 h*( ) = h*( ) = sgn g( ) 1 f g ( ) < 0 [ ] y =-1 3

4 Lnear Dscrmnants & Separable Data We have a classfcatn errr f We have a classfcatn errr f y = 1 and g() < 0 r y = -1 and g() > 0 e.e., f y g() < 0 We have a crrect classfcatn f y = 1 and g() > 0 r y = -1 and g() < 0.e., f y g() > 0 Nte that, f the data s lnearly separable, gven a tranng set D = {( 1,y 1 ),..., ( n,y n )} e can have zer tranng errr. he necessary & suffcent cndtn fr ths s that ( + ) > 0, = 1,, y b n 4

5 he Margn he margn s the dstance frm the bundary t the clsest pnt y=1 γ = mn + b here ll be n errr n the tranng set f t s strctly greater than zer: ( + ) > 0, > 0 y b γ Nte that ths s ll-defned n the sense that γ des nt change f bth and b b are scaled by a cmmn scalar λ We need a nrmalzatn y=-1 g ( ) 5

6 Supprt Vectr Machne (SVM) A cnvenent nrmalzatn s t make g() = 1 fr the clsest pnt,.e. mn + b 1 y=1 under hch γ = 1 y=-1 he Supprt Vectr Machne (SVM) s the lnear dscrmnant classfer that mamzes the margn subject t these cnstrants: b b, 2 ( ) mn subject t y + b 1 g ( ) 6

7 Dualty We must slve an ptmzatn prblem th cnstrants here s a rch thery n h t slve such prblems We ll nt get nt t here (take 271B f nterested) he man result s that e can ften frmulate a dual prblem hch s easer t slve In the dual frmulatn e ntrduce a vectr f Lagrange multplers α > 0, ne fr each cnstrant, and slve here 0 α 0 { L b α } ma q( α ) = ma mn (,, ) α L(, b, α ) s the Lagrangan 1 2 = α b 2 [ ( y + ) 1] 7

8 he Dual Optmzatn Prblem Fr the SVM, the dual prblem can be smplfed nt ma + α 0 subject t y α = 0 α 1 αα jyy j j 2 j Once ths s slved, the vectr * = αy s the nrmal t the mamum margn hyperplane Nte: the dual slutn des nt determne the ptmal b*, snce b drps ut hen e slve mn Lbα (,, ) 8

9 he Dual Prblem here are varus pssbltes fr determnng b*. Fr eample: Pck ne pnt + n the margn n the y = 1 sde and ne pnt - n margn n the y = -1 sde hen use the margn cnstrant Nte: + = 1 ( + ) b* = + b= b he mamum margn slutn guarantees that there s alays at least ne pnt n the margn n each sde If nt, e culd mve the hyperplane and get an even e larger margn (see fgure n the rght) 1/ 1/ * 1/ * 9

10 Supprt Vectrs It turns ut that: An nactve cnstrant alays has zer Lagrange multpler α hat s, ) α > 0 and y (* + b*) = 1 r ) α = 0 and y (* + b*) > 1 Hence α > 0 nly fr pnts * + b* = 1 hch are thse that le at a dstance equal t the margn (.e., thse that are n the margn ) ). hese pnts are the Supprt Vectrs α >0 α =0 α =0 10

11 Supprt Vectrs he pnts th α > 0 supprt the ptmal hyperplane (*,b*). hs hy they are called Supprt Vectrs Nte that the decsn rule s α =0 f( ) = sgn * + b* * + = sgn yα 2 * = sgn yα SV here SV = { α* > 0} ndees the set f supprt vectrs α >0 α =0 11

12 Supprt Vectrs and the SVM Snce the decsn rule s + * + f( ) = sgn yα SV 2 here + and - are supprt vectrs, e see that e nly need the supprt vectrs t cmpletely defne the classfer! We can lterally ll thr aay all ther pnts!! he Lagrange multplers can als be seen as a measure f mprtance f each pnt α >0 α =0 α =0 Pnts th α = 0 have n nfluence a small perturbatn des nt change the slutn 12

13 he Rbustness f SVMs We talked a lt abut the curse f fdmensnalty In general, the number f eamples requred t acheve certan precsn f pdf estmatn, and pdf-based classfcatn, s epnental n the number f dmensns It turns ut that SVMs are remarkably rbust t the dmensnalty f the feature space Nt uncmmn t see successful applcatns n 1,000D+ spaces man reasns fr ths: 1) All that the SVM has t d s t learn a hyperplane. Althugh the number f dmensns may be large, the number f parameters s relatvely small and there s nt much rm fr verfttng In fact, d+1 pnts are enugh t specfy the decsn rule n R d!! 13

14 Rbustness: SVMs as Feature Selectrs he secnd reasn fr rbustness s that the data/feature space effectvely s nt really that large 2) hs s because the SVM s a feature selectr see ths let s lk at the decsn functn f = y + b ( ) * sgn α * SV hs s a threshldng f the quantty SV y α * Nte that each f the terms s the prjectn (actually, nner prduct) f the vectr hch e sh t classfy,, nt the tranng (supprt) vectr 14

15 SVMs as Feature Selectrs Defne z t be the vectr f the prjectn f nt all f the supprt vectrs ( ) z( ) =,, 1 he decsn functn s a hyperplane n the z-space th * * f ( ) = sgn y α + b * = sgn k zk ( ) + b * SV k hs means that ( * * α,, ) α * = y,, y 1 1 he classfer perates nly n the span f the supprt vectrs! he SVM perfrms feature selectn autmatcally. k k k 15

16 SVMs as Feature Selectrs Gemetrcally, e have: 1) Prjectn f ne data pnt n the span f the supprt vectrs 2) Classfcatn n ths (sub)space z() * * (*,b b*) * = ( α y,, α y ) 1 1 k k he effectve dmensn s SV and, typcally, SV << n!! 16

17 Summary f the SVM SVM tranng: SVM tranng: 1) Slve the ptmzatn prblem: 1 ma αα yy + α 0 2 j subject t y α = 0 α j j j 2) hen cmpute the parameters f the large margn lnear dscrmnant functn: * * α y SV 1 2 = * + b* = yα ( + ) SVM Lnear Dscrmnant Decsn Functn: * ( ) = sgn α + b* SV SV f y b 17

18 Nn-Separable Prblems S far e have assumed lnearly separable classes hs s rarelythecasenpractce A separable prblem s easy mst classfers ll d ell We need t be able t etend the SVM t the nn-separable case Basc dea: Wth class verlap e cannt enfrce a ( hard ) margn. But e can enfrce a sft margn Fr mst pnts there s a margn. But there are a fe utlers that crss-ver, r are clser t the bundary than the margn. S h d e handle the latter set f pnts? 18

19 Sft Margn Optmzatn Mathematcally ths s dne by ntrducng slack varables Rather than slvng the hard margn prblem b, 2 ( ) mn subject t y + b 1 nstead e slve the sft margn prblem 1/ * 1/ * 2 ( ) mn subject t y + b 1 ξ 1/ *, ξ, b he ξ are called slack varables ξ 0, 1/ * Bascally, the same ptmzatn as befre but ξ / * pnts th ξ > 0 are alled t vlate the margn 19

20 Sft Margn Optmzatn Nte that, as t stands, the prblem s nt ell defned By makng ξ arbtrarly large, 0 s a slutn! herefre, e need t penalze large values f ξ hus, nstead e slve the penalzed, r regularzed, ptmzatn prblem: mn, ξ, b 2 1/ * ξ 1/ ( ) + ξ ξ / * + C ξ 1/ * subject t y b 1 ξ 0, he quantty C ξ s the penalty, r regularzatn, term. he pstve parameter C cntrls h harsh t s. 20

21 he Sft Margn Dual Prblem he dual ptmzatn prblem: 1 ma α α yy + α 0 2 j subject t y α = 0, 0 α j j j α he nly dfference th respect t the hard margn case s the b cnstrant n the Lagrange multplers α Gemetrcally e ehave eths C α =0 0 < α < C α = 0 * * α = C 21

22 Supprt Vectrs hey are the pnts th α > 0 As befre, the decsn rule s * f ( ) = sgn yα + b* SV here SV = { α* >0} } and b* s chsen s.t. y g( ) =1 1, fr all ll s.t. 0< α <C he b cnstrant n the Lagrange multplers: makes ntutve sense as t prevents any sngle supprt vectr utler frm havng an unduly large mpact n the decsn rule. 22

23 Summary f the sft-margn SVM SVM tranng: SVM tranng: 1) Slve the ptmzatn prblem: 1 ma αα jyy j j + α α 0 2 j subject t y α = 0, 0 α C 2) hen cmpute the parameters f the large margn lnear dscrmnant functn: * * α y SV 1 2 = * + b* = yα ( + ) SVM Lnear Dscrmnant Decsn Functn: * ( ) = sgn α + b * SV SV f y b 23

24 he Kernel rck What f e ant a nn-lnear bundary? Cnsder the fllng transfrmatn f the feature space: Intrduce a mappng t a better (.e., lnearly separable) feature space φ:x Z here, generally, dm(z) >dm(x) dm(x). If a classfcatn algrthm nly depends n the data thrugh nner prducts then, n the transfrmed space, t depends n ( ), ( ) ( ) ( ) j j φ φ = φ φ 1 2 n 2 3 Φ 1 24

25 he Inner Prduct Implementatn In the transfrmed space, the learnng algrthms nly requres nner prducts φ ( ),φ ( j ) = φ ( j ) φ ( ) Nte that e d nt need t stre the φ ( j ), but nly the n 2 (scalar) cmpnent values f the nner prduct matr Interestngly, ths hlds even f φ () takes ts value n an nfnte dmensnal space. We get a reductn frm nfnty t n 2! here s, hever, stll ne prblem: When φ ( j) s nfnte dmensnal the cmputatn f the nner prduct φ ( ),φ ( j ) lks mpssble. 25

26 he Kernel rck Instead f defnng φ ( ), then cmputng φ ( ) fr each, and then cmputng φ ( ),φ ( j ) fr each par (,j), smply defne a kernel functn def and rk th t drectly. K (, z) = φ( ), φ ( z ) K(,z) s called an nner prduct r dt-prduct kernel Snce e nly use the kernel, hy bther t defne φ ( )? Just defne the kernel K(,z) drectly! hen e never have t deal th the cmplety f φ ( ). hs s usually called the kernel trck 26

27 Kernel Summary 1. D nt easy t deal th n X, apply feature transfrmatn φ :X Z, such that dm(z) >> dm(x) 2. Cnstructng and cmputng φ() drectly s t epensve: Wrte yur learnng algrthm n nner prduct frm hen, nstead f φ(), e nly need φ ( ),φ φ ( j ) fr all and j, hch e can cmpute by defnng an nner prduct kernel K(, z) = φ( ), φ( z) and cmputng K(, j ),j drectly Nte: the matr M K = LK( (, zj ) L M s called the Kernel matr r Gram matr 3. Mral: Frget abut φ() ) and nstead use K(,z) frm the start! t! 27

28 Questn? What s a gd nner prduct kernel? hs s a dffcult questn (see Prf. Lenckret s rk) In practce, the usual recpe s: Pck a kernel frm a lbrary f knn kernels sme eamples the lnear kernel K(,z) = z the Gaussan famly z σ K(, z) = e the plynmal famly 2 ( k z) { } K (, z ) = 1 + z, k 1, 2, L 28

29 Kernelzatn f the SVM Nte that all SVM equatns depend nly n j he kernel trck s trval: replace by K(, j ) 1) ranng: 1 ma αα yyk (, ) + α 0 2 j α j j j subject t yα = 0, 0 α C 1 b = yα K + K + ( ) ( ) ( ) * *,, 2 SV 2) Decsn functn: f y K( ) b * ( ) = sgn α, + * SV 2 1 n 2 3 φ 1 29

30 Kernelzatn f the SVM Ntes: As usual, nthng e dd really requres us t be n R d. We culd have smply used <, j > t dente fr the nner prduct n a nfnte dmensnal space and all the equatns uld stll hld he nly dfference s that e can n lnger recver * eplctly thut determnng the feature transfrmatnφ, snce ( ) y * * = α y φ SV hs can be an nfnte dmensnal bject. E.g., t s a sum f Gaussans ( lves n an nfnte dmensnal functn space) hen e use the Gaussan kernel Luckly, e dn t need *, nly the SVM decsn functn f y K( ) b * ( ) = sgn α, + * SV 30

31 Lmtatns f the SVM he SVM s appealng, but there are sme lmtatns: A majr prblem s the selectn f an apprprate kernel. here s n generc ptmal prcedure t fnd the kernel r ts parameters Usually e pck an arbtrary kernel, e.g. Gaussan hen, determne kernel parameters, e.g. varance, by tral and errr C cntrls the mprtance f utlers (larger C = less nfluence) Nt really ntutve h t chse C SVM s usually tuned and perfrmance-tested usng crss-valdatn. here s a need t crss-valdate th respect t bth C and kernel parameters 31

32 Practcal Implementatn f the SVM In practce, e need an algrthm fr slvng the ptmzatn prblem f the tranng stage hs s a cmple prblem here has been a large amunt f research n ths area herefre, rtng yur n algrthm s nt gng t be cmpettve Luckly there are varus packages avalable, e.g.: lbsvm: SVM lght: SVM fu: percent natn.mt.edu/svmfu/ varus thers (see here are als many papers and bks n algrthms (see e.g. B. Schölkpf and A. Smla. Learnng th Kernels. MI Press, 2002) 32

33 END 33