The support vector machine. Nuno Vasconcelos ECE Department, UCSD

Transcription

1 he supprt vectr machne Nun Vascncels ECE Department UCSD

2 Outlne e have talked abut classfcatn and lnear dscrmnants then e dd a detur t talk abut kernels h d e mplement a nn-lnear bundar n the lnear dscrmnant framerk ths led us t RHS the reprducng therem regularzatn the dea that all these learnng prblems bl dn t the slutn f an ptmzatn prblem e have seen h t slve ptmzatn prblems th and thut cnstrants equaltes nequaltes tda e ll put everthng tgether b studng the SVM

3 Classfcatn a classfcatn prblem has t tpes f varables e.g. X - vectr f bservatns (features n the rld Y - state (class f the rld X R = (fever bld pressure Y = {dsease n dsease} X Y related b a (unknn functn f (. = f ( gal: desgn a classfer h: X Y such hthat th( h( = f( 3

4 Lnear classfer mplements the decsn rule f g ( > h ( = = sgn[g(] f g ( < has the prpertes t dvdes X nt t half-spaces bundar s the plane th: nrmal dstance t the rgn b/ g(/ s the dstance frm pnt t the bundar g( = fr pnts n the plane th g( > n the sde pnts t ( pstve sde g( < n the negatve sde g ( b = + b g ( 4

5 Lnear classfer e have a classfcatn errr f = and g( < r = - and g( >.e.g( < and a crrect classfcatn f = and g( > r = - and g( <.e.g( > nte that fr a lnearl separable tranng set D = {(... ( n n } e can have zer emprcal rsk the necessar and suffcent cndtn s that ( + b > 5

6 he margn s the dstance frm the bundar t the clsest pnt = γ = mn + b there ll be n errr f t s strctl greater than zer ( + b > γ > nte that ths s ll-defned n the sense that γ des nt change f bth and b are scaled b λ e need a nrmalzatn b =- g ( 6

7 Mamzng the margn ths s smlar t hat e have seen fr Fsher dscrmnants lets assume e have selected sme nrmalzatn e.g. = the net questn s: hat s the cst that e are gng t ptmze? there are several planes that separate the classes hch ne s best? recall that n the case f the Perceptrn e have seen that the margn determnes the cmplet f the learnng prblem the Perceptrn cnverges n less than (k/γ teratns t sunds lke mamzng the margn s a gd dea. 7

8 Mamzng the margn ntutn : thnk f each pnt n the tranng set as a sample frm a prbablt denst centered n t f e dra anther sample e ll nt get the same pnts each pnt s reall a pdf th a certan varance ths s a kernel denst estmate f e leave a margn f γ n the tranng set e are safe aganst ths uncertant (as lng as the radus f supprt f the pdf s smaller than γ the larger γ the mre rbust the classfer! γ 8

9 Mamzng the margn ntutn : thnk f the plane as an uncertan estmate because t s learned frm a sample dran at randm snce the sample changes frm dra t dra the plane parameters are randm varables f nn-zer varance nstead f a sngle plane e have a prbablt dstrbutn ver planes the larger the margn the larger the number f planes that ll nt rgnate errrs the larger γ the larger the varance alled n the plane parameter estmates! 9

10 Nrmalzatn e ll g ver a frmal prf n a fe classes fr n let s lk at the nrmalzatn natural chce s = the margn s mamzed b slvng = ma mn b + b subect t = ths s smehat cmple =- need t fnd the pnts that acheve the mn thut knng hat and b are ptmzatn prblem th quadratc cnstrants

11 Nrmalzatn a mre cnvenent nrmalzatn s t make g( = fr the clsest pnt.e. under hch mn + b γ = = =- the SVM s the classfer that mamzes the margn under these cnstrants mn bb ( + b subect t b g (

12 Supprt vectr machne mn b ( + b subect t last tme e prved the fllng therem herem: (strng dualt Cnsder the prblem = arg mnf ( subect t e d X here X and f are cnve and the ptmal value f fnte. hen there s at least ne Lagrange multpler vectr and there s n dualt gap ths means the SVM prblem can be slved b slvng ts dual

13 he dual prblem fr the prmal ( b b + t subect mn the dual prblem s { } ( ma ma b L q mn ( = here the Lagrangan s ( [ ] ( b b L settng dervatves t zer ( [ ] - + = ( b b L g = = = = b L L ( 3 = = b L

14 he dual prblem pluggng back e get the Lagrangan = = g g ( [ ] + = b b L ( - = - = + b = 4

15 he dual prblem and the dual prblem s ma subect t nce ths s slved the vectr = = + s the nrmal t the mamum margn plane nte: the dual slutn des nt determne the ptmal b snce b drps ff hen e take dervatves 5

16 he dual prblem determnng b varus pssbltes fr eample pck ne pnt + n the margn n the = sde and ne pnt - n the =- sde use the margn cnstrant nte: b = ( + = + b + b = the mamum margn slutn guarantees that there s alas at least ne pnt n the margn n each sde f nt e culd mve the plane and get a larger margn / / / 6

17 Supprt vectrs anther pssblt s t average ver all pnts n the margn these are called supprt vectrs frm the cndtns a nnactve cnstrant has zer Lagrange multpler. hat s > and ( + b = r = and ( + b hence >nl fr pnts + b = hch are thse that le at a dstance equal t the margn > = = 7

18 Supprt vectrs pnts th > supprt the ptmal plane (b. fr ths the are called supprt vectrs nte that the decsn rule s [ + b ] f ( = sgn = sgn + b = sgn + b SV here SV = { >} s the set f supprt vectrs > = = 8

19 Supprt vectrs snce the decsn rule s f ( = sgn + b SV e nl need the supprt vectrs t cmpletel defne the classfer e can lterall thr aa all ther pnts! the Lagrange multplers can als be seen as a measure f mprtance f each pnt = > = pnts th = have n nfluence small perturbatn des nt change slutn 9

20 Perceptrn learnng nte the smlartes th the dual Perceptrn set = b = set R = ma d { } fr = :n { n f + b then { = } = + b = b + R } untl n errrs In ths case: = means that the pnt as never msclassfed ths means that e have an eas pnt far frm the bundar ver unlkel t happen fr a supprt vectr but the Perceptrn des nt mamze the margn!

21 he rbustness f SVMs n SLI e talked a lt abut the curse f dmensnalt number f eamples requred t acheve certan precsn s epnental n the number f dmensns t turns ut that SVMs are remarkabl rbust t dmensnalt nt uncmmn t see successful applcatns n D+ spaces t man reasns fr ths: all that the SVM des s t learn a plane. Althugh h the number f dmensns ma be large the number f parameters s relatvel small and there s n much rm fr verfttng In fact d+ pnts are enugh t specf the decsn rule n R d!

22 SVMs as feature selectrs the secnd reasn s that the space s nt reall that large the SVM s a feature selectr see ths let s lk at the decsn functn f ( = sgn SV + b. hs s a threshldng f the quantt SV nte that each f the terms s the prectn f the vectr t classf ( nt the tranng vectr

23 SVMs as feature selectrs defnng z as the vectr f the prectn nt all supprt vectrs ( z ( = L the decsn functn s a plane n the z-space f ( = sgn + b = sgn + k z k ( b SV k th ( L = ths means that the classfer perates n the span f the supprt vectrs! k the SVM perfrms feature selectn autmatcall k k 3

24 SVMs as feature selectrs gemetrcall e have: prectn n the span f the supprt vectrs classfer n ths space ( = L k k z( (b the effectve dmensn s SV and tpcall SV << n 4

25 In summar SVM tranng: slve ma subect then cmpute SV t = + = + b = ( + decsn functn: SV f ( = sgn + b SV 5

26 Practcal mplementatns n practce e need an algrthm fr slvng the ptmzatn prblem f the tranng stage ths s stll a cmple prblem there has been a large amunt f research n ths area cmng up th ur n algrthm s nt gng t be cmpettve luckl there are varus packages avalable e.g.: lbsvm: SVM lght: SVM fu: percent natn edu/svmfu/ varus thers (see als man papers and bks n algrthms (see e.g. B. Schölkpf and A. Smla. Learnng th ernels. MI Press 6

27 ernelzatn nte that all equatns depend nl n the kernel trck s trval: replace b ( tranng ( ma + k ( = t subect Φ decsn functn: ( ( ( + + = b SV decsn functn: ( + = sgn ( b f SV 3 n 7 SV

28 ernelzatn ntes: as usual ths flls frm the fact that nthng f hat e dd reall requres us t be n R d. e culd have smpl used the ntatn < > fr the dt prduct and all the equatns uld stll hld the nl dfference s that e can n lnger recver eplctl thut determnng the feature transfrmatn Φ snce = SV ( Φ ths culd have nfnte dmensn e.g. e have seen that t s a sum f fgaussans hen e use the Gaussan kernel but luckl e dn t reall need nl the decsn functn f ( = sgn SV ( + b 8

29 Input space nterpretatn hen e ntrduce a kernel hat s the SVM dng n the nput space? let s lk agan at the decsn functn th nte that ( + b f ( = sgn SV ( ( + + ( b = SV + and - are supprt vectrs assumng that the kernel as reduced supprt hen cmpared t the dstance beteen supprt vectrs 9

30 Input space nterpretatn nte that assumng that the kernel as reduced supprt hen cmpared t the dstance beteen supprt vectrs b = + ( ( + ( SV [ ( ( ] here e have als assumed that + ~ - these assumptns are nt crucal but smplf hat flls namel the decsn functn s ( f ( = sgn SV 3

31 Input space nterpretatn r ( = ( SV f f rerttng ( < = ( SV f f g ( ( ( < > = SV ths s ( ( f ( ( = < therse f f ( 3

32 Input space nterpretatn r ( ( = < f f ( β π th = < < therse f ( < = = < β π hch s the same as < ( π ( ( = < < h f f ( β π 3 therse

33 Input space nterpretatn nte that ths s the Baesan decsn rule fr class th lkelhd and prr π ( class th lkelhd and prr < β ( these lkelhd functns < / / are a kernel denst estmate f k(. s a vald pdf pecular kernel estmate that nl places kernels arund the supprt vectrs all ther pnts are gnred 33

34 Input space nterpretatn ths s a dscrmnant frm f denst estmatn cncentrate mdelng per here t matters the mst.e. near classfcatn bundar smart snce pnts aa frm the bundar are alas ell classfed even f denst estmates n ther regn are pr the SVM can be seen as a hghl hl effcent cmbnatn f the BDR th kernel denst estmates recall that ne mar prblem f kernel estmates s the cmplet f the decsn functn O(n. th the SVM the cmplet s nl O(SV but nthng s lst 34

35 Input space nterpretatn nte n the apprmatns made: ths result as derved assumng b~ n practce b s frequentl left as a parameter hch s used t trade-ff false pstves fr msses here that can be dne b cntrllng the BDR threshld f ( ( ( π f = β < therse hence there s reall nt much practcal dfference even hen the assumptn f b= des nt hld! 35

36 36