LECTURE 21: Support Vector Machines

Transcription

1 LECURE 2: Support Vector Maches Emprcal Rsk Mmzato he VC dmeso Structural Rsk Mmzato Maxmum mar hyperplae he Laraa dual problem Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty

2 Itroducto () Cosder the famlar problem of lear a bary classfcato problem from data Assume a ve a dataset (X,Y){(x,y ),(x 2,y 2 ), (x,y )}, here the oal s to lear a fucto yf(x) that ll correctly classfy usee examples Ho do e fd such fucto? By optmz some measure of performace of the leared model What s a ood measure of performace? As e sa Lecture 4, a ood measure s the expected rsk [ ] C( f( x),y) dp( x, y) R f here C(f,y) s a sutable cost fucto, such as the squared error C(f,y)(f(x)-y) 2 Ufortuately, the rsk caot be measured drectly sce the uderly pdf s uko. Istead, e typcally use the rsk over the tra set, also ko as the emprcal rsk R emp [] f C( f( x ), y ) Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 2

3 Itroducto (2) Emprcal Rsk Mmzato A formal term for a smple cocept: fd the fucto f(x) that mmzes the averae rsk o the tra set Mmz the emprcal rsk s ot a bad th to do, provded that suffcet tra data s avalable, sce the la of lare umbers esures that the emprcal rsk ll asymptotcally covere to the expected rsk for Hoever, for small samples, oe caot uaratee that ERM ll also mmze the expected rsk. hs s the all too famlar ssue of eeralzato Ho do e avod overftt? By cotroll model complexty. Itutvely, e should prefer the smplest model that explas the data (Occam s razor) Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Müller et al., 200] 3

4 he VC dmeso () he Vapk-Chervoeks dmeso s a measure of the complexty (or capacty) of a class of fuctos f(α) he VC dmeso measures the larest umber of examples that ca be explaed by the famly f(α) he basc arumet s that hh capacty ad eeralzato propertes are at odds If the famly f(α) has eouh capacty to expla every possble dataset, e should ot expect these fuctos to eeralze very ell O the other had, f fuctos f(α) have small capacty but they are able to expla our partcular dataset, e have stroer reasos to beleve that they ll also ork ell o usee data Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Crsta ad Schölkopf, 2002] 4

5 he VC dmeso (2) Shatter a set of examples Assume a bary classfcato problem th examples R D ad cosder the set of 2 possble dchotomes For stace, th 3 examples, the set of all possble dchotomes s {(000), (00), (00), (0), (00), (0), (0), ()} A class of fuctos f(α) s sad to shatter the dataset f, for every possble dchotomy, there s a fucto f(α) that models t he VC dmeso he VC dmeso VC(f) s the sze of the larest dataset that ca be shattered by the set of fuctos f(α) If the VC dmeso of f(α) s h, the there exsts at least oe set of h pots that ca be shattered by f(α), but eeral t ll ot be true that every set of h pots ca be shattered Oe may eve fd a set of <h pots that caot be shattered by ths set of fuctos Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Mtchel, 997] 5

6 he VC dmeso (3) Cosder a bary classfcato problem R 2, ad let f(α) be the famly of oreted hyperplaes (e.., perceptros) For 3, oe ca perform a lear separato of all pots for every possble class assmet (see examples belo) For 4, a hyperplae caot separate all possble class assmets (e.., cosder the XOR problem) herefore, the VC dmeso of the set of oreted les R 2 s three It ca be sho that the VC dmeso of the famly of oreted separat hyperplaes R D s at least D+ Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 6

7 he VC dmeso (4) he VC dmeso ad the umber of free model parameters Oe may tutvely expect that models th a lare umber of free parameters ould have hh VC dmeso, hereas models th fe parameters ould have lo VC dmesos Couter example Cosder the oe-parameter fucto f(x,α)s(s(αx)), x,α R You choose a arbtrary umber h (as lare as you at) I choose the set of examples x 0 -, h You choose ay labels you lke y, y 2, y h ; x {-,+} I choose α to be h y 0 α π + 2 Despte hav oly oe parameter, the fucto f(x,α) shatters a arbtrarly lare umber of pots chose accord to the outled procedure Ad, at the same tme, oe ca fd four pots that caot be shattered by ths fucto! So hat do e make of ths? ( ) he VC dmeso s a more sophstcated measure of model complexty tha dmesoalty or umber of free parameters [Pardo,2000] Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 7

8 Structural Rsk Mmzato () Why s the VC dmeso relevat? Because the VC dmeso provdes bouds o the expected rsk as a fucto of the emprcal rsk ad the umber of avalable examples It ca be sho that, th probablty -η, the follo boud holds () R () f R f emp ( ( 2/h) + ) l( η/4) h l VC cofdece Eq. () here h s the VC dmeso of f(α), s the umber of tra examples, ad >h As the rato /h ets larer, the VC cofdece becomes smaller ad the actual rsk becomes closer to the emprcal rsk herefore, ths expresso s cosstet th the tuto that ERM s oly sutable he suffcet data s avalable hs ad other results are part of the feld ko as Statstcal Lear heory or Vapk-Chervoeks heory, from hch Support Vector Maches orated Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 8

9 Structural Rsk Mmzato (2) Structural Rsk Mmzato Aother formal term for a tutve cocept: the optmal model s foud by strk a balace betee the emprcal rsk ad the VC dmeso he SRM prcple proceeds as follos Costruct a ested structure for famly of fucto classes F F 2 F k th o-decreas VC dmesos (h h 2 h k ) For each class F, compute the soluto f that mmzes the emprcal rsk Choose the fucto class F, ad the correspod soluto f, that mmzes the rsk boud o the RHS of equato () I other ords ra a set of maches, oe for each subset For a ve subset, tra to mmze the emprcal rsk Choose the mache hose sum of emprcal rsk ad VC cofdece s mmum Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Bures, 998] 9

10 Structural Rsk Mmzato (3) Uderftt Overftt Expected rsk VC cofdece Emprcal Rsk VC dmeso S S 2 S k Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998] 0

11 he VC dmeso practce Ufortuately, comput a upper boud o the expected rsk s ot practcal varous stuatos he VC dmeso caot be accurately estmated for o-lear models such as eural etorks Implemetato of Structural Rsk Mmzato may lead to a o-lear optmzato problem he VC dmeso may be fte (e.., k earest ehbor), requr fte amout of data or he upper boud may sometmes be trval (e.., larer tha oe) Fortuately, Statstcal Lear heory ca be rorously appled the realm of lear models Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998; Müller et al., 200]

12 Optmal separat hyperplaes () Cosder the problem of fd a separat hyperplae for a learly separable dataset {(x,y ),(x 2,y 2 ),,(x,y )}, x R D, y {-,+} Whch of the fte hyperplaes should e choose? Itutvely, a hyperplae that passes too close to the tra examples ll be sestve to ose ad, therefore, less lkely to eeralze ell for data outsde the tra set Istead, t seems reasoable to expect that a hyperplae that s farthest from all tra examples ll have better eeralzato capabltes herefore, the optmal separat hyperplae ll be the oe th the larest mar, hch s defed as the mmum dstace of a example to the decso surface x 2 x 2 Optmal hyperplae Maxmum mar x x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Cherkassky ad Muler, 998] 2

13 Optmal separat hyperplaes (2) Ho does ths tutve result relate to the VC dmeso? It ca be sho [Vapk, 998] that the VC dmeso of a separat hyperplae th a mar m s bouded as follos R m m h 2,D + here D s the dmesoalty of the put space, ad R s the radus of the smallest sphere cota all the put vectors herefore, by maxmz the mar e are fact mmz the VC dmeso Ad, sce the separat hyperplae has zero emprcal error (t correctly separates all the tra examples), maxmz the mar ll also mmze the upper boud o the expected rsk Cocluso he separat hyperplae th maxmum mar ll also mmze the structural rsk 2 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 3

14 Optmal separat hyperplaes (3) o further uderstad the relatoshp betee mar ad capacty, cosder the to separat hyperplaes depcted belo A sky oe (small mar), hch ll be able to adopt may oretatos A fat oe (lare mar), hch ll have lmted flexblty A larer mar ecessarly results loer capacty We ormally thk of complexty as be a fucto of the umber of parameters Istead, Statstcal Lear heory tells us that f the mar s suffcetly lare, the complexty of the fucto ll be lo eve f the dmesoalty s very hh! x 2 Small mar Lare mar x 2 x x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Beett ad Campbell, 2000] 4

15 Optmal separat hyperplaes (4) Sce e at to maxmze the mar, let s express t as a fucto of the eht vector ad bas of the separat hyperplae From basc troometry, the dstace betee a pot x ad a plae (,b) s x + b otc that the optmal hyperplae has fte solutos by smply scal the eht vector ad bas, e choose the soluto for hch the dscrmat fucto becomes oe for the tra examples closest to the boudary x + b hs s ko as the caocal hyperplae herefore, the dstace from the closest example to the boudary s x 2 2 Ad the mar becomes x + b b x + b m 2 x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 5

16 (Dstace betee a plae ad a pot) x 2 A r θ AB B x A r r r r AB AB AB cosθ AB AB ( x x,x x ) (, ) A x A B b 67 x 2A 8 B 2B x A + b 2 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 6

17 Optmal separat hyperplaes (5) herefore, the problem of maxmz the mar s equvalet to mmze subject to J y ( ) 2 ( x + b) 2 otce that J() s a quadratc fucto, hch meas that there exsts a sle lobal mmum ad o local mma o solve ths problem, e ll use classcal Laraa optmzato techques We frst preset the Kuh-ucker heorem, hch provdes a essetal result for the terpretato of Support Vector Maches Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 7

18 (Kuh-ucker heorem) Gve a optmzato problem th covex doma Ω R mmze subject to f h ( z) ( z) z Ω 0,...,k ( z) 0,..., m th f C covex ad, h affe, ecessary ad suffcet codtos for a ormal pot z* to be a optmum are the exstece of α*, β* such that L L α ( z*,α*,β *) z ( z*,α*,β *) β * ( z *) ( z *) 0 α * ,...,k,...,k,...,k here L k ( z,α,β) f( z) + α ( z) + β h ( z) m L(z,α,β) s ko as a eeralzed Laraa fucto he thrd codto s ko as the Karush-Kuh-ucker (KK) complemetary codto. It mples that for actve costrats α 0; ad for actve costrats α 0 As e ll see a mute, the KK codto allos us to detfy the tra examples that defe the larest mar hyperplae. hese examples ll be ko as Support Vectors. Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Crsta ad Shae-aylor, 2000] 8

19 he Laraa dual problem () Costraed mmzato of J()/2 2 s solved by troduc the Laraa L P 2 (,b,α) - α [ y ( x + b) ] 2 hch yelds a ucostraed optmzato problem that s solved by: mmz L P th respect to the prmal varables ad b, ad maxmz L P th respect to the dual varables α 0 (the Larae multplers) hus, the optmum s defed by a saddle pot (see belo for llustrato) hs s ko as the Laraa prmal problem A saddle pot Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Hayk, 999] 9

20 Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 20 he Laraa dual problem (2) o smplfy the prmal problem, e elmate the prmal varables (,b) us the frst Kuh-ucker codto J/ z0 Dfferetat L P (,b,α) th respect to ad b, ad sett to zero yelds Expaso of L P yelds Us the optmalty codto J/ 0, the frst term L P ca be expressed as he secod term L P ca be expressed the same ay he thrd term L P s zero by vrtue of the optmalty codto J/ b0 ( ) ( ) P P 0 α y 0 b,b,α L α y x 0,b,α L ( ) + P α α y b x α y - 2,b,α L j j j j j j j j x x α α y y x x α y α y x α y α y x From [Hayk, 999]

21 he Laraa dual problem (3) Mer these expressos toether e obta L ( α) α j Subject to the (smpler) costrats α 0 ad D 2 α α y y x j j x αy 0 j hs s ko as the Laraa dual problem Commets We have trasformed the problem of fd a saddle pot for L P (,b) to the easer oe of maxmz L D (α) otce that L D (α) depeds o the Larae multplers α, ot o (,b) he prmal problem scales th dmesoalty ( has oe coeffcet for each dmeso), hereas the dual problem scales th the amout of tra data (there s oe Larae multpler per example) Moreover, L D (α) the tra data appears oly as dot products x x j As e ll see the ext lecture, ths property ca be cleverly exploted to perform the classfcato a hher (e.., fte) dmesoal space Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty From [Hayk, 999] 2

22 Support Vectors he K complemetary codto states that, for every pot the tra set, the follo equalty must hold α [ y ( x + b) ] 0... herefore, for each example, ether α 0 or y ( x +b-)0 must hold hose pots for hch α >0 must the le o oe of the to hyperplaes that defe the larest mar (oly at these hyperplaes the term y ( x +b-) becomes zero) hese pots are ko as the Support Vectors All the other pots must have α 0 ote that oly the support vectors cotrbute to def the optmal hyperplae J (,b,α) 0 OE: the bas term b s foud from the KK complemetary codto o the support vectors herefore, the complete dataset could be replaced by oly the support vectors, ad the separat hyperplae ould be the same α y x x 2 Support Vectors (α>0) x Itroducto to Patter Aalyss Rcardo Guterrez-Osua exas A&M Uversty 22