Support Vector Machines

Transcription

1 Suppo Veco Machine CSL 3 ARIFICIAL INELLIGENCE SPRING 4

2 Suppo Veco Machine O, Kenel Machine Diciminan-baed mehod olean cla boundaie Suppo veco coni of eample cloe o bounday Kenel compue imilaiy beeen eample omap inance pace o a highe-dimenional pace hee (hopefully) linea model uffice Chooing he igh kenel i cucial Kenel machine among be-pefoming leane CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR

3 Suppo Veco Machine O = uppo veco magin CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 3

4 Opimal Sepaaing Hypeplane X, find and hee uch ha if if C C fo hich can be fo eien a Noe e an +, no Wan inance ome diance fom hypeplane CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 4

5 Magin Diance fom inance o hypeplane + ( ) o Diance fom hypeplane o cloe inance i he magin magin CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 5

6 Opimal Sepaaing Hypeplane min ubjec o, Quadaic opimizaion poblem ih compleiy polynomial in d (#feaue) Kenel ill evenually map d-dimenional pace o highe-dimenional pace Pefe compleiy no baed on #dimenion CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 7

7 Opimal Sepaaing Hypeplane Conve opimizaion poblem o depend on numbe of aining eample N (no d) osill polynomial in N Bu opimizaion ill depend only on cloe eample (uppo veco) oypically N CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 8

8 Lagange Muliplie Reie quadaic opimizaion poblem uing Lagange muliplie α, N CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 9 N N N p L, ubjec o min

9 Lagange Muliplie Equivalenly, e can maimize L p ubjec o he conain: Plugging hee ino L p CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR N p N p L L

10 Lagange Muliplie Maimize L d ih epec o α only CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR and o ubjec L d,

11 Suppo Veco Machine Mo α = oi.e., ( + ) > ( lie ouide magin) Suppo veco: uch ha α > oi.e., ( + ) = ( lie on magin) = Σ α = fo any uppo veco oypically aveage ove all uppo veco Reuling diciminan i called he uppo veco machine (SVM) CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR

12 Sof Eo magin O = uppo veco CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 3

13 Sof Magin Hypeplane Daa no linealy epaable Find hypeplane ih lea eo Define lack vaiable ξ oing deviaion fom he magin CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 4

14 Sof Eo Coecly claified eample fa fom magin (ξ = ) Coecly claified eample, bu inide he magin ( < ξ < ) Incoecly claified eample (ξ ) Sof eo = CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 5

15 Sof Magin Hypeplane Lagangian equaion ih lack vaiable C i penaly faco μ, ne e of Lagange muliplie Wan o minimize L p CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 6 p C L

16 Sof Magin Hypeplane L d ubjec o C i a egulaizaion paamee ohigh C high penaly fo non-epaable eample (ovefi) olo C le penaly (undefi) odeemine uing validaion e (C= ypical) and C, CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 9

17 Kenel o ue peviou appoache, daa mu be nea linealy epaable If no, pehap a anfomaion φ() ill help φ() ae bai funcion φ CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR

18 Kenel anfom d-dimenional pace o k- dimenional z pace uing bai funcion φ() z=φ() hee z j = φ j (), j=,,k Inead of, aume z = φ () CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR k j j j g g ) ( ) ( ) ( ) ( φ z z

19 Sof Magin ih Kenel Replace inne poduc of bai funcion φ( ) φ( ) ih kenel funcion K(, ) CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR p C L ) ( φ, and ubjec o ) ( C L d φ φ d K L ), (

20 Kenel Funcion Kenel K(, ) compue z-pace poduc φ( ) φ( ) in -pace Mai of kenel value K, hee K = K(, ), called he Gam mai K hould be ymmeic and poiive emidefinie CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 3 K g g φ φ φ φ z,

21 Kenel Funcion Polynomial kenel of degee q If q=, hen ue oiginal feaue Fo eample, hen q= and d= CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 4 q K, y y y y y y y y K,,,,,, y y

22 Kenel Funcion Polynomial kenel of degee magin O = uppo veco CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 5

23 Kenel Funcion Radial bai funcion (Gauian kenel) K, ep i he cene i he adiu Lage implie moohe boundaie CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 6

24 Kenel Funcion: Radial Bai Funcion CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 7

25 K> Clae Lean K diffeen kenel machine g i () oeach ue one cla a poiive, emaining clae a negaive ochooe cla i uch ha i=agma j g j () obe appoach in pacice CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 3

26 K> Clae Lean K(K-)/ kenel machine oeach ue one cla a poiive and anohe cla a negaive oeaie (fae) leaning pe kenel machine CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 3

27 Summay: Kenel Machine Seek opimal epaaing hypeplane Suppo veco machine (SVM) find hypeplane uing only cloe eample Kenel funcion allo SVM o opeae in highe dimenion Chooing coec kenel i cucial Kenel machine among be-pefoming leane CSL 3 ARIFICIAL INELLIGENCE, INDIAN INSIUE OF ECHNOLOGY ROPAR 33