Support Vector Machines. More Generally Kernel Methods

Transcription

1 Support Vector Machnes More Generall Kernel Methods

2 Important Concepts Seek lnear decson boundar wth two new concepts Optmzaton based on mamzng margns not GD Kernel mappng for better lnear separaton massagng data PR ANN & ML

3 Two Class Problem: Lnear Separable Case Class Class Man decson boundares can separate these two classes Whch one should we choose? 3

4 Eample of Bad Decson Boundares Class Class Class Class 4

5 Lnear classfers: Whch Hperplane? Lots of possble solutons for abc. Some methods fnd a separatng hperplane but not the optmal one [accordng to some crteron of epected goodness] E.g. perceptron GD Support Vector Machne SVM fnds an optmal soluton. Mamzes the dstance between the hperplane and the dffcult ponts close to decson boundar One ntuton: f there are no ponts near the decson surface then there are no ver uncertan classfcaton decsons Ths lne represents the decson boundar: a + b - c = 0 5 5

6 Another ntuton If ou have to place a fat separator between classes ou have less choces and so the capact of the model has been decreased 6 6

7 Support Vector Machne SVM SVMs mamze the margn around the separatng hperplane. A.k.a. large margn classfers The decson functon s full specfed b a subset of tranng samples the support vectors. Quadratc programmng problem Seen b man as most successful current tet classfcaton method Support vectors Mamze margn 7 7

8 SVM wth Kernel Mappng Orgnal feature space mght not be well condtoned X -> fx n a hgher dmensonal space PR ANN & ML 8

9 Non-lnear SVMs Datasets that are lnearl separable wth some nose work out great: 0 But what are we gong to do f the dataset s just too hard? 0 How about mappng data to a hgher-dmensonal space: 0 9 9

10 Non-lnear SVMs: Feature spaces General dea: the orgnal feature space can alwas be mapped to some hgherdmensonal feature space where the tranng set s lkel separable: Φ: φ 0 0

11 Transformaton to Feature Space Hgh computaton burden due to hgh-dmensonalt and hard to get a good estmate Kernel trcks for effcent computaton: Onl the nner products of feature vectors are used mappng s not eplctl computed effcenc n tme and space Input space. Feature space

12 Kernel-Induced Feature Spaces Recall from Perceptron learnng rule weght s a combnaton of feature vectors those that are dffcult to handle or classfed wrongl w k w k w c k w k 0 or otherwse PR ANN & ML

13 Kernel-Induced Feature Spaces Or the decson rule nvolves onl nner product of features not features themselves h w The same holds true for mapped space Dmenson s not mportant ma not know the mappng Inner products can be evaluated at the low-dmensonal space h w PR ANN & ML 3

14 Kernels A functon that returns the value of dot product of mappng of two arguments k The same perceptron stle learnng can be appled just replace all dot products wth kernels; however SVM has more sophstcated learnng rules PR ANN & ML 4

15 5 Eample : 3 k R F

16 6 More Eample : 4 k R F

17 7 Even More Eample The nterpretaton of mappng s not unque even wth a sngle k functon : 3 k R F

18 Gaussan Kernel Identcal to the Radal Bass Functon z k z z ep Recall that e 0! The feature dmenson s nfntel hgh n ths case Embeddng s not eplctl computed mpossble onl nner product s needed n SVM 8

19 SVM Cost Functon Smlar to logstc functon but pecewse lnear no penalt for z> = and z<- =0 PR ANN & ML 9

20 Margn for w T to be > for = and <- for =- w must be larger n the left fgure than the rght. So large margn mpl small w w w PR ANN & ML 0

21 Numercal Methods Mamze margn or mnmze w cannot be done b gradent descent but b quadratc programmng DON'T tr t ourself fnd a readl avalable mplementaton SVM-lght SVMlb h b h K b s the number of support vectors s the -th support vector a s the weght and b s a bas PR ANN & ML

22 The Optmzaton Problem Let {... n } be our data set and let {-} be the class label of The decson boundar should classf all ponts correctl hard magn A constraned optmzaton problem w = w T w

23 Optmzaton fx no constrants f = 0 f= + PR ANN & ML 3

24 Optmzaton fx equalt gx =0: constrants Mn f +λg or f + λ g = 0 f f= + g g: += PR ANN & ML 4

25 Optmzaton fx nequalt constrants hx<=0: Mn f +λh or f + λ h = 0 Necessar condton for optmalt: f=0 Outsde feasble regon hx>0 not a soluton Insde feasble regon a soluton hx<0 h does not matter λ = 0 On the boundar of feasble regon hx=0 h behaves just lke g an equalt constrant For mnmzaton problem f must pont nsde h must pont outsde f = λ h λ > 0 5

26 KKT condton h 0 λ > 0 f + λ h = 0 PR ANN & ML 6

27 Lagrangan of Orgnal Problem The Lagrangan s Lagrangan multplers Note that w = w T w Settng the gradent of 0 w.r.t. w and b to zero L w = 0 L b = 0 7

28 L= wt w + α w T + b = wt w + α w T α +b α = wt w + α w T w =- wt w + α = - α T α j j j + α = α - α α j j j PR ANN & ML 8

29 The Dual Optmzaton Problem We can transform the problem to ts dual Dot product of X Ths s a conve quadratc programmng QP problem Global mamum of can alwas be found well establshed tools for solvng ths optmzaton problem e.g. cple s New varables Lagrangan multplers 9

30 Quadratc Programmng The cost s quadratc Wthout constrant there s a unque mnmum An descent search technque e.g. gradent descent should eventuall fnd the mnmum Added twst The search s wth constrant PR ANN & ML 30

31 A Geometrcal Interpretaton Class 5 =0 8 =0.6 0 =0 7 =0 =0 Support vectors s wth values dfferent from zero the hold up the separatng plane! 4 =0 9 =0 Class 3 =0 6 =.4 =0.8 3

32 Classfcaton wth SVMs Gven a new pont we can score ts projecton onto the hperplane normal: In dms: score = w +w +b. I.e. compute score: w + b = Σα T + b Set confdence threshold t. Score > t: es Score < -t: no Else: don t know

33 Soft Margn Classfcaton If the tranng set s not lnearl separable slack varables ξ can be added to allow msclassfcaton of dffcult or nos eamples. Allow some errors Let some ponts be moved to where the belong at a cost Stll tr to mnmze tranng set errors and to place hperplane far from each class large margn ξ ξ j 33 33

34 Soft margn We allow error n classfcaton; t s based on the output of the dscrmnant functon w T +b appromates the number of msclassfed samples New objectve functon: Class C : tradeoff parameter between error and margn; chosen b the user; large C means a hgher penalt to errors Class 34

35 Lnear Non-Separable Case When classes are not separable Introduce slack varables and rela constrants w 0 Mnmze the objectve functons Allow some samples nto the buffer zone but mnmze the number and the amount of protruson L w C C : 0 weght PR ANN & ML 35

36 36 PR ANN & ML Wolfe Dual C C C L j j j j w w

37 37 PR ANN & ML General Case C k C C L j j j j w w

38 38 PR ANN & ML Eample Xor X ] [ L K k T j j j j k L

39 39 PR ANN & ML Eample cont ] [ X X X X X w >=0 w SVM can solve Xor!

40 Eample: 5 D data ponts Value of dscrmnant functon class class class

41 5 D data ponts Eample = = 3 =4 4 =5 5 =6 wth 6 as class and 4 5 as class = = 3 =- 4 =- 5 = We use the polnomal kernel of degree K = + C s set to 00 We frst fnd = 5 b 4

42 Eample B usng a QP solver we get =0 =.5 3 =0 4 = =4.833 Verf at home that the constrants are ndeed satsfed The support vectors are { = 4 =5 5 =6} The dscrmnant functon s b s recovered b solvng f= or b f5=- or b f6= as 4 5 le on f and all gve b=9 wth 4

43 Software A lst of SVM mplementaton can be found at Some mplementaton such as LIBSVM can handle mult-class classfcaton SVMLght s among one of the earlest mplementaton of SVM Several Matlab toolboes for SVM are also avalable 43

44 Evaluaton: Classc Reuters Data Set Most overused data set 578 documents 9603 tranng 399 test artcles 8 categores An artcle can be n more than one categor Learn 8 bnar categor dstnctons Average document: about 90 tpes 00 tokens Average number of classes assgned.4 for docs wth at least one categor Onl about 0 out of 8 categores are large Common categores #tran #test Earn Acqustons Mone-f Gran Crude Trade 3699 Interest Shp Wheat 7 Corn

45 Reuters Tet Categorzaton data set Reuters-578 document <REUTERS TOPICS="YES" LEWISSPLIT="TRAIN" CGISPLIT="TRAINING-SET" OLDID="98" NEWID="798"> <DATE> -MAR-987 6:5:43.4</DATE> <TOPICS><D>lvestock</D><D>hog</D></TOPICS> <TITLE>AMERICAN PORK CONGRESS KICKS OFF TOMORROW</TITLE> <DATELINE> CHICAGO March - </DATELINE><BODY>The Amercan Pork Congress kcks off tomorrow March 3 n Indanapols wth 60 of the natons pork producers from 44 member states determnng ndustr postons on a number of ssues accordng to the Natonal Pork Producers Councl NPPC. Delegates to the three da Congress wll be consderng 6 resolutons concernng varous ssues ncludng the future drecton of farm polc and the ta law as t apples to the agrculture sector. The delegates wll also debate whether to endorse concepts of a natonal PRV pseudorabes vrus control and eradcaton program the NPPC sad. A large trade show n conjuncton wth the congress wll feature the latest n technolog n all areas of the ndustr the NPPC added. Reuter 45 &#3;</BODY></TEXT></REUTERS> 45

46 New Reuters: RCV: docs Top topcs n Reuters RCV 46 46

47 Actual Class Good practce department: Confuson matr Ths j entr means 53 of the docs actuall n class were put n class j b the classfer. Class assgned b classfer 53 In a perfect classfcaton onl the dagonal has non-zero entres 47 47

48 Per class evaluaton measures Recall: Fracton of docs n class classfed correctl: Precson: Fracton of docs assgned class that are actuall about class : Correct rate : - error rate Fracton of docs classfed correctl: c c j j j c c c j j c j 48 48

49 Dumas et al. 998: Reuters - Accurac Roccho NBaes Trees LnearSVM earn 9.9% 95.9% 97.8% 98.% acq 64.7% 87.8% 89.7% 9.8% mone-f 46.7% 56.6% 66.% 74.0% gran 67.5% 78.8% 85.0% 9.4% crude 70.% 79.5% 85.0% 88.3% trade 65.% 63.9% 7.5% 73.5% nterest 63.4% 64.9% 67.% 76.3% shp 49.% 85.4% 74.% 78.0% wheat 68.9% 69.7% 9.5% 89.7% corn 48.% 65.3% 9.8% 9.% Avg Top % 8.5% 88.4% 9.4% Avg All Cat 6.7% 75.% na 86.4% Recall: % labeled n categor among those stores that are reall n categor Precson: % reall n categor among those stores labeled n categor Break Even: Recall + Precson / 49 49

50 Reuters ROC - Categor Gran Recall LSVM Decson Tree Naïve Baes Fnd Smlar Precson Recall: % labeled n categor among those stores that are reall n categor 50 Precson: % reall n categor among those stores labeled n categor 50

51 ROC for Categor - Crude Recall LSVM Decson Tree Naïve Baes Fnd Smlar Precson 5 5

52 Lnear Programmng Standard ma Standard mn PR ANN & ML 5

53 Standard Ma Eample : #pants #shrts bb: #buttons #zppers cc: proft/pant proft/shrt : $/button $/zpper <- shadow prce Aj: :buttons zppers j:pants shrts use ma c c Subject to bn bn zp zp b b mn b 0 0 b bn bn Subject to zp zp c c 0 0 PR ANN & ML 53

54 Standard Ma Ma s prmal Make pants/shrts b ourself Ma profts for ou Sta wthn avalable buttons and zppers Mn s dual Sell our materal to a buer Mn costs for buer Buer must offer more than what ou can earn b makng thngs ourself PR ANN & ML 54

55 Standard Mn Eample : pound of meat pound of vegg bb: $/pound meat $/pound vegg cc: requred calore requred proten Aj: :meat vegg j:calore proten provded : $/ calore supplement $/ proten supplement <- shadow prce mn b b c/pmeat p/pmeat Subject to c c c/pvegg p/pvegg 0 0 ma c c c/meat p/meat Subject to c/vegg p/vegg b b 0 0 PR ANN & ML 55

56 Standard Mn Mn s prmal Bu meat and vegg to provde calore and proten Mn cost for ou Suppl enough nutrent Ma s dual Alens sell ou magc calore and proten plls Ma profts for seller seller must offer less than what ou pa to bu for ourself PR ANN & ML 56

57 Some Facts about LP All lnear programmng problems can be transformed nto ether standard ma or standard mn If ma f feasble then mn f feasble and vce versa Optmal ma s also the optmal mn no gap PR ANN & ML 57

58 Converson from Prmal to Dual mn b b a a Subject to a a c c 0 0 c-a*-a* 0 c-a*-a* 0 ma mn b*+b* +*c-a*-a* ma mn c + c +*c-a*-a* +*b-a*-a* +*b-a*-a* ma c c Subject to a a a a b b 0 0 PR ANN & ML 58