17 Support Vector Machines

Transcription

1 17 We now dscuss an nfluental and effectve classfcaton algorthm called (SVMs). In addton to ther successes n many classfcaton problems, SVMs are responsble for ntroducng and/or popularzng several mportant deas to machne learnng, namely, kernel methods, maxmum margn methods, convex optmzaton, and sparsty/support vectors. Unlke the mostly-bayesan treatment that we have gven n ths course, SVMs are based on some very sophstcated Frequentst arguments (based on a theory called Structural Rsk Mnmzaton and VC-Dmenson) whch we wll not dscuss here, although there are many close connectons to Bayesan formulatons Maxmzng the margn Suppose we are gvenn tranng vectors{(x,y )}, wherex R D,y { 1,1}. We want to learn a classfer f(x) = w T φ(x)+b (1) so that the classfer s output for a newxs sgn(f(x)). Suppose that our tranng data are lnearly-separable n the feature space φ(x),.e., as llustrated n Fgure 1, the two classes of tranng exemplars are suffcently well separated n the feature space that one can draw a hyperplane between them (e.g., a lne n 2D, or plane n 3D). If they are lnearly separable then n almost all cases there wll be many possble choces for the lnear decson boundary, each one of whch wll produce no classfcaton errors on the tranng data. Whch one should we choose? If we place the boundary very close to some of the data, there seems to be a greater danger that we wll msclassfy test data, especally when the tranng data are alsmot certany nosy. Ths motvates the dea of placng the boundary to maxmze the margn, that s, the dstance from the hyperplane to the closest data pont n ether class. Ths can be thought of havng the largest margn for error f you are drvng a fast car between a scattered set of obstacles, t s safest to fnd a path that stays as far from them as possble. More precsely, n a maxmum margn method, we want to optmze the followng objectve functon: max w,b mn dst(x,w,b) (2) such that, for all,y (w T φ(x )+b) 0 (3) where dst(x, w, b) s the Eucldean dstance from the feature pont φ(x) to the hyperplane defned bywandb. Wth ths objectve functon we are maxmzng the dstance from the decson boundary w T φ(x) + b = 0 to the nearest pont. The constrants force us to fnd a decson boundary that classfes all tranng data correctly. That s, for the classfer a tranng pont correctly y and w T φ(x )+b should have the same sgn, n whch case ther product must be postve. Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 115

2 y= f = 1 y= f = 0 y= f = 11 y= 1 f = 1 y= f = 0 y= 1 f = 1 margn Fgure 1: Left: the margn for a decson boundary s the dstance to the nearest data pont. Rght: In SVMs, we fnd the boundary wth maxmum margn. (Fgure from Pattern Recognton and Machne Learnng by Chrs Bshop.) It can be shown that the dstance from a pont φ(x ) to a hyperplane w T φ(x)+b = 0 s gven by wt φ(x )+b, or, snce y w tells us the sgn of f(x ), y (w T φ(x )+b). Ths can be seen ntutvely by w wrtng the hyperplane n the form f(x) = w T (φ(x ) p), where p s a pont on the hyperplane such thatw T p = b. The vector fromφ(x ) to the hyperplane projected ontow/ w gves a vector from the hyperplane to the the pont; the length of ths vector s the desred dstance. Substtutng ths expresson for the dstance functon nto the above objectve functon, we get: y max w,b mn (w T φ(x )+b) w (4) such that, for all,y (w T φ(x )+b) 0 (5) Note that, because of the normalzaton by w n (4), the scale of w s arbtrary n ths objectve functon. That s, f we were to multply w and b by some real scalar α, the factors of α n the numerator and denomnator wll cancel one another. Now, suppose that we choose the scale so that the nearest pont to the hyperplane, x, satsfes y (w T φ(x )+b) = 1. Wth ths assumpton the mn n Eqn (4) becomes redundant and can be removed. Thus we can rewrte the objectve functon and the constrant as 1 max w,b w (6) such that, for all,y (w T φ(x )+b) 1 (7) Fnally, as a last step, snce maxmzng 1/ w s the same as mnmzng w 2 /2, we can re-express the optmzaton problem as 1 mn w,b 2 w 2 (8) such that, for all,y (w T φ(x )+b) 1 (9) Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 116

3 Ths objectve functon s a quadratc program, or QP, because the objectve functon s quadratc n the unknowns, and all of the constrants are lnear n the unknowns. A QP has a sngle global mnma, whch can be found effcently wth current optmzaton packages. In order to understand ths optmzaton problem, we can see that the constrants wll be actve for only a few dataponts. That s, only a few dataponts wll be close to the margn, thereby constranng the soluton. These ponts are called the support vectors. Small movements of the other data ponts have no effect on the decson boundary. Indeed, the decson boundary s determned only by the support vectors. Of course, movng ponts to wthn the margn of the decson boundary wll change whch ponts are support vectors, and thus change the decson boundary. Ths s n constrast to the probablstc methods we have seen earler n the course, n whch the postons of all data ponts affect the locaton of the decson boundary Slack Varables for Non-Separable Datasets Many datasets wll not be lnearly separable. As a result, there wll be no way to satsfy all the constrants n Eqn. (9). One way to cope wth such datasets and stll learn useful classfers s to loosen some of the constrants by ntroducng slack varables. Slack varables are ntroduced to allow certan constrants to be volated. That s, certan tranng ponts wll be allowed to be wthn the margn. We want the number of ponts wthn the margn to be as small as possble, and of course we want ther penetraton of the margn to be as small as possble. To ths end, we ntroduce a slack varable ξ, one for each datapont. (ξ s the Greek letter x, pronounced ks. ). The slack varable s ntroduced nto the optmzaton problem n two ways. Frst, the slack varable ξ dctates the degree to whch the constrant on the th datapont can be volated. Second, by addng the slack varable to the energy functon we are amng to smultaneously mnmze the use of the slack varables. Mathematcally, the new optmzaton problem can be expressed as mn ξ +λ 1 w,b,ξ 1:N 2 w 2 (10) such that, for all,y (w T φ(x )+b) 1 ξ andξ 0 (11) As dscussed above, we am to both maxmze the margn and mnmze volaton of the margn constrants. Ths objectve functon s stll a QP, and so can be optmzed wth a QP lbrary. However, t does have a much larger number of optmzaton varables, namely, one slack varable ξ must now be optmzed for each datapont. In practce, SVMs are normally optmzed wth specal-purpose optmzaton procedures desgned specfcally for SVMs. Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 117

4 ξ > 1 f y= = 11 y= f = 0 f y= = 1 ξ < 1 ξ = 0 ξ = 0 Fgure 2: The slack varablesξ 1 for msclassfed ponts, and0 < ξ < 1 for ponts close to the decson boundary. (Fgure from Pattern Recognton and Machne Learnng by Chrs Bshop.) Loss Functons In order to better understand the behavor of SVMs, and how they compare to other methods, we wll analyze them n terms of ther loss functons. 1 In some cases, ths loss functon mght come from the problem beng solved: for example, we mght pay a certan dollar amount f we ncorrectly classfy a vector, and the penalty for a false postve mght be very dfferent for the penalty for a false negatve. The rewards and losses due to correct and ncorrect classfcaton depend on the partcular problem beng optmzed. Here, we wll smply attempt to mnmze the total number of classfcaton errors, usng a penalty s called the 0-1 Loss: L 0 1 (x,y) = { 1 yf(x) < 0 0 otherwse (12) (Note that yf(x) > 0 s the same as requrng that y and f(x) have the same sgn.) Ths loss functon says that we pay a penalty of 1 when we msclassfy a new nput, and a penalty of zero f we classfy t correctly. Ideally, we would choose the classfer to mnmze the loss over the new test data that we are gven; of course, we don t know the true labels, and nstead we optmze the followng surrogate objectve functon over the tranng data: E(w) = L(x,y )+λr(w) (13) 1 A loss functon specfes a measure of the qualty of a soluton to an optmzaton problem. It s the penalty functon that tell us how badly we want to penalze errors n a models ablty to ft the data. In probablstc methods t s typcally the negatve log lkelhood or the negatve log posteror. Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 118

5 E(z) z Fgure 3: Loss functons, E(z), for learnng, for z = y f(x). Black: 0-1 loss. Red: LR loss. Green: Quadratc loss ((z 1) 2 ). Blue: Hnge loss. (Fgure from Pattern Recognton and Machne Learnng by Chrs Bshop.) wherer(w) s a regularzer meant to prevent overfttng (and thus mprove performance on future test data). The basc assumpton s that loss on the tranng set should correspond to loss on the test set. If we can get the classfer to have small loss on the tranng data, whle also beng smooth, then the loss we pay on new data ought to not be too bg ether. Ths optmzaton framework s equvalent to MAP estmaton as dscussed prevously 2 ; however, here we are not at all concerned wth probabltes. We only care about whether the classfer gets the rght answers or not. Unfortunately, optmzng a classfer for the 0-1 loss s very dffcult: t s not dfferentable everywhere, and, where t s dfferentable, the gradent s zero everywhere. There are a set of algorthms called Perceptron Learnng whch attempt to do ths; of these, the Voted Perceptron algorthm s consdered one of the best. However, these methods are somewhat complex to analyze and we wll not dscuss them further. Instead, we wll use other loss functons that approxmate 0-1 loss. We can see that maxmum lkelhood logstc regresson s equvalent to optmzaton wth the followng loss functon: L LR = ln ( 1+e yf(x)) (14) whch s the negatve log-lkelhood of a sngle data vector. Ths functon s a poor approxmaton to the 0-1 loss, and, f all we care about s gettng the labels rght (and not the class probabltes), then we ought to search for a better approxmaton. SVMs mnmze the slack varables, whch, from the constrants, can be seen to gve the hnge loss: { 1 yf(x) 1 yf(x) > 0 L hnge = (15) 0 otherwse 2 However, not all loss functons can be vewed as the negatve log of a vald lkelhood functon, although all negatve-log lkelhoods can be vewed as loss functons for learnng. Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 119

6 That s, when a data pont s correctly classfed and further from the decson boundary than the margn, then the loss s zero. In ths way t s nsenstve to correctly-classfed ponts far from the boundary. But when the pont s wthn the margn or ncorrectly classfed, then the loss s smply the magntude of the slack varable,.e. ξ = 1 yf(x), wheref(x) = w T φ(x)+b. The hnge loss therefore ncreases lnearly for msclassfed ponts, whch s not nearly as quckly as the LR loss The Lagrangan and the Kernel Trck We now use the Lagrangan to transform the SVM problem n a way that wll lead to a powerful generalzaton. For smplcty here we assume that the dataset s lnearly separable, and so we drop the slack varables. The Langrangan allows us to take the constraned optmzaton problem above n Eqn. (9) and re-express t as an unconstraned problem. The Lagrangan for the SVM objectve functon n Eqn. (9), wth Lagrange multplers a 0, s: L(w,b,a 1:N ) = 1 2 w 2 a ( y ( w T φ(x )+b ) 1 ) (16) The mnus sgn wth the secon term s used because we are mnmzng wth respect to the frst term, but maxmzng the second. Settng the dervatve of dl dl = 0 and = 0 gves the followng constrants on the soluton: dw db w = a y φ(x ) (17) y a = 0 (18) Usng (17) we can substtute for w n 16. Then smplfyng the result, and makng use of the next constrant (17), one can derve what s often called the dual Lagrangan: L(a 1:N ) = a 1 a a j y y j φ(x ) T φ(x j ) (19) 2 Whle ths objectve functon s actually more expensve to evaluate than the prmal Lagrangan (.e., 16), t does lead to the followng modfed form j L(a 1:N ) = a 1 a a j y y j k(x,x j ) (20) 2 where k(x,x j ) = φ(x ) T φ(x j ) s called a kernel functon. For example, f we used the basc lnear features,.e.,φ(x) = x, then k(x,x j ) = x T x j. The advantage of the kernel functon representaton s that t frees us from thnkng about the features drectly; the classfer can be specfed solely n terms of the kernel. Any kernel that Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 120 j

7 satsfes a specfc techncal condton 3 s a vald kernel. For example, one of the most commonlyused kernels s the RBF kernel : k(x,z) = e γ x z 2 (21) whch corresponds to a vector of features φ(x) wth nfnte dmensonalty! (Specfcally, each element ofφs a Gaussan bass functon wth vanshng varance). Note that, just as most constrants n the Eq. (9) are not actve, the same wll be true here. That s, only some constrants wll be actve (e the support vectors), and for all other constrants, a = 0. Hence, once the model s learned, most of the tranng data can be dscarded; only the support vectors and ther a values matter. The one fnal thng we need to do s estmate the bas b. We now know the values for a for all support vectors (.e., for data constrants that are consdered actve), and hence we know w. Accordngly, for all support vectors we know, by assumpton above, that From ths one can easly solve forb. f(x ) = w T φ(x )+b = 1. (22) Applyng the SVM to new data. For the kernel representaton to be useful, we need to be able to classfy new data wthout needng to evaluate the weghts. Ths can be done as follows: f(x new ) = w T φ(x new )+b (23) ( T = a y φ(x )) φ(x new )+b (24) = a y k(x,x new )+b (25) Generalzng the kernel representaton to non-separable datasets (.e., wth slack varables) s straghtforward, but wll not be covered n ths course Choosng parameters To determne an SVM classfer, one must select: The regularzaton weght λ The parameters to the kernel functon The type of kernel functon These values are typcally selected ether by hand-tunng or cross-valdaton. 3 Specfcally, suppose one s gven N nput ponts x 1:N, and forms a matrx K such that K,j = k(x,x j ). Ths matrx must be postve semdefnte (.e., all egenvalues non-negatve) for all possble nput sets for k to be a vald kernel. Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 121

8 Fgure 4: Nonlnear classfcaton boundary learned usng kernel SVM (wth an RBF kernel). The crcled ponts are the support vectors; curves are socontours of the decson functon (e.g., the decson boundary f(x) = 0, etc.) (Fgure from Pattern Recognton and Machne Learnng by Chrs Bshop.) 17.6 Software Lke many methods n machne learnng there s freely avalable software on the web. For SVM classfcaton and regresson there s well-known software developed by Thorsten Joachms, called SVMlght, (URL: ). Copyrght c 2015 Aaron Hertzmann, Davd J. Fleet and Marcus Brubaker 122