SVM for Statisticians

Transcription

1 SVM for Statisticias Youyi Fog Fred Hutchiso Cacer Research Istitute November 13, / 21

2 Primal Problem ad Pealized Loss Fuctio Miimize J over b, β ad ξ uder some costraits J = 1 2 β 2 + C ξ i (1) y i (b + x i β) 1 ξ i (2) ξ i 0 (3) Rewrite (2) as Combiig with (3), we have ξ i 1 y i (b + x i β) ξ i max (1 y i (b + x i β), 0) = {1 y i (b + x i β)} + Whe J is miimized, ξ i should be equal to its lower boud simply because we are miimizig J over b, β ad ξ. Thus effectively we are miimizig J = C {1 y i (b + x i β)} + + β 2 (4) The pealized loss form is coveiet mathematically, but icoveiet for optimizatio because of the hige loss fuctio () + Primal formulatio is coveiet for optimizatio through the use of slack variables ξ i 2 / 21

3 Dual Problem Because the costrait (2) ivolves multiple parameters, it is diffi cult to hadle. We dualize with respect to costraits (2) ad (3). The primary Lagragia is L p = 1 2 β 2 + C ξ i α i {y i (b + x i β) 1 + ξ i } γ i ξ i. (5) L p has to be miimized with respect to β, b ad ξ i ad maximized with respect to o-egative Lagrage multipliers α i ad γ i. Takig derivatives with respect to the primal space variables, we get β = α i y i x i (6) α i y i = 0 (7) C = α i + γ i (8) 3 / 21

4 Dual Problem (cot d) Pluggig (6)-(8) back ito (5), we get the dual variables Lagragia L d = 1 ) 2 β 2 + (C α i γ i ) ξ i α i y i (b + x T i β + α i = 1 2 βt β α i y i x T i β + α i by (7) ad (8) ( ) 1 = 2 βt α i y i x T i β + α i = 1 2 y i y j α i α j x T i x j + α i by (6) i,j=1 We are to maximize L d uder two costraits that come from (7) ad (8) ad with the o-egativity of Lagrage multipliers α i y i = 0 (9) 0 α i C (10) Comparig this set of costraits to the cotraits of the primal problem, (2), we see that they ivolve oe variable at a time. This allows for simple decompositio algorithms to work 4 / 21

5 KKT Coditios Statioarity Primal ad dual feasibility Complemetary slackess α i {y i (b + x i β) 1 + ξ i } = 0 (C α i ) ξ i = 0 5 / 21

6 Itercept-free Model If we do t wat b to be i the model, as i AUC work, we ca set b = 0. As a result, the costrait (7)/(9) do ot apply b is also kow as the bias term 6 / 21

7 Weighted SVM Suppose we wat to maximize The primary Lagragia becomes J = C w i {1 y i (b + x i β)} + + β 2, Costrait (8) becomes Dual variables Lagragia becomes L p = 1 2 β 2 + C w i ξ i α i {y i (b + x i β) 1 + ξ i } γ i ξ i Cw i = α i + γ i L d = 1 ) 2 β 2 + (Cw i α i γ i ) ξ i α i y i (b + x T i β + α i = 1 2 y i y j α i α j x T i x j + α i by (6) i,j=1 uder the costrait α i y i = 0 0 α i Cw i The KKT complemetary coditios are α i {y i (b + x i β) 1 + ξ i } = 0 (w i C α i ) i ξ i = 0 7 / 21

8 Decompositio Algorithm A decompositio algorithm chooses a subset of variables to optimize at each iteratio Workig set size ad selectio strategy SVMlight defaults to 10 accordig to steepest gradiet, while satisfyig all costraits (Joachims 1998) oce i a while, select a somewhat radom workig set to escape dead zoe (svmlight code) libsvm oly supports 2. First var is chose based o gradiet, ad the secod var is chose based o secod order iformatio (Fa et al 2005) Burges (1998) metios cojugate gradiet I our experiece, set size of 2 works better tha set size of 1 (as illustrated o the ext slide) 8 / 21

9 slope: 1 slope: 2 val oe var two var val oe var two var sweep/iter sweep/iter slope: 5 slope: 20 oe var two var oe var two var val val sweep/iter sweep/iter 9 / 21

10 Decompositio Algorithm (cot d) Other heuristics/tricks Shrikig (Joachims, 1998). Oly choose workig set from a subset of total variables Cachig. This happes at several levels. Burges (1998) To optimize a workig set is to solve a costraied quadratic problem. May optimizers ca be used. SVMlight uses Hideo s optimizer miquad (explaied i the ext few slides) 10 / 21

11 11 / 21

12 12 / 21

13 13 / 21

14 14 / 21

15 15 / 21

16 16 / 21

17 17 / 21

18 SVM software svmlight libsvm svmw aucm mai advatage is the subproblem is ot restricted to two variables implemets ull bias/itercept klar R package hadles more types of svm models (but oe of the exteded model solves α T Qα + b T α, where b = 1 does ot implemet ull bias/itercept e1071 R package 18 / 21

19 Oe-class SVM Usual SVM miimize subject to w 2 + C ξ i y i ( Φ (x i ), w + b) 1 ξ i ξ i 0 Oe-class SVM miimizes subject to w 2 ρ + C ξ i y i ( Φ (x i ), w + b) ρ ξ i ξ i 0 19 / 21

20 Ackowledgemet Shuxi Yi Krisztia Sebestye 20 / 21

21 Refereces Be-Hur, A., Og, C., Soeburg, S., Scholkopf, B., ad Ratsch, G. (2008), Support vector machies ad kerels for computatioal biology, PLoS computatioal biology, 4, e Burges, C. (1998), A tutorial o support vector machies for patter recogitio, Data miig ad kowledge discovery, 2, Chag, C. ad Li, C. (2011), LIBSVM: a library for support vector machies, ACM Trasactios o Itelliget Systems ad Techology (TIST), 2, 27. Joachims, T. (1998), Makig Large-Scale SVM Learig Practical, Advaces i Kerel Methods Support Vector Learig, pp Li, C. ad Wag, S. (2002), Fuzzy support vector machies, Neural Networks, IEEE Trasactios o, 13, Moguerza, J. ad Muñoz, A. (2006), Support vector machies with applicatios, Statistical Sciece, pp Wag, L. (2005), Support Vector Machies: theory ad applicatios, vol. 177, Spriger Verlag. 21 / 21