Understanding SVM (and associated kernel machines) through the development of a Matlab toolbox

Transcription

1 Understanding SVM (and associated kernel machines) through the development of a Matlab toolbox Stephane Canu To cite this version: Stephane Canu. Understanding SVM (and associated kernel machines) through the development of a Matlab toolbox. Engineering school. Introduction to Support Vector Machines (SVM), Sao Paulo, 204, pp.33. <cel > HAL Id: cel Submitted on 8 Jun 204 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 Lecture 8: Multi Class SVM Stéphane Canu Sao Paulo 204 April 0, 204

3 Roadmap Multi Class SVM 3 different strategies for multi class SVM Multi Class SVM by decomposition Multi class SVM Coupling convex hulls

4 3 different strategies for multi class SVM Decomposition approaches one vs all: winner takes all one vs one: max-wins voting pairwise coupling: use probability c SVDD 2 global approach (size c n), formal (different variations) min f H,α 0,ξ IR n 2 with and c f l 2 H + C p l= n c ξ p il i= l=,l y i f yi (x i )+b yi f l (x i )+b l + 2 ξ il ξ il 0 for i =,..., n; l =,..., c; l y i non consistent estimator but practically useful structured outputs 3 A coupling formulation using the convex hulls

5 3 different strategies for multi class SVM Decomposition approaches one vs all: winner takes all one vs one: max-wins voting pairwise coupling: use probability best results c SVDD 2 global approach (size c n), formal (different variations) min f H,α 0,ξ IR n 2 with and c f l 2 H + C p l= n c ξ p il i= l=,l y i f yi (x i )+b yi f l (x i )+b l + 2 ξ il ξ il 0 for i =,..., n; l =,..., c; l y i non consistent estimator but practically useful structured outputs 3 A coupling formulation using the convex hulls

6 Multiclass SVM: complexity issues n training data n = 60, 000 for MNIST c class c = 0 for MNIST problem number of approach discrimination rejection size sub problems vs. all n c n c(c ) vs. c n c SVDD c c - ++ all together n c ++ - coupling CH n + +

7 Roadmap Multi Class SVM 3 different strategies for multi class SVM Multi Class SVM by decomposition Multi class SVM Coupling convex hulls

8 Multi Class SVM by decomposition One-Against-All Methods winner-takes-all strategy One-vs-One: pairwise methods max-wins voting directed acyclic graph (DAG) error-correcting codes post process probabilities Hierarchical binary tree for multi-class SVM mas622j/projects/aisen-project/

9 SVM and probabilities (Platt, 999) The decision function of the SVM is: sign ( f(x)+b ) log IP(Y = x) IP(Y = x) should have (almost) the same sign as f(x)+b log IP(Y = x) IP(Y = x) = a (f(x)+b)+a 2 IP(Y = x) = +exp a(f(x)+b)+a2 max L a,a 2 with L = a et a 2 estimated using maximum likelihood on new data n IP(Y = x i ) y i +( IP(Y = x i )) ( y i) i= and log L = n i= y i log(ip(y = x i ))+( y i )log( IP(Y = x i )) = n i= y i log ( IP(Y= x i ) IP(Y= x i )) + log( IP(Y = xi )) = n i= y ) i( a (f(x i )+b)+a 2 log(+exp a (f(x i )+b)+a 2 ) = n i= y ) i( a z i log(+exp a z i) Newton iterations: a new a old H logl

10 SVM and probabilities (Platt, 999) n max log L = ( y i a ) z i log(+exp a z i ) a IR 2 i= Newton iterations a new a old H logl logl = = n z y i z i expa +exp z a z i n ( yi IP(Y = x i ) ) z i = Z (y p) i= i= H = n i= z i z i IP(Y = x i ) ( IP(Y = x i ) ) = Z WZ Newton iterations a new a old +(Z WZ) Z (y p)

11 SVM and probabilities: practical issues y t = ε + = n + + n ε = n + 2 if y i = if y i = in: X,y, f /out: p 2 t 3 Z 4 loop until convergence p +exp a z 2 W diag ( p( p) ) 3 a new a old +(Z WZ) Z (t p)

12 SVM and probabilities: pairwise coupling From pairwise probabilities IP(c l, c j ) to class probabilities p l = IP(c l x) ( )( ) Q e p e = 0 µ min p ( 0 The global procedure : ) c l IP(c l, c j ) 2 (p l p j ) 2 l= j= (Xa, ya, Xt, yt) split(x, y) 2 (Xl, yl, Xp, yp) split(xa, ya) 3 loop for all pairs (c i, c j ) of classes { IP(cl, c with Q lj = j ) 2 l j i IP(c l, c i ) 2 l = j model i,j train_svm(xl, yl,(c i, c j )) 2 IP(c i, c j ) estimate_proba(xp, yp, model) % Platt estimate 4 p post_process(xt, yt, IP) % Pairwise Coupling Wu, Lin & Weng, 2004, Duan & Keerti, 05

13 SVM and probabilities Some facts SVM is universally consistent (converges towards the Bayes risk) SVM asymptotically implements the bayes rule but theoretically: no consistency towards conditional probabilities (due to the nature of sparsity) to estimate conditional probabilities on an interval (typically[ 2 η, 2 +η]) to sparseness in this interval (all data points have to be support vectors) Bartlett & Tewari, JMLR, 07

14 SVM and probabilities (2/2) An alternative approach g(x) ε (x) IP(Y = x) g(x)+ε + (x) with g(x) = +4 f (x) α 0 non parametric functions ε and ε + have to verify: with a = log 2 and a 2 = 0 g(x)+ε + (x) = exp a( f(x) α0)++a2 g(x) ε (x) = exp a(+f(x)+α0)++a2 Grandvalet et al., 07

15 Roadmap Multi Class SVM 3 different strategies for multi class SVM Multi Class SVM by decomposition Multi class SVM Coupling convex hulls

16 Multi class SVM: the decision function One hyperplane by class f l (x) = w l x+b l l =, c Winner takes all decision function ( D(x) = Argmax w x+b, w2 x+b 2,..., wl x+b l,..., w ) c x+b c l=,c We can revisit the 2 classes case in this setting c (d + ) unknown variables (w l, b l ); l =, c

17 Multi class SVM: the optimization problem The margin in the multidimensional case m = min l y i ( v yi x i a yi v l x i + a l ) = v yi x i + a yi max l y i ( v l x i + a l ) The maximal margin multiclass SVM max m v l,a l with vy i x i + a yi vl x i a l m for i =, n; l =, c; l y i c and 2 v l 2 = l= The multiclass SVM c min w l,b 2 w l 2 l l= with x i (w yi w l )+b yi b l for i =, n; l =, c; l y i

18 Multi class SVM: KKT and dual form: The 3 classes case min w l,b 2 l 3 w l 2 l= with w y i x i + b yi w l x i + b l + for i =, n; l =, 3; l y i min w l,b 2 w w w 3 2 l with w x i + b w2 x i + b 2 + for i such that y i = w x i + b w3 x i + b 3 + for i such that y i = w2 x i + b 2 w x i + b + for i such that y i = 2 w2 x i + b 2 w3 x i + b 3 + for i such that y i = 2 w3 x i + b 3 w x i + b + for i such that y i = 3 w3 x i + b 3 w2 x i + b 2 + for i such that y i = 3 L = 2 ( w 2 + w w 3 2 ) α 2 (X (w w 2 )+b b 2 ) α 3 (X (w w 3 )+b b 3 ) α 2 (X 2(w 2 w )+b 2 b ) α 23 (X 2(w 2 w 3 )+b 2 b 3 ) α 3 (X 3(w 3 w )+b 3 b ) α 32 (X 3(w 3 w 2 )+b 3 b 2 )

19 Multi class SVM: KKT and dual form: The 3 classes case L = 2 w 2 α (XMw+Ab ) with I I 0 w I 0 I w = w 2 IR 3d M = M I = I I 0 w 3 0 I I I 0 I 0 I I a 6d 3d matrix where I the identity matrix and X X X = 0 0 X X X X 3 a 2n 6d matrix with input data X = X X 2 n d X 3

20 Multi class SVM: KKT and dual form: The 3 classes case KKT Stationality conditions = w L = w M X α b L = A α The dual min α IR 2 n 2 α Gα e α with Ab = 0 and 0 α With G = XMM X = X(M I)(M I) X = X(MM I)X = (MM I). XX = (MM I). I K I and M =

21 Multi class SVM and slack variables (2 variants) A slack for all (Vapnik & Blanz, Weston & Watkins 998) c n c min w l,b l,ξ IR cn 2 w l 2 + C ξ il l= i= l=,l y i with wy i x i + b yi wl x i b l ξ il and ξ il 0 for i =, n; l =, c; l y i The dual min α IR 2 n 2 α Gα e α with Ab = 0 and 0 α C Max error, a slack per training data (Cramer and Singer, 200) c n min w l,b l,ξ IR n 2 w l 2 + C ξ i l= i= with (w yi w l ) x i ξ i for i =, n; l =, c; l y i i= and ξ i 0 for i =, n

22 Multi class SVM and Kernels c min f H,α 0,ξ IR cn 2 f l 2 H + C with l= n c ξ il i= l=,l y i f yi (x i )+b yi f l (x i ) b l ξ il n i= and ξ il 0 for i =, n; l =, c; l y i The dual min α IR 2 n 2 α Gα e α with Ab = 0 and 0 α C where G is the multi class kernel matrix

23 Other Multi class SVM Lee, Lin & Wahba, 2004 c λ min f H 2 f l 2 H + n l= c with f l (x) = 0 l= n c (f l (x i )+ c ) + l=,l y i i= x Structured outputs = Cramer and Singer, 200 MSVMpack : A Multi-Class Support Vector Machine Package Fabien Lauer & Yann Guermeur

24 Roadmap Multi Class SVM 3 different strategies for multi class SVM Multi Class SVM by decomposition Multi class SVM Coupling convex hulls

25 One more way to derivate SVM Minimizing the distance between the convex hulls min u v 2 α with u(x) = α i (x i x), v(x) = and {i y i =} {i y i =} α i =, {i y i = } α i =, 0 α i {i y i = } α i (x i x) i =, n

26 The multi class case min α with and c c u l u l 2 l= l = u l (x) = α i,l (x i x), l =, c {i y i =l} α i,l =, 0 α i,l i =, n;l =, c {i y i =l}

27 Bibliography Estimating probabilities Platt, J. (2000). Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in large margin classifiers. MIT Press. T. Lin, C.-J. Lin, R.C. Weng, A note on Platt s probabilistic outputs for support vector machines, Mach. Learn. 68 (2007) Multiclass SVM K.-B. Duan & S. Keerthi (2005). "Which Is the Best Multiclass SVM Method? An Empirical Study". T.-F. Wu, C.-J. Lin, R.C. Weng, Probability estimates for multi-class classification by pairwise coupling, JMLR. 5 (2004) K. Crammer & Y. Singer (200). "On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines". JMLR 2: Lee, Y.; Lin, Y.; and Wahba, G. (200). "Multicategory Support Vector Machines". Computing Science and Statistics Stéphane Canu (INSA Rouen - LITIS) April 0, / 25