A Unified View on Multi-class Support Vector Classification Supplement

Transcription

1 Journal of Mahine Learning Researh??) Submitted 7/15; Published?/?? A Unified View on Multi-lass Support Vetor Classifiation Supplement Ürün Doğan Mirosoft Researh Tobias Glasmahers Institut für Neuroinformatik Ruhr-Universität Bohum, Germany Christian Igel Department of Computer Siene University of Copenhagen, Denmark udogan@mirosoft.om tobias.glasmahers@ini.rub.de igel@diku.dk Editor: Ingo Steinwart A. Aggregation Operators As Linear Programs Aggregation operators, whih ombine the d margin violations into a single ost value, an be understood as omputing the value of the linear program v, y) = min ξ r R y ξ r s.t. p P y : ξ syp) v p fx), y) for the variables ξ = ξ r ) r Ry, where P y Y, R y is an index set, and s y : P y R y is surjetive. The set P y lists all margin violations that enter the loss, R y lists the slak variables, and s y assigns slak variables to margin omponents, depending on the partiular loss in use. Table 1 lists the onfigurations of the linear programs orresponding to the different aggregation operators. linear program definition aggregation operator P y R y s y self {y} { } p o-max Y \ {y} { } p t-max Y { } p t-sum Y Y id o-sum Y \ {y} Y \ {y} id Table 1: Aggregation operators and the orresponding linear programs, expressed in terms of the sets P y and R y, and the assignment s y : P y R y, for eah y Y. As for the margin funtion definition based on the sparse oeffiients ν y,p,m, the true underlying degrees of freedom for aggregation operators are far more restrited than it? Ürün Doğan, Tobias Glasmahers and Christian Igel.

2 Doğan, Glasmahers and Igel seems, in partiular if lasses are treated symmetrially. For symmetry reasons, the sets P y an take only the three values {y}, Y, and Y \ {y}, sine all lasses y are to be treated the same way. The same argument implies that s y either has to be injetive or onstant, restrited to the atomi invariant subsets {y} and Y \ {y}. This again leaves only few hoies for R y under the restrition that s y is surjetive. The hinge loss L hinge µ) = max{0, 1 µ} an also be expressed as a linear program, namely L hinge µ) = min u u s.t. u 1 µ u 0. The two linear programs an be ombined into one: Lfx), y) = min ξ r R y ξ r s.t. p P y : ξ syp) 1 µ p fx), y) r R y : ξ r 0 The first onstraint an be rewritten as µ p fx), y) = m ν y,p,m f m x) 1 ξ syp). Thus, the deision funtion values enter a multi-lass loss based on the hinge loss as parameters of a linear program. B. Deriving the Uniform Dual Problems For deriving the dual problem from the primal, we introdue Lagrange multipliers α 0, β i,r 0, η H, and τ R orresponding to the onstraints of the primal problem, and ompute the Lagrangian L = 1 w 2 + C ξ i,r 2 i,r + [ α γ yi,p ] ) ν yi,p, w, φx i ) + b ξi,syi p) β i,r ξ i,r i,r + η, w + τ b, 2

3 A Unified View on Multi-ategory Support Vetor Classifiation: Supplement with derivatives L = w α ν yi,p,φx i ) + η = 0 w = α ν yi,p,φx i ) η 1) w L = α ν yi,p, + τ = 0 α ν yi,p, = τ b L = C α β i,r = 0 α C. ξ i,r p P r y The sets Py r, r R y, are defined as Py r = s 1 y {r}) = {p P y s y p) = r}. They form a partition of the set P y of onstraints. To derive the dual in the absene of the sum-to-zero onstraint we just set the dual variables η and τ to zero. Then the first derivative above gives us an expression of w in terms of α. In the ase with sum-to-zero onstraint we get 0 = w = p P r y α ν yi,p,φx i ) η η = α 1 d ν yi,p, ) φx i ) and thus w = α [ m δ m, 1 ) ] ν yi,p,m φx i ). d To get to the dual problem, we plug this expression into the Lagrangian using the identity δ m, 1 ) δ n, 1 ) = δ m,n 1 ). d d d C. Proof of Theorem 5 In the following, we outline a proof of Theorem 5. Let Lfx), y) denote either the loss funtion used by the AMO mahine or the loss funtion used by the ATM mahine, that is, the loss resulting from appliation of either the max-over-others or the total-max operator to absolute margins: { } Lfx), y) = max v abs fx), y) = [ { 1 + max f x) }] AMO) Y \{y} Y \{y} + or { } Lfx), y) = max v abs fx), y) Y { [ ] = max 1 + max {f x)}, [ 1 f y x) ] } Y \{y} + + ATM) Then Theorem 5 states that the minimizer f of the orresponding risk R = E[Lfx), y)], subjet to the sum-to-zero onstraint Y f x) = 0, satisfies:. 3

4 Doğan, Glasmahers and Igel If there exists a majority lass y Y suh that P y > d 1)/d, then f y x) = d 1 and f x) = 1 for all Y \ {y}. If P y < d 1)/d for all y Y, then f x) = 0. Proof We demonstrate the proof for the AMO loss funtion. Following Liu 2007), we argue that f x) 1 for all Y. Suppose f x) < 1, then it is easy to see that f defined as f x) = 1 and f e x) = f e x) + f x) + 1)/d 1) fulfills R x f) R x f) ontraditing the optimality of f. Restriting the solution spae to f x) 1 allows us to write the point-wise risk as R x = y Y P y 1 + max { f x) } ) Y \ {y}. Now we pik y arg max{f x) Y } and treat the value f y x) 0 whih is non-negative beause of the sum-to-zero onstraint) as fixed from now on. We write the point-wise risk as R x = P y 1 + max { f x) } ) Y \ {y} + P 1 + f y x) ). y The best we an do to keep this risk low is to set all omponents f x), y, to the same value: f x) = y f x)/d 1) = f y x)/d 1) for all Y \ {y}. It holds R x = P y 1 f ) y) + P 1 + f y x) ) = P y 1 f y) d 1 d 1 y = 1 P y fy) d P y) f y x) = d d 1 P y ) f y x). ) + 1 P y ) 1 + f y x) ) For P y > d 1)/d this expression is a dereasing funtion of f y x), resulting in the optimum f y x) = d 1 and f x) = 1 for y, whih maximizes f y x) under the onstraints x f x) = 0 and : f x) 1. In ontrast, for P y < d 1)/d the risk is lower bounded by one. In this ase f x) = 0 minimizes the expression yielding R x = 1. The analogous result for the ATM loss funtion an be proven with exatly the same arguments. 4

5 A Unified View on Multi-ategory Support Vetor Classifiation: Supplement D. Data Sets The desriptive statistis of the 12 UCI data sets used in both the linear as well as nonlinear SVM experiments are given in Table 2. The additional data sets used in the linear SVM experiments are desribed in Table 3. Data set d l train l test p Abalone Car Glass Iris Opt. digits Page bloks Sat Segment Soy bean Vehile Red wine White wine Table 2: Desriptive statistis of the 12 UCI data sets used in the non-linear SVM study. The olumns d, l train, l test, and p ontain the number of lasses, the number of training examples, the number of test examples, and the input spae dimension number of features), respetively. Data set d l train l test p Covertype 7 406, , Letter 26 15,000 5, News ,935 3,993 62,061 Setor 105 6,412 3,207 55,197 Usps 10 7,291 2, Table 3: Desriptive statistis of the additional data sets used in the linear SVM experiments. The olumns d, l train, l test, and p ontain the number of lasses, the number of training examples, the number of test examples, and the input spae dimension number of features), respetively. 5

6 Doğan, Glasmahers and Igel E. Model Seletion Results The best parameter onfigurations C, γ) for the non-linear SVMs are found in Table 4. The values of the parameter C for the linear SVM experiments are listed in Table 5. OVA MMR WW CS LLW AMO ATS ATM RM Abalone Car Glass Iris Opt. digits Page bloks Sat Segment Soy bean Vehile Red wine White wine Table 4: Best hyperparameter values, ) found by the model seletion proedure. Referenes Y. Liu. Fisher onsisteny of multiategory support vetor mahines. In M. Meila and X. Shen, editors, Eleventh International Conferene on Artifiial Intelligene and Statistis AISTATS), volume 2 of JMLR W&P, pages ,

7 A Unified View on Multi-ategory Support Vetor Classifiation: Supplement OVA MMR WW CS LLW AMO ATS ATM RM Cover type Letter News Setor Usps Abalone Car Glass Iris Opt. digits Page bloks Sat Segment Soybean Vehile Red wine White wine Table 5: Best hyperparameter values ) for linear models found by the model seletion proedure. 7