Bilinear Formulated Multiple Kernel Learning for Multi-class Classification Problem

Transcription

1 Bilinear Formulated Multiple Kernel Learning for Multi-lass Classifiation Problem Takumi Kobayashi and Nobuyuki Otsu National Institute of Advaned Industrial Siene and Tehnology, -- Umezono, Tsukuba, Japan Abstrat. In this paper, we propose a method of multiple kernel learning (MKL) to inherently deal with multi-lass lassifiation problems. The performanes of kernel-based lassifiation methods depend on the employed kernel funtions, and it is diffiult to predefine the optimal kernel. In the framework of MKL, multiple types of kernel funtions are linearly integrated with optimizing the weights for the kernels. However, the multi-lass problems are rarely inorporated in the formulation and the optimization is time-onsuming. We formulate the multi-lass MKL in a bilinear form and propose a sheme for omputationally effiient optimization. The sheme makes the method favorably appliable to large-saled samples in the real-world problems. In the experiments on multi-lass lassifiation using several datasets, the proposed method exhibits the favorable performane and low omputation time ompared to the previous methods. Keywords: Kernel methods, multiple kernel learning, multi-lass lassifiation, bilinear form. Introdution The kernel-based methods have attrated keen attentions, exhibiting the stateof-the-art performanes, suh as in support vetor mahines (SVM) [0] and kernel multivariate analyses [8]. These methods are applied in various real-world tasks, e.g., in the fields of omputer vision and signal proessing. In the kernelbased methods, the input vetors are impliitly embedded in a high dimensional spae (alled kernel feature spae) via kernel funtions whih effiiently ompute inner produts of those vetors in the kernel feature spae. Thus, the performane of the kernel-based methods depends on how to onstrut the kernel funtions. In reent years, Lankriet et al. [5] proposed the method to integrate different kernel funtions with optimizing the weights for the kernels, whih is alled multiple kernel learning (MKL). By ombining multiple types of kernels, the heterogeneous information, whih is omplementary to eah other, an be effetively inorporated, possibly improving the performane. The omposite kernel is suessfully applied to, for example, objet reognition []. In MKL, the weights for ombining the kernels are obtained via the optimization proesses based on a ertain riterion, mainly for lassifiation. Sine the riterion an be defined in different formula, various methods for MKL have been K.W. Wong et al. (Eds.): ICONIP 00, Part II, LNCS 6444, pp , 00. Springer-Verlag Berlin Heidelberg 00

2 00 T. Kobayashi and N. Otsu proposed by treating different optimization problems in different approahes; e.g., semi-definite programming [5] and semi-infinite linear program [9,3]. Most of the methods, however, are intended for lassifying binary lasses, while realworld problems ontain multi lasses in general. In addition, for appliation to pratial problems, the optimization proess should be omputationally effiient. In this paper, we propose a MKL method for multi-lass lassifiation problems. Without deomposing the multi-lass problem into several binary lass problems, the proposed method inherently deals with it based on the formulation of Crammer & Singer [] whih first proposed multi-lass SVM using a single kernel. The ontributions of this paper are as follows: We extend the formulation of multi-lass lassifiation in [] to ope with multiple kernel funtions, and formulate multi-lass MKL (MC-MKL) in a bilinear form. In the formulation, the optimal weights for kernel funtions are obtained in respetive lasses. We propose a sheme to effetively optimize the bilinear formulated problem, whih makes the method appliable to large-saled samples. In the experiments on various datasets, we demonstrate the effetiveness of the proposed method, ompared to the existing MKL methods [7,3]. While Zien & Ong [3] proposed the method of MC-MKL based on a similar formulation, we employ a different riterion for the margin of the multi-lass lassifiers and propose a more effiient optimization sheme. Bilinear Formulation for MC-MKL To onsider multiple kernels, we introdue multiple types of features x (r) (r {,.., R}, wherer is the number of feature types). The inner produts of those features an be replaed with respetive types of kernels via kernel triks: x (r) (r) i x j k r (x (r) i, x (r) j ). Crammer & Singer [] have proposed a formulation for multilass SVM, onsidering only a single type of feature x. We extend the formulation to inorporate the multiple types of features (kernels). We additionally introdue the weights v for feature types as well as the weights w within features similarly in MKL methods [3]. These two kinds of weights are mathematially integrated into the following bilinear form to onstitute multi-lass lassifiation []: { R } =arg max w (r) x (r) = w Xv = X, w v F, () {,..,C} r= v (r) where C is the number of lasses,, F indiates Frobenius inner produt, v (r) is a weight for the r-th type of feature, w (r) is a lassifier vetor for the r-th type of feature vetor in lass, and these variables are onatenated into long vetors, respetively; [ ] x () 0 0 v v (),,v (R), X x (R), w w ().. w (R). ()

3 Bilinear Formulated Multiple Kernel Learning for Multi-lass Problem 0 Then, we an onsider the margin of the above-defined bilinear lassifiers (projetions) by using the Frobenius norm w v F, and define the following optimization problem based on a large margin riterion: OP: min {w,v },{ξ i} C w v F + κ ξ i (3) s.t. i, X i, w yi v y i F X i, w v F + δ yi, ξ i, where N is the number of samples, y i indiates the lass label of the i-th sample; y i {,.., C}, δ yi, equalstoif = y i and 0 otherwise, ξ i is a slak variable for soft margin, and κ is a parameter to ontrol the trade-off between the margin and the training errors. Note that we minimize the Frobenius norm, not squared one in SVM. Sine this problem is diffiult to diretly optimize, we employ the upper bound of the Frobenius norm: w v F = w v ( w + v ). (4) Therefore, the optimization problem OP is modified to ( C ) P : min w + v + κ ξ i (5) {w,v },{ξ i} s.t. i, w y i X i v yi w X i v + δ yi, ξ i. The weights w and v separately emerge as standard squared norm, whih failitates the optimization. It an be shown that the OP and P have the idential optimum solution by using resaling tehnique desribed in Se In the problem P, if the optimal weights v are obtained, the optimal lassifier vetors are represented as w = N τ i X iv,whereτ i are the optimal dual variables []. Thus, the multi-lass bilinear lassifier in Eq.() results in w X v = τ iv X i X v = R τ i v (r) r= R τ i v (r) r= k r (x (r) i, x (r) ), x (r) i x (r) (6) where k r (x (r) i, x (r) ) is a kernel funtion on behalf of the inner-produt of the r-th type of features, x (r) i x (r), in kernel triks. Note that the kernel funtions an be differently defined for respetive feature types. The squared weights v (r) play a role to weight the kernel funtions as in MKL, and produe the omposite kernel funtion speialized to lass. In this ase, we an introdue alternative nonnegative variables d (r) = v (r) 0 without loss of generality. The variables d are the weights for kernel funtions, and therefore the above bilinear formulation is appliable to MC-MKL. The primal problem P is reformulated to

4 0 T. Kobayashi and N. Otsu P: min {w,d },{ξ i} ( C w + d ) + κ ξ i (7) s.t. i, w y i X i d yi w X i d + δ yi, ξ i, d 0, where d =[d (),.., d (R) ],andd is a omponent-wise square root of the vetor d. In the problem P, non-negativity onstraint is additionally introdued to the problem P (or OP). The bilinear lassifier is finally obtained by w X d = R (r) τ i d r= k r (x (r) i, x (r) ). (8) We desribe the sheme to effiiently optimize P in the following setion. 3 Optimization Methods The primal problem P in Eq.(7) has the following dual form, similarly to []: max {τ i} e y i τ i, s.t. i τ i κe yi, τ i =0, r, k r (x (r) i, x (r) j )τ i τ j κ. i,j where τ i is the i-th C-dimensional dual variable, e yi is a C-dimensional vetor in whih only the y i -th element is and the others are 0, and is a C-dimensional vetor of whih all elements are. This is a onvex problem having the global optimum. However, it is atually solved by seond order one programming, whih requires exhaustive omputational ost, and it is not appliable to largesaled samples. Therefore, we take an alternative sheme to optimize the primal problem P in a manner similar to [7,]. The sheme is based on the iterative optimization for w and d, with applying projeted gradient desent. 3. Optimization with Respet to w At the t-th iteration with fixing the variable d to d [t], in a manner similar to [], the problem P results in the dual form: max {τ i} D w :max {τ i} C (v [t] i,j= i,j= X i X j v [t] )τ i τ j + e y i τ i τ i Λ ijτ j + e y i τ i, s.t. i τ i κe yi, τ i =0, (9) where Λ ij is a C-dimensional diagonal matrix, {Λ ij } = R r= d(r)[t] k r (x (r) i, x (r) j ). In this dual problem, the onstants derived from d [t] are omitted. This is optimized by iteratively solving the deomposed small subproblem [,3], as follows.

5 Bilinear Formulated Multiple Kernel Learning for Multi-lass Problem 03 Algorithm. Optimization for subproblem SubD w Require: Reindex b = b λ and λ suh that b are sorted in dereasing order Initialize =, ζ num = λ b κ, ζ den = λ. while C, ζ = ζnum ζ den b do ζ num ζ num + λ b, ζ den ζ den + λ, +. end while Output: τ =min(b,ζλ ), τ =min{κδ y,,λ (ζ β )}. The dual problem D w is deomposed into N small subproblems; the i-th subproblem fouses on the dual variable τ i assoiated with the i-th sample, while fixing the others τ j (j i): SubD w :max τ i τ i Λ ii τ i β τ i γ, s.t. τ i κe yi, τ i =0, (0) where β = Λ ij τ j e yi, γ = τ j Λ jk τ k e y j τ j. j i j i,k i j i For optimization in D w, the proess to solve SubD w works in rounds for all i and the dual variables τ i are updated until onvergene. The subproblems are rather more omplex than those in [] sine they inlude not salar value x i x j but the diagonal matrix Λ ij derived from multiple features (kernels). However, it is noteworthy that they are solved at a quite low omputational ost, as follows. Optimization for Subproblem SubD w In the following, we omit the index i for simpliity. By ignoring the onstant, the subproblem SubD w in Eq.(0) is reformulated to min τ τ, s.t. τ κλ y e y + Λ β, λ τ = λ Λ β, where τ = Λ τ + Λ β, λ is a C-dimensional vetor omposed of diagonal elements of Λ,andλ y is the y-th element of the vetor λ. Byusingb = κλ y e y + Λ β, the onstraints are rewritten as s.t. τ b, λ τ = λ b κ. () The Lagrangian for this problem is L = τ α (b τ ) ζ(λ τ λ b + κ), () where α 0, ζare Lagrangian multipliers. When the subproblem is optimized, the followings hold: L τ = τ + α ζλ =0, KKT: α (b τ )=0. (3)

6 04 T. Kobayashi and N. Otsu Therefore, we obtain α =0 τ = ζλ, ζ b λ, α > 0 τ = b,ζ> b λ. (4) By using the above, the seond onstraint in Eq.() results in λ τ =ζ α =0 λ + α >0 λ b = C α λ b κ, ζ = λ =0 b κ. (5) α =0 λ Thus, for solving the subproblem, we only seek ζ satisfying Eq.(4) and (5), and the simple algorithm is onstruted in Algorithm. The optimization of D w is the ore and most exhaustive proess for the whole optimization in P. Therefore, the effetive algorithm (Algorithm ) to solve the subproblem SubD w makes the whole optimization proess omputationally effiient. 3. Optimization with Respet to d Then, the optimization of P is performed with respet to d. In this study, we simply employ projeted gradient desent approah, although the other method suh as in [] would be appliable. In this approah, the objetive ost funtion is minimized by a line searh [6] along the projeted gradient under the onstraints d 0. Based on the priniple of strong duality, the primal P is represented by using D w in Eq.(9) with the optimal dual variables τ [t] as min {d } {( C ) d θ d + e y i τ [t] i } = W (d), s.t. d 0, i,j τ (r)[t] where θ is a R-dimensional vetor of θ (r) = i τ (r)[t] j k r (x (r) i, x (r) j ). In this ase, W is differentiable with respet to d (ref. [7]), and thus the gradients are obtained as W = θ. Thereby, the optimization in P is performed by using projeted gradient desent, d [t+] = d [t] ɛ W.Weapplyalinesearh[6] to greedily seek the parameter ɛ suh that W, i.e., the objetive ost funtion in P, is minimized while ensuring d 0. Note that, in this greedy searh, the ost funtion is evaluated several times via optimization of D w. 3.3 Resaling After optimization of D w with the fixed d, the ost funtion is further dereased by simply resaling the variables of τ and d, so as to reah the lower bound in Eq.(4). The resaling, ˆτ i = s τ i, ˆd = s d, does not affet the bilinear projetion in Eq.(8) and thus the onstraints in P are kept: ŵ X ˆd = R s τ i r= d (r) k r (x, x i )=w s Xd, (6)

7 Bilinear Formulated Multiple Kernel Learning for Multi-lass Problem 05 while the first term in the ost funtion is transformed to C ŵ + ˆd = C s w + d. (7) s The optimal resaling that minimizes the above is analytially obtained as s = d / w, and Eq.(7) equals to the lower bound (the Frobenius norm): C ŵ + ˆd = C d w = C w d F. (8) Although the resaled ˆτ is not neessarily the solution of the problem D w with the resaled ˆd, in the greedy optimization for d, the gradients using ˆτ are employed as the approximation for W ( ˆd). This resaling ontributes to fast onvergene. 4 Experimental Result We show the lassifiation performanes and omputation time of the proposed methods in omparison with the other MKL methods [7,3] on various datasets. We employed the -vs-all version of [7] to ope with multi-lass problems. The proposed method is implemented by using MATLAB with C-mex on Xeon 3GHz PC. For the methods of [7,3], we used the MATLAB odes provided by the authors and ombined them with libsvm [] and MOSEK optimization toolbox in order to speed up those methods as muh as possible. In this experiment, the parameter values in the all methods are set as follows: κ is determined from κ {0.5,, 0} based on 3-fold ross validation and the maximum number of iterations is set to 40 iterations for fair omparison of omputation time. These methods almost onverge on various datasets within 40 iterations. First, we used four benhmark datasets: waveform from UCI Mahine Learning Repository, satimage andsegment from the STATLOG projet, and USPS [4]. The multiple RBF kernels with 0 σ s (uniformly seleted on the logarithmi sale over [0, 0 ]) were employed. We drew 000 random training samples and lassified the remained samples. The trial is repeated 0 times and the average performane is reported. Fig. (a,b) shows the lassifiation results (error rates) and omputation time on those datasets. While the performanes of the proposed method are ompetitive to the others, the omputation time is muh more redued; espeially, more than 0 times faster than the method of [3]. Next, we applied the proposed method to the other pratial lassifiation problems in ell biology. The task is to predit the sub-ellular loalizations of proteins, and in this ase it results in multi-lass lassifiation problems. We employed a total of 69 kernels of whih details are desribed in [3]. MKL would

8 06 T. Kobayashi and N. Otsu Error rate (%) waveform (3) [000] satimage (6) [000] segment (7) [000] Benhmark dataset Bilinear MCMKL simplemkl [7] MCMKL SILP [3] USPS (0) [000] Computation time (se) [log-sale] waveform (3) [000] satimage (6) [000] Bilinear MCMKL simplemkl [7] MCMKL SILP [3] segment (7) [000] (a) Classifiation performane (b) Computation time Biologial dataset Error rate (%) plant (4) [376] nonplant (3) [093] Bilinear MCMKL simplemkl [7] MCMKL SILP [3] psort+ (4) [7] psort- (5) [578] () Classifiation performane Computation time (se) [log-sale] plant (4) [376] nonplant (3) [093] psort+ (4) [7] USPS (0) [000] Bilinear MCMKL simplemkl [7] MCMKL SILP [3] (d) Computation time psort- (5) [578] Fig.. The lassifiation performanes (error rates) and omputation time on the four benhmark datasets. The number of lasses is indiated in parentheses and that of training samples is in brakets. The left bar shows the result of the proposed methods. be effetively applied to these substantial types of kernel. In this experiment, we used four biologial datasets [3]: plant, nonplant, psort+, andpsort-. We randomly split the dataset into 40% for training and 60% for testing. The trial is repeated 0 times and the average performane is reported. The results are shown in Fig. (,d), demonstrating that the proposed method is quite effetive; the proposed method is superior and faster to the methods of [7,3]. The experimental result shows that the proposed method effetively and effiiently ombines a lot of heterogeneous kernel funtions. 5 Conlusion We have proposed a multiple kernel learning (MKL) method to deal with multilass problems. In the proposed method, the multi-lass lassifiation using multiple kernels is formulated in the bilinear form, and the omputationally effiient optimization sheme is proposed in order to be appliable to large-saled samples. In the experiments on the benhmarks and the biologial datasets, the proposed method exhibited the favorable performanes and omputation time ompared to the previous methods of MKL.

9 Bilinear Formulated Multiple Kernel Learning for Multi-lass Problem 07 Referenes. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vetor mahines (00), Software available at Crammer, K., Singer, Y.: On the algorithmi implementation of multilass kernelbased vetor mahines. Journal of Mahine Learning Researh, 65 9 (00) 3. Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: A library for large linear lassifiation. Journal of Mahine Learning Researh 9(6), (008) 4. Hull, J.: A database for handwritten text reognition researh. IEEE Trans. Pattern Analysis and Mahine Intelligene 6(5), (994) 5. Lankriet, G., Cristianini, N., Bartlett, P., Ghaoui, L.E., Jordan, M.I.: Learning the kernel matrix with semidefinite programming. Journal of Mahine Learning Researh 5, 7 7 (004) 6. Noedal, J., Wright, S. (eds.): Numerial optimization. Springer, New York (999) 7. Rakotomamonjy, A., Bah, F.R., Canu, S., Grandvalet, Y.: Simplemkl. Journal of Mahine Learning Researh 9, 49 5 (008) 8. Shölkopf, B., Smola, A.: Learning with Kernels. MIT Press, Cambridge (00) 9. Sonnenburg, S., Rätsh, G., Shäfer, C., Shölkopf, B.: Large sale multiple kernel learning. Journal of Mahine Learning Researh 7, (006) 0. Vapnik, V.: Statistial Learning Theory. Wiley, Chihester (998). Varma, M., Ray, D.: Learning the disriminative power-invariane trade-off. In: Proeedings of IEEE th International Conferene on Computer Vision (007). Xu, Z., Jin, R., King, I., Lyu, M.R.: An extended level method for effiient multiple kernel learning. In: Advanes in Neural Information Proessing Systems, vol., pp (008) 3. Zien, A., Ong, C.S.: Multilass multiple kernel learning. In: Proeedings of the 4th International Conferene on Mahine Learning (007)