SimpleMKL. hal , version 1-26 Jan Abstract. Alain Rakotomamonjy LITIS EA 4108 Université de Rouen Saint Etienne du Rouvray, France

Transcription

1 SipleMKL Alain Rakotoaonjy LITIS EA 48 Université de Rouen 768 Saint Etienne du Rouvray, France Francis Bach INRIA - Willow project Départeent d Inforatique, Ecole Norale Supérieure 45, Rue d Ul 7523 Paris, France Stéphane Canu LITIS EA 48 INSA de Rouen 768 Saint Etienne du Rouvray, France Yves Grandvalet CNRS/IDIAP Research Institute, Centre du Parc, Av. des Prés-Beudin 2 92 Martigny, Switzerland Abstract alain.rakotoaonjy@insa-rouen.fr francis.bach@ines.org stephane.canu@insa-rouen.fr yves.grandvalet@idiap.ch Multiple kernel learning ais at siultaneously learning a kernel and the associated predictor in supervised learning settings. For the support vector achine, an efficient and general ultiple kernel learning (MKL) algorith, based on sei-infinite linear progaing, has been recently proposed. This approach has opened new perspectives since it akes the MKL approach tractable for large-scale probles, by iteratively using existing support vector achine code. However, it turns out that this iterative algorith needs nuerous iterations for converging towards a reasonable solution. In this paper, we address the MKL proble through an adaptive 2-nor regularization forulation that encourages sparse kernel cobinations. Apart fro learning the cobination, we solve a standard SVM optiization proble, where the kernel is defined as a linear cobination of ultiple kernels. We propose an algorith, naed SipleMKL, for solving this MKL proble and provide a new insight on MKL algoriths based on ixed-nor regularization by showing that the two approaches are equivalent. Furtherore, we show how SipleMKL can be applied beyond binary classification, for probles like regression, clustering (one-class classification) or ulticlass classification. Experiental results show that the proposed algorith converges rapidly and that its efficiency copares favorably to other MKL algoriths. Finally, we illustrate the usefulness of MKL for soe regressors based on wavelet kernels and on soe odel selection probles related to ulticlass classification probles. A SipleMKL Toolbox is available at

2 . Introduction During the last few years, kernel ethods, such as support vector achines (SVM) have proved to be efficient tools for solving learning probles like classification or regression (Schölkopf and Sola, 2). For such tasks, the perforance of the learning algorith strongly depends on the data representation. In kernel ethods, the data representation is iplicitly chosen through the so-called kernel K(x,x ). This kernel actually plays two roles: it defines the siilarity between two exaples x and x, while defining an appropriate regularization ter for the learning proble. Let {x i,y i } l i= be the learning set, where x i belongs to soe input space X and y i is the target value for pattern x i. For kernel algoriths, the solution of the learning proble is of the for l f(x) = α i K(x,x i ) + b, () i= where α i and b are soe coefficients to be learned fro exaples, while K(, ) is a given positive definite kernel associated with a reproducing kernel Hilbert space (RKHS) H. In soe situations, a achine learning practitioner ay be interested in ore flexible odels. Recent applications have shown that using ultiple kernels instead of a single one can enhance the interpretability of the decision function and iprove perforances (Lanckriet et al., 24a). In such cases, a convenient approach is to consider that the kernel K(x,x ) is actually a convex cobination of basis kernels: K(x,x ) = M d K (x,x ), with d, = M d =, where M is the total nuber of kernels. Each basis kernel K ay either use the full set of variables describing x or subsets of variables steing fro different data sources (Lanckriet et al., 24a). Alternatively, the kernels K can siply be classical kernels (such as Gaussian kernels) with different paraeters. Within this fraework, the proble of data representation through the kernel is then transferred to the choice of weights d. Learning both the coefficients α i and the weights d in a single optiization proble is known as the ultiple kernel learning (MKL) proble. For binary classification, the MKL proble has been introduced by Lanckriet et al. (24b), resulting in a quadratically constrained quadratic prograing proble that becoes rapidly intractable as the nuber of learning exaples or kernels becoe large. What akes this proble difficult is that it is actually a convex but non-sooth iniization proble. Indeed, Bach et al. (24a) have shown that the MKL forulation of Lanckriet et al. (24b) is actually the dual of a SVM proble in which the weight vector has been regularized according to a ixed (l 2,l )-nor instead of the classical squared l 2 -nor. Bach et al. (24a) have considered a soothed version of the proble for which they proposed a SMO-like algorith that enables to tackle ediu-scale probles. Sonnenburg et al. (26) reforulate the MKL proble of Lanckriet et al. (24b) as a sei-infinite linear progra (SILP). The advantage of the latter forulation is that the algorith addresses the proble by iteratively solving a classical SVM proble with a single kernel, for which any efficient toolboxes exist (Vishwanathan et al., 23, Loosli et al., = 2

3 25, Chang and Lin, 2), and a linear progra whose nuber of constraints increases along with iterations. A very nice feature of this algorith is that is can be extended to a large class of convex loss functions. For instance, Zien and Ong (27) have proposed a ulticlass MKL algorith based on siilar ideas. In this paper, we present another forulation of the ultiple learning proble. We first depart fro the prial forulation proposed by Bach et al. (24a) and further used by Bach et al. (24b) and Sonnenburg et al. (26). Indeed, we replace the ixed-nor regularization by a weighted l 2 -nor regularization, where the sparsity of the linear cobination of kernels is controlled by a l -nor constraint on the kernel weights. This new forulation of MKL leads to a sooth and convex optiization proble. By using a variational forulation of the ixed-nor regularization, we show that our forulation is equivalent to the ones of Lanckriet et al. (24b), Bach et al. (24a) and Sonnenburg et al. (26). The ain contribution of this paper is to propose an efficient algorith, naed SipleMKL, for solving the MKL proble, through a prial forulation involving a weighted l 2 -nor regularization. Indeed, our algorith is siple, essentially based on a gradient descent on the SVM objective value. We iteratively deterine the cobination of kernels by a gradient descent wrapping a standard SVM solver, which is SipleSVM in our case. Our schee is siilar to the one of Sonnenburg et al. (26), and both algoriths iniize the sae objective function. However, they differ in that we use reduced gradient descent in the prial, whereas Sonnenburg et al. s SILP relies on cutting planes. We will epirically show that our optiization strategy is ore efficient, with new evidences confiring the preliinary results reported in Rakotoaonjy et al. (27). Then, extensions of SipleMKL to other supervised learning probles such as regression SVM, one-class SVM or ulticlass SVM probles based on pairwise coupling are proposed. Although it is not the ain purpose of the paper, we will also discuss the applicability of our approach to general convex loss functions. This paper also presents several illustrations of the usefulness of our algorith. For instance, in addition to the epirical efficiency coparison, we also show, in a SVM regression proble involving wavelet kernels, that autoatic learning of the kernels leads to far better perforances. Then we depict how our MKL algorith behaves on soe ulticlass probles. The paper is organized as follows. Section 2 presents the functional settings of our MKL proble and its forulation. Details on the algorith and soe analysis like discussion about convergence and coputational coplexity are given in Section 3. Extensions of our algorith to other SVM probles are discussed in Section 4 while experiental results dealing with coputational coplexity or with coparison with other odel selection ethods are presented in Section 5. A SipleMKL toolbox based on Matlab code is available at fr/enseignants/~arakoto/code/klindex.htl. This toolbox is an extension of our SipleSVM toolbox (Canu et al., 23). 2. Multiple Kernel Learning Fraework In this section, we present our MKL forulation and derive its dual. In the sequel, i and j are indices on exaples, wheras is the kernel index. In order to lighten notations, 3

4 we oit to specify that suations on i and j go fro to l, and that suations on go fro to M. 2. Functional fraework Before entering into the details of the MKL optiization proble, we first present the functional fraework adopted for ultiple kernel learning. Assue K, =,...,M are M positive definite kernels on the sae input space X, each of the being associated with an RKHS H endowed with an inner product,. For any, let d be a non-negative coefficient and H be the Hilbert space derived fro H as follows: endowed with the inner product H = {f f H : f H d < }, f,g H = d f,g. In this paper, we use the convention that x = if x = and otherwise. This eans that, if d = then a function f belongs to the Hilbert space H only if f = H. In such a case, H is restricted to the null eleent of H. Within this fraework, H is a RKHS with kernel K(x,x ) = d K (x,x ) since f H H, f(x) = f( ),K (x, ) = d f( ),d K (x, ) = f( ),d K (x, ) H. Now, if we define H as the direct su of the spaces H, i.e., H = M H, = then, a classical result on RKHS (Aronszajn, 95) says that H is a RKHS of kernel K(x,x ) = M d K (x,x ). = Owing to this siple construction, we have built a RKHS H for which any function is a su of functions belonging to H. In our fraework, MKL ais at deterining the set of coefficients {d } within the learning process of the decision function. The ultiple kernel learning proble can thus be envisioned as learning a predictor belonging to an adaptive hypothesis space endowed with an adaptive inner product. The forthcoing sections explain how we solve this proble. 4

5 2.2 Multiple kernel learning prial proble In the SVM ethodology, the decision function is of the for given in equation (), where the optial paraeters α i and b are obtained by solving the dual of the following optiization proble: in f,b,ξ 2 f 2 H + C i ξ i s.t. y i (f(x i ) + b) ξ i i ξ i i. In the MKL fraework, one looks for a decision function of the for f(x) + b = f (x) + b, where each function f belongs to a different RKHS H associated with a kernel K. According to the above functional fraework and inspired by the ultiple soothing splines fraework of Wahba (99, chap. ), we propose to address the MKL SVM proble by solving the following convex (see proof in appendix) proble, which we will be referred to as the prial MKL proble: in {f },b,ξ,d s.t. f 2 H 2 d + C i y i f (x i ) + y i b ξ i ξ i ξ i i d =, d, where each d controls the squared nor of f in the objective function. The saller d is, the soother f (as easured by f H ) should be. When d =, f H has also to be equal to zero to yield a finite objective value. The l -nor constraint on the vector d is a sparsity constraint that will force soe d to be zero, thus encouraging sparse basis kernel expansions. 2.3 Connections with ixed-nor regularization forulation of MKL The MKL forulation introduced by Bach et al. (24a) and further developed by Sonnenburg et al. (26) consists in solving an optiization proble expressed in a functional for as in {f},b,ξ s.t. ( ) 2 f H + C ξ i 2 i y i f (x i ) + y i b ξ i i ξ i i. Note that the objective function of this proble is not sooth since f H is not differentiable at f =. However, what akes this forulation interesting is that the ixed-nor penalization of f = f is a soft-thresholding penalizer that leads to a sparse solution, for which the algorith perfors kernel selection (Bach, 27). We have stated in the previous section that our proble should also lead to sparse solutions. In the following, we show that the forulations (2) and (3) are equivalent. i (2) (3) 5

6 For this purpose, we siply show that the variational forulation of the ixed-nor regularization is equal to the weighted 2-nor regularization, (which is a particular case of a ore general equivalence proposed by Micchelli and Pontil (25)) i.e., by Cauchy-Schwartz inequality, for any vector d in the siplex: ( ) 2 f H = ( ( f H d /2 d /2 f 2 H d ) 2 ) ( ) ( d = f 2 H d which leads to ( ) f 2 2 in d P H = f H. (4), d= d Moreover, equality in the Cauchy-Schwartz inequality, leads to the following optial vector d: d = f H / f q Hq. (5) q ) Hence, owing to this variational forulation, the non-sooth ixed-nor objective function of proble (3) has been turned into a sooth objective function in proble (2). Although the nuber of variables has increased, we will see that this proble can be solved ore efficiently. 2.4 The MKL dual proble The dual proble is a key point for deriving MKL algoriths and for studying their convergence properties. Since our prial proble (2) is equivalent to the one of Bach et al. (24a), they lead to the sae dual. However, our prial forulation being convex and differentiable, it provides a siple derivation of the dual, that does not use conic duality. The Lagrangian of proble (2) is L = f 2 H 2 d + C ξ i + ( ) α i ξ i y i f (x i ) y i b ν i ξ i i i i ( ) +λ d η d, (6) where α i and ν i are the Lagrange ultipliers of the constraints related to the usual SVM proble, whereas λ and η are associated to the constraints on d. When setting to zero the gradient of the Lagrangian with respect to the prial variables, we get the following (a) (b) f ( ) = d i α i y i = i (c) C α i ν i =, i α i y i K (,x i ), (7) (d) 2 f 2 H d 2 + λ η =,. 6

7 We note again here that f ( ) has to go to if the coefficient d vanishes. Plugging these optiality conditions in the Lagrangian gives the dual proble ax α i λ α i,λ i s.t. α i y i = i (8) α i C i α i α j y i y j K (x i,x j ) λ,. 2 i,j This dual proble is difficult to optiize due to the last constraint. This constraint ay be oved to the objective function, but then, the latter becoes non-differentiable causing new difficulties (Bach et al., 24a). Hence, in the forthcoing section, we propose an approach based on the iniization of the prial. In this fraework, we benefit of the proble differentiability that allows an efficient derivation of approxiate prial solutions, whose accuracy will be onitored by the duality gap. 3. Algorith for solving the MKL prial proble One possible approach for solving proble (2) is to use the alternate optiization algorith applied by Grandvalet and Canu (999, 23) another contexts. In the first step, proble (2) is optiized with respect to f, b and ξ, considering that d is fixed. Then, in the second step, the weight vector d is updated to decrease the objective function of proble (2), with f, b and ξ being fixed. In Section 2.3, we showed that the second step can be carried out in closed for. However, this approach lacks convergence guarantees and ay lead to nuerical probles, in particular when soe eleents of d approach zero (Grandvalet, 998). Note that these nuerical probles can be handle by introducing a perturbed version of the alternate algorith as shown by Argyriou et al. (28). Instead of using an alternate optiization algorith, we prefer to consider here the following constrained optiization proble: where in d J(d) such that in {f},b,ξ J(d) = s.t. M d =, d (9) = f 2 H 2 d + C i y i f (x i ) + y i b ξ i ξ i i. We show below how to solve proble (9) on the siplex by a siple gradient ethod. We will first note that the objective function J(d) is actually an optial SVM objective value. We will then discuss the existence and coputation of the gradient of J( ), which is at the core of the proposed approach. ξ i i (). Note that Bach et al. (24a) forulation differs slightly, in that the kernels are weighted by soe pre-defined coefficients that were not considered here. 7

8 3. Coputing the optial SVM value and its derivatives The Lagrangian of proble () is identical to the first line of equation (6). By setting to zero the derivatives of this Lagrangian according to the prial variables, we get conditions (7) (a) to (c), fro which we derive the associated dual proble ax α i α j y i y j d K (x i,x j ) + α 2 i,j i with α i y i = i C α i i, α i () which is identified as the standard SVM dual forulation using the cobined kernel K(x i,x j ) = d K (x i,x j ). Function J(d) is defined as the optial objective value of proble (). Because of strong duality, J(d) is also the objective value of the dual proble: J(d) = α i 2 α j y iy j d K (x i,x j ) + α i, (2) i i,j where α axiizes (). Note that the objective value J(d) can be obtained by any SVM algorith. Our ethod can thus take advantage of any progress in single kernel algoriths. In particular, if the SVM algorith we use is able to handle large-scale probles, so will our MKL algorith. Thus, the overall coplexity of SipleMKL is tied to the one of the single kernel SVM algorith. Fro now on, we assue that each Gra atrix (K (x i,x j )) i,j is positive definite, with all eigenvalues greater than soe η > (to enforce this property, a sall ridge ay be added to the diagonal of the Gra atrices). This iplies that, for any adissible value of d, the dual proble is strictly concave with convexity paraeter η (Learéchal and Sagastizabal, 997). In turn, this strict concavity property ensures that α is unique, a characteristic that eases the analysis of the differentiability of J( ). The existence and coputation of the derivatives of optial value functions such as J( ) have been largely discussed by Bonnans and Shapiro (998). For our purpose, the appropriate reference, which is Theore 4. in Bonnans and Shapiro (998), which has already been applied by Chapelle et al. (22) for the special case of squared-hinge loss SVM. This theore is reproduced here in the appendix for self-containedness. In a nutshell, this theore says that differentiability of J(d) is ensured by the unicity of α, and by the differentiability of the objective function that gives J(d). Furtherore, the derivatives of J(d) can be coputed as if α were not to depend on d. Thus, by siple differentiation of the dual function () with respect to d, we have: J d = 2 α i α jy i y j K (x i,x j ). (3) i,j We will see in the sequel that the applicability of this theore can be extended to other SVM probles. 8

9 3.2 Reduced gradient algorith The optiization proble we have to deal with in (9) is a non-linear objective function with constraints over the siplex. With our positivity assuption on the kernel atrices, J( ) is convex and differentiable with Lipschitz gradient (Learéchal and Sagastizabal, 997). The approach we use for solving this proble is a reduced gradient ethod, which converges for such functions (Luenberger, 984). Once the gradient of J(d) is coputed, d is updated by using a descent direction ensuring that the equality constraint and the non-negativity constraints on d are satisfied. We handle the equality constraint by coputing the reduced gradient (Luenberger, 984, Chap. ). Let d µ be a non-zero entry of d, the reduced gradient of J(d), denoted red J, has coponents: [ red J] = J J µ, and [ red J] µ = ( J J ). d d µ d d µ µ We chose µ to be the index of the largest coponent of vector d, for better nuerical stability (Bonnans, 26). The positivity constraints have also to be taken into account in the descent direction. Since we want to iniize J( ), red J is a descent direction. However, if there is an index such that d = and [ red J] >, using this direction would violate the positivity constraint for d. Hence, the descent direction for that coponent is set to. Eventually, the descent direction for updating d is: D = if d = and J d J d µ > J d + J d µ if d > and µ ( J J ) for = µ. d ν d µ g µ,d ν> and the updating schee is the following: d d + γd. Our updating schee, detailed in Algorith, goes one step beyond: once a descent direction D has been coputed, we first look for the axial adissible step size in that direction and check whether the objective value decreases or not. The axial adissible step size corresponds to a coponent, say d ν, set to zero. If the objective value decreases, d is updated, we set D ν = and noralize D to coply with the equality constraint. This procedure is repeated until the objective value stops decreasing. At this point, we look for the optial step size γ, which is deterined by using a one-diensional line search, with proper stopping criterion, such as Arijo s rule, to ensure global convergence. In this algorith, the gradient of the cost function is not coputed after each update of the weight vector d. Instead, we take advantage of an easily updated descent direction as long as decrease in the objective value is possible. We will see in the nuerical experients that this approach saves a substantial aount of coputation tie. Note that we have also investigated a plain gradient projection algorith (Bertsekas, 999, Chap 2.3) for solving 9

10 Algorith SipleMKL algorith set d = M for =,...,M while stopping criterion not et do solve the classical SVM proble with K = d K copute J d for =,...,M and descent direction D set µ = argax d, ν = argin d /D and γ ax = d ν /D ν { D <} while J(d + γ ax D) < J(d) do d d + γ ax D D µ D µ D ν, D ν, copute ν = argin d /D and γ ax = d ν /D ν { D <} end while Search for the optial step γ [,γ ax ] d d + γd end while proble (9). The resulting update schee was epirically observed to to be slightly less efficient than the proposed approach, and we will not report its results. Also, note that the projection and the line-search techniques involve several querying of the objective function and thus it requires the coputation of several single kernel SVM with sall variations of d. This ay be very costly but it can be speeded up by initializing the SVM algorith with previous values of α (DeCoste and Wagstaff., 2). The above described steps of the algorith are perfored until a stopping criterion is et. This stopping criterion can be either based on the duality gap, the KKT conditions, the variation of d between two consecutive steps or, even ore siply, on a axial nuber of iterations. Our ipleentation, based on the duality gap, is detailed in the forthcoing section. 3.3 Optiality conditions In a convex constrained optiization algorith like the one we are considering, we have the opportunity to check for proper optiality conditions such as the KKT conditions or the duality gap. Since our forulation of the MKL proble is convex, convergence can be onitored through the duality gap (the difference between prial and dual objective values), which should be zero at the optiu. Fro the prial and dual objectives provided respectively in (2) and (8), the MKL duality gap is DualGap = J(d ) i α i + 2 ax α i α j y iy j K (x i,x j ), i,j where d and {α i } are optial prial and dual variables, and J(d ) depends iplicitly on optial prial variables {f }, b and {ξ i }. If J(d ) has been obtained through the dual proble (), then this duality gap can also be related to the one of the single kernel SVM

11 algorith DG SVM. Indeed, substituting J(d) with its expression in (2) yields DualGap = DG SVM α i α 2 jy i y j d K (x i,x j ) + 2 ax α i α jy i y j K (x i,x j ). i,j i,j Hence, the MKL duality gap can be obtained with a sall additional coputational cost copared to the SVM duality gap. In iterative procedures, it is coon to stop the algorith when the optiality conditions are respected up to a tolerance threshold ε. Obviously, SipleMKL has no ipact on DG SVM, hence, one ay assue, as we did here, that DG SVM needs not to be onitored. Consequently, we terinate the algorith when ax α iα jy i y j K (x i,x j ) α i α jy i y j d K (x i,x j ) ε. (4) i,j i,j For soe of the other MKL algoriths that will be presented in in Section 4, the dual function ay be ore difficult to derive. Hence, it ay be easier to rely on approxiate KKT conditions as a stopping criterion. For the general MKL proble (9), the first order optiality conditions are obtained through the KKT conditions: J d + λ η = η d =, where λ and {η } are the Lagrange ultipliers of respectively the equality and inequality constraints of (9). These KKT conditions iply J d = λ if d > J d λ if d =. However, as Algorith is not based on the Lagrangian forulation of proble (9), λ is not coputed. Hence, we derive approxiate necessary optiality conditions to be used for terination criterion. Let s define dj in and dj ax as dj in = in {d d >} J d and dj ax = ax {d d >} J d, then, the necessary optiality conditions are approxiated by the following terination conditions: dj in dj ax ε and J d dj ax if d = In other words, we consider to be at the optiu when the gradient coponents for all positive d lie in a ε-tube and when all gradient coponents for vanishing d are above this tube. Note that these approxiate necessary optiality conditions are available right away for any differentiable objective function J(d).

12 Figure : Illustrating three iterations of the SILP algorith and a gradient descent algorith for a one-diensional proble. This diensionality is not representative of the MKL fraework, but our ai is to illustrate the typical oscillations of cutting planes around the optial solution (with iterates d to d 3 ). Note that coputing an affine lower bound at a given d requires a gradient coputation. Provided the step size is chosen correctly, gradient descent converges directly towards the optial solution without overshooting (fro d to d ). 3.4 Cutting Planes, Steepest Descent and Coputational Coplexity As we stated in the introduction, several algoriths have been proposed for solving the original MKL proble defined by Lanckriet et al. (24b). All these algoriths are based on equivalent forulations of the sae dual proble; they all ai at providing a pair of optial vectors (d, α). In this subsection, we contrast SipleMKL with its closest relative, the SILP algorith of Sonnenburg et al. (25, 26). Indeed, fro an ipleentation point of view, the two algoriths are alike, since they are wrapping a standard single kernel SVM algorith. This feature akes both algoriths very easy to ipleent. They however differ in coputational efficiency, because the kernel weights d are optiized in quite different ways, as detailed below. Let us first recall that our differentiable function J(d) is defined as: ax α i α j y i y j d K (x i,x j ) + α i α 2 J(d) = i,j i with α i y i =, C α i i, i and both algoriths ai at iniizing this differentiable function. However, using a SILP approach in this case, does not take advantage of the soothness of the objective function. The SILP algorith of Sonnenburg et al. (26) is a cutting plane ethod to iniize J with respect to d. For each value of d, the best α is found and leads to an affine lower bound on J(d). The nuber of lower bounding affine functions increases as ore (d, α) pairs are coputed, and the next candidate vector d is the iniizer of the current lower bound on 2

13 J(d), that is, the axiu over all the affine functions. Cutting planes ethod do converge but they are known for their instability, notably when the nuber of lower-bounding affine functions is sall: the approxiation of the objective function is then loose and the iterates ay oscillate (Bonnans et al., 23). Our steepest descent approach, with the proposed line search, does not suffer fro instability since we have a differentiable function to iniize. Figure illustrates the behaviour of both algoriths in a siple case, with oscilations for cutting planes and direct convergence for gradient descent. Section 5 evaluates how these oscillations ipact on the coputational tie of the SILP algorith on several genuine exaples. These experients show that our algorith needs less costly gradient coputations. Conversely, the line search in the gradient base approach requires ore SVM retrainings in the process of querying the objective function. However, the coputation tie per SVM training is considerably reduced, since the gradient based approach produces estiates of d on a sooth trajectory, so that the previous SVM solution provides a good guess for the current SVM training. In SILP, with the oscillating subsequent approxiations of d, the benefit of war-start training severely decreases. 3.5 Convergence Analysis In this paragraph, we briefly discuss the convergence of the algorith we propose. We first suppose that proble () is always exactly solved, which eans that the duality gap of such proble is. With such conditions, the gradient coputation in (3) is exact and thus our algorith perfors reduced gradient descent on a continuously differentiable function J( ) (reeber that we have assued that the kernel atrices are positive definite) defined on the siplex {d d =,d }, which does converge to the global iniu of J (Luenberger, 984). However, in practice, proble () is not solved exactly since ost SVM algoriths will stop when the duality gap is saller than a given ε. In this case, the convergence of our projected gradient ethod is no ore guaranteed by standard arguents. Indeed, the output of the approxiately solved SVM leads only to an ε-subgradient (Bonnans et al., 23, Bach et al., 24a). This situation is ore difficult to analyze and we plan to address it thoroughly in future work (see for instance D Aspreont (26) for an exaple of such analysis in a siilar context). 4. Extensions In this section, we discuss how the algorith we propose can be siply extended to other SVM algoriths like SVM regression, one-class SVM or pairwise ulticlass SVM algoriths. More generally, we will discuss other loss functions that can be used within our MKL algoriths. 4. Extensions to other SVM Algoriths The algorith we described in the previous section focuses on binary classification SVMs, but it is worth noting that our MKL algorith can be extended to other SVM algoriths with only little changes. For SVM regression with the ε-insensitive loss, or clus- 3

14 tering with the one-class soft argin loss, the proble only changes in the definition of the objective function J(d) in (). For SVM regression (Vapnik et al., 997, Schölkopf and Sola, 2), we have J(d) = in f,b,ξ i 2 s.t. y i d f 2 H + C i f (x i ) b ε + ξ i f (x i ) + b y i ε + ξi ξ i,ξ i i, (ξ i + ξ i ) i i (5) and for one-class SVMs (Schölkopf and Sola, 2), we have: J(d) = in f 2 H f,b,ξ i 2 d + ξ i b νl i s.t. f (x i ) b ξ i ξ i. Again, J(d) can be defined according to the dual functions of these two optiization probles, which are respectively ax (β i α i )y i ε (β i + α i ) (β i α i )(β j α j ) d K (x i,x j ) α,β 2 i i i,j J(d) = with (β i α i ) = i α i, β i C, i, (7) and ax α i α j d K (x i,x j ) α 2 i,j J(d) = with α i i (8) νl α i =, i where {α i } and {β i } are Lagrange ultipliers. Then, as long as J(d) is differentiable, a property strictly related to the strict concavity of its dual function, our descent algorith can still be applied. The ain effort for the extension of our algorith is the evaluation of J(d) and the coputation of its derivatives. Siilarly to the binary classification SVM, J(d) can be coputed by eans of efficient off-the-shelf SVM solvers and the gradient of J(d) is easily obtained through the dual probles. For SVM regression, we have: J d = 2 (6) (βi α i )(βj α j)k (x i,x j ), (9) i,j 4

15 and for one-class SVM, we have: J d = 2 α i α j K (x i,x j ), (2) i,j where α i and β i are the optial values of the Lagrange ultipliers. These exaples illustrate that extending SipleMKL to other SVM probles is rather straighforward. This observation is valid for other SVM algoriths (based for instance on the ν paraeter, a squared hinge loss or squared-ε tube) that we do not detail here. Again, our algorith can be used provided J(d) is differentiable, by plugging in the algorith the function that evaluates the objective value J(d) and its gradient. Of course, the duality gap ay be considered as a stopping criterion if it can be coputed. 4.2 Multiclass Multiple Kernel Learning With SVMs, ulticlass probles are custoarily solved by cobining several binary classifiers. The well-known one-against-all and one-against-one approaches are the two ost coon ways for building a ulticlass decision function based on pairwise decision functions. Multiclass SVM ay also be defined right away as the solution of a global optiization proble (Weston and Watkins, 999, Craer and Singer, 2), that ay also be addressed with structured-output SVM (Tsochantaridis et al., 25). Very recently, an MKL algorith based on structured-output SVM has been proposed by Zien and Ong (27). This work extends the work of Sonnenburg et al. (26) to ulticlass probles, with an MKL ipleentation still based on a QCQP or SILP approach. Several works have discussed the copared perforances of ulticlass SVM algoriths (Duan and Keerthi, 25, Hsu and Lin, 22, Rifkin and Klautau, 24). In this subsection, we do not deal with this aspect; we explain how SipleMKL can be extended to pairwise SVM ulticlass ipleentations. The proble of applying our algorith to structuredoutput SVM will be briefly discussed later. Suppose we have a ulticlass proble with P classes. For a one-against-all ulticlass SVM, we need to train P binary SVM classifiers, where the p-th classifier is trained by considering all exaples of class p as positive exaples while all other exaples are considered negative. For a one-against-one ulticlass proble, P(P )/2 binary SVM classifiers are built fro all pairs of distinct classes. Our ulticlass MKL extension of SipleMKL differs fro the binary version only in the definition of a new cost function J(d). As we now look for the cobination of kernels that jointly optiizes all the pairwise decision functions, the objective function we want to optiize according to the kernel weights {d } is: J(d) = p P J p (d), where P is the set of all pairs to be considered, and J p (d) is the binary SVM objective value for the classification proble pertaining to pair p. Once the new objective function is defined, the lines of Algorith still apply. The gradient of J(d) is still very siple to obtain, since owing to linearity, we have: J d = 2 α i,p α j,p y iy j K (x i,x j ), (2) p P i,j 5

16 where α j,p is the Lagrange ultiplier of the j-th exaple involved in the p-th decision function. Note that those Lagrange ultipliers can be obtained independently for each pair. The abovedescribed approach ais at finding the cobination of kernels that jointly optiizes all binary classification probles: this one set of features should axiize the su of argins. Another possible and straightforward approach consists in running independently SipleMKL for each classification task. However, this choice is likely to result in as any cobinations of kernels as there are binary classifiers. 4.3 Other loss functions Multiple kernel learning has been of great interest and since the seinal work of Lanckriet et al. (24b), several works on this topic have flourished. For instance, ultiple kernel learning has been transposed to least-square fitting and logistic regression (Bach et al., 24b). Independently, several authors have applied ixed-nor regularization, such as the additive spline regression odel of Grandvalet and Canu (999). This type of regularization, which is now known as the group lasso, ay be seen as a linear version of ultiple kernel learning (Bach, 27). Several algoriths have been proposed for solving the group lasso proble. Soe of the are based on projected gradient or on coordinate descent algorith. However, they all consider the non-sooth version of the proble. We previously entioned that Zien and Ong (27) have proposed an MKL algorith based on structured-output SVMs. For such proble, the loss function, which differs fro the usual SVM hinge loss, leads to an algorith based on cutting planes instead of the usual QP approach. Provided the gradient of the objective value can be obtained, our algorith can be applied to group lasso and structured-output SVMs. The key point is whether the theore of Bonnans et al. (23) can be applied or not. Although we have not deeply investigated this point, we think that any probles coply with this requireent, but we leave these developents for future work. 4.4 Approxiate regularization path SipleMKL requires the setting of the usual SVM hyperparaeter C, which usually needs to be tuned for the proble at hand. For doing so, a practical and useful technique is to copute the so-called regularization path, which describes the set of solutions as C varies fro to. Exact path following techniques have been derived for soe specific probles like SVMs or the lasso (Hastie et al., 24, Efron et al., 24). Besides, regularization paths can be sapled by predictor-corrector ethods (Rosset, 24, Bach et al., 24b). For odel selection purposes, an approxiation of the regularization path ay be sufficient. This approach has been applied for instance by Koh et al. (27) in regularized logistic regression. Here, we copute an approxiate regularization path based on a war-start technique. Suppose, that for a given value of C, we have coputed the optial (d,α ) pair; the idea of a war-start is to use this solution for initializing another MKL proble with a different value of C. In our case, we iteratively copute the solutions for decreasing values of C 6

17 (note that α has to be odified to be a feasible initialization of the ore constrained SVM proble). The war-start technique can be used as is, but soe siple tricks can be used for further iproving the initialization. For instance, suppose we have coputed the pair (d,α ) that optiizes an MKL proble for a given C. For a different hyperparaeter value, say C, the first step our algorith (see Algorith ) consists in coputing a SVM solution with paraeter C and with a single kernel obtained as the weighted according to d su of kernels. Hence this step can be solved by coputing the regularization path of an SVM with fixed kernel. Then, once the new optial value of α (corresponding to the hyperparaeter C ) has been coputed, the kernel weights can be updated accordingly. Indeed, according to equations (5) and (7), we have:, d = f H with f ( ) = d α i f y ik (x i, ), H i which corresponds to the optial weight values if the new α are also optial. This is probably not the case, but it leads to another possible and probably better initialization values for our war-start techniques. 5. Nuerical experients In this experiental section, we essentially ai at illustrating three points. The first point is to show that our gradient descent algorith is efficient. This is achieved by binary classification experients, where SipleMKL is copared to the SILP approach of Sonnenburg et al. (26). Then, we illustrate the usefulness of a ultiple kernel learning approach in the context of regression. The exaples we use are based on wavelet-based regression in which the ultiple kernel learning fraework naturally fits. The final experient ais at evaluating the ultiple kernel approach in a odel selection proble for soe ulticlass probles. 5. Coputation tie The ai of this first set of experients is to assess the running ties of SipleMKL. 2 First, we copare with SILP regarding the tie required for coputing a single solution of MKL with a given C hyperparaeter. Then, we copute an approxiate regularization path by varying C values. We finally provide hints on the expected coplexity of SipleMKL, by easuring the growth of running tie as the nuber of exaples or kernels increases. 5.. Tie needed for reaching a single solution In this first benchark, we put SipleMKL and SILP side by side, for a fixed value of the hyperparaeter C (C = ). This procedure, which does not take into account a proper odel selection procedure, is not representative of the typical use of SVMs. It is however relevant for the purpose of coparing algorithic issues. 2. All the experients have been run on a Pentiu D-3 GHz with 3 GB of RAM. 7

18 The evaluation is ade on five datasets fro the UCI repository: Liver, Wpbc, Ionosphere, Pia, Sonar (Blake and Merz, 998). The candidate kernels are: Gaussian kernels with different bandwidths σ, on all variables and on each single variable; polynoial kernels of degree to 3, again on all and each single variable. All kernel atrices have been noralized to unit trace, and are precoputed prior to running the algoriths. Both SipleMKL and SILP wrap an SVM dual solver based on SipleSVM, an active constraints written in Matlab (Canu et al., 23). The descent procedure of SipleMKL is also ipleented in Matlab, whereas the linear prograing involved in SILP is ipleented in the publicly available toolbox LPSOLVE (Berkelaar et al., 24). For a fair coparison, we use the sae stopping criterion for both algoriths. They halt when, either the duality gap is lower than., or the nuber of iterations exceeds 2. Quantitatively, the displayed results differ fro the preliinary version of this work, where the stopping criterion was based on the stabilization of the weights, but they are qualitatively siilar (Rakotoaonjy et al., 27). For each dataset, the algoriths were run 2 ties with different train and test sets (7% of the exaples for training and 3% for testing). Training exaples were noralized to zero ean and unit variance. In Table, we report different perforance easures: accuracy, nuber of selected kernels and running tie. As the latter is ainly spent in querying the SVM solver and in coputing the gradient of J with respect to d, the nuber of calls to these two routines is also reported. Both algoriths are nearly identical in perforance accuracy. Their nuber of selected kernels are of sae agnitude, although SipleMKL tends to select to 2% ore kernels. As both algoriths address the sae convex optiization proble, with convergent ethods starting fro the sae initialization, the observed differences are only due to the inaccuracy of the solution when the stopping criterion is et. Hence, the trajectories chosen by each algorith for reaching the solution, detailed in Section 3.4, explain the differences in the nuber of selected kernels. The updates of d based on the descent algorith of SipleMKL are rather conservative (sall steps departing fro /M for all d ), whereas the oscilations of cutting planes are likely to favor extree solutions, hitting the edges of the siplex. This explanation is corroborated by Figure 2, which copares the behavior of the d coefficients trough tie. The instability of SILP is clearly visible, with very high oscillations in the firsts iterations and a noticeable residual noise in the long run. In coparison, the trajectories for SipleSVM are uch soother. If we now look at the overall difference in coputation tie reported in Table, clearly, on all data sets, SipleSVM is faster than SILP, with an average gain factor of about 5. Furtherore, the larger the nuber of kernels is, the larger the speed gain we achieve. Looking at the last colun of Table, we see that the ain reason for iproveent is that SipleMKL converges in fewer iterations (that is, gradient coputations). It ay see surprising that this gain is not conterbalanced by the fact that SipleMKL requires any 8

19 Table : Average perforance easures for the two MKL algoriths. Liver l = 24 M = 9 Algorith # Kernel Accuracy Tie (s) # SVM eval # Gradient eval SILP.6 ± ± ± ± ± 2 SipleMKL.2 ± ± ± ± ± 26 Pia l = 538 M = 7 Algorith # Kernel Accuracy Tie (s) # SVM eval # Gradient eval SILP.6 ± ± ± ± ± 3 SipleMKL 4.7 ± ± ± 3 34 ± ± 4.8 Ionosphere l = 246 M = 442 Algorith # Kernel Accuracy Tie (s) # SVM eval # Gradient eval SILP 2.6 ± ± ± 5 43 ± ± 53 SipleMKL 23.6 ± ± ± 46 7 ± ± 25 Wpbc l = 36 M = 442 Algorith # Kernel Accuracy Tie (s) # SVM eval # Gradient eval SILP 3.7 ± ± ± ± ± 44 SipleMKL 5.8 ± ± ± ± ± Sonar l = 46 M = 793 Algorith # Kernel Accuracy Tie (s) # SVM eval # Gradient eval SILP 33.5 ± ± ± ± ± 87 SipleMKL 36.7 ± ± ± ± 56 5 ± 66 d k SipleMKL d k SipleMKL d k SILP Iterations Pia d k SILP Iterations Ionosphere Figure 2: Evolution of the five largest weights d for SipleMKL and SILP; left row: Pia; right row: Ionosphere. 9

20 3.5 x 4 SipleMKL SILP 5 SipleMKL SILP Objective value Objective value Iterations Pia Iterations Ionosphere Figure 3: Evolution of the objective values for SipleSVM and SILP; left row: Pia; right row: Ionosphere. ore calls to the SVM solver (on average, about 4 ties). As we stated in Section 3.4, when the nuber of kernels is large, coputing the gradient ay be expensive copared to SVM retraining with war-start techniques. To understand why, with this large nuber of calls to the SVM solver, SipleMKL is still uch faster than SILP, we have to look back at Figure 2. On the one hand, the large variations in subsequents d values for SILP, entail that subsequent SVM probles are not likely to have siilar solutions: a war-start call to the SVM solver does not help uch. On the other hand, with the sooth trajectories of d in SipleMKL, the previous SVM solution is often a good guess for the current proble: a war-start call to the SVM solver results in uch less coputation than a call fro scratch. To end this first series of experients, Figure 3 depicts the evolution of the objective function for the data sets that were used in Figure 2. Besides the fact that SILP needs ore iterations for achieving a good approxiation of the final solution, it is worth noting that the objective values rapidly reach their steady state while still being far fro convergence, when d values are far fro being settled. Thus, onitoring objective values is not suitable to assess convergence Tie needed for getting an approxiate regularization path In practice, the optial value of C is unknown, and one has to solve several SVM probles, spanning a wide range of C values, before choosing a solution according to soe odel selection criterion like the cross-validation error. Here, we further pursue the coparison of the running ties of SipleMKL and SILP, in a series of experients that include the search for a sensible value of C. In this new benchark, we use the sae data sets as in the previous experients, with the sae kernel settings. The task is only changed in the respect that we now evaluate the running ties needed by both algoriths to copute an approxiate regularization path. 2

21 nuber of selected kernels C nuber of selected kernels C d k C Pia d k C Wpbc Figure 4: Regularization paths for d and the nuber of selected kernels versus C; left row: Pia; right row: Wpbc. For both algoriths, we use a siple war-start technique, which consists in using the optial solutions {d } and {α i } obtained for a given C to initialize a new MKL proble with C + C (DeCoste and Wagstaff., 2). As described in Section 4.4, we start fro the largest C and then approxiate the regularization path by decreasing its value. The set of C values is obtained by evenly sapling the interval [.,] on a logarithic scale. Figure 4 shows the variations of the nuber of selected kernels and the values of d along the regularization path for the Pia and Wpbc datasets. The nuber of kernels is not a onotone function of C: for sall values of C, the nuber of kernels is soewhat constant, then, it rises rapidly. There is a sall overshooth before reaching a plateau corresponding to very high values of C. This trend is siilar for the nuber of leading leading ters in the kernel weight vector d. Both phenoenon were observed consistently over the datasets we used. Table 2 displays the average coputation tie (over runs) required for building the approxiate regularization path. As previously, SipleMKL is ore efficient than SILP, with a gain factor increasing with the nuber of kernels in the cobination. The range 2

22 Table 2: Average coputation tie (in seconds) for getting an approxiate regularization path. For the Sonar data set, SILP was extreely slow, so that regularization path was coputed only once. Dataset SipleMKL SILP Ratio Liver 48 ± ± Pia 3 ± ± Ionosphere 29 ± ± Wpbc 88 ± 6 24 ± Sonar 625 ± (*) 243 of gain factors, fro 5.9 to 23, is even ore ipressive than in the previous benchark. SipleMKL benefits fro the continuity of solutions along the regularization path, whereas SILP does not take advantage of war starts. Even provided with a good initialization, it needs any cutting planes to stabilize More on SipleMKL running ties Here, we provide an epirical assessent of the expected coplexity of SipleMKL on different data sets fro the UCI repository. We first look at the situation where kernel atrices can be pre-coputed and stored in eory, before reporting experients where the eory are too high, leading to repeated kernel evaluations. In a first set of experients, we use Gaussian kernels, coputed on rando subsets of variables and with rando width. These kernels are precoputed and stored in eory, and we report the average CPU running ties obtained fro 2 runs differing in the rando draw of training exaples. The stopping criterion is the sae as in the previous section: a relative duality gap less than ε =.. The first two rows of Figure 5 depicts the growth of coputation tie as the nuber of kernel increases. We observe a nearly linear trend for the four learning probles. This growth rate could be expected considering the linear convergence property of gradient techniques, but the absence of overhead is valuable. The last row of Figure 5 depicts the growth of coputation tie as the nuber of exaples increases. Here, the nuber of kernels is set to. In these plots, the observed trend is clearly superlinear. Again, this trend could be expected, considering that SVM expected training ties are superlinear in the nuber of training exaples. As we already entioned, the coplexity of SipleMKL is tightly linked to the one of SVM training (for soe exaples of single kernel SVM running tie, one can refer to the work of Loosli and Canu (27)). When all the kernels used for MKL cannot be stored in eory, one can resort to a decoposition ethod. Table 3 reports the average coputation ties, over runs, in this ore difficult situation. The large-scale SVM schee of Joachis (999) has been ipleented, with basis kernels recoputed whenever needed. This approach is coputationally expensive but goes with no eory liit. For these experients, the stopping criterion is 22