Margin and Radius Based Multiple Kernel Learning

Transcription

1 Margn and Radus Based Multple Kernel Learnng Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro Unversty of Geneva, Computer Scence Department 7, route de Drze, Battelle batment A, 1227 Carouge, Swtzerland {Huyen.Do, Alexandros.Kalouss, Adam.Woznca, Abstract. A serous drawbac of ernel methods, and Support Vector Machnes (SVM) n partcular, s the dffculty n choosng a sutable ernel functon for a gven dataset. One of the approaches proposed to address ths problem s Multple Kernel Learnng (MKL) n whch several ernels are combned adaptvely for a gven dataset. Many of the exstng MKL methods use the SVM objectve functon and try to fnd a lnear combnaton of basc ernels such that the separatng margn between the classes s maxmzed. However, these methods gnore the fact that the theoretcal error bound depends not only on the margn, but also on the radus of the smallest sphere that contans all the tranng nstances. We present a novel MKL algorthm that optmzes the error bound tang account of both the margn and the radus. The emprcal results show that the proposed method compares favorably wth other state-of-the-art MKL methods. Key words: Learnng Kernel Combnaton, Support Vector Machnes, convex optmzaton 1 Introducton Over the last few years ernel methods [1, 2], such as Support Vector Machnes (SVM), have proved to be effcent machne learnng tools. They wor n a feature space mplctly defned by a postve sem-defnte ernel functon, whch allows the computaton of nner products n feature spaces usng only the objects n the nput space. The man lmtaton of ernel methods stems from the fact that n general t s dffcult to select a ernel functon, and hence a feature mappng, that s sutable for a gven problem. To address ths problem several several attempts have been recently made to learn ernel operators drectly from the data [3 12]. The proposed methods dffer n the objectve functons (e.g. CV rs, margn based, algnment, etc.) as well as n the classes of ernels that they consder (e.g. combnaton of fnte or nfnte set of basc ernels). The most popular approach n the context of ernel learnng consders a fnte set of predefned basc ernels whch are combned so that the margnbased objectve functon of SVM s optmzed. The learned ernel K s a lnear

2 2 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro combnaton of basc ernels K,.e. K(x, x ) = µ K (x, x ), µ 0, where M s the number of basc ernels, and x and x are nput objects. The weghts µ of the ernels are ncluded n the margn-based objectve functon. Ths settng s commonly referred to as the Multple Kernel Learnng (MKL). The MKL formulaton has been ntroduced n [3] as a sem-defnte programmng problem, whch scaled well only for small problems. [7] extended that wor and proposed a faster method based on the conc dualty of MKL and solved the problem usng Sequental Mnmal Optmzaton (SMO). [5] reformulated the MKL problem as sem-nfnte lnear problem. In [6] the authors proposed an adjustment n the cost functon of [5] to mprove predctve performance. Although the MKL approach to ernel learnng has some lmtatons (e.g. one has to choose the basc ernels), t s wdely used because of ts smplcty, nterpretablty and good performance. The MKL methods that use the SVM objectve functon do not explot the fact that the error bound of SVM depends not only on the separatng margn, but also on the radus of the smallest sphere that encloses the data. In fact even the standard SVM algorthms do not explot the latter, because for a gven feature space the radus s fxed. However n the context of MKL the radus s not fxed but s a functon of the weghts of the basc ernels. In ths paper we propose a novel MKL method that taes account of both radus and margn to optmze the error bound. Followng a number of transformatons, these problems are cast n a form that can be solved by the two step optmzaton algorthm gven n [6]. The paper s organzed as follows. In Secton 2 we ntroduce the general MKL framewor. Next, n Secton 3 we dscuss the varous error bounds that motvate the use of the radus. The man contrbuton of the wor s presented n Secton 4 where we propose a new method for multple ernel learnng that ams to optmze the margn- and radus-dependent error bound. In Secton 5 we present the emprcal results on several benchmar datasets. Fnally, we conclude wth Secton 6 where we also present ponters to future wor. 2 Multple ernel learnng problem Consder a mappng of nstances x X, to a new feature space H x Φ (x) H (1) Ths mappng can be performed by a ernel functon K (x,x ) whch s defned as the nner product of the mages of two nstances x and x n H,.e. K (x,x ) = Φ (x),φ (x ) ; H may have even nfnte dmensonalty. Typcally, the computaton of the nner product n H s done mplctly,.e. wthout havng to compute explctly the mages Φ (x) and Φ (x ). 2.1 Orgnal problem formulaton of MKL Gven a set of tranng examples S = {(x 1, y 1 ),..., (x l, y l )} and a set of basc ernel functons, Z = {K (x,x ) := 1,...M}, the goal of MKL s to optmze

3 Margn and Radus Based Multple Kernel Learnng 3 a cost functon Q(f(Z, µ)(x,x ), S) where f(z, µ)(x,x ) s some postve semdefnte functon of the set of the bass ernels, parametrzed by µ; most often a lnear combnaton of the form: f(z, µ)(x,x ) = µ K (x,x ), µ 0, µ = 1 (2) To smplfy notaton we wll denote f(z, µ) by K µ. In the remanng part of ths wor we wll only focus on the normalzed versons of K, defned as: K (x,x ) := K (x,x ) K (x,x) K (x,x ). (3) If K µ s a lnear combnaton of ernels then ts feature space H µ s gven by the mappng: x Φ µ (x) = ( µ 1 Φ 1 (x),..., µ M Φ M (x)) T H µ (4) where Φ (x) s the mappng to the H feature space assocated wth the K ernel, as ths was gven n Formula 1. In prevous wor wthn the MKL context the cost functon, Q, has taen dfferent forms such as the Kernel Target Algnment, whch measures the goodness of a ernel for a gven learnng tas [9], or the typcal SVM cost functon combnng classfcaton error and the margn [3, 5,6], or as n [4] any of the above wth an added regularzaton term for the complexty of the combned ernel. 3 Margn and radus based error bounds There are a number of theorems n statstcal learnng that bound the expected classfcaton error of the thresholded lnear classfer, that corresponds to the maxmum margn hyperplane, by quanttes that are related to the margn and the radus of the smallest sphere that encloses the data. Below we gve two of them that are applcable on lnearly separable and non-separable tranng sets, respectvely. Theorem 1 [10], Gven a tranng set S = {(x 1, y 1 ),..., (x l, y l )} of sze l, a feature space H and a hyperplane (w, b), the margn γ(w, b, S) and the radus R(S) are defned by y ( w,φ(x ) + b) γ(w, b, S) = mn (x,y ) S w R(S) = mn max Φ(x ) a a The maxmum margn algorthm L l : (X Y) l H R taes as nput a tranng set of sze l and returns a hyperplane n feature space such that the margn

4 4 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro γ(w, b, S) s maxmzed. Note that assumng the tranng set s separable means that γ > 0. Under ths assumpton, for all probablty measures P underlyng the data S, the expectaton of the msclassfcaton probablty has the bound p err (w, b) = P(sgn( w,φ(x) + b) Y ) E{p err (L l 1 (Z))} 1 { l E R 2 } (Z) γ 2 (L l (Z), Z) The expectaton s taen over the random draw of a tranng set Z of sze l 1 for the left hand sde and l for the rght hand sde. The followng theorem gves a smlar result for the error bound of the lnearly non-separable case. Theorem 2 [13], Consder thresholdng real-valued lnear functons L wth unt weght vectors on an nner product space H and fx γ R +. There s a constant c, such that for any probablty dstrbuton D on H {, } wth support n a ball of radus R around the orgn, wth probablty 1 δ over l random examples S, any hypothess f L has error no more than: err(f) D c + ξ 2 l (R2 2 γ 2 log 2 l + log 1 ), (5) δ where ξ = ξ(f, S, γ) s the margn slac vector wth respect to f and γ. It s clear from both theorems that the bound on the expected error depends not only on the margn but also on the radus of the data, beng a functon of the R 2 /γ 2 rato. Nevertheless standard SVM algorthms can gnore the dependency of the error bound on the radus because for a fxed feature space the radus s constant and can be smply gnored n the optmzaton procedure. However n the MKL scenaro where the H µ feature space s not fxed but depends on the parameter vector µ the radus s no longer fxed but t s a functon of µ and thus should not be gnored n the optmzaton procedure. The radus of the smallest sphere that contans all nstances n the H feature space defned by the Φ(x) mappng s computed by the followng formula [14]: mn R,Φ(x R2 (6) 0) s.t. Φ(x ) Φ(x 0 ) 2 R 2, It can be shown that f K µ s a lnear combnaton of ernels, of the form gven n Formula 2, then for the R µ radus of ts H µ feature space the followng nequaltes hold: max (µ R 2 ) R2 µ M µ R 2 max (R 2 ), (7) s.t. µ = 1

5 Margn and Radus Based Multple Kernel Learnng 5 where R s the radus of the component feature space H assocated wth the K ernel. The proof of the above statement s gven n the appendx. 4 MKL wth margn and radus optmzaton In the next sectons we wll show how we can mae drect use of the dependency of the error bound both on the margn and the radus n the context of the MKL problem n an effort to decrease even more the error bound than what s possble by optmzng only over the margn. 4.1 Soft margn MKL The standard l 2 -soft margn SVM s based on theorem 2 and learns maxmal margn hyperplanes whle controllng for the l 2 norm of the slac vector n an effort to optmze the error bound gven n equaton 5; as already mentoned prevously the radus although t appears n the error bound s not consdered n the optmzaton problem due to the fact that for a gven feature space t s fxed. The exact optmzaton problem solved by the l 2 -soft margn SVM s [13]: mn w,b,ξ 1 2 w,w + C 2 ξ 2 (8) s.t. y ( w,φ(x ) + b) 1 ξ, The soluton hyperplane (w, b ) of ths problem realzes the maxmum margn classfer wth geometrc margn γ = 1 w. When nstead of a sngle ernel we learn wth a combnaton of ernels K µ then the radus of the resultng feature space H µ depends on the parameters µ whch are also learned. We can proft from ths addtonal dependency and optmze not only for the margn but also for the radus, as Theorems 1 and 2 suggest, n the hope of reducng even more the error bounds than what would be possble by just focusng on the margn. A straghtforward way to do so s to alter the cost functon of the above optmzaton problem so that t also ncludes the radus. Thus we defne the prmal form of soft margn MKL optmzaton problem as follows: mn w,b,ξ,µ 1 2 w,w R2 µ + C 2 ξ 2 (9) s.t. y ( w,φ(x ) + b) 1 ξ,

6 6 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro Accountng for the form Φ µ of the feature space H µ, as t s gven n equaton 4, ths optmzaton problem can be rewrtten as: mn w,b,ξ,µ 1 2 w,w Rµ 2 + C 2 ξ 2 (10) s.t. y ( w, µ Φ (x ) + b) 1 ξ, where the w s the same as that of Formula 9 and equal to (w 1,...,w M ), R 2 µ can be computed by equaton 6. By lettng w := µ.w then w := µ w we can rewrte equaton 10 as 1 : mn w,b,ξ,µ 1 2 w,w µ R 2 µ + C 2 s.t. y ( w,φ (x ) + b) 1 ξ, µ = 1, µ 0, =1 ξ 2 (11) The non-negatvty of µ s requred to guarantee that the ernel combnaton s a vald ernel functon; the constrant =1 µ = 1 s added to mae the soluton nterpretable (ernel wth bgger weght can be nterpreted as more mportant one) and to get a specfc soluton (note that f µ s soluton of 11 (wthout the constrant =1 µ = 1), then λµ, λ R + s also ts soluton). We wll denote the cost functon of equaton 11 by F(w, b, ξ, µ). F s not a convex functon, ths s probably the man reason why n current MKL algorthms the radus s smply removed from the orgnal cost functon, therefore they do not really optmze the generalzaton error bound. From the set of nequaltes gven n equaton 7 we have R 2 µ µ R 2 and from ths we can get: F(w, b, ξ, µ) = 1 2 ( w,w w,w µ R 2 µ + C 2 l ξ2 (12) µ µ R 2 + C l 2 ξ2 = l 2 P M µ R 2 ξ2 ) µ R 2 l ξ2 ) = F(w, b, ξ, µ) w,w C µ + ( 1 2 w,w C µ + 2 P M µ R 2 The last nequalty holds because n the context we examne we have µ R 2 1. Ths s a result of the fact that we wor wth the normalzed feature spaces, 1 Note that f µ = 0 then from the dual form we have w = 0. In ths case, we use the conventon that 0 0 = 0.

7 Margn and Radus Based Multple Kernel Learnng 7 usng the normalzed ernels as these were defned n equaton 3, thus we have R 2 1 and snce µ = 1 t holds that: µ R 2 1. Snce F s an upper bound of F and moreover t s convex 2 we are gong to use t as our objectve functon. As a result, we propose to solve, nstead of the orgnal soft margn optmzaton problem gven n equaton 11, the followng upper boundng convex optmzaton problem: mn w,b,ξ,µ 1 2 s.t. y ( w,w C + µ 2 µ R 2 w,φ (x ) + b) 1 ξ, µ = 1, µ 0, =1 The dual functon of ths optmzaton problem s: W s (α, µ) = = 1 2 ξ 2 (13) M α α j y y j µ K (x,x j ) (14) j j α µ R 2 α, α 2C α α j y y j ( µ K (x,x j ) + µ R 2 δ j ) + C where δ j s the Kronecer δ defned to be 1 f = j and 0 otherwse. The dual optmzaton problem s gven as: max W s(α, µ) (15) α,µ s.t. α y = 0, α 0, α α j y y j K (x,x j ) = R2 C ξ2 ( µ R 2, )2 j In the next secton we wll show how we can solve ths new optmzaton problem. 4.2 Algorthm The dual functon 14 s quadratc wth respect to α and lnear wth respect to µ. One way to solve the optmzaton problem 13 s to use a two step teratve 2 The convexty of ths new functon can be easly proved by showng that the Hessan matrx s postve sem-defnte. α

8 8 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro algorthm such as the ones descrbed n [6], [10]. Followng such a two step approach, n the frst step we wll solve a quadratc problem that optmzes over (w, b), whle eepng µ fxed; as a consequence the resultng dual functon s a smple quadratc functon of α whch can be optmzed easly. In the second step we wll solve a lnear problem that optmzes over µ. More precsely, the formulaton of the optmzaton problem wth the twostep approach taes the followng form: where mn J(µ) (16) µ s.t. µ = 1, µ 0, =1 J(µ) = { 1 M w,w C l mnw,b 2 µ + 2 P M µ R 2 ξ2 s.t. y ( w (17),Φ (x ) + b) 1 ξ To solve the outer optmzaton problem,.e. mn µ J(µ), we use gradent descent method. At each teraton, we fx µ, compute the value of J(µ) and then compute the gradent of J(µ) wth respect to µ. The dual functon of Formula 17 s the W s (α, µ) functon already gven n Formula 14. Snce µ s fxed we now optmze only over α (the resultng dual optmzaton problem s much smpler compared to the orgnal soft margn dual optmzaton problem gven n Formula 15): max α W s(α, µ) s.t. α y = 0, α 0, whch has the same form as the SVM quadratc optmzaton problem, the only C dfference s that the C parameter here s equal to. P M µ R 2 For the strong dualty, at the optmal soluton α, the values of dual cost functon and prmal cost functon are equal. Thus the value of W s (α, µ), and the J(µ) value, s gven by: W s (α, µ) = 1 2 α α j y y j ( µ K (x,x j ) j + µ R 2 δ j ) + C The last step of the algorthm s to compute the gradent of the J(µ) functon, Formula 17, wth respect to µ. As [6] have ponted out, we can use the theorem α

9 Margn and Radus Based Multple Kernel Learnng 9 of Bonnans and Shapro [15] to compute gradents of such functons. Hence, the gradent s n the followng form: J(µ) µ = 1 2 j α α j y y j (K (x,x j ) + R2 C δ j) To compute the optmal step n the gradent descent we used lne search. The complete two-step procedure s gven n Algorthm 1. Algorthm 1 R-MKL Intalze µ 1 = 1 for = 1,..., M M repeat Set Rµ 2 = P M µt R 2 compute P J(µ t ) as the soluton of a quadratc optmzaton problem wth K := M µt K compute J µ for = 1,..., M compute optmal step γ t µ t+1 µ t J(µ) + γ t µ untl stopcrtera s true 4.3 Computatonal complexty At each step of the teraton we have to compute the soluton of a standard SVM, wth ernel K = =1 µ C K, and C equal to P M, whch s a quadratc programmng problem wth a complexty of O(n 3 ) where n s number of nstances. µ R 2 Moreover, when µ s updated we have to recompute the approxmaton of Rµ 2 ; the complexty of ths procedure s lnear n the number of ernels, O(M). 5 Experments We expermented wth ten dfferent datasets. Sx of them were taen from the UCI repostory (Ionosphere, Lver, Sonar, Wdbc, Wpbc, Mus1), whle four come from the doman of genomcs and proteomcs (ColonCancer, CentralNervousSystem, FemaleVsMale, Leuema) [16]; these four are characterzed by small sample and hgh dmensonalty morphology. A short descrpton of the datasets s gven n Table 1. We expermented wth two dfferent types of basc ernels,.e. polynomals and Gaussans, and performed two sets of experments. In the frst set of experments we used both types of ernels and n the second one we focused only on Gaussans ernels. For each set of experments the total number of basc ernels was 20; for the frst set we used polynomal ernels of degree one, two, and three and 17 Gaussans wth bandwdth δ that ranged from 1 to 17 wth

10 10 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro Table 1. Short descrpton of the classfcaton datasets used Dataset #Inst #Attr #Class1 #Class2 Ionosphere Lver Sonar Wdbc Wpbc Mus ColonCancer CentralNervous FemaleVSMale Leuema a step of one; for the second set of experments we only used Gaussan ernels wth bandwdth δ that ranged from 1 to 20 wth a step of one. We compared our MKL algorthm (denoted as R-MKL) wth two state-of-theart MKL algorthms: Support Kernel Machne (SKM) [7], and SmpleMKL [6]. We estmate the classfcaton error usng 10-fold cross valdaton. For comparson purposes we also provde the performances of the best sngle ernel (BK) and the majorty classfer (MC); the latter always predcts the majorty class. The performance of the BK s that of the best sngle ernel estmated also by 10-fold cross valdaton, snce t s the best result after seeng the performance of all ndvdual ernels on the avalable data t s optmstcally based. We tuned the parameter C n an nner-loop 10-fold cross-valdaton choosng the values from the set {0.1, 1, 10, 100}. All algorthms termnate when the dualty gap s smaller than All nput ernel matrces are normalzed by equaton 3. We compared the sgnfcance level of the performance dfferences of the algorthms wth McNemar s test [17], where the level of sgnfcance s set to We also establshed a ranng schema of the examned MKL algorthms based on the results of the parwse comparsons [18]. More precsely, f an algorthm s sgnfcantly better than another t s credted wth one pont; f there s no sgnfcant dfference between two algorthms then they are credted wth 0.5 ponts; fnally, f an algorthm s sgnfcantly worse than another t s credted wth zero ponts. Thus, f an algorthm s sgnfcantly better than all the others for a gven dataset t has a score of two. We gve the full results n Tables 2, 3. For each algorthm we report a trplet n whch the frst element s the estmated classfcaton error, the second s the number of selected ernels, and the last s the above descrbed ran. Our ernel combnaton algorthm does remarably well n the frst set of experments, Table 2, n whch t s sgnfcantly better than both other algorthms n four datasets and sgnfcantly worse n two; for the four remanng datasets there are no sgnfcant dfferences. Note that n the cases of Wpbc and CentralNervousSystem all algorthms have a performance that s smlar to that of the majorty classfer,.e. the learned models do not have any dscrmnatory power. By examnng the classfcaton performances of the ndvdual ernels on these datasets we see that none of them had a performance that was bet-

11 Margn and Radus Based Multple Kernel Learnng 11 ter than that of the majorty classfer; ths could explan the bad behavor of the dfferent ernel combnaton schemata. Overall, for ths set of experments R-MKL gets 12 sgnfcance ponts over the dfferent datasets, SKM 10.5, and SmpleMKL 7.5. The performance mprovements of R-MKL over the two other methods are are qute mpressve on those datasets on whch R-MKL performs well; more precsely ts classfcaton error s around 30%, 50%, and 40%, of that of the other algorthms for Wdbc, Mus1, and Leuema datasets respectvely. In the second set of experments,table 3, all methods perform very poorly n seven out of ten datasets; ther classfcaton performance s smlar to that of the majorty classfer. In the remanng datasets, wth the excepton of ColonCancer for whch SKM faled (we used the mplementaton provded by the authors of the algorthm and t returns wth some errors), there s no sgnfcant dfference between the three algorthms. The collectvely bad performance n the last seven databases s explaned by the fact that none of the basc ernels had a classfcaton error that was better than that of the majorty classfer. Overall, for ths set of experments SKM scores 9 ponts, and Smple MKL and R-MKL score 10.5 ponts each. Comparng the number of selected ernels by the dfferent ernel combnaton methods usng a pared t-test (sgnfcance level of 0.05) revealed no statstcally sgnfcant dfferences between the three algorthms on both sets of experments. In an effort to get an emprcal estmaton of the qualty of the approxmaton of the radus that we used to mae the optmzaton problems convex, we P M µ R 2 R2 µ computed the approxmaton error defned as R. We computed ths 2 µ error over the dfferent folds of the ten-fold cross-valdaton for each dataset. The average approxmaton error over the dfferent datasets was We also computed ths error over 1000 random values of µ for each dataset and the average error was Thus, the emprcal evdence seems to ndcate that the Rµ 2 µ R 2 bound s relatvely tght, at least for the datasets we examned. 6 Concluson and future wor In ths paper we presented a new ernel combnaton method that ncorporates n ts cost functon not only the margn but also the radus of the smallest sphere that encloses the data. Ths dea s a drect mplementaton of well nown error bounds from statstcal learnng theory. To the best of our nowledge ths s the frst wor n whch the radus s used together wth the margn n an effort to mnmze the generalzaton error. Even though the resultng optmzaton problems were non-convex and we had to use an upper bound on the radus to get convex forms, the emprcal results were qute encouragng. In partcular, our method competed wth other state-of-the-art methods for ernel combnaton, thus demonstratng the beneft and the potental of the proposed technque. Fnally, we menton that t s stll a challengng research drecton to fully explot the examned generalzaton bound. In future wor we would le to examne optmzaton technques for drectly solvng the non-convex optmzaton problem presented n Formula 11. In partc-

12 12 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro Table 2. Results for the frst experments, where both polynomal and Gaussan ernels are used. Each trplet x,y,z gves respectvely the classfcaton error, the number of selected ernels, and the number of sgnfcance pont that the algorthm scores for the gven experment set and dataset. Columns BK and MC gve the errors of the best sngle ernel and the majorty classfer, respectvely. D. Set SKM Smple R-MKL BK MC Ionos ,02, ,02, ,02, Lver 33.82,05, ,13, ,03, Sonar 15.50,01, ,01, ,01, Wdbc 11.25,03, ,18, ,18, Wpbc 23.68,17, ,01, ,01, Mus ,07, ,18, ,01, Colon ,18, ,18, ,18, CentNe ,17, ,17, ,17, Female ,20, ,18, ,18, Leue ,18, ,18, ,18, Table 3. Results for the second set of experments, where only Gaussan ernels are used. The table contans the same nformaton as the prevous one. D. Set SKM Smple R-MKL BK MC Ionos ,02, ,04, ,03, Lver 33.53,03, ,20, ,16, Sonar 15.50,01, ,01, ,01, Wdbc 37.32,03, ,19, ,20, Wpbc 23.68,20, ,19, ,20, Mus ,14, ,19, ,20, Colon. NA 35.00,20, ,20, CentNe ,20, ,20, ,20, Female ,20, ,20, ,20, Leue ,20, ,20, ,20, ular, we wll examne whether t s possble to decompose the cost functon as a sum convex and concave functons, or to represent t as d.m functons (dfference of two monotonc functons) [19, 20]. Addtonally, we plan to analyze the bound Rµ 2 µ R 2 and see how t relates wth the real optmal value. Acnowledgments The wor reported n ths paper was partally funded by the European Commsson through EU projects DropTop (FP ), DebugIT (FP ) and e-lico (FP ). The support of the Swss NSF (Grant /1) s also gratefully acnowledged. References 1. Shawe-Taylor, J., Crstann, N.: Kernel methods for pattern analyss. Cambrdge Unversty Press (2004)

13 Margn and Radus Based Multple Kernel Learnng Schölopf, B., Smola, A.J.: Learnng wth Kernels: Support Vector Machnes, Regularzaton, Optmzaton, and Beyond. MIT Press, Cambrdge, MA, USA (2001) 3. Lancret, G., Crstann, N., Bartlett, P., Ghaou, L.E.: Learnng the ernel matrx wth semdefnte programmng. Journal of Machne Learnng Research 5 (2004) Ong, C.S., Smola, A.J., Wllamson, R.C.: Learnng the ernel wth hyperernels. Journal of Machne Learnng Research 6 (2005) Sonnenburg, S., Ratsch, G., Schafer, C.: A general and effcent multple ernel learnng algorthm. Journal of Machne Learnng Research 7 (2006) Bach, F., Raotomamonjy, A., Canu, S., Grandvalet, Y.: SmpleMKL. Journal of Machne Learnng Research (2008) 7. Bach, F.R., Lancret, G.R.G., Jordan, M.I.: Multple ernel learnng, conc dualty, and the smo algorthm. In: ICML 04: Proceedngs of the twenty-frst nternatonal conference on Machne learnng, New Yor, NY, USA, ACM (2004) 6 8. Lancret, G., Be, T.D., Crstann, N., I.Jordan, M., Noble, W.S.: A statstcal framewor for genomc data fuson. Bonformatcs 20 (2004) 9. Crstann, N., Shawe-Taylor, J., Elsseeff, A.: On ernel-target algnment. Journal of Machne Learnng Research (2002) 10. Chapelle, O., Vapn, V., Bousquet, O., Muherjee, S.: Choosng multple parameters for support vector machnes. Machne Learnng 46(1-3) (2002) Crammer, K., Keshet, J., Snger, Y.: Kernel desgn usng boostng. In: Advances n Neural Informaton Processng Systems 14, Cambrdge, MA, MIT Press (2002) 12. Bousquet, O., Herrmann, D.: On the complexty of learnng the ernel matrx. In: Advances n Neural Informaton Processng Systems 14, Cambrdge, MA, MIT Press (2003) 13. Crstann, N., Shawe-Taylor, J.: An ntroducton to Support Vector Machnes. Cambrdge Unversty Press (2000) 14. Vapn, V.: Statstcal learnng theory. Wley Interscence (1998) 15. Bonnans, J., A.Shapro.: Optmzaton problems wth perturbaton: A guded tour. SIAM Revew 40(2) (1998) Kalouss, A., Prados, J., Hlaro, M.: Stablty of feature selecton algorthms: a study on hgh dmensonal spaces. Knowledge and Informaton Systems 12(1) (2007) McNemar, Q.: Note on the samplng error of the dfference between correlated proportons or percentages. Psychometra 12 (1947) Kalouss, A., Theohars, T.: Noemon: Desgn, mplementaton and performance results for an ntellgent assstant for classfer selecton. Intellgent Data Analyss Journal 3 (1999) Leo Lbert, N.M., ed.: Global Optmzaton - From Theory to Implementaton. Sprnger (2006) 20. R. Collobert, J.W., Bottou., L.: Tradng convexty for scalablty. Proceedngs of the 23th Conference on Machne Learnng (2006.) 21. Stephen Boyd, L.V., ed.: Convex optmzaton. Cambrge Unversty Press (2004)

14 14 Huyen Do, Alexandros Kalouss, Adam Woznca, and Melane Hlaro Appendx Proof of Inequalty 7. If K(x,x ) s the ernel functon assocated wth the Φ(x) mappng then the computaton of the radus n the dual form s gven n [1]: max β β j R 2 = s.t. β K(x,x ) β = 1, β 0 β β j K(x,x j ) (18) j If β s the optmal soluton of (18) when K = K µ = 1 µ K, and ˆβ s the optmal soluton of (18) when K = K,.e. : Rµ 2 = µ ( β K (x,x ) R 2 = =1 βˆ K (x,x ),j=1 β βj K (x,x j )),j=1 βˆ ˆ β jk (x,x j ) then β K (x,x ) βˆ K (x,x ) Therefore: R 2 µ =1 µ R 2 β β j K (x,x j ),j=1,j=1 βˆ ˆ β j K (x,x j ) Proof of convexty of R-MKL (Eq.13) To prove that 13 s convex, t s enough to show that functons x2 µ, where x R, µ R+ ξ, and P 2 M, where ξ α µ R, µ, α R + are convex. The frst s quadratc-over-lnear functon whch s convex. The second s convex because ts epgraph s a convex set [21].