arxiv: v1 [cs.lg] 15 May 2015

Transcription

1 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio arxiv: v [cs.lg] 5 May 205 Harish G. Ramaswamy Idia Isiue of Sciece, Bagalore, INIA Ambu Tewari Uiversiy of Michiga, A Arbor, USA Shivai Agarwal Idia Isiue of Sciece, Bagalore, INIA Absrac We cosider he problem of -class classificaio ( 2, where he classifier ca choose o absai from makig predicios a a give cos, say, a facor α of he cos of misclassificaio. esigig cosise algorihms for such -class classificaio problems wih a reec opio is he mai goal of his paper, hereby exedig ad geeralizig previously kow resuls for = 2. We show ha he Crammer- Siger surrogae ad he oe vs all hige loss, albei wih a differe predicor ha he sadard argmax, yield cosise algorihms for his problem whe α = 2. More ieresigly, we desig a ew covex surrogae ha is also cosise for his problem whe α = 2 ad operaes o a much lower dimesioal space (log( as opposed o. We also geeralize all hree surrogaes o be cosise for ay α [0, 2 ].. Iroducio I classificaio problems, oe ofe ecouers cases where i would be beer for he classifier o ake o decisio ad absai from predicig raher ha makig a wrog predicio. For example, i he problem of medical diagosis wih iexpesive ess as feaures, a coclusive decisio is good, bu i he face of uceraiy i is beer o o make a predicio ad go for coslier ess. HARISH GURUP@CSA.IISC.ERNET.IN TEWARIA@UMICH.EU SHIVANI@CSA.IISC.ERNET.IN For he case of biary acios, his problem has bee called classificaio wih a reec opio (Barle & Wegkamp, 2008; Yua & Wegkamp, 200; Gradvale e al., 2008; Fumera & Roli, 2002; 2004; Fumera e al., 2000; 2003; Golfarelli e al., 997. Yua ad Wegkamp (200 show ha may sadard covex opimizaio based procedures for biary classificaio like logisic regressio, leas squares classificaio ad expoeial loss miimizaio (Adaboos yield cosise algorihms for his problem. Bu as Barle ad Wegkamp (2008 show, he algorihm based o miimizig he hige loss (SVM requires a modificaio o be cosise. The suggesed modificaio is raher simple use a double hige loss wih hree liear segmes isead of he wo segmes i sadard hige loss, he raio of slopes of he wo o-fla segmes depeds o he cos of absaiig α. I he case of muliclass classificaio however here exis o such resuls ad i is o sraighforward o geeralize he double hige loss o his seig. To he bes of our kowledge, here has bee oly empirical ad heurisic work o muliclass versio of his problem, (Zou e al., 20; Simeoe e al., 202; Wu e al., I his paper, we give a formal reame of he muliclass problem wih a reec opio ad provide cosise algorihms for his problem. The reec opio is accommodaed io he problem of -class classificaio hrough he evaluaio meric. We ow seek a fucio h : X {, 2,...,, + }, where X is he isace space, ad he classes are deoed by {, 2,..., } = [] ad + deoes he acio of absaiig or he reec opio. The loss icurred by such a fucio o a example (x, y wih h(x = is give by if y ad + l α (y, = α if = + 0 if = y where α [0, ] deoes he cos of absaiig. We will call his loss he absai(α loss. I ca be easily show ha he Bayes opimal risk for he above loss is aaied by he fucio h α : X {,..., + } give by h α(x = { argmax y [] p x (y + ( if max y [] p x (y α Oherwise (2

2 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio where p x (y = P (Y = y X = x. The above ca be see as a aural exesio of he Chow s rule (Chow, 970 for he biary case. I ca also be see ha he ieresig rage of values for α is [0, ] as for all α > he Bayes opimal classifier for he absai(α loss ever absais. For example, i biary classificaio, oly α 2 is meaigful, as higher values of α imply i is ever opimal o absai. For small α, he classifier h α acs as a high-cofidece classifier ad would be useful i applicaios like medical diagosis. For example, if oe wishes o lear a classifier for diagosig a illess wih 80% cofidece, ad recommed furher medical ess if i is o possible, he ideal classifier would be h 0.2, which is he miimizer of he absai(0.2 loss. If α = 2, he Bayes classifier h α has a very appealig srucure a class y [] is prediced oly if he class y has a simple maoriy. The absai(α loss is also useful i applicaios where a greaer ha α codiioal probabiliy deecor ca be used as a black box. For example a greaer ha 2 codiioal probabiliy deecor plays a crucial role i hierarchical classificaio (Ramaswamy e al., 205. (eails i supplemeary maerial. Absai(α loss wih α = 2 will be he mai focus of our paper ad will be he defaul whe he absai loss is referred o wihou ay referece o α. (As will be he case i Secios 3, 4, 5 ad 7. As i ca be see ha he Bayes classifier h α depeds oly o he codiioal disribuio of Y X, ay algorihm ha gives a cosise esimaor of he codiioal probabiliy of he classes, e.g. miimizig he oe vs all squared loss, (Ramaswamy & Agarwal, 202; Vere e al., 20, ca be made io a cosise algorihm (wih a suiable chage i he decisio for his problem. However smooh surrogaes ha esimae he codiioal probabiliy do much more ha wha is ecessary o solve his problem. Cosise piecewise liear surrogae miimizig algorihms, o he oher had do oly wha is eeded ad ca be expeced o be more successful. For example, leas squares classificaio, logisic regressio ad SVM are all cosise for sadard biary classificaio, bu he SVM (which miimizes a piecewise liear hige loss surrogae is arguably he mos widely used mehod. Piecewise liear surrogaes have oher advaages like easier opimizaio ad sparsiy (i he dual as well, hece fidig cosise piecewise liear surrogaes for he absai loss is a impora ad ieresig ask. We show ha he -dimesioal muliclass surrogae of Crammer ad Siger (Crammer & Siger, 200 ad he simple oe vs all hige surrogae loss (Rifki & Klauau, 2004 boh yield a cosise algorihm for he absai ( 2 loss. I is ieresig o oe ha boh hese surrogaes are o cosise for he sadard muliclass classificaio problem (Tewari & Barle, 2007; Lee e al., 2004; Zhag, More ieresigly, we cosruc a ew covex piecewise liear surrogae, which we call he biary ecoded predicios (BEP surrogae ha operaes o a log 2 ( dimesioal space, ad yields a cosise algorihm for he - class absai ( 2 loss. Whe opimized over comparable fucio classes, his algorihm is more efficie ha he Crammer-Siger ad oe vs all algorihms due o requirig o oly fid log 2 ( fucios over he isace space, as opposed o fucios. This resul is surprisig because, i has bee show ha oe eeds o miimize a leas a dimesioal covex surrogae o ge a cosise algorihm for he sadard -class problem, i.e. wihou he reec opio (Ramaswamy & Agarwal, 202. Also he oly kow geeric way of geeraig cosise surrogae miimizig algorihms for a give loss marix (Ramaswamy & Agarwal, 202, whe applied o he -class absai loss would give a -dimesioal surrogae here. I is impora o oe he role of α he cos of absaiig. While codiioal probabiliy esimaio based surrogaes ca be used for desigig cosise algorihms for he - class problem wih he reec opio wih ay α (0,, he Crammer-Siger surrogae, he oe vs all hige ad he BEP surrogae ad heir correspodig varias all yield cosise algorihms oly for α [0, 2 ]. While his may seem resricive, we coed ha hese form a ieresig ad useful se of problems o solve. We also suspec ha, absai(α problems wih α > 2 are fudameally more difficul ha hose wih α 2, for he reaso ha evaluaig he Bayes classifier h α(x ca be doe for α 2 wihou fidig he maximum codiioal probabiliy us check if ay class has codiioal probabiliy greaer ha ( α as here ca oly be oe. This is also evideced by he more complicaed pariios of he simplex iduced by he Bayes opimal classifier for α > 2 as show i Figure... Overview We sar wih some prelimiaries ad oaio i Secio 2. I Secio 3 we give excess risk bouds relaig he excess Crammer-Siger muliclass surrogae risk ad oe vs all hige surrogae risk o he excess absai ( 2 risk. I Secio 4 we give our log 2 ( dimesioal BEP surrogae, ad give similar excess risk bouds. I Secio 5, we frame he learig problem wih he BEP surrogae as a opimizaio problem, derive is dual ad give a block coordiae desce syle algorihm for solvig i. I Secio 6 we give geeralizaios of he Crammer-Siger, oe vs all hige ad BEP surrogaes ha are cosise for absai(α loss for α [ 0, 2]. I Secio 7 we iclude experimeal

3 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio resuls for all hree algorihms. We coclude i Secio 8 wih a summary. 2. Prelimiaries Le he isace space be X, he fiie se of class labels be Y = [] = {,..., }, ad he fiie se of arge labels be give by T = [ + ] = {,..., + }. Give raiig examples (X, Y,..., (X m, Y m draw i.i.d. from a disribuio o X Y, he goal is o lear a predicio model h : X T. For ay give α [0, ], he performace of a predicio model h : X T is measured via he absai(α loss l α : Y T R + from Equaio (. l α (y, deoes he loss icurred o predicig whe he ruh is y. We will fid i coveie o represe he loss fucio l α : Y T R + as a loss marix L α R (+ + wih elemes L α y = l α (y, for y [], [ + ], ad colum vecors l α = (L α,..., L α R + for [ + ]. The absai(α loss marix ad a schemaic represeaio of he Bayes classifier for various values of α give by equaio (2 are give i Figure for = 3. Specifically, he goal is o lear a model h : X T wih low expeced loss or l α -error er lα [h] = E (X,Y [l α (Y, h(x]. Ideally, oe was he l α -error of he leared model o be close o he opimal l α -error er lα, = if h:x T erlα [h]. A algorihm, which oupus a (radom model h m : X T o beig give a radom raiig sample as above, is said o be cosise w.r.. l α if he l α -error of he leared model h m coverges i probabiliy o he opimal for all disribuios : er lα [h m] P er l,. Here he covergece i probabiliy is over he leared classifier h m as a fucio of he raiig sample disribued i.i.d. accordig o. However, miimizig he discree l α -error direcly is compuaioally difficul; herefore oe uses isead a surrogae loss fucio ψ : Y R d R + (where R + = [0, ], for some d Z +, ad lears a model f : X R d by miimizig (approximaely, based o he raiig sample he ψ-error er ψ [f] = E (X,Y [ψ(y, f(x]. Predicios o ew isaces x X are he made by applyig he leared model f ad mappig back o predicios i he arge space T via some mappig pred : R d T, givig h(x = pred(f(x. Uder suiable codiios, algorihms ha approximaely miimize he ψ-error based o a raiig sample are kow o be cosise wih respec o ψ, i.e. o coverge i probabiliy o he opimal ψ-error er ψ, = if f:x R d er ψ [f]. Also, whe ψ is covex i is secod argume, he resulig opimizaio problem is covex ad ca be efficiely solved. Hece, we seek a surrogae ad a predicor (ψ, pred, wih ψ covex over is secod argume, ad saisfyig a boud of he followig form holdig for all f : X R d ( er lα [pred f] er lα, ξ er ψ [f] erψ, where ξ : R R is icreasig, coiuous a 0 ad ξ(0 = 0. A surrogae ad a predicor (ψ, pred, saisfyig such a boud, kow as a excess risk rasform boud, would immediaely give a algorihm cosise w.r.. l α from a algorihm cosise w.r.. ψ. We derive such bouds w.r.. he l 2 loss for he Crammer-Siger surrogae, he oe vs all hige surrogae, ad he BEP surrogae, wih ξ as a liear fucio. 3. Excess Risk Bouds for he Crammer-Siger ad Oe vs All Hige Surrogaes I his secio we give a excess risk boud relaig he absai loss l, ad he Crammer-Siger surrogae ψ CS (Crammer & Siger, 200 ad also he oe vs all Hige loss. efie he surrogae ψ CS : [] R R + ad predicor pred CS : R [ + ] as ψ CS (y, u = (max u u y + + y { pred CS (u = argmax i [] u i if u ( u (2 > + oherwise where (a + = max(a, 0, u (i is he i h eleme of he compoes of u whe sored i descedig order ad (0, is a hreshold parameer. We proceed furher ad also defie he surrogae ad predicor for he oe vs all hige loss. The surrogae ψ OVA : [] R R + ad predicor pred OVA : R [ + ] are defied as ψ OVA (y, u = (y = i( u i + + (y i( + u i + i= pred OVA (u = { argmax i [] u i + if max u > oherwise

4 0 α 0 α 0 α Cosise Algorihms for Muliclass Classificaio wih a Reec Opio ( 2 3, 3, 0 (0,, 0 2 ( 3, 2 3, 0 (0, 2 3, 3 4 (, 0, 0 ( 2 (0, 0, 3, 0, 3 ( 3, 0, (0, 3, 2 3 (, 0, 0 ( 2, 0, 2 (0, 0, (a (b (c (d Figure. (a The absai(α loss marix (wih = 3; (b,c,d he pariio of he simplex 3, depicig he opimal predicio for differe codiioal probabiliies, iduced by he Bayes classifier for he absai(, absai( ad absai( 3 losses respecively ( 2, 2, 0 (0,, (0, 2, 2 (0,, 0 2 ( 2, 2, 0 ( 5, 2 5, 2 5 ( 2 5, 2 5, 5 4 ( 2 5, 5, 2 5 (0, 2, 2 (, 0, 0 ( 2, 0, 2 (0, 0, 3 where (a + = max(a, 0 ad (, is a hreshold parameer, ad ies are broke arbirarily, say, i favor of he label y wih he smaller idex. The followig is he mai resul of his secio, he proof of which is i Appedix A ad B. Theorem. Le N, CS (0, ad OVA (,. The for all f : X R er l [pred CS CS f] er l, er l [pred OVA OVA f] er l, (er ψcs [f], erψcs 2 mi( CS, CS (er ψova [f] erψova, 2( OVA Remark: I has bee poied ou previously by Zhag (2004, ha if he daa disribuio is such ha max y p x (y > 0.5 for all x X, he Crammer-Siger surrogae ψ CS ad he oe vs all hige loss are cosise wih he zero-oe loss whe used wih he sadard argmax predicor. Our Theorem implies he above observaio. However i also gives more i he case ha he disribuio does o saisfy he domia class assumpio, he model leared by usig he surrogae ad predicor (ψ CS, pred CS or (ψova, pred OVA asympoically sill gives he righ aswer for isaces havig a domia class, ad fails i a graceful maer by absaiig for isaces ha do o have a domia class. 4. Excess Risk Bouds for he BEP Surrogae The Crammer-Siger surrogae ad he oe vs all hige surrogae, us like surrogaes desiged for codiioal probabiliy esimaio, are defied over a -dimesioal domai. Thus ay algorihm ha miimizes hese surrogaes mus lear real valued fucios over he isace space. I his secio, we cosruc a log 2 ( dimesioal covex surrogae, which we call as he biary ecoded predicios (BEP surrogae ad give a excess risk boud relaig his surrogae ad he absai loss. I paricular hese resuls show ha he BEP surrogae is calibraed w.r.. he absai loss; his i ur implies ha he covex calibraio dimesio (CC-dimesio (Ramaswamy & Agarwal, 202 of he absai loss is a mos log 2 (. For he purpose of simpliciy le us assume = 2 d for some posiive ieger d. Le B : [] {+, } d be ay oe-oe ad oo mappig, wih a iverse mappig B : {+, } d []. efie he BEP surrogae ψ BEP : [] R d R + ad is correspodig predicor : R d [ + ] as pred BEP ψ BEP (y, u = (max [d] B (yu + + pred BEP (u = { + B (sig( u if mi i [d] u i Oherwise where sig(u is he sig of u, wih sig(0 = ad (0, is a hreshold parameer. efie he ses U,..., U+, where Uk = {u Rd : (u = k}. Which evaluaes o pred BEP Uy = {u R d : max B (yu < } U+ = {u R d : mi u } for y [] To make he above defiiio clear we will see how he surrogae ad predicor look like for he case of = 4 ad = 2. We have d = 2. Le us fix he mappig B such ha B(y is he sadard d-bi biary represeaio of (y, wih i he place of 0. The we have, ψ BEP (, u = (max( u, u ψ BEP (2, u = (max( u, u ψ BEP (3, u = (max(u, u ψ BEP (4, u = (max(u, u if u > 2, u 2 > 2 if u > 2, u 2 < 2 pred BEP (u = 3 if u < 2 2, u 2 > 2 4 if u < 2 2 < 2 5 oherwise If is o a power of 2, us add eough dummy classes ha ever occur.

5 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio U 3 U ( 2, 2 ( 2, 2 u 2 ( 2, 2 U 5 (0, 0 ( 2, 2 U 4 U 2 u Primal problem: mi w,...,w d,ξ,...,ξ m such ha ual problem: max α R m (d+ m ξ i + λ 2 i= i [m], [d] d w 2 = ξ i B (y i w x i + ξ i 0 m i= α i,0 2λ m i= i = such ha i [m], [d] {0} m x i, x i µ i,i (α Figure 2. The pariio of R 2 iduced by pred BEP 2 Figure 2 gives he pariio iduced by he predicor pred BEP. 2 The followig is he mai resul of his secio, he proof of which is i Appedix C Theorem 2. Le N ad (0,. Le = 2 d. The for all f : X R d er l [pred BEP f] er l, (er ψbep [f] erψbep, 2 mi(, Remark: The excess risk bouds for he CS, OVA, ad BEP surrogaes sugges ha = 2 is he bes choice for CS ad BEP surrogaes, while = 0 is he bes choice for he OVA surrogae. However, iuiively is he hreshold coverig cofidece values o predicios, ad so i makes sese o use values closer o 0 (or i he case of OVA o predic aggressively i low-oise siuaios, ad use larger o predic coservaively i oisy siuaios. Pracically, i makes sese o choose he parameer via cross-validaio. 5. BEP Surrogae Opimizaio Algorihm I his secio we frame he problem of fidig he liear (vecor valued fucio ha miimizes he BEP surrogae loss over a raiig se {(x i, y i } m i=, wih x i R a ad y i [], as a covex opimizaio problem. Oce agai, for simpliciy we assume ha he size of he label space = 2 d for some d Z +. The primal ad dual of he resulig opimizaio problem wih a orm squared regularizer is give below: α i, 0; d α i, =. =0 where µ i,i (α = d = B (y i B (y i α i α i,. We opimize he dual as i ca be easily exeded o work wih kerels. The srucure of he cosrais i he dual leds iself easily o a block co-ordiae asce algorihm, where we opimize over {α i, : {0,..., d}} ad fix every oher variable i each ieraio. Such mehods have bee recely prove o have expoeial covergece rae for SVM-ype problems (Wag & Li, 204, ad we expec resuls of hose ype o apply o our problem as well. The problem o be solved a every ieraio reduces o a l 2 proecio of a vecor g i R d o o he se S i = {g R d : g b i }, where b i {±} d is such ha b i = B (y i. The proecio problem is a simple varia of proecig a vecor o he l ball of radius, which ca be solved efficiely i O(d ime (uchi e al., The vecor g i is such ha for ay [d], g i = λ b i x i, x i λ m i =;i i 6. Absai(α Loss for α < 2 x i, x i α i,b (y i. The excess risk bouds derived for he CS, OVA hige loss ad BEP surrogaes apply oly o he absai ( 2 loss. Bu i is possible o derive such excess risk bouds for absai(α wih α [ 0, 2] wih sligh modificaios o he CS, OVA ad BEP surrogaes. efie ψ CS,α : [] R R +, ψ OVA,α : [] R R + ad ψ BEP,α : [] R d R +, wih = 2 d as

6 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio ( ψ CS,α (y, u = 2 max α max γ(u u y, y ( α max γ(u u y + 2α y ( ( ψ OVA,α (y, u = 2 (y = iα( u i + i= + (y i( α( + u i + ( ψ BEP,α (y, u = 2 max α max γ(b (yu, [d] ( α max γ(b (yu + 2α [d] where, γ(a = max(a, ad B : [] {, } d is ay biecio. Noe ha ψ CS, 2 = ψ CS, ψ OVA, 2 = ψ OVA ad ψ BEP, 2 = ψ BEP. Oe ca show he followig heorem which is a geeralizaio of Theorems ad 2. The proof proceeds alog he same lies as he proofs of Theorems ad 2 ad is hece omied. [ Theorem ] 3. Le N, (0,, (, ad α 0, 2. Le = 2 d. The for all f : X R d, g : X R er lα [pred CS g] er lα, er lα [pred OVA 2 mi(, g], erlα 2( er lα [pred BEP f] er lα, (er ψcs,α [g] erψcs,α, (er ψova,α [g] er ψova,α, 2 mi(, (er ψbep,α [f] er ψbep,α, Remark: Whe = 2, he Crammer-Siger surrogae, he oe vs all hige ad he BEP surrogae all reduce o he hige loss ad α is resriced o be a mos 2 o esure he relevace of he absai opio. Applyig he above exesio for α 2 o he hige loss, we ge he geeralized hige loss of Barle ad Wegkamp ( Experimeal Resuls I his secio give our experimeal resuls for he algorihms proposed o boh syheic ad real daases. 7.. Syheic aa We opimize he Crammer-Siger surrogae, he oe vs all hige surrogae ad he BEP surrogae, over appropriae kerel spaces o a 2-dimesioal 8 class syheic daa se ad show ha he he absai ( 2 loss icurred by he raied model for all hree algorihms approaches he Bayes opimal uder various hresholds. The daase we used was geeraed as follows. We radomly sample 8 prooype vecors v,..., v 8 R 2, wih each v y draw idepedely from a zero mea ui variace 2-Gaussia, N (0, I 2 disribuio. These 8 prooype vecors correspod o he 8 classes. Each example (x, y is geeraed by firs pickig y from oe of he 8 classes uiformly a radom, ad he isace x is se as x = v y u, where u is idepedely draw from N (0, I 2. We geeraed 2800 such (x, y pairs for raiig, ad aoher 0000 isaces, for esig. The CS, OVA, BEP surrogaes were all opimized over a reproducig kerel Hilber Space (RKHS wih a Gaussia kerel ad he sadard orm-squared regularizer. The kerel widh parameer ad he regularizaio parameer were chose by grid search usig a separae validaio se. 2 As Figure 3 idicaes, he expeced absai risk icurred by he raied model approaches he Bayes risk wih icreasig raiig daa for all hree algorihms ad iermediae values. The excess risk bouds i Theorems ad 2 breakdow whe he hreshold parameer {0, } for he CS ad BEP surrogaes, ad whe {, } for he OVA surrogae. This is suppored by he observaio ha, i Figure 3 he curves correspodig o hese hresholds perform poorly. I paricular, usig = 0 for he CS ad BEP algorihms implies ha he resulig algorihms ever absai. Though all hree surrogae miimizig algorihms we cosider are cosise w.r.. absai loss, we fid ha he BEP ad OVA algorihms use less compuaio ime ad samples ha he CS algorihm o aai he same error. However, he BEP surrogae performs poorly whe opimized over a liear fucio class (experimes o show here, due o is much resriced represeaio power Real aa We ra experimes o real muliclass daases from he UCI reposiory, he deails of which are i Table 2. I each of hese daases if a rai/es spli is o idicaed i he daase we make oe ourselves by spliig a radom. All hree algorihms (CS, OVA ad BEP were opimized over a RKHS wih a Gaussia kerel ad he sadard orm-squared regularizer. The kerel widh ad regularizaio parameers were chose hrough validaio 0- fold cross-validaio i he case of saimage, yeas, vehicle ad image daases, ad a spli of he 2 We used Joachims SVM-ligh package (Joachims, 999 for he OVA ad CS algorihms.

7 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio Expeced absai loss =0 =0.25 =0.5 =0.75 = Bayes risk Expeced absai loss = = 0.5 =0 =0.5 = Bayes risk Expeced absai loss =0 =0.25 =0.5 =0.75 = Bayes risk Traiig size (a Traiig size (b Traiig size (c Figure 3. (a Performace of he CS surrogae for various hresholds as a fucio of raiig size (b Performace of he OVA surrogae for various hresholds as a fucio of raiig size (c Performace of he BEP surrogae for various hresholds as a fucio of raiig size Table. Error perceages of he hree algorihms whe he reecio perceage is fixed a 0%, 20% ad 40%. Reec: 0% 20% 40% Algorihm: CS OVA BEP CS OVA BEP CS OVA BEP saimage yeas leer vehicle image coverype Table 2. eails of daases used. # Trai # Tes # Fea # Class saimage 4,435 2, yeas, leer 6,000 4, vehicle image 2, coverype 5,20 565, rai se io rai ad validaio for he leer ad coverype daases. For simpliciy we se = 0 (or = for OVA durig he validaio phase. The resuls of he experime wih he CS, OVA ad BEP algorihms is give i Table 2. The reecio rae is fixed a some give level by choosig he hreshold for each algorihm ad daase appropriaely. As ca be see from he Table, he BEP algorihm s performace is comparable o he OVA, ad is beer ha he CS algorihm. However, Table 3, which gives he raiig imes for he algorihms, reveals ha he BEP algorihm rus he fases, hus makig he BEP algorihm a good opio for large daases. The mai reaso for he observed speedup of he BEP is ha i lears oly log 2 ( fucios for a -class problem ad hece he speedup facor of he BEP over he OVA would Table 3. Time ake for learig fial model ad makig predicios o es se (does o iclude validaio ime Algorihm CS OVA BEP saimage 253s 76s 44s yeas 5s 7s 2s leer 9608s 055s 33s vehicle 3s 3s s image 222s 6s 6s coverype 47974s 23709s 6786s poeially be beer for larger. 8. Coclusio The muliclass classificaio problem wih a reec opio, is a powerful absracio ha capures corollig he uceraiy of he classifier ad is very useful i applicaios like medical diagosis. We formalized his problem via a evaluaio meric, called he absai loss, ad gave excess risk bouds relaig he absai loss o he Crammer-Siger surrogae, he oe vs all hige surrogae ad also o he BEP surrogae which is a ew surrogae ad operaes o a much smaller dimesio. Exedig hese resuls for oher such evaluaio merics, i paricular he absai(α loss for α > 2, is a ieresig fuure direcio.

8 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio Refereces Barle, P. L. ad Wegkamp, M. H. Classificaio wih a reec opio usig a hige loss. Joural of Machie Learig Research, 9: , Chow, C. O opimum recogiio error ad reec radeoff. IEEE Trasacios o Iformaio Theory, 6:4 46, 970. Crammer, K. ad Siger, Y. O he algorihmic implemeaio of muliclass kerel-based vecor machies. Joural of Machie Learig Research, 2: , 200. uchi, J., Shalev-Shwarz, S., Siger, Y., ad Chadra, T. Efficie proecios oo he l -ball for learig i high dimesios. I Ieraioal Coferece o Machie Learig, Fumera, G. ad Roli, F. Supppor vecor machies wih embedded reec opio. Paer Recogiio wih Suppor Vecor Machies, pp , Fumera, G. ad Roli, F. Aalysis of error-reec rade-off i liearly combied muliple classifiers. Paer Recogiio, 37: , Fumera, G., Roli, F., ad Giacio, G. Reec opio wih muliple hresholds. Paer Recogiio, 33: , Fumera, G., Pillai, I., ad Roli, F. Classificaio wih reec opio i ex caegorisaio sysems. I IEEE Ieraioal Coferece o Image Aalysis ad Processig, pp , Golfarelli, M., Maio,., ad Maloi,. O he errorreec rade-off i biomeric verificaio sysems. IEEE Trasacios o Paer Aalysis ad Machie Ielligece, 9: , 997. Gradvale, Y., Rakoomamoy, A., Keshe, J., ad Cau, S. Suppor vecor machies wih a reec opio. I Neural Iformaio Processig Sysems, Ramaswamy, H. G., Tewari, A., ad Agarwal, S. Covex calibraed surrogaes for hierarchical classificaio. I Ieraioal Coferece o Machie Learig, 205. Rifki, R. ad Klauau, A. I defese of oe-vs-all classificaio. Joural of Machie Learig Research, 5: 0 4, Simeoe, P., Marrocco, C., ad Tororella, F. esig of reec rules for ECOC classificaio sysems. Paer Recogiio, 45: , 202. Tewari, A. ad Barle, P. L. O he cosisecy of muliclass classificaio mehods. Joural of Machie Learig Research, 8: , Vere, E., Williamso, R. C., ad Reid, M.. Composie muliclass losses. I Neural Iformaio Processig Sysems, 20. Wag, P.-W. ad Li, C.-J. Ieraio complexiy of feasible desce mehods for covex opimizaio. Joural of Machie Learig Research, 5: , 204. Wu, Q., Jia, C., ad Che, W. A ovel classificaioreecio sphere SVMs for muli-class classificaio problems. I IEEE Ieraioal Coferece o Naural Compuaio, Yua, M. ad Wegkamp, M. Classificaio mehods wih reec opio based o covex risk miimizaio. Joural of Machie Learig Research, : 30, 200. Zhag, T. Saisical aalysis of some muli-caegory large margi classificaio mehods. Joural of Machie Learig Research, 5:225 25, Zou, C., hui Zheg, E., wei Xu, H., ad Che, L. Cossesiive muli-class SVM wih reec opio: A mehod for seam urbie geeraor faul diagosis. Ieraioal Joural of Compuer Theory ad Egieerig, 20. Joachims, T. Makig large-scale svm learig pracical. I Schlkopf, B., Burges, C., ad Smola, A. (eds., Advaces i Kerel Mehods - Suppor Vecor Learig. MIT-Press, 999. Lee, Y., Li, Y., ad Wahba, G. Mulicaegory suppor vecor machies: Theory ad applicaio o he classificaio of microarray daa. Joural of he America Saisical Associaio, 99(465:67 8, Ramaswamy, H. G. ad Agarwal, S. Classificaio calibraio dimesio for geeral muliclass losses. I Advaces i Neural Iformaio Processig Sysems 25, pp

9 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio Appedix We break he Proof of Theorem io wo pars cosisig of he proof of excess risk bouds for he CS surrogae ad he OVA surrogae respecively. A. Proof of Excess Risk Bouds for he Crammer Siger Surrogae efie he ses U,..., U + such ha U i is he se of vecors u i R, for which pred CS (u = i U y = {u R : u y > u + for all y}; y [] U + = {u R : u ( u (2 + }. The followig lemma gives some crucial, bu sraighforward o prove, (iequaliies saisfied by he Crammer- Siger surrogae. Lemma 4. y [], p p ψ CS (e y = 2( p y, (3 p ψ CS (0 =, (4 u R, y argmax i u i, y / argmax i u i ψ CS (y, u u (2 u ( +, (5 ψ CS (y, u u ( u (2 +, (6 where e y is he vecor i R wih i he y h posiio ad 0 everywhere else. The par of Theorem proved here is resaed below. Theorem. Le N ad (0,. The for all f : X R (er ψcs [f], erψcs er l [pred CS f] er l, 2 mi(, Proof. We will show ha p ad all u R d p ψ CS (u if u R p ψ(u 2 mi(, ( p l pred CS (u mi p l. (7 The Theorem simply follows from lieariy of expecaio. Case : p y 2 for some y []. We have ha y argmi p l. Case a: u U y The RHS of equaio (7 is zero, ad hece becomes rivial. Case b: u U + We have ha u ( u (2. Le q = i argmax u p i. We he have p ψ CS (u p ψ CS (e y (3 = p i ψ CS (i, u + i:u i=u ( (5,(6 i:u i<u ( p y ψ CS (y, u 2( p y (2q (u (2 u ( + 2p y (2p y (.(8 The las iequaliy follows from u (2 u ( ad he followig observaios. If q > p y he u ( = u (2, ad if q < p y we have q < 2. p l pred CS (u mi p l = p l + p l y = p y 2 (9 From Equaios (8 ad (9 we have p ψ CS (u if u R p ψ(u 2( ( p l pred CS (u mi p l (0 Case c: u R \ (Uy U+ We have pred CS (u = y y. Also p y p y 2 ad u ( = u y > u (2 +. p ψ CS (u p ψ CS (e y (3 = p i ψ CS (i, u + p y ψ CS (y, u i=;i y (5,(6 2( p y ( 2p y (u y u (2 + p y 2(p y p y (From Case c ( We also have ha p l pred CS (u mi p l = p l y p l y = p y p y From Equaios ( ad (2 we have p ψ CS (u if u R p ψ CS (u 2 ( p l pred CS (u mi p l Case 2: p y < 2 for all y [] (2 (3

10 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio We have ha + argmi p l Case 2a: u U+ (or pred CS (u = + The RHS of equaio (7 is zero, ad hece becomes rivial. Case 2b: u R \ U+ (or pred CS (u + Le pred CS (u = argmax i u i = y. We have ha u ( = u y > u (2 + ad p y < 2. p ψ CS (u p ψ CS (0 (4 = p i ψ CS (i, u + p y ψ CS (y, u i=;i y (5,(6 ( 2p y (u ( u (2 ( 2p y ( (From Case 2b (4 We also have ha p l pred CS (u mi p l = p l y p l + = 2 p y (5 From Equaios (4 ad (5 we have p ψ CS (u if u R p ψ CS (u 2 ( p l pred CS (u mi p l (6 Equaio (7, ad hece he Theorem, follows from Equaios (0, (3 ad (6. B. Proof of Excess Risk Bouds for he Oe vs All Hige Surrogae efie he ses U,..., U + such ha U i is he se of vecors u i R, for which pred OVA (u = i U y U + = {u R : u y >, y = argmax i [] u i }, y [] = {u R : u for all []}. The followig lemma gives some crucial, bu sraighforward o prove, (iequaliies saisfied by he OVA hige surrogae. Lemma 5. y [], p, u R p ψ OVA (2 e y = 4( p y (7 p ψ OVA ( = 2 (8 ψ OVA (y, u u 2u y + (9 [] where e y is he vecor i R wih i he y h posiio ad 0 everywhere else. The par of Theorem proved here is resaed below. Theorem. Le N ad (0,. The for all f : X R er l [pred OVA f] er l, 2( (er ψova [f] erψova, Proof. We will show ha p ad all u [, ] d p ψ OVA (u if u R p ψ OVA (u 2( ( p l pred OVA (u mi p l (20 he Theorem simply follows from he observaio ha for all u R clippig he compoes of u o [, ] does o icrease ψ OVA (y, u for ay y, ad by lieariy of expecaio. Case : p y 2 for some y []. We have ha y argmi p l. Case a: u [, ] U y The RHS of equaio (20 is zero, ad hece becomes rivial. Case b: u [, ] U + We have ha max u. p ψ OVA (u 9 i= ( 2p i u i + i []\{y} ( 2p i u i + (2p y ( + (2p i + (2p y ( + i [] = (2p y ( + 2 Ad hece we have We also have p ψ OVA (u p ψ OVA (2 e y 7 (2py ( + 2 4( p y = (2p y ( (2 p l pred OVA (u mi p l = p l + p l y = p y 2 (22 From Equaios (2 ad (22 we have for all u [, ] U+ p ψ OVA (u if u R p ψ OVA (u 2( ( p l pred OVA (u mi p l (23

11 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio Case c: u [, ] \ (U y U + We have pred OVA (u = y y. Also p y 2 ; u y > ad u y u y. p ψ OVA (u ( 9 ( 2p i u i + i= i []\{y } ( 2p i u i + ( 2p y ( + (2p i + ( 2p y ( + +. i [] Ad hece we have, p ψ OVA (u p ψ OVA (2 e y ( 2py ( p y = ( 2p y ( + + 2(2p y ( 2p y ( + + ( + (2p y = 2( + (p y p y. (24 We also have ha p l pred OVA (u mi p l = p l y p l y = p y p y (25 From Equaios (24 ad (25 we have for all u [, ] \ (Uy U+ p ψ OVA (u if u R p ψ OVA (u 2( + ( p l pred OVA (u mi p l Case 2: p y < 2 for all y [] We have ha + argmi p l Case 2a: u U + (26 The RHS of equaio (7 is zero, ad hece becomes rivial. Case 2b: u [, ] \ U + Le pred OVA (u = argmax i u i = y. We have ha u y ad p y < 2. p ψ OVA (u p ψ OVA ( ( (8 = p ψ OVA (u 2 ( (9 ( 2p i u i + 2 i= i []\{y} ( 2p i u i + ( 2p y ( + 2 (2p i + ( 2p y ( i [] = ( 2p y ( + (27 We also have ha p l pred OVA (u mi p l = p l y p l + = 2 p y (28 From Equaios (27 ad (28 we have for all u [, ] \ U+ p ψ OVA (u if u R p ψ OVA (u 2( + ( p l pred OVA (u mi p l (29 Equaio (20, ad hece he Theorem, follows from Equaios (23, (26 ad (29. C. Proof of Excess Risk Bouds for he BEP Surrogae The followig lemma gives some crucial, bu sraighforward o prove, (iequaliies saisfied by he BEP surrogae. Lemma 6. y, y [], p, u R, y B (sig( u p ψ BEP ( B(y = 2( p y (30 p ψ BEP (0 = (3 ψ BEP (B (sig( u, u mi u + (32 er l [pred BEP ψ BEP (y, u mi u + (33 Theorem. Le N ad (0,. Le = 2 d. The for all f : X R d (er ψbep [f] erψbep, f] er l, 2 mi(, Proof. We will show ha p ad all u R d p ψ BEP (u if u R d p ψ BEP (u 2 mi(, (p l pred BEP (u mi p l (34

12 Cosise Algorihms for Muliclass Classificaio wih a Reec Opio The heorem follows by lieariy of expecaio. Case : p y 2 for some y [] We have ha y argmi p l Case a: u U y (or pred BEP (u = y The RHS of equaio (34 is zero, ad hece becomes rivial. Case b: u U + (or pred BEP (u = + Le y = B (sig( u. We have mi u. p ψ BEP (u = p y ψ BEP (y, u + (32,(33 i []\{y } p i ψ BEP (i, u p y ( mi [d] u + ( p y (mi [d] u + (2p y ( +. The las iequaliy i he above follows from he observaio ha if y y, he p y < p y 2. We hus have We also have ha p ψ BEP (u p ψ BEP ( B(y (30 (2p y (. (35 p l pred BEP (u mi p l = p l + p l y = p y 2 (36 From Equaios (35 ad (36 we have ha p ψ BEP (u if u R d p ψ BEP (u 2( (p l pred BEP (u mi p l (37 Case c: u R d \ (U y U + Le B (sig( u = pred(u = y for some y y. We have p y p y 2, ad mi u > ad We also have ha p l pred BEP (u mi p l = p l y p l y = p y p y (39 From Equaios (38 ad (39 we have ha p ψ BEP (u if u R d p ψ BEP (u Case 2: p y < 2 for all y [] 2((p l pred BEP (u mi p l (40 We have ha + argmi p l Case 2a: u U + The RHS of equaio (34 is zero, ad hece becomes rivial. Case 2b: u R d \ U + Le B (sig( u = y = pred BEP (u for some y []. We have p y < 2 ad mi u >. p ψ BEP (u p ψ BEP (0 (3 = p y ψ BEP (y, u + (32,(33 i=;i y p i ψ BEP (i, u p y mi u + ( p y mi u ( 2p y (4 We also have ha p l pred BEP (u mi p l = p l y p l + = 2 p y (42 From Equaios (4 ad (42 we have ha p ψ BEP (u if u R d p ψ BEP (u 2(p l pred BEP (u mi p l (43 Equaio (34, ad hece he Theorem, follows from equaios (37, (40 ad (43. p ψ BEP (u = p y ψ BEP (y, u + (32,(33 Hece we ge i=;i y p i ψ BEP (i, u p y ( mi u + ( p y (mi u + ( 2p y + (From case c p ψ BEP (u p ψ BEP ( B(y (30 2(p y p y (38