Multiclass Boosting for Weak Classifiers

Transcription

1 Journal of Machne Learnng Research ) Submed 6/03; Revsed 7/04; Publshed 2/05 Mulclass Boosng for Weak Classfers Günher Ebl Karl Peer Pfeffer Deparmen of Bosascs Unversy of Innsbruck Schöpfsrasse 4, 6020 Innsbruck, Ausra GUEHER.EIBL@UIBK.AC.A KARL-PEER.PFEIFFER@UIBK.AC.A Edor: Rober Schapre Absrac AdaBoos.M2 s a boosng algorhm desgned for mulclass problems wh weak base classfers. he algorhm s desgned o mnmze a very loose bound on he ranng error. We propose wo alernave boosng algorhms whch also mnmze bounds on performance measures. hese performance measures are no as srongly conneced o he expeced error as he ranng error, bu he derved bounds are gher han he bound on he ranng error of AdaBoos.M2. In expermens he mehods have roughly he same performance n mnmzng he ranng and es error raes. he new algorhms have he advanage ha he base classfer should mnmze he confdence-raed error, whereas for AdaBoos.M2 he base classfer should mnmze he pseudo-loss. hs makes hem more easly applcable o already exsng base classfers. he new algorhms also end o converge faser han AdaBoos.M2. Keywords: boosng, mulclass, ensemble, classfcaon, decson sumps. Inroducon Mos papers abou boosng heory consder wo-class problems. Mulclass problems can be eher reduced o wo-class problems usng error-correcng codes Allwen e al., 2000; Deerrch and Bakr, 995; Guruswam and Saha, 999) or reaed more drecly usng base classfers for mulclass problems. Freund and Schapre 996 and 997) proposed he algorhm AdaBoos.M whch s a sraghforward generalzaon of AdaBoos usng mulclass base classfers. An exponenal decrease of an upper bound of he ranng error rae s guaraneed as long as he error raes of he base classfers are less han /2. For more han wo labels hs condon can be oo resrcve for weak classfers lke decson sumps whch we use n hs paper. Freund and Schapre overcame hs problem wh he nroducon of he pseudo-loss of a classfer h : X Y [0,] : ) ε = h x,y )+ 2 Y h x,y). y y In he algorhm AdaBoos.M2, each base classfer has o mnmze he pseudo-loss nsead of he error rae. As long as he pseudo-loss s less han /2, whch s easly reachable for weak base classfers as decson sumps, an exponenal decrease of an upper bound on he ranng error rae s guaraneed. In hs paper, we wll derve wo new drec algorhms for mulclass problems wh decson sumps as base classfers. he frs one s called GrPloss and has s orgn n he graden descen c 2005 Günher Ebl and Karl-Peer Pfeffer.

2 EIBL AD PFEIFFER framework of Mason e al. 998, 999). Combned wh deas of Freund and Schapre 996, 997) we ge an exponenal bound on a performance measure whch we call pseudo-loss error. he second algorhm was movaed by he aemp o make AdaBoos.M work for weak base classfers. We nroduce he maxlabel error rae and derve bounds on. For boh algorhms, he bounds on he performance measures decrease exponenally under condons whch are easy o fulfll by he base classfer. For boh algorhms he goal of he base classfer s o mnmze he confdence-raed error rae whch makes hem applcable for a wde range of already exsng base classfers. hroughou hs paper S = {x,y ); =,...,)} denoes he ranng se where each x belongs o some nsance or measuremen space X and each label y s n some label se Y. In conras o he wo-class case, Y can have Y 2 elemens. A boosng algorhm calls a gven weak classfcaon algorhm h repeaedly n a seres of rounds =,...,. In each round, a sample of he orgnal ranng se S s drawn accordng o he weghng dsrbuon D and used as ranng se for he weak classfcaon algorhm h. D ) denoes he wegh of example of he orgnal ranng se S. he fnal classfer H s a weghed majory voe of he weak classfers h where α s he wegh assgned o h. Fnally, he elemens of a se M ha maxmze and mnmze a funcon f are denoed argmax m M 2. Algorhm GrPloss fm) and arg mn m M fm) respecvely. In hs secon we wll derve he algorhm GrPloss. Mason e al. 998, 999) embedded AdaBoos n a more general heory whch sees boosng algorhms as graden descen mehods for he mnmzaon of a loss funcon n funcon space. We ge GrPloss by applyng he graden descen framework especally for mnmzng he exponenal pseudo-loss. We frs consder slghly more general exponenal loss funcons. Based on he graden descen framework, we derve a graden descen algorhm for hese loss funcons n a sragh forward way n Secon 2.. In conras o he general framework, we can addonally derve a smple updae rule for he samplng dsrbuon as exss for AdaBoos.M and AdaBoos.M2. Graden descen does no provde a specal choce for he sep sze α. In Secon 2.2, we defne he pseudo-loss error and derve α by mnmzaon of an upper bound on he pseudo-loss error. Fnally, he algorhm s smplfed for he specal case of decson sumps as base classfers. 2. Graden Descen for Exponenal Loss Funcons Frs we brefly descrbe he graden descen framework for he wo-class case wh Y = {,+}. As usual a ranng se S = {x,y ); =,...,)} s gven. We are consderng a funcon space F = lnh ) conssng of funcons f : X R of he form fx; α, β) = α h x;β ), h : X {±} wh α = α,...,α ) R, β = β,...,β ) and h H. he parameers β unquely deermne h herefore α and β unquely deermne f. We choose a loss funcon L f) = E y,x [l fx),y)] = E x [E y [ly fx))]] l : R R 0 90

3 MULICLASS BOOSIG FOR WEAK CLASSIFIERS where for example he choce of l fx),y) = e y fx) leads o L f) = he goal s o fnd f = argmnl f). f F he graden n funcon space s defned as: = e y fx ). L f)x) := L f + e x) L f + e x ) L f) e=0 = lm e e 0 e where for wo arbrary uples v and ṽ we denoe { ṽ = v v ṽ) = 0 ṽ v. A graden descen mehod always makes a sep n he drecon of he negave graden L f)x). However L f)x) s no necessarly an elemen of F, so we replace by an elemen h of F whch s as parallel o L f)x) as possble. herefore we need an nner produc, : F F R, whch can for example be chosen as f, f = fx ) fx ). = hs nner produc measures he agreemen of f and f on he ranng se. Usng hs nner produc we can se β := argmax L f ),h ; β) β and h := h ; β ). he nequaly L f ),hβ ) 0 means ha we can no fnd a good drecon hβ ), so he algorhm sops, when hs happens. he resulng algorhm s gven n Fgure. Inpu: ranng se S, loss funcon l, nner produc, : F F R, sarng value f 0. := Loop: whle L f ),hβ ) > 0 β := argmax L f ),hβ) β α := argmn α L f + αh β ))) f = f + α h β ) Oupu: f, L f ) Fgure : Algorhm graden descen n funcon space ow we go back o he mulclass case and modfy he graden descen framework n order o rea classfers f of he form f : X Y R, where fx,y) s a measure of he confdence, ha an 9

4 EIBL AD PFEIFFER objec wh measuremens x has he label y. We denoe he se of possble classfers wh F. For graden descen we need a loss funcon and an nner produc on F. We choose f, ˆf := Y = y= fx,y) ˆfx,y), whch s a sraghforward generalzaon of he defnon for he wo-class case. he goal of he classfcaon algorhm GrPloss s o mnmze he specal loss funcon he erm L f) := [ l f,) wh l f,) := exp 2 fx,y) fx,y )+ y y Y fx,y )+ y y )] fx,y). ) Y compares he confdence o label he example x correcly wh he mean confdence of choosng one of he wrong labels. ow we consder slghly more general exponenal loss funcons where he choce l f,) = exp[v f,)] wh exponen loss v f,) = v 0 +v y ) fx,y), y v 0 = 2 and v y) = { 2 y = y y y 2 Y ) leads o he loss funcon ). hs choce of he loss funcon leads o he algorhm gven n Fgure 2. he properes summarzed n heorem can be shown o hold on hs algorhm. Inpu: ranng se S, maxmum number of boosng rounds Inalsaon: f 0 := 0, :=, : D ) :=. Loop: For =,..., do h = argmn D )vh,) h If D )vh,) v 0 : :=, goo oupu. Choose α. Updae f = f + α h and D + ) = Z D )lα h,) Oupu: f, L f ) Fgure 2: Graden descen for exponenal loss funcons 92

5 MULICLASS BOOSIG FOR WEAK CLASSIFIERS heorem For he nner produc f,h = Y = y= and any exponenal loss funcons l f,) of he form fx,y)hx,y) l f,) = exp[v f,)] wh v f,) = v 0 +v y ) fx,y) y where v 0 and v y ) are consans, he followng saemens hold: ) he choce of h ha maxmzes he projecon on he negave graden h = argmax L f ),h h s equvalen o ha mnmzng he weghed exponen-loss wh respec o he samplng dsrbuon h = argmn h D )vh,) D ) := l f,) l f, ) = ) he soppng creron of he graden descen mehod L f ),hβ ) 0 l f,) Z. leads o a sop of he algorhm, when he weghed exponen-loss ges posve D )vh,) v 0. ) he samplng dsrbuon can be updaed n a smlar way as n AdaBoos usng he rule where we defne Z as a normalzaon consan D + ) = Z D )lα h,), Z := D )lα h,), whch ensures ha he updae D + s a dsrbuon. In conras o he general framework, he algorhm uses a smple updae rule for he samplng dsrbuon as exss for he orgnal boosng algorhms. oe ha he algorhm does no specfy he choce of he sep sze α, because graden descen only provdes an upper bound on α. We wll derve a specal choce for α n he nex secon. 93

6 EIBL AD PFEIFFER Proof. he proof bascally consss of hree seps: he calculaon of he graden, he choce for base classfer h ogeher wh he soppng creron and he updae rule for he samplng dsrbuon. ) Frs we calculae he graden, whch s defned by L f)x,y) := lm k 0 L f + k x,y) ) L f) k { for x,y) x,y ) = x,y)=x,y ) 0 x,y) x,y ). So we ge for x = x : ] L f + k x y) = [v exp 0 +v y ) fx,y )+kv y ) y Subsuon n he defnon of L f) leads o hus ow we nser 2) no L f ),h and ge l f,)e kvy) ) L f)x,y) = lm = l f,)v y ). k 0 k = l f,)ekv y). { 0 x x L f)x,y) =. 2) l f,)v y ) x = x L f ),h = l f,)v y )hx,y) = y l f,)vh,) v 0 ). 3) If we defne he samplng dsrbuon D up o a posve consan C by we can wre 3) as L f ),h = C D ) := C l f,), 4) D )vh,) v 0 ) = C D )vh,) v 0 ). 5) Snce we requre C o be posve, we ge he choce of h of he algorhm h = argmax h L f ),h = argmn ) One can verfy he soppng creron of Fgure 2 from 5) h D )vh,). L f ),h 0 D )vh,) v 0. ) Fnally, we show ha we can calculae he updae rule for he samplng dsrbuon D. D + ) = C l f,) = C l f + α h,) = C l f,)lα h,) = C C D )lα h,). 94

7 MULICLASS BOOSIG FOR WEAK CLASSIFIERS hs means ha he new wegh of example s a consan mulpled wh D )lα h,). By comparng hs equaon wh he defnon of Z we can deermne C C = C Z. Snce l s posve and he weghs are posve one can show by nducon, ha also C s posve, whch we requred before. 2.2 Choce of α and Resulng Algorhm GrPloss he algorhm above leaves he sep lengh α, whch s he wegh of he base classfer h, unspecfed. In hs secon we defne he pseudo-loss error and derve α by mnmzaon of an upper bound on he pseudo-loss error. Defnon: A classfer f : X Y R makes a pseudo-loss error n classfyng an example x wh label k, f fx,k) < Y fx,y). y k he correspondng ranng error rae s denoed by plerr: plerr := = I fx,y ) < Y y y fx,y) he pseudo-loss error couns he proporon of elemens n he ranng se for whch he confdence fx,k) n he rgh label s smaller han he average confdence n he remanng labels fx,y)/ Y ). hus s a weak measure for he performance of a classfer n he sense y k ha can be much smaller han he ranng error. ow we consder he exponenal pseudo-loss. he consan erm of he pseudo-loss leads o a consan facor whch can be pu no he normalzng consan. So wh he defnon he updae rule can be wren n he shorer form u f,) := fx,y ) Y y y fx,y) D + ) = Z D )e α uh,)/2, wh Z := = ) D )e α uh,)/2. We presen our nex algorhm, GrPloss, n Fgure 3, whch we wll derve and jusfy n wha follows. ) Smlar o Schapre and Snger 999) we frs bound plerr by he produc of he normalzaon consans o prove 6), we frs noce ha plerr plerr. Z. 6) e u f,)/2. 7) 95

8 EIBL AD PFEIFFER Inpu: ranng se S = {x,y ),...,x,y ); x X, y Y }, Y = {,..., Y }, weak classfcaon algorhm wh oupu h : X Y [0,] Oponally : maxmal number of boosng rounds Inalzaon: D ) =. For =,..., : ran he weak classfcaon algorhm h wh dsrbuon D, where h should maxmze U := D )uh,). If U 0: goo oupu wh := Se Updae D: +U α = ln U ). D + ) = Z D )e α uh,)/2. where Z s a normalzaon facor chosen so ha D + s a dsrbuon) Oupu: fnal classfer Hx): Hx) = argmax y Y fx,y) = argmax α h x,y) y Y Fgure 3: Algorhm GrPloss ) ow we unravel he updae rule D + ) = Z e α uh,)/2 D ) = Z Z e α uh,)/2 e α uh,)/2 D ) =... = D ) e α uh,)/2 Z = exp = e u f,)/2 α uh,)/2 Z ) where he las equaon uses he propery ha u s lnear n h. Snce = D + ) = 96 Z e u f,)/2 Z

9 MULICLASS BOOSIG FOR WEAK CLASSIFIERS we ge Equaon 6) by usng 7) and he equaon above plerr e u f,)/2 = ) Dervaon of α : ow we derve α by mnmzng he upper bound 6). Frs, we plug n he defnon of Z ) Z = D )e α uh,)/2. ow we ge an upper bound on hs produc usng he convexy of he funcon e αu beween and + from hx,y) [0,] follows ha u [,+]) for posve α : ) Z D ) 2 [ uh,))e + 2 α ++uh,))e 2 α ]. 8) ow we choose α n order o mnmze hs upper bound by seng he frs dervave wh respec o α o zero. o do hs, we defne U := D )uh,). Snce each α occurs n exacly one facor of he bound 8) he resul for α only depends on U and no on U s, s, more specfcally ) +U α = ln. U oe ha U has s values n he nerval [,], because u [,+] and D s a dsrbuon. ) Dervaon of he upper bound of he heorem: ow we subsue α back n 8) and ge afer some sraghforward calculaons Z U 2. Usng he nequaly x 2 x) e x/2 for x [0,] we can ge an exponenal bound on Z [ ] Z exp U 2 /2. If we assume ha each classfer h fulflls U δ, we fnally ge Z e δ2 /2. v) Soppng creron: he soppng creron of he slghly more general algorhm drecly resuls n he new soppng creron o sop, when U 0. However, noe ha he bound depends on he square of U nsead of U leadng o a formal decrease of he bound even when U > 0. Z. 97

10 EIBL AD PFEIFFER We summarze he foregong argumen as a heorem. heorem 2 If for all base classfers h : X Y [0,] of he algorhm GrPloss gven n Fgure 3 U := D )uh,) δ holds for δ > 0 hen he pseudo-loss error of he ranng se fulflls plerr U 2 e δ2 /2. 9) 2.3 GrPloss for Decson Sumps So far we have consdered classfers of he form h : X Y [0,]. ow we wan o consder base classfers ha have addonally he normalzaon propery hx,y) = 0) y Y whch we dd no use n he prevous secon for he dervaon of α. he decson sumps we used n our expermens fnd an arbue a and a value v whch are used o dvde he ranng se no wo subses. If arbue a s connuous and s value on x s a mos v hen x belongs o he frs subse; oherwse x belongs o he second subse. If arbue a s caegorcal he wo subses correspond o a paron of all possble values of a no wo ses. he predcon hx,y) s he proporon of examples wh label y belongng o he same subse as x. Snce proporons are n he nerval [0,] and for each of he wo subses he sum of proporons s one our decson sumps have boh he former and he laer propery 0). ow we use hese properes o mnmze a gher bound on he pseudo-loss error and furher smplfy he algorhm. ) Dervaon of α : o ge α we can sar wh plerr Z = ) D )e α uh,)/2 whch was derved n par ) of he proof of he prevous secon. Frs, we smplfy uh,) usng he normalzaon propery and ge uh,) = Y Y hx,y ) Y ) In conras o he prevous secon, uh,) [ Y,] for hx,y ) [0,], whch we wll ake no accoun for he convexy argumen: plerr = ) D ) hx,y )e α/2 + h x,y ))e α /2 Y )) 2) 98

11 MULICLASS BOOSIG FOR WEAK CLASSIFIERS Seng he frs dervave wh respec o α o zero leads o 2 Y ) Y )r α = ln Y r where we defned r := = D )h x,y ). ) Upper bound on he pseudo-loss error: ow we plug α n 2) and ge ) Y )/ Y ) ) r r Y ) / Y plerr r + r ). 3) r Y ) r ) Soppng creron: As expeced for r = / Y he correspondng facor s. he soppng creron U 0 can be drecly ranslaed no r / Y. Lookng a he frs and second dervave of he bound one can easly verfy ha has a unque maxmum a r = / Y. herefore, he bound drops as long as r > / Y. oe agan ha snce r = / Y s a unque maxmum we ge a formal decrease of he bound even when r > / Y. v) Updae rule: ow we smplfy he updae rule usng ) and nser he new choce of α and ge D + ) = D ) e α h x,y ) / Y ) Z for ), ) Y )r α := ln r Also he goal of he base classfer can be smplfed, because maxmzng U s equvalen o maxmzng r. We wll see n he nex secon, ha he resulng algorhm s a specal case of he algorhm BoosMA of he nex chaper wh c = / Y. 3. BoosMA he am behnd he algorhm BoosMA was o fnd a smple modfcaon of AdaBoos.M n order o make work for weak base classfers. he orgnal dea was nfluenced by a frequenly used argumen for he explanaon of ensemble mehods. Assumng ha he ndvdual classfers are uncorrelaed, majory vong of an ensemble of classfers should lead o beer resuls han usng one ndvdual classfer. hs explanaon suggess ha he wegh of classfers ha perform beer han random guessng should be posve. hs s no he case for AdaBoos.M. In AdaBoos.M he wegh of a base classfer α s a funcon of he error rae, so we red o modfy hs funcon so ha ges posve, f he error rae s less han he error rae of random guessng. he resulng classfer AdaBoos.MW showed good resuls n expermens Ebl and Pfeffer, 2002). Furher heorecal consderaons led o he more elaborae algorhm whch we call BoosMA whch uses confdence-raed classfers and also compares he base classfer wh he unnformave rule. In AdaBoos.M2, he samplng weghs are ncreased for nsances for whch he pseudo-loss exceeds /2. Here we wan o ncrease he weghs for nsances, where he base classfer h : 99

12 EIBL AD PFEIFFER X Y [0,] performs worse han he unnformave or wha we call he maxlabel rule. he maxlabel rule labels each nsance as he mos frequen label. As a confdence-raed classfer, he unnformave rule has he form maxlabel rule : X Y [0,] : hx,y) := y, where y s he number of nsances n he ranng se wh label y. So seems naural o nvesgae a modfcaon where he updae of he samplng dsrbuon has he form D + ) = D ) e α h x,y ) c) Z, wh Z := = D )e α h x,y ) c), where c measures he performance of he unnformave rule. Laer we wll se c := y Y ) 2 y and jusfy hs seng. Bu up o ha pon we le he choce of c open and jus requre c 0,). We now defne a performance measure whch plays he same role as he pseudo-loss error. Defnon Le c be a number n 0,). A classfer f : X Y [0,] makes a maxlabel error n classfyng an example x wh label k, f fx,k) < c. he maxlabel error for he ranng se s called mxerr: mxerr := = Ifx,y ) < c). he maxlabel error couns he proporon of elemens of he ranng se for whch he confdence fx,k) n he rgh label s smaller han c. he number c mus be chosen n advance. he hgher c s, he hgher s he maxlabel error for he same classfer f ; herefore o ge a weak error measure we se c very low. For BoosMA we choose c as he accuracy for he unnformave rule. When we use decson sumps as base classfers we have he propery hx,y) [0,]. By normalzng α,...,α, so ha hey sum o one, we ensure fx,y) [0,] Equaon 5). We presen he algorhm BoosMA n Fgure 4 and n wha follows we jusfy and esablsh some properes abou. As for GrPloss he modus operand consss of fndng an upper bound on mxerr and mnmzng he bound wh respec o α. ) Bound of mxerr n erms of he normalzaon consans Z : Smlar o he calculaons used o bound he pseudo-loss error we begn by boundng mxerr n erms of he normalzaon consans Z : We have = = D + ) = s Z s s= D ) e α h x,y ) c) Z =... e α sh s x,y ) c) = s Z s e fx,y ) c s α s ). 200

13 MULICLASS BOOSIG FOR WEAK CLASSIFIERS Inpu: ranng se S = {x,y ),...,x,y ); x X, y Y }, Y = {,..., Y }, weak classfcaon algorhm of he form h : X Y [0,]. Oponally : number of boosng rounds Inalzaon: D ) =. For =,..., : ran he weak classfcaon algorhm h wh dsrbuon D, where h should maxmze If r c: goo oupu wh := r = D )h x,y ) Se α = ln ) c)r. c r ) Updae D: D + ) = D ) e α h x,y ) c) Z. where Z s a normalzaon facor chosen so ha D + s a dsrbuon) Oupu: ormalze α,...,α and se he fnal classfer Hx): Hx) = argmax y Y fx,y) = argmax y Y α h x,y) Fgure 4: Algorhm BoosMA ) So we ge Usng Z = e fx,y ) c α ). 4) fx,y ) < c e fx,y ) cα ) > 5) α and 4) we ge mxerr Z. 6) ) Choce of α : ow we bound Z and hen we mnmze, whch leads us o he choce of α. Frs we use he defnon of Z and ge Z = D )e α h x,y ) c) 20 ). 7)

14 EIBL AD PFEIFFER ow we use he convexy of e α h x,y ) c) for h x,y ) beween 0 and and he defnon and ge mxerr = r := D )h x,y ) D ) h ) x,y )e α c) + h x,y ))e α c r e α c) + r )e α c ). We mnmze hs by seng he frs dervave wh respec o α o zero, whch leads o ) c)r α = ln. c r ) ) Frs bound on mxerr: o ge he bound on mxerr we subsue our choce for α n 7) and ge mxerr c)r c r )) c ) ) c r ) h x,y ) D ). 8) c)r ow we bound he erm c r ) c)r ) h x,y ) by use of he nequaly x a a+ax for x 0 and a [0,], whch comes from he convexy of x a for a beween 0 and and ge c r ) c)r ) h x,y ) h x,y )+h x,y ) c r ) c)r. Subsuon n 8) and smplfcaons lead o r c mxerr r ) c ) c) c c c. 9) he facors of hs bound are symmerc around r = c and ake her maxmum of here. herefore f r > c s vald he bound on mxerr decreases. v) Exponenal decrease of mxerr: o prove he second bound we se r = c+δ wh δ 0, c) and rewre 9) as mxerr δ ) c + δ c. c c) We can bound boh erms usng he bnomal seres: all erms of he seres of he frs erm are negave, we sop afer he erms of frs order and ge δ ) c δ. c 202

15 MULICLASS BOOSIG FOR WEAK CLASSIFIERS he seres of he second erm has boh posve and negave erms, we sop afer he posve erm of frs order and ge hus Usng +x e x for x 0 leads o + δ c) c +δ. mxerr δ 2 ). mxerr e δ2. We summarze he foregong argumen as a heorem. heorem 3 If all base classfers h wh h x,y) [0,] fulfll r := D )h x,y ) c+δ for δ 0, c) and he condon c 0,)) hen he maxlabel error of he ranng se for he algorhm n Fgure 4 fulflls r c mxerr r ) c ) c) c c c e δ2. 20) Remarks:.) Choce of c for BoosMA: snce we use confdence-raed base classfcaon algorhms we choose he ranng accuracy for he confdence-raed unnformave rule for c, whch leads o c = ) 2 y. 2) y = = y y ;y =y = y Y 2.) For base classfers wh he normalzaon propery 0) we can ge a smpler expresson for he pseudo-loss error. From we ge y k fx,y) = y k α h x,y) = α h x,k)) = α fx,k) fx,k) < Y fx,y) fx,k) < y k α Y. 22) ha means ha f we choose c = / Y for BoosMA he maxlabel error s he same as he pseudoloss error. For he choce 2) of c hs s he case when he group proporons are balanced, because hen c = y Y ) 2 ) y 2 = = Y y Y Y Y 2 = Y. For hs choce of c he updae rule of he samplng dsrbuon for BoosMA ges D + ) = D ) ) e α h x,y ) / Y ) Y )r and α = ln, Z r 203

16 EIBL AD PFEIFFER whch s jus he same as he updae rule GrPloss for decson sumps. Summarzng hese wo resuls we can say ha for base classfers wh he normalzaon propery, he choce 2) for c of BoosMA and daa ses wh balanced labels, he wo algorhms GrPloss and BoosMA and her error measures are he same. 3.) In conras o GrPloss he algorhm does no change when he base classfer addonally fulflls he normalzaon propery 0) because he algorhm only uses h x,y ). 4. Expermens In our expermens we focused on he derved bounds and he praccal performance of he algorhms. 4. Expermenal Seup o check he performance of he algorhms expermenally we performed expermens wh 2 daa ses, mos of whch are avalable from he UCI reposory Blake and Merz, 998). o ge relable esmaes for he expeced error rae we used relavely large daa ses conssng of abou 000 cases or more. he expeced classfcaon error was esmaed eher by a es error rae or 0-fold cross-valdaon. A shor overvew of he daa ses s gven n able. Daabase Labels Varables Error Esmaon Labels car * CV unbalanced dgbreman es error balanced leer es error balanced nursery * CV unbalanced opdgs es error balanced pendgs es error balanced samage * es error unbalanced segmenaon CV balanced waveform es error balanced vehcle CV balanced vowel es error balanced yeas * CV unbalanced able : Properes of he daabases For all algorhms we used boosng by resamplng wh decson sumps as base classfers. We used AdaBoos.M2 by Freund and Schapre 997), BoosMA wh c = y Y y /) 2 and he algorhm GrPloss for decson sumps of Secon 2.3 whch corresponds o BoosMA wh c = / Y. For only four daabases he proporons of he labels are sgnfcanly unbalanced so ha GrPloss and BoosMA should have greaer dfferences only for hese four daabases marked wh a *). As dscussed by Bauer and Kohav 999) he ndvdual samplng weghs D ) can ge very small. Smlar o was done here, we se he weghs of nsances whch were below 0 0 o

17 MULICLASS BOOSIG FOR WEAK CLASSIFIERS We also se a maxmum number of 2000 boosng rounds o sop he algorhm f he soppng creron s no sasfed. 4.2 Resuls he expermens have wo man goals. From he heorecal pon of vew one s neresed n he derved bounds. For he praccal use of he algorhms, s mporan o look a he ranng and es error raes and he speed of he algorhms DERIVED BOUDS Frs we look a he bounds on he error measures. For he algorhm AdaBoos.M2, Freund and Schapre 997) derved he upper bound Y )2 ε ε ) 23) on he ranng error. We have hree dfferen bounds on he pseudo-loss error of Grploss: he erm Z 24) whch was derved n he frs par of he proof of heorem 2, he gher bound 9) of heorem 2 and he bound 3) for he specal case of decson sumps as base classfers. In Secon 3, we derved wo upper bounds on he maxlabel error for BoosMA: erm 24) and he gher bound 20) of heorem 3. For all algorhms her respecve bounds hold for all me seps and for all daa ses. Bound 23) on he ranng error of AdaBoos.M2 s very loose even exceeds for egh of he 2 daa ses, whch s possble due o he facor Y able 2). In conras o he bound on he ranng error of AdaBoos.M2, he bounds on he pseudo-loss error of GrPloss and he maxlabel error of BoosMA are below for all daa ses and all boosng rounds. In ha sense, hey are gher han he bounds on he ranng error of AdaBoos.M2. As expeced, bound 3) derved for he specal case of decson sumps as base classfers on he pseudo-loss error s smaller han bound 9) of heorem 2 whch doesn use he normalzaon propery 0) of he decson sumps. For boh GrPloss and BoosMA, bound 24) s he smalles bound snce conans he fewes approxmaons. For BoosMA, erm 24) s a bound on he maxlabel error and for GrPloss erm 24) s a bound on he pseudo-loss error. For unbalanced daa ses, he maxlabel error s he harder error measure han he pseudo-loss error, so for hese daa ses bound 24) s hgher for BoosMA han for GrPloss. For balanced daa ses he maxlabel error and he pseudo-loss error are he same. Bound 9) for GrPloss s hgher for hese daa ses han bound 20) of BoosMA. hs suggess ha bound 9) for GrPloss could be mproved by more sophscaed calculaons COMPARISO OF HE ALGORIHMS ow we wsh o compare he algorhms wh one anoher. Snce GrPloss and BoosMA dffer only for he four unbalanced daa ses, we focus on he comparson of GrPloss wh AdaBoos.M2 and make only a shor comparson of GrPloss and BoosMA. For he subsequen comparsons we ake 205

18 EIBL AD PFEIFFER AdaBoos.M2 GrPloss BoosMA ranng error [%] pseudo-loss error [%] maxlabel error [%] Daabase rerr BD23 plerr BD24 BD3 BD9 mxerr BD24 BD20 car * dgbreman leer nursery * opdgs pendgs samage * segmenaon vehcle vowel waveform yeas * able 2: Performance measures and her bounds n percen a he boosng round wh mnmal ranng error. rerr, BD23: ranng error of AdaBoos and s bound 23); plerr, BD24,BD3,BD9: pseudo-loss error of GrPloss and s bounds 24), 3) and 9); mxerr, BD24, BD20: maxlabel error of BoosMA and s bounds 24) and 20). all error raes a he boosng round wh mnmal ranng error rae as was done by Ebl and Pfeffer 2002). Frs we look a he mnmum acheved ranng and es error raes. he heory suggess AdaBoos.M2 o work bes n mnmzng he ranng error. However, GrPloss seems o have roughly he same performance wh maybe AdaBoos.M2 leadng by a slgh edge ables 3 and 4, Fgure 5). he dfference n he ranng error manly carres over o he dfference n he es error. Only for he daa ses dgbreman and yeas do he ranng and he es error favor dfferen algorhms able 4). Boh he ranng and he es error favor AdaBoos.M2 for sx daa ses and GrPloss for four daa ses wh wo draws able 4). Whle GrPloss and AdaBoos.M2 were que close for he ranng and es error raes, hs s no he case for he pseudo-loss error. Here, GrPloss s he clear wnner agans AdaBoos.M wh egh wns and four draws able 4). he reason for hs mgh be he fac ha bound 3) on he pseudo-loss error of GrPloss s gher han bound 23) on he ranng error of AdaBoos.M2 able 2). For he daa se nursery, bound 3) on he pseudo-loss error of GrPloss 0.5%) s smaller han he pseudo-loss error of AdaBoos.M2.9%). So for hs daa se, bound 3) can explan he superory of GrPloss n mnmzng he pseudo-loss error. Due o he fac ha only four daa ses are sgnfcanly unbalanced, s no easy o assess he dfference beween GrPloss and BoosMA. GrPloss seems o have a lead regardng he ranng and es error raes ables 3 and 5). For he expermens, he consan c of BoosMA was chosen as he ranng accuracy for he confdence-raed unnformave rule 2). For he unbalanced daa ses, hs c exceeds / Y, whch s he correspondng choce for GrPloss 22). A change of c maybe even adapvely durng he run could possbly mprove he performance. We wsh o make furher 206

19 MULICLASS BOOSIG FOR WEAK CLASSIFIERS ranng error es error Daabase AdaM2 GrPloss BoosMA AdaM2 GrPloss BoosMA car * dgbreman leer nursery * opdgs pendgs samage * segmenaon vehcle vowel waveform yeas * able 3: ranng and es error a he boosng round wh mnmal ranng error; bold and alc numbers correspond o hgh>5%) and medum>.5%) dfferences o he smalles of he hree error raes GrPloss vs. AdaM2 Daabase rerr eserr plerr speed car * o o o + dgbreman leer nursery * opdgs o o o - pendgs samage * segmenaon - - o + vehcle vowel - - o + waveform yeas * oal able 4: Comparson of GrPloss wh AdaBoos.M2: wn-loss-able for he ranng error, es error, pseudo-loss error and speed of he algorhm +/o/-: wn/draw/loss for GrPloss) nvesgaons abou a sysemac choce of c for BoosMA. Boh algorhms seem o be beer n he mnmzaon of her correspondng error measure able 5). he small dfferences beween GrPloss and BoosMA occurrng for he nearly balanced daa ses can no only come from he small 207

20 EIBL AD PFEIFFER car dgbreman leer nursery samage vowel opdgs segmenaon pendgs vehcle waveform yeas Fgure 5: ranng error curves: sold: AdaBoos.M2, dashed: GrPloss, doed: BoosMA dfferences n he group proporons, bu also from dfferences n he resamplng sep and from he paron of a balanced daa se no unbalanced ranng and es ses durng cross-valdaon. Performng a boosng algorhm s a me consumng procedure, so he speed of an algorhm s an mporan opc. Fgure 5 ndcaes ha he ranng error rae of GrPloss s decreasng faser han he ranng error rae of AdaBoos.M2. o be more precse, we look a he number of boosng rounds needed o acheve 90% of he oal decrease of he ranng error rae. For 0 of he 2 daa ses, AdaBoos.M2 needs more boosng rounds han GrPloss, so GrPloss seems o lead o a faser decrease n he ranng error rae able 4). Besdes he number of boosng rounds, he me for he algorhm s also heavly nfluenced by he me needed o consruc a base classfer. In our program, ook longer o consruc a base classfer for AdaBoos.M2 because he mnmzaon of he pseudo-loss whch s requred for AdaBoos.M2 s no as sraghforward as he maxmzaon of r requred for GrPloss and BoosMA. However, he me needed o consruc a base classfer srongly depends on programmng deals, so we do no wsh o over-emphasze hs aspec. 208

21 MULICLASS BOOSIG FOR WEAK CLASSIFIERS GrPloss vs. BoosMA Daabase rerr eserr plerr mxerr speed car * o - nursery * + + o o + samage * o yeas * oal able 5: Comparson of GrPloss wh BoosMA for he unbalanced daa ses: wn-loss-able for he ranng error, es error, pseudo-loss error, maxlabel error and speed of he algorhm +/o/-: wn/draw/loss for GrPloss) 5. Concluson We proposed wo new algorhms GrPloss and BoosMA for mulclass problems wh weak base classfers. he algorhms are desgned o mnmze he pseudo-loss error and he maxlabel error respecvely. Boh have he advanage ha he base classfer mnmzes he confdence-raed error nsead of he pseudo-loss. hs makes hem easer o use wh already exsng base classfers. Also he changes o AdaBoos.M are very small, so one can easly ge he new algorhms by only slgh adapon of he code of AdaBoos.M. Alhough hey are no desgned o mnmze he ranng error, hey have comparable performance as AdaBoos.M2 n our expermens. As a second advanage, hey converge faser han AdaBoos.M2. AdaBoos.M2 mnmzes a bound on he ranng error. he oher wo algorhms have he dsadvanage of mnmzng bounds on performance measures whch are no conneced so srongly o he expeced error. However he bounds on he performance measures of GrPloss and BoosMA are gher han he bound on he ranng error of AdaBoos.M2, whch seems o compensae for hs dsadvanage. References Ern L. Allwen, Rober E. Schapre, Yoram Snger. Reducng mulclass o bnary: A unfyng approach for margn classfers. Machne Learnng, :3 4, Erc Bauer, Ron Kohav. An emprcal comparson of vong classfcaon algorhms: baggng, boosng and varans. Machne Learnng, 36:05 39, 999. Caherne Blake, Chrsopher J. Merz. UCI Reposory of machne learnng daabases [hp:// mlearn/mlreposory.hml]. Irvne, CA: Unversy of Calforna, Deparmen of Informaon and Compuer Scence, 998 homas G. Deerrch, Ghulum Bakr. Solvng mulclass learnng problems va error-correcng oupu codes. Journal of Arfcal Inellgence Research 2: , 995. Günher Ebl, Karl Peer Pfeffer. Analyss of he performance of AdaBoos.M2 for he smulaed dg-recognon-example. Machne Learnng: Proceedngs of he welfh European Conference, 09 20,

22 EIBL AD PFEIFFER Günher Ebl, Karl Peer Pfeffer. How o make AdaBoos.M work for weak classfers by changng only one lne of he code. Machne Learnng: Proceedngs of he hreenh European Conference, 09 20, Yoav Freund, Rober E. Schapre. Expermens wh a new boosng algorhm. Machne Learnng: Proceedngs of he hreenh Inernaonal Conference, 48 56, 996. Yoav Freund, Rober E. Schapre. A decson-heorec generalzaon of onlne-learnng and an applcaon o boosng. Journal of Compuer and Sysem Scences, 55):9 39, 997. Venkaesan Guruswam, Am Saha. Mulclass learnng, boosng, and error-correcng codes. Proceedngs of he welfh Annual Conference on Compuaonal Learnng heory, 45 55, 999. Llew Mason, Peer L. Barle, Jonahan Baxer. Drec opmzaon of margns mproves generalzaon n combned classfers. Proceedngs of IPS 98, , 998. Llew Mason, Peer L. Barle, Jonahan Baxer, Marcus Frean. Funconal graden echnques for combnng hypoheses. Advances n Large Margn Classfers, , 999. Ross Qunlan. Baggng, boosng, and C4.5. Proceedngs of he hreenh aonal Conference on Arfcal Inellgence, , 996. Gunnar Räsch, Bernhard Schölkopf, Alex J. Smola, Sebasan Mka, akash Onoda, Klaus R. Müller. Robus ensemble learnng. Advances n Large Margn Classfers, , 2000a. Gunnar Räsch, akash Onoda, Klaus R. Müller. Sof margns for AdaBoos. Machne Learnng 423): , 2000b. Rober E. Schapre, Yoav Freund, Peer L. Barle, Wee Sun Lee. Boosng he margn: A new explanaon for he effecveness of vong mehods. Annals of Sascs, 265):65 686, 998. Rober E. Schapre, Yoram Snger. Improved boosng algorhms usng confdence-raed predcons. Machne Learnng 37: ,