CROSS ENTROPY METHOD FOR MULTICLASS SUPPORT VECTOR MACHINE

Transcription

1 CROSS ENTROPY METHOD FOR MULTICLASS SUPPORT VECTOR MACHINE Bud Sanosa Deparmen of Indusral Engneerng Insu Teknolog Sepuluh Nopember (ITS) Surabaya ITS Campus Sukollo, Surabaya 60 Indonesa ABSTRACT In hs paper, an mporance samplng mehod cross enropy mehod s presened o deal wh solng suppor ecor machnes (SVM) problem for mulclass classfcaon cases. Usng one-agans-res (OAR) and one-agans-one (OAO) approaches, seeral bnary sm classfers are consruced and combned o sole mulclass classfcaon problems. For each bnary SVM classfer, he cross enropy mehod s appled o sole dual Lagrange SVM opmzaon problem o fnd he opmal or a leas near opmal soluon whch s Lagrange mulplers, n he feaure space hrough kernel map. For he meanme only RBF kernel funcon s nesgaed nensely. Expermens are done on four real world daa ses. The resuls show one-agans-res produces beer resuls han one-agans-one n erms of compung me and generalzaon error. In addon, applyng cross enropy mehod on mulclass SVM produces comparable resuls o he sandard quadrac programmng SVM n erms of generalzaon error. Keywords: cross enropy, generalzaon error, one agans res, one agans one, mulclass, SVM INTRODUCTION The Cross Enropy (CE) mehod s one of he mos sgnfcan deelopmens n sochasc opmzaon and smulaon n recen years. Frsly, CE was proposed as an adape algorhm for rareeen smulaon [4]. I was soon realzed ha he underlyng deas of CE had wder range of applcaon han us rare-een smulaon. In he nex deelopmen, CE could also be appled for solng combnaoral and mul-exremal opmzaon problems [5]. The exenson of CE has been appled n daa mnng such as cluserng, ecor quanzaon, classfcaon and oher applcaons such as relably and resourceconsraned schedulng asks. Cross enropy mehod has been appled o sole dual Lagrange SVM for bnary classfcaon problems [9]. The resuls showed ery promsng performance. In hs paper, cross enropy mehod s presened o deal wh solng mulclass suppor ecor machne problem. Cross enropy also appled on L 0 -norm SVM wh a good generalzaon error whle mnmzng he number of suppor ecors [3]. Two approaches are used o ackle mulclas classfcaon problem: oneagans-res (OAR) and one-agans-one (OAO). By hese wo approaches, seeral bnary sm classfers are consruced and combned o sole mulclass classfcaon problems. For each bnary SVM classfer, he cross enropy mehod s appled o sole dual Lagrange SVM opmzaon problem o fnd he opmal or a leas near opmal soluon, whch s Lagrange mulplers, α. To udge he resuls, he approaches are appled o some real world daases wh mullabel oupu such as Irs, Dermaology, Balance Scale and Glass. The expermens show promsng resuls of OAR approach n erms of compuaonal me and he generalzaon error compared o he sandard mulclass SVM usng quadrac programmng algorhm. Ths paper s organzed as follows. The second secon reews SVM and mulclass classfcaon. Secon 3 recapulaes he CE mehod and descrbes he applcaon of cross enropy mehod n bnary SVM. In secon 4, we descrbe our proposed algorhm and he Malab code. Secon 5 explanes he expermenal seng and dscusses he resuls. In secon 6, we conclude he resuls of hs research.. 2 SVM AND MULTICLASS CLASSIFICATION Suppor Vecor Machnes (SVMs) are an algorhm nroduced by Vapnk. For many years a lo of aenons gen o SVM n erms of her applcaons and how o sole SVM problems hrough dfferen echnques n more effcen manner. SVM s based on he followng dea: npu 99

2 00 The 5 h Inernaonal Conference on Informaon & Communcaon Technology and Sysems pons are mapped o a hgh dmensonal feaure space where a lnear separang hyper plane can be found. To fnd a separang hyper plane, SVMs work by choosng one ha maxmzes he dsance from he closes paerns of wo dfferen classes. Ths dsance s called he margn beween wo classes. Ths s acheed by formulang he problem no a quadrac programmng problem whch s hen usually soled wh opmzaon rounes from numercal lbrares. The mahemacal formulaon of SVM whch fs a quadrac programmng problem, ye sll suffers from hgh compung me, especally for large scale problems. Orgnally, SVM was deeloped on he cases of bnary classfcaon.in he las few years, SVM has been exended o ackle mulclass classfcaon problems. There are wo man approaches for dealng mulclass SVM (MSVM) problems. One s by consrucng and combnng seeral bnary classfers. The oher one s by drecly consderng all daa n one opmzaon formulaon. The frs approach where seeral bnary classfers are consruced and combned appears n wo mehods: one-agans-res (OAR), and one-agans-one (OAO) mehod respecely ([2],[8]). Some research ulzng he second approach s proposed by Vapnk [2] and Weson and Wakns [3]. Hereby, he approach where seeral bnary classfers are consruced and combned s adoped. 2. One-Agans-Res (OAR) Mehod Suppose we are gen m ranng daa (x, y ),.., (x m, y m ) where x R n, =, 2..,m and y S = {,.., k} s he class label of x.. In hs mehod, for he k-class classfcaon problem, we consruc k classfers where k s he number of classes. Le s call our classfer ρ where =,2,..,k. Then, ρ s raned wh all of he examples n he h class wh pose labels (+) and all oher examples wh negae labels (-). In a hree-class classfcaon problem, for example, when we ran ρ, all pons n class are labeled wh + and he oher pons from class 2 and 3 are labeled wh -. Lkewse, when we ran ρ 2 all pons n class 2 are labeled wh + and all oher pons from class and 3 are labeled wh -. We do hs for all =, 2, 3. Then he class of a esng pon x s deermned by he larges alue of he hree decson funcons ealuaed a x. Then he h classfer soles he followng bnary SVM opmzaon problem [4] mn 2 w, b, ( w ) T Subec o w + C = () ( w ) x + b, f y T ( w ) x + b +, f y 0 =,... T =,, Afer solng (), here are k decson funcons w (x) + b,w 2 (x) +b 2,...,w k (x)+b k o consder. Then, he class of pon x s deermned by he larges alue of he decson funcon: = class of x = arg max w x + b, where S. =,.., k 2.2 One-Agans-One (OAO) Mehod Ths mehod consrucs k(k )/2 classfers where each one s raned on daa from wo classes. For example, f we hae a hree-class classfcaon problem we hae o consruc 3 classfers: ρ 2, ρ 3 and ρ 23. When we ran ρ 2, all pons from class are labeled wh + and all pons from class 2 are labeled wh. The same procedure s appled when we ran ρ 3 and ρ 23. For ranng daa from h and h classes, we sole he followng SVM bnary classfcaon problem [2]: T mn ( w ) w + C r w, b, 2 r (2) Subec o T ( w ) xr + b r, f yr =, T ( w ) xr + b + r, f yr =, r 0 Afer all classfers k(k )/2 are consruced, here are dfferen mehods for dong he fuure esng. One sraegy s max-oe. Based on hs sraegy, for classfer ρ f a new pon or paern x s n h class, hen he oe for he h class s added by one. Oherwse, he oe for he h class s ncreased by one. We repea hs sep for all classfers. Then, we predc paern x as beng n he class wh he larges oe. In he case where wo classes hae dencal oes, we selec he one wh he smaller ndex. 3 THE CROSS ENTROPY METHOD Cross enropy s a que new approach n opmzaon and learnng algorhm. In hs secon we oerew he prncples of he CE mehod. Rubnsen [7], Rubnsen and Kroese. [6], de-boer e al. [] prode complee descrpon on cross

3 C7 - Cross enropy mehod for mulclass suppor ecor machne Bud Sanosa 0 enropy mehod.. The basc dea of he CE mehod s o ransform he orgnal (combnaoral) opmzaon problem o an assocaed sochasc opmzaon problem, and hen o handle he sochasc problem effcenly by an adape samplng algorhm. Through hs process one consrucs a random sequence of soluons whch conerges (probablscally) o he opmal or a leas a reasonable soluon. Once he assocaed sochasc opmzaon s defned, he CE mehod follows hese wo phases:. Generaon of a sample of random daa (raecores, ecors, ec.) accordng o a specfed random mechansm. 2. Updae of he parameers of he random mechansm, on he bass of he daa, n order o produce a beer sample n he nex eraon. CE mehod now can be presened as follows. Suppose we wsh o mnmze some cos funcon S(z) oer all z n some se Z. Le us denoe he mnmum by γ*, hus γ * = mn S(z) (3) z Ζ We randomze our deermnsc problem by defnng a famly of auxlary pdfs {f( ; ), V} on Z and we assocae wh Eq. (3) he followng esmaon problem for a gen scalar γ: P u (S(Z) γ) = E u [I {S(Z) γ} ] where u s some known (nal) parameer. We consder he een cos s low o be he rare een I{S(Z) γ} of neres. To esmae hs een, he CE mehod generaes a sequence of uples {( ˆ γ, ˆ )}, ha conerge (wh hgh probably) o a small neghborhood of he opmal uple (γ*, *), where γ* s he soluon of he program (3) and * s a pdf ha emphaszes alues n Z wh a low cos. We noe ha ypcally he opmal * s degeneraed as concenraes on he opmal soluon (or a small neghborhood hereof). Le ρ denoe he fracon of he bes samples used o fnd he hreshold γ. The process ha s based on sampled daa s ermed he sochasc counerpar snce s based on sochasc samples of daa. The number of samples n each sage of he sochasc counerpar s denoed by N, whch s a predefned parameer. The followng s a sandard CE procedure for mnmzaon borrowed from de-boer e al. [] We nalze by seng ˆ 0 = 0 = u and choose a no ery small ρ, say 0 2 ρ. We hen proceed eraely as follows:. Adape updang of γ. For a fxed, le γ be a ρ00%-percenle of S(Z) under. Tha s, γ sasfes P (S(Z) γ ) ρ and P (S(Z) γ ) ρ where Z ~ f( ; ). A smple esmaor γˆ of γ can be obaned by akng a random sample Z(),..., Z(N) from he pdf f( ; ), calculang he performances S(Z(l)) for all l, orderng hem from smalles o bgges as S()... S(N) and fnally ealuang he ρ00% sample percenle as γˆ = S ([ρn]). 2. Adape updang of. For a fxed γ and, dere from he soluon of he program max D( ) = max Ε I { S ( Z ) γ} log f ( Z; ) (4) The sochasc counerpar of (4) s as follows: for fxed γˆ and ˆ, dere ˆ from he followng program: N ˆ ( ) maxd( ) = max I log f ( Z ; ) (5) { S ( Z ) ˆ γ} N = We noe ha f f belongs o he Naural Exponenal Famly (e.g., Gaussan, Bernoull), hen Eq. (5) has a closed form soluon (see de-boer e al.[],rubnsen and Kroese [6]). The CE opmzaon algorhm s summarzed n Algorhm 2. In hs paper we wll assume ha f belongs o a Gaussan famly. In our case Z {0,C} n and s an n dmensonal ecor of numbers beween 0 and, where C s consan defned by users. The consan C s upper bound of Lagrange mulpler αs whch we seek for. The updae formula of he k h elemen n (Eq. (5)) n hs case smply becomes: ˆ ( k) N I { S ( Z ) ˆ γ} = = N = I I k { S ( Z ) ˆ γ} { Z = }. Ths formula has he nerpreaon ha couns how many mes a alue of (n I {Z(l) k =} ) led o a sgnfcan resul (maches wh he ndcaor I {S(Z(l)) γˆ }), how many mes a alue of 0 led o a sgnfcan resul, and normalze he alue of he parameer accordngly. Insead of he updang he parameer ecor drecly a he soluon of Eq. (5) we use he followng smoohed erson ˆ ˆ = β ˆ + ( β), (6) where ˆ s he parameer ecor obaned from he soluon of (5), and β s a smoohng parameer,

4 02 The 5 h Inernaonal Conference on Informaon & Communcaon Technology and Sysems wh 0.7 < β <. The reason for usng he smoohed (6) nsead of he orgnal updang rule s o smooh ou he alues of ˆ, and o reduce he probably ha some componen ˆ, of ˆ wll be zeros or unes a an early sage, and he algorhm wll ge suck n a local maxma. Noe ha for 0 < β < we always hae ha ˆ, > 0, whle for β = one mgh hae (een a he frs eraons) ha eher ˆ = 0 ˆ,, or = for some ndces. As a resul, he algorhm may conerge o a wrong soluon. Fgure 2 prodes he CE mehod for he sochasc opmzaon problem. Algorhm The CE Mehod for Sochasc Opmzaon. Choose some ˆ 0. Se = (leel couner). 2. Generae a sample Z(),...,Z(N) from he densy f( ; ) and compue he sample ρ00%- percenle γˆ of he sample scores. 3. Use he same sample Z(),...,Z(N) and sole he sochasc program (4). Denoe he soluon by ~. ~. 4. Apply (6) o smooh ou he ecor 5. If for some d, say d = 3, γˆ = ˆ γ = = γˆ d hen sop; oherwse se = + and reerae from sep 2. Fgure. A prooypcal CE algorhm. I s found emprcally [6] ha he CE mehod s robus wh respec o he choce of s parameers N, ρ and β, as long as ρ s no oo small, β <, and N s large enough. Typcally hose parameers sasfy ha 0.0 ρ 0., 0.5 β 0.9, and N 3n, where n s he number of parameers. Improemens of Algorhm 2 nclude he Fully Adape CE (FACE) aran, where he parameers N and ρ are updaed onlne. See de-boer e al. [] for more deals. Sanosa [9] has deeloped Cross Enropy mehod for Lagrange Suppor Vecor Machne for bnary cases. The approach prodes a good generalzaon and fas compung ye. In he proposed approach, Cross Enropy was used o sole he dual formulaon of SVM by generang ecor α, whch are Lagrange Mulplers and updang hrough a specfc mechansm o produce mnmum alue of Lagrange problem of SVM. The followng Lagrange problem was soled usng cross enropy o fnd α. Mnmmze m m m (7) L( α ) = y αα K( x, x ) α 2 = = = wh respec o he followng consrans = y α = 0 0 α C =,... m where K(x,x ) s kernel marx and C s consan cos. The followng s he CE algorm for bnary SVM [9]. Algorma 2, Cross Enropy-SVM. Sae he number of samples, cosnan cos C, compue kernel marx of npued daa 2. Defne number of ele sample, Ne 3. Generae random sample as nal α alues and parameers μ and σ 4. Adus α so ha locaed n he neral [0,C] Unl max of σ less hen epslon do seps 5 o Calculae he score of hese α alues by npung o he score funcon (fness funcon). In hs case, use funcon L n Eq (7) 6. Sor hese alues and ake he α alues prodng Ne smalles score of L 7. Compue he mean and sandard deaon of Ne α 8. Updae α based on sep 7 resuls and reurn o sep 4 Fgure. 2. Cross Enropy for bnar y SVM 4 THE PROPOSED METHOD In hs paper, CE s used o sole bnary SVM (Eq (7)) and hen appled on OAR as n Eq.() and OAO as n Eq (2) o sole muclass SVM. If OAR s chosen hen here are k dual Lagrange SVM problems requred o sole. Whle f OAO s chosen, hen here are k(k )/2 dual Lagrange SVM problems needed o sole. The pseudocode code for solng bnary SVM usng Cross Enropy mehod can be found n [9].

5 C7 - Cross enropy mehod for mulclass suppor ecor machne Bud Sanosa 03 The followng s Malab code for one-agans-res and one-agans-one mulclass SVM usng Cross Enropy mehod. funcon [alpha0,yo]=mulsc_enropy(x,y,ker,p ar,c); %Ths code s o compue alpha0 for mulclass CE-SVM %wh one-agans-res approach %npu : x - npu sample % y - npu label % ker- kernel funcon ( rbf ) % par - kernel parameer % C - consan cos % oupu: alpha0 - lagrange mulpler % xsup0=[]; alpha0=[]; b0=[]; s = cpume; Mmax=max(y); %label:, 2, 3,..., M Mmn=mn(y); %label:, 2, 3,..., M M = Mmax-Mmn + ; yo=[]; for =:M yone=(y==)+(y~=)*(-);%changng label o and - yo=[yo yone]; [alpha,ns,b]=sm_cross_enropy(x,yon e,c,ker,par); %call bnary CE-SVM alpha=alpha'; alpha0=[alpha0;alpha]; end; fprnf('compuaon-me: %4.f seconds\n',cpume - s); funcon[alpha0,nbdaa,xsup,ysup]=sco neaone_en(x,y,ker,par,c); % Ths code s o compue alpha0 for mulclass kernel sm % wh one-agans-one approach %npu : x - npu sample % y - npu label % ker- kernel funcon ( rbf ) % par - kernel parameer % C - consan cos % oupu:alpha0 - lagrange mulpler % nbdaa - number of daa for each % classfer % xsup-daa afer all classfers % raned alpha0=[]; xsup=[]; ysup=[]; b0=[]; classfer=[]; Mmax=max(y); %label:, 2, 3,..., M Mmn=mn(y); %label:, 2, 3,..., M M = Mmax-Mmn + ; nbdaa=zeros(,m*(m-)/2); s = cpume; k=; for =:M- for =+:M nd=fnd(y==); nd=fnd(y==); xapp=[x(nd,:); x(nd,:)]; yone=[ones(lengh(nd),);- ones(lengh(nd),)]; [alpha]=sc(xapp,yone,c,ker,par); xsup=[xsup;xapp]; ysup=[ysup;yone]; n2=sze(xapp,); alpha=alpha'; n=sze(alpha,2); nbdaa(k)=n2; alpha0=[alpha0;alpha']; k=k+; end; end; fprnf('compuaon me: %4.f seconds\n',cpume - s); funcon... y=sconeaone_en(xsup,ysup,sx,nbda a,ker,par,alpha0); %Ths s o predc he label of daa pon wh SVM %npu : xsup-from sconeaone ranng % sx - npu sample for esng se % alpha0-from runnng smoneaone % nbdaa - number of daa for each classfer % oupu y: predced label for esng se [n,n2]=sze(sx); nbclass=(+ sqr(+4*2*lengh(nbdaa)))/2; oe=zeros(n,nbclass); k=; nbdaa=[0 nbdaa]; aux=cumsum(nbdaa); for =:nbclass-; for =+:nbclass

6 04 The 5 h Inernaonal Conference on Informaon & Communcaon Technology and Sysems xaux=xsup(aux(k)+:aux(k)+nbdaa(k+ ),:); yaux=ysup(aux(k)+:aux(k)+nbdaa(k+ ),:); alphaaux=alpha0(aux(k)+:aux(k)+nbda a(k+)); H = kernel(sx',xaux',ker,par); [m,n]=sze(h); for rw=:m for r=:n H(rw,r) = yaux(r)*h(rw,r); end end y=(alphaaux'*h'); nd=fnd(y>=0); nd=fnd(y<0); oe(nd,)=oe(nd,)+; oe(nd,)=oe(nd,)+; k=k+; end end [max,y]=max(oe'); y=y'; Afer obanng α from from runnng hs code, for each x n he esng se, we apply he followng classfer funcon f ( x) = yα K( x, x ) SV In he OAR approach, he label of a new paern x s beng defned usng he procedure explaned n Secon 2.. In he OAO approach, he label of a new paern x s beng deermned usng max- oe as explaned n Secon EXPERIMENTS AND RESULTS We apply he cross enropy mulclass SVM on seeral daa ses and usng he wo aforemenoned procedures: One-agans-res (OAR) and One-agans-one (OAO). The daa used for he expermens are four ses aken from real worl daa Irs, Balance Scale, Dermaology, Glass. The deal nformaon abou he daa s shown n Table. Table. Four real world daases Name Feaures Number of class Number of Insances Irs Dermaology Balance Scale Glass The daa ses are spled no wo ses: ranng se and esng se.for each daa se, he rao of ranng sze and he esng sze s abou 70:30. Ten dfferen ranng and esng samples are chosen randomly from each daa se. The expermens are done by usng an RBF kernel. The resuls of he expermens are presened n Tables 2. I conans compung me and aerage of generalzaon error. For comparson purposes, he mulclass SVM (MSVM) usng quadrac programmng soler wh one agans res and one agans one are appled on he same daa ses. Table 2. Compung me (CPU) and Msclassfcaon Error on dfferen daases Irs Dermaology Mehod CE-SVM (OAR) CE-SVM (OAO) MSVM (OAR) MSVM (OAO) Mehod CE-SVM (OAR) CE-SVM (OAO) MSVM (OAR) MSVM (OAO) CPU Error CPU Error Tme Tme Glass Balance Scale CPU Error CPU Error Tme Tme Table 2 shows ha CE-SVM wh OAR producng he bes resuls n erms of generalzaon error and compuaon me for hree daa se, Irs, Dermaology and Glass. Only for Balance Scale daa where mul SVM wh boh OAR and OAO are slghly beer han CE-SVM n erms of generalzaon error. Bu hey need hgher compuaon me han CE-SVM wh OAR. 6 CONCLUSIONS We hae presened an algorhm whch soles mulclass SVM problem by usng cross enropy mehod. Cross Enropy mehod s used o sole bnary dual Lagrange SVM. Two approaches, one agans res (OAR) and one agans one (OAO), are used o fnd maxmum margn classfers n he

7 C7 - Cross enropy mehod for mulclass suppor ecor machne Bud Sanosa 05 feaure space for mulclass classfcaon. One agans res s he bes approach o ackle mulclass classfcaon problems n erms of compuaonal me and generalzaon error usng Cross Enropy mehod. The man adanage of applyng CE on SVM s he generalzaon performance s comparable o SVM whle he compung me s sgnfcanly lower. The mehod does no requre any opmzaon roune o fnd an opmal or near opmal soluon. Tesng on four real world daases proe ha he proposed mehod produced promsng resuls. Applyng on more daases s expeced o srenghen hs concluson. More nesgaon by applyng oher kernel funcons raher han RBF mgh reeal neresng nsgh on applcaon of CE on SVM. Acknowledgmens. Ths research s suppored by Hgher Educaon Drecorae of Indonesa Educaon Mnsry REFERENCES [] De Boer, Peer Terk, D.P. Kroese, She Mannor and Reuen Rubnsen (2005), A Tuoral on he Cross-enropy mehod, Annals of Operaons Research, ol. 34, No., pp [2] Hsu, C.W., and Ln, C.J. (2002)., A comparson of mehods for mul-class suppor ecor machnes, IEEE Transacons on Neural Neworks, 3, [3] Mannor,S. Peleg, D., and Rubnsen, R.Y(2005)., The cross enropy mehod for classfcaon, Proceedngs of he 22 nd Inernaonal Conference on Machne Learnng, Bonn, Germany. [4] Rubnsen, R.Y. (997), Opmzaon of compuer smulaon models wh rare eens, European Journal of Operaons Research, 99,89-2. [5] Rubnsen, R.Y. (999)., The Cross-Enropy Mehod for Combnaoral and Connuous Opmzaon, Mehodology and Compung n Appled Probably, ol, Kluwer Academc Publshers, Boson. [6] Rubnsen, R., & Kroese., D. (2004)., The cross-enropy mehod: A unfed approach o combnaoral opmzaon, Mone-Carlo smulaon, and machne-learnng, Sprnger- Verlag, [7] Rubnsen, R.Y. (2005) Sochasc Mnmum Cross-Enropy Mehod for Combnaoral Opmzaon and Rare-Een, Esmaon Mehodology and Compung n Appled Probably, 7, 5 50, [8] Sanosa, Bud and Trafals, TB. (2007), Compuaonal Opmzaon and Applcaons, 38, Issue 2, [9] Sanosa, Bud (2009). Applcaon of he Cross-Enropy Mehod o Suppor Vecor Machnes, The 5h Inernaonal Conference on Adanced Daa Mnng and Applcaons (ADMA2009), Beng, Augus [0] Schölkopf, B. and Smola, A. (2002). Learnng wh Kernels, The MIT Press, Cambrdge,Massachuses [] Uc Reposory, hp:// ml, (2009) [2] Vapnk, V., (998).Sascal Learnng Theory, Wley, [3] Weson, J. and Wakns,C. (999). Suppor ecor machnes for mul-class paern recognon. In: Proceedng of he Seenh European Symposum on Arcal Neural Neworks, pp [4] Zhou, H., Wang, J. and Qu, Y. (2008). Applcaon of he Cross Enropy Mehod o he Cred Rsk Assessmen n an Early Warnng Sysem, Inernaonal Symposums on Informaon Processng.

8 06 The 5 h Inernaonal Conference on Informaon & Communcaon Technology and Sysems [Ths page nenonally lef blank]