Regularzed Mulple-Crera Lnear Programmng Va Second Order Cone Programmng Formulaons Zhquan Q Research Cener on Fcous Economy & Daa Scence Chnese Academy of Scences Bejng 0090 Chna qzhquan@gucas.ac.cn Yngje Tan Research Cener on Fcous Economy & Daa Scence Chnese Academy of Scences Bejng 0090 Chna yj@gucas.ac.cn Yong Sh Research Cener on Fcous Economy & Daa Scence Chnese Academy of Scences Bejng 0090 Chna ysh@gucas.ac.cn ABSTRACT Regularzed mulple-crera lnear programmng (RMCLP model s a new powerful mehod for classfcaon and has been used n varous real-lfe daa mnng problems. So far curren RMCLP mplcly assumes he ranng daa o be known exacly. However n pracce here are usually many measuremen and sascal errors n he ranng daa. In hs paper we propose a Robus Regularzed Mulple- Crera Lnear Programmng (called R-RMCLP va second order cone programmng formulaons for classfcaon. Prelmnary numercal expermens show he robusness of our mehod. Caegores and Subjec Descrpors H..8 [Informaon Sysems]: Daabase Managemen Daa mnng General Terms Theory Algorhms Performance Keywords RMCLP; second order cone programmng; SVM. INTRODUCTION For he las decade he researchers have exensvely developed varous opmzaon echnques o deal wh classfcaon problem n daa mnng or machne learnng. Suppor Vecor Machne(SVM [8 6] s one of he mos popular mehods. However Applyng opmzaon echnques o solve classfcaon has seveny years hsory. Lnear Dscrmnan Analyss(LDA[8] was frs proposed n 936. Mangasaran [0] has proposed a large margn classfer based on correspondng auhor correspondng auhor Permsson o make dgal or hard copes of all or par of hs work for personal or classroom use s graned whou fee provded ha copes are no made or dsrbued for prof or commercal advanage and ha copes bear hs noce and he full caon on he frs page. To copy oherwse o republsh o pos on servers or o redsrbue o lss requres pror specfc permsson and/or a fee. DM-IKM Augus - 6 0 Bejng Chna Copyrgh 0 ACM 978--4503-55-7//08...$5.00. lnear programmng n 960 s. From 980aŕs o 990aŕs Glover proposed a number of lnear programmng models o solve dscrmnan problems wh a small sample sze of daa [9 0]. Sh and hs colleagues[] exend Glover s mehod no classfcaon va mulple crera lnear programmng (MCLP and hen varous mproved algorhms were proposed one afer he oher [5 4 6 8 3 9 9 3 4 3]. These mahemacal programmng approaches o classfcaon have been appled o handle many real world daa mnng problems such as cred card porfolo managemen [6 4] bonformacs [30 7] fraud managemen [] nformaon nruson and deecon [5] frm bankrupcy[7] and ec. For he mehods menoned above he parameers n he ranng ses are mplcly assumed o be known exacly. However n real world applcaons he parameers have perurbaons snce hey are esmaed from he daa subjec o measuremen and sascal errors[]. Goldfarb e al. poned ou ha he soluons o opmzaon problems are ypcally sensve o parameer perurbaons errors n he npu parameers end o ge amplfed n he decson funcon whch ofen resuls n msclassfcaon. For an nsance for he fxed examples orgnal dscrmnans can correcly separae hem(see Fg (a. When each example s allowed o move n a sphere orgnal decson funcon canno separae samples n he wors case (see Fg (b. So he our goal s o explore a robus model whch can deal wh daa se wh measuremen or sascal errors[ 5](see Fg (c. In hs paper we proposed a Robus formulaon of Regularzed Mulple Crera Lnear Programmng(called Robus- RMCLP whch s represened as a second-order cone programng(socp. Ths mehod can deal wh daa wh measuremen nose and oban a robus decson funcon whch s an useful exenson of RMCLP[5]. The remanng pars of he paper are organzed as follows. Secon nroduces he basc formulaon of MCLP and RMCLP; Secon 3 descrbes n deal our proposed Algorhms: Robus-RMCLP; All expermen resuls are shown n secon 4; In he las secon he conclusons are gven.. REGULARIZED MCLP FOR DATA MIN- ING We gve a bref nroducon of MCLP n he followng.
Fgure : (a The orgnal examples and dscrmnans; (b The effec of measuremen noses; (c The resul of robus model. For classfcaon abou he ranng daa T = {(x y (x l y l } (R n Y l ( where x R n y Y = { } = l daa separaon can be acheved by wo oppose objecves. The frs objecve separaes he observaons by mnmzng he sum of he devaons (MSD among he observaons. The second maxmzes he mnmum dsances (MMD of observaons from he crcal value[0]. However s dffcul for radonal lnear programmng o opmze MMD and MSD smulaneously. Accordng o he concep of Pareo opmaly we can seek he bes rade-off of he wo measuremens[6 4]. So MCLP model can be descrbed as follows: mn uv Ce T ξ ( e T ξ ( ( s.. (w x (ξ ( = b for { y = } (3 (w x +(ξ ( = b for { y = }(4 ξ ( ξ ( 0 (5 where CD > 0 and e R l be vecor whose all elemens are w and b are unresrced u s he overlappng and v he dsance from he ranng sample x o he dscrmnaor (w x = b (classfcaon separang hyperplane. AloofemprcalsudeshaveshownhaMCLP sapowerful ool for classfcaon. However we canno ensure hs model always has a soluon under dfferen knds of ranng samples. To ensure he exsence of soluon recenly Sh e al. proposed a RMCLP model by addng wo regularzed ems wt Hw and ξ(t Qξ ( on MCLP as follows(more heorecal explanaon of hs model can be found n [5]: mn z wt Hw + ξ(t Qξ ( +Ce T ξ ( e T ξ ( (6 s.. (w x (ξ ( = b for { y = } (7 (w x +(ξ ( = b for { y = }(8 ξ ( ξ ( 0 (9 where z = (w T ξ (T ξ (T b T R n+l+l+ H R n n Q R l l are symmerc posve defne marces. Obvously he regularzed MCLP s a convex quadrac programmng. Compared wh radonal SVM we can fnd ha he RMCLP model s smlar o he Suppor Vecor Machne model n erms of he formaon by consderng mnmzaon of overlappng of he daa. However RMCLP res o measure all possble dsances v from he ranng samples x o separang hyperplane whle SVM fxes he dsance as (hrough boundng planes (w x = b± from he suppor vecors. Alhough he nerpreaon can vary RMCLP addresses more conrol parameers han he SVM whch may provde more flexbly for beer separaon of daa under he framework of he mahemacal programmng. In addon dfferen wh SVM RMCLP consders all he samples o solve classfcaon problem. These make RMCLP have sronger nsensvy o oulers. 3. ROBUST REGULARIZED MULTIPLE CRI- TERIA LINEAR PROGRAMMING(ROBUST- RMCLP 3. Lnear Robus-RMCLP We frsly gve he formal represenaon of robus classfcaon learnng problem. Gven a ranng se T = {(X y (X l y l } (0 where y Y = { } = l and npu se X s a sphere whn r radus of he x cener: X = { x x = x +r u } = l u ( x s he rue value of he ranng daa u R n r s a gven consan. The goal s o nduce a real-valued funcon y = sgn(g(x ( o nfer he label y correspondng o any example x n R n space. Generally Such problem s caused by measuremen errors where r reflecs he measuremen accuracy. In order o oban he opmzaon decson funcon of (0 we mnmze he maxmum msclassfcaon of each examples whn her correspondng confdence balls. In hs case we choose HQ o be deny marx and add slack varable ξ (3 (6 (9 can be wren as he followng robus opmzaon problem: mn wbξ ( ξ ( ξ (3 w + ξ( +C = ξ ( = ξ ( (3 s.. y ((w (x +r u b = ξ ( (4 u = l ξ ( ξ ( 0 = l (5
where C D are respecvely gven consans. Snce mn{y r (w u u } = r w (6 problem (3 (5 can be convered o mn wbξ ( ξ ( ξ (3 w + ξ( +C = ξ ( = ξ ( (7 s.. y ((w x b r w = ξ ( (8 u = l ξ ( ξ ( 0 = l. (9 By nroducng new varables and seng w ξ ( The above problem becomes mn wbξ ( ξ ( ξ (3 + +C = ξ ( = ξ ( (0 s.. y ((w x b r = ξ ( ( u = l ( ξ ( ξ ( 0 = l. (3 w ξ (. (4 For replacng nhe objecve funcon(0 we nroduce new varables u u v v and sasfy he lnear consrans u + v = = and second order cone consrans +v u. Therefore problem (0 (4 can be reformulaed as he followng Second Order Cone Program (SOCP: (ξ (j mn z (u v+ (u v+c = ξ ( = ξ ( (5 s..y ((w x b r = ξ ( (6 u = l (7 ξ ( ξ ( 0 = l. (8 u +v = (9 u +v = (30 +v u (3 +v u (3 w ξ ( (33 where z = (w T bξ (T ξ (T u u v v T ξ (j = ξ(j l j = he penaly parameers CD > 0. Now we derve he dual problem of problem (5 (33. By nroducng s Lagrange funcon L = (u v+ (u v+c = = ξ ( = ξ ( α (y ((w x b r ξ ( +ξ ( = η ( ξ ( = η ( ξ ( β (u +v β (u +v z u u z v v γ z u u z v v γ z z T ww z z T ξ (ξ( (34 where αη ( η ( R n and β β z u z v γ z u z v γ z z wz z ξ ( R n are lagrange mulplers. In he followng we seek for he mnmum value of L abou wbξ ( ξ ( u u v v separaely: u L = 0 u L = 0 v L = 0 v L = 0 b L = 0(35 w L = 0 ξ ( L = 0 ξ (L = 0. (36 From (35 (36 we ge β +z u = β +zu = (37 β +z v = β +zv = (38 y α = 0 (39 = α y x = z w (40 = α r = γ +z γ +z = 0 (4 = C +α η ( z ξ ( = 0 = l (4 D +α +η ( = 0 = l. (43 Aferdelengvarablesz z z wz ξ (η ( hedualproblem of (5 (33 can be reformulaed as max z β +β (44 s..β +z u = β +zu = (45 β +z v = β +zv = (46 y α = 0 (47 = γ r α l = = j= α α jy y j(x x j (48 γ l (C +α η ( (49 = γ +z v z u (50 γ +z v z u (5 η ( 0 = l (5 where z = (β β z u z v γ z u z v γ α T η (T T. Theorem 3.. Suppose ha z s a soluon abou he dual problem (44 (5 where z = (ββ z u zv γz u zv γα T η (T T. If here exss ξ ( j = 0 we wll oban
he soluon (w b o he prmal problem(5 (33: w γ = l = α y x (53 (γ α r b = = γ (γ l = α r α y (x x j+y jr jγ. (54 = Proof. Inroduce he dual problem s lagrange funcon L = β β ( α r γ w T ( y α x ( γ = = = ξ ( (C +α η ( +u (β +z u + u (β +z u +v(β +zv + +v(β +zv + +b( y α = = ρ η ( z z u z z v z γ γ z 3z u z 4z v z γ γ. (55 Accordng o he KKT condons n he nfne-dmensonal space [3] we know ha here exs lagrange mulplers sasfyng: u v η ( r y (w x ξ ( +b y = 0 (56 +u +v = 0 +u +v = 0 (57 ξ ( ρ = 0 (58 = z z z γ u v = z 3 z 4 z γ (59 β +z u = β +z u = (60 β +zv = β +zv = (6 l = yα = 0 (6 0 ρ 0 ρ η ( = 0 = l (63 ( T ( l w = α r γ l = α y x ( w ( ξ ( ( ξ ( u v u v T = 0 (64 L n+ R w R n (65 T ( γ C +α η ( = 0 (66 L n+ R ξ ( R n (67 z u z v T γ z u z v γ = 0 = 0 u v u v L 3 (68 L 3 (69 Fromabovecondons(56 (69 weeaslygeha(w T b ξ (T ξ (T u u v v T s a feasble soluon o problem (5 (33. Snce ( w ( ξ ( L n+ L n+ ( l = α r γ l = α y x ( γ C +α η ( accordng o Lemma5 of [] we can oban L n+ (70 L n+ (7 ( l = α r γ+ l = α y (w x = 0 (7 ( l = α y l x +( = α r γw = 0 (73 ( γ +(ξ ( T (C +α η ( = 0 (74 (C +α η ( +( γ ξ ( = 0. (75 Furhermore accordng o Lemma5 of [] (68 (69 are equvalen o he followng formulas u zu +vz v + γ = 0 (76 ( u( z v γ +zu v = 0. (77 By (7 (77 (57 (60 and (6 we can evenually ge u = zu v = zv = γ (78 w γ = α y x. (79 (γ α r = = By (56 (69 (78 and (79 we can ge (u v + (u v +C = β +β. = ξ ( ξ ( = = (80 Equaon (80 shows ha he objec funcon value of prmal problem(5 (33abou(w T bξ (T ξ (T u u v v T s equal o he one of dual problem (44 (5 abou (β β z u z v γ z u z v γ α T η (T T. Accordng o Theorem4n[7] wecange(w T bξ (T ξ (T u u v v T s he soluon o he prmal problem. On he oher hand accordng o dualy heory we can oban ha w sheunquesoluonoprmalproblem(5 (33. Hence equaon (79 s proved. If here exs a ranng pon x j classfed correcly hen ξ ( j = 0 accordng o (56 we oban he correspondng b = γ (γ l = α r α y (x x j+y jr jγ. (8 = Now we are n a poson o esablsh he followng Algorhm based on he heorem above. 3. Nonlnear Robus-RMCLP The above dscusson s resrced o he lnear case. Here we wll analyze nonlnear Robus-RMCLP by nroducng
Algorhm Lnear Robus-RMCLP Algorhm Nonlnear Robus-RMCLP Inalze: Gven a ranng se (0; Choose approprae penaly parameers C D > 0; Choose Q and H o be deny marxes; Process:. Solve he dual problem (44 (5 and ge s soluon (β β z u z v γ z u z v γ α T η (T T. Compung b : Choose a ranng pon classfed correcly jj l and Compue b = γ (γ = α r α y (x x j+y jr jγ; = Oupu: Oban he decson funcon f(x = sgn( Gaussan kernel funcon γ (γ = α r α y (x x+b. = K(xx = exp( x x /σ (8 where σ s a real parameer and he correspondng ransformaon: x = Φ(x (83 where x H H s he Hlber space. So he ranng se (33 becomes T = {(X y (X l y l } (84 where X = {Φ( x x s n he sphere of he radus r and he cener x }. So when x x r we have Φ( x Φ(x = (Φ( x Φ(x (Φ( x Φ(x (85 where = K( x x K( x x +K(x x (86 = exp( x x /σ (87 r (88 r = exp( x x /σ. (89 Thus X becomes a sphere of he cener Φ(x and he radus r X = { x x Φ(x r }. (90 Ths leads o he followng algorhm. 4. NUMERICAL EXPERIMENT Our algorhm code was wroe n MATLAB 00. The expermen envronmen: Inel Core I5 CPU GB memory. The SeDuM sofware s employed o solve he second cone programmng problem relaed o hs paper. hp://sedum.e.lehgh.edu/ We only modfy Algorhm as follows:. Inroduce o Gauss kernel funcon K(xx = exp( x x /σ. The nner produc (x x j and (x x n Algorhm are replaced by K(x x j and K(x x;. r n he second order cone Programmng (5 (33 becomes r = exp( x x /σ. To demonsrae he capables of our algorhm we repor resuls on sx daa ses from he UCI daases. They are respecvely Hepas BUPA lver Hear-Salog Hear-c Voes and WPBC. In he expermens he daases are normalzed o and. For smplcy we se all r n ( o be a consan r. The nose u s generaed randomly from he normal dsrbuon and scaled on he un sphere. we add many noses for he daases by x = x +ru. The numercal resuls are gven n Table and Fg.. The esng accuraces for our mehod are compued usng sandard 0-fold cross valdaon [6]. The lnear kernel parameer C and he RBF kernel parameer σ s seleced from he se { = 7 7}((CD n RMCLP and R-RMCLP models are also seleced n he same range by 0-fold cross valdaon on he unng se comprsng of random 0% of he ranng daa. Once he parameers are seleced he unng se was reurned o he ranng se o learn he fnal decson funcon. For comparson he resuls correspondng o he orgnal RMCLP and Robus SVM[] are also lsed n hese ables. From he resuls of Table and Fg. we can fnd ha he performance of he Robus-RMCLP and Robus SVM s conssenly beer han ha of he orgnal RMCLP. The accuracy of Robus-RMCLP s comparable wh Robus SVM. In addon wh he ncrease of he nose he accuracy for he orgnal model s much lower han he Robus model. 5. CONCLUSION In hs paper a new Robus Regularzed Mulple Crera Lnear Programmng has been proposed. All expermens n daases wh noses show ha he performance of he robus model s beer han ha of he orgnal model. However snce Robus-RMCLP need o solve Second Order Cone Programmng s compung speed s slower han orgnal RMCLP. In he fuure work we wll develop more effcen robus algorhm for classfcaon. 6. ACKNOWLEDGMENT Ths work has been parally suppored by grans from Naonal Naural Scence Foundaon of Chna( NO.70906 NO.060064 he CAS/SAFEA Inernaonal Parnershp Program for Creave Research Teams Major Inernaonal(Ragonal Jon Research Projec(NO.700706 he Presden Fund of GUCAS and he Naonal Technology Suppor Program 009BAH4B0. 7. ADDITIONAL AUTHORS
Fgure : The percenage of enfold esng correcness for daases wh noses n he case of lnear kernel. Table : The percenage of enfold esng correcness for daases wh noses n he case of rbf kernel Daase Ins Dm Model r 0.0 0.0 0.03 0.04 0.05 RMCLP 0.788 0.7768 0.7754 0.753 0.754 Hepas 55 9 R-SVM 0.789 0.78 0.7765 0.76 0.76 R-RMCLP 0.7934 0.790 0.7790 0.768 0.766 RMCLP 0.64 0.6390 0.5980 0.587 0.587 BUPA lver 345 6 R-SVM 0.650 0.6567 0.6488 0.6488 0.6488 R-RMCLP 0.6555 0.6555 0.6547 0.643 0.634 RMCLP 0.80 0.7845 0.7765 0.7549 0.7474 Hear-Salog 70 4 R-SVM 0.83 0.83 0.8080 0.793 0.793 R-RMCLP 0.85 0.847 0.88 0.805 0.8076 RMCLP 0.838 0.84 0.84 0.7865 0.7474 Hear-c 303 4 R-SVM 0.85 0.833 0.833 0.83 0.83 R-RMCLP 0.8467 0.8457 0.834 0.845 0.85 RMCLP 0.9 0.980 0.980 0.8900 0.8876 Voes 435 6 R-SVM 0.93 0.93 0.93 0.948 0.90 R-RMCLP 0.9467 0.9348 0.989 0.956 0.94 RMCLP 0.88 0.7999 0.7865 0.779 0.7436 WPBC 98 34 R-SVM 0.878 0.834 0.890 0.8006 0.7856 R-RMCLP 0.84 0.896 0.8 0.794 0.78 8. REFERENCES [] F. Alzadeh and D. Goldfarb. Second-order cone programmng. Mahmacal Programmng 95:3 5 003. [] A. Ben-al and A. Nemrovsk. Robus convex opmzaon. Mahemacs of Operaons Research 3:769 805 998. [3] J. Borwen. Opmzaon wh respec o paral orderngs. Jesus College Oxford Unversy 974. [4] W. Chen and Y. Tan. Kernel regularzed mulple crera lnear programmng. pages 345 35. 3rd Inernaonal Symposum on Opmzaon and Sysems Bology 009. [5] V. K. Chopra and W. T. Zemba. The effec of errors n means varances and covarances on opmal porfolo choce. The Journal of Porfolo Managemen 9(:6 993. [6] N. Deng and Y. Tan. Suppor vecor machnes: Theory Algorhms and Exensons. Scence Press Chna 009. [7] L. Faybusovch and T. Tsuchya. Prmal-dual algorhms and nfne-dmensonal jordan algebras of fne rank. Mahmacal Programmng 97:47 493 003. [8] R. A. Fsher. The Use of Mulple Measuremens n
Taxonomc Problems. Annals of Eugencs 7:79 88 936. [9] N. Freed and F. Glover. Smple bu powerful goal programmng models for dscrmnan problems. European Journal of Operaonal Research 7(:44 60 98. [0] N. Freed and F. Glover. Evaluang alernave lnear programmng models o solve he wo-group dscrmnan problem. Decson Scence 7:5 6 986. [] D. Goldfarb and G. Iyengar. Robus porfolo selecon problems. Mahemacs of Operaons Research 8: 38 00. [] D. Goldfarb and G. Iyengar. Robus convex quadracally consraned programs. Mahemacal Programmng 97:495 55 00. [3] G. Kou X. Lu Y. Peng Y. Sh M. Wse and W. Xu. Mulple crera lnear programmng o daa mnng: Models algorhm desgns and sofware developmens. Opmzaon Mehods and Sofware 8 (4:453 473 003. [4] G. Kou Y. Peng Y. Sh M. Wse and W. Xu. Dscoverng cred cardholdersaŕ behavor by mulple crera lnear programmng. Annals of Operaons Research 35 (:6 74 005. [5] G. Kou Y. Peng and N. Yan. Nework nruson deecon by usng mulple-crera lnear programmng. volume 7 of Inernaonal Conference on Servce Sysems and Servce Managemen pages 9 004. [6] G. Kou Y. Sh and S. Wang. Mulple crera decson makng and decson suppor sysems - gues edor s nroducon. Decson Suppor Sysems 5:47 49 0. [7] W. Kwak Y. Sh S. W. Eldrdge and G. Kou. Bankrupcy predcon for japanese frms: usng mulple crera lnear programmng daa mnng approach. Inernaonal Journal of Busness Inellgence and Daa Mnng (4:40 46 006. [8] A. L Y. Sh and J. He. A daa classfcaon mehod based on fuzzy lnear programmng 008. [9] J. L L. We G. L and W. Xu. An evoluon-sraegy based mulple kernels mul-crera programmng approach: The case of cred decson makng. Decson Suppor Sysem 5(:9-98 0. [0] O. L. Mangasaran and D. R. Muscan. Generalzed suppor vecor machnes. In Advances n Large Margn Classfers pages 35 46 000. [] D. Olson and Y. Sh. Inroducon o busness daa mnng. Irwn-McGraw-Hll Seres: Operaons and Decson Scences. McGraw-Hll 006. [] Y. Peng G. Kou A. Sabaka J. Maza Z. Chen D. Khazanch and Y. Sh. Applcaon of classfcaon mehods o ndvdual dsably ncome nsurance fraud deecon. In Y. Sh G. D. van Albada J. Dongarra and P. M. A. Sloo edors Inernaonal Conference on Compuaonal Scence (3 volume 4489 of Lecure Noes n Compuer Scence pages 85 858. Sprnger 007. [3] Y. Peng G. Kou Y. Sh and Z. Chen. A mul-crera convex quadrac programmng model for cred daa analyss. Decson Suppor Sysems 44:06 030 March 008. [4] Y. Sh Y. Peng W. Xu and X. Tang. Daa Mnng va Mulple Crera Lnear Programmng: Applcaons n Cred Card Porfolo Managemen. Inernaonal Journal of Informaon Technology and Decson Makng (:3 5 Mar. 00. [5] Y. Sh Y. Tan X. Chen and P. Zhang. Regularzed mulple crera lnear programs for classfcaon. Scence n Chna Seres F: Informaon Scences 5(0:8 80 009. [6] Y. Sh W. Wse and M. Lou. Mulple crera decson makng n cred card porfolo managemen. pages 47 436. Mulple Crera Decson Makng n New Mllennum 00. [7] Y. Sh X. Zhang J. Wan Y. Wang and W. Yn. Predcng he dsance beween anbodyaŕs nerface resdue and angen o recognze angen ypes by suppor vecor machne. Inernaonal Journal of Compuer Mahemacs 84:690 707 004. [8] V. N. Vapnk. The Naure of Sascal Learnng Theory. Sprnger New York 996. [9] D. Zhang T. Yngje and Y. Sh. Knowledge-ncorporaed mclp classfer. Proceedngs of Conference on Mul-crera Decson Makng 008. [30] J. Zhang W. Zhuang and N. Yan. Classfcaon of hv- medaed neuronal dendrc and synapc damage usng mulple crera lnear programmng. Neuronformacs :303 36 004. [3] Y. Zhang P. Zhang and Y. Sh. Kernel-based mulple crera lnear program. Proceedngs of Conference on Mul-crera Decson Makng 008.