A Novel Ordinal Regression Method with Minimum Class Variance Support Vector Machine

Transcription

1 Intenatonal Confeence on Mateals Engneeng and Infomaton echnology Applcatons (MEIA 05) A ovel Odnal Regesson Method wth Mnmum Class Vaance Suppot Vecto Machne Jnong Hu,, a, Xaomng Wang and Zengx Huang School of Compute and Soft Engneeng, Xhua Unvesty, Chengdu 60039, Chna Key Laboatoy of Patten Recognton and Intellgent Infomaton Pocessng, Chengdu Unvesty, Chengdu 6006, Chna a dewh@hotmal.com Keywods: Machne leanng, Odnal egesson, Suppot vecto machne, Suppot vecto odnal egesson. Abstact. In the pape, we popose a novel odnal egesson method called mnmum class vaance suppot vecto odnal egesson (MCVSVOR). MCVSVOR s deved fom mnmum class vaance suppot vecto machne (MCVSVM) whch s a vaant of SVM, and so nhets the latte s chaactestcs such as takng the dstbuton of the categoes nto consdeaton and good genealzaton pefomance. Fnally, the expemental esults valdate the effectveness of MCVSVOR and ndcate ts supeo genealzaton pefomance ove SVOR.. Intoducton In the pactcal applcatons of machne leanng, a stuaton s fequently nvolved,.e. exhbtng an ode among the dffeent categoes. hs type of supevsed leanng poblems s efeed to as odnal egesson whch pedcts categoes of odnal scale [-4]. Dffeent fom tadtonal metc egesson poblems, ts gades ae usually dscete and fnte. Also, t dffes fom tadtonal classfcaton poblems n that thee s an odnal elatonshp among dffeent classes. In fact, odnal egesson shows esemblance to both egesson and classfcaton because labels ae dscete and odnal []. In the past decade, many methods have been poposed to deal wth the odnal egesson poblems [, 9, 3]. Suppot vecto odnal egesson (SVOR) s a poweful method whch s desgned to tackle the odnal egesson poblems and ognated n suppot vecto machne (SVM). Howeve, SVM s actually a local method n the sense that soluton s exclusvely detemned by suppot vectos wheeas all othe data ponts ae elevant to the decson hypeplane,.e., the SVM soluton does not take nto consdeaton the dstbuton of the categoes and may esult n a non-obust soluton [6]. In ode to ovecome the dawback of SVM, a modfed class of SVM called mnmum class vaance suppot vecto machne (MCVSVM) was pesented n [6]. hs method s nsped fom the optmzaton of Fshe s dscmnant ato [5]. Smla to SVM, MCVSVM mplements the lage magn pncple [5]. Howeve, unlke SVM, the soluton of MCVSVM takes nto consdeaton both the samples n the boundaes and the dstbuton of the categoes and gves a obust soluton. In ths pape, we popose a novel odnal egesson leanng method called mnmum class vaance odnal egesson (MCVSVOR) n whch the dstbuton of the categoes s explctly consdeed and the lage magn pncple s emboded. Followng the basc dea of SVOR, we defne the MCVSVOR optmzaton poblem. Snce MCVSVOR s deved fom MCVSVM, t nhets the latte s chaactestcs such as takng fully the dstbuton of the categoes nto consdeaton and embodyng the lage magn pncple. At the same tme, we also develop the lnea and nonlnea cases of MCVSVOR and analyze the elatonshp between MCVSVOR and SVR. he elatonshp shows that MCVSVOR can be solved usng the exstng SVOR softwae, whch makes the soluton easy to be computed. Fnally, the expemental esults ndcate that MCVSVOR s effectve and can get supeo genealzaton pefomance ove SVOR. 05. he authos - Publshed by Atlants Pess 894

2 . Related wok In ths pape, we consde an odnal egesson poblem wth odeed categoes whch ae denoted by consecutve nteges Y = {,, } to keep the known ank nfomaton. he tanng d dataset s epesented by D = {( x,y ) x R,y Y}, whee categoy, and y epesents the coespondng ank of the nput data pont dmensonalty of sample vecto. he dataset composed of x efes to the th sample n the -th = x. Hee d efes to the = sample ponts. Hee s the numbe of tanng samples n the -th categoy. And, we set X=[ x,, x, x,, x,, x,, x ]=[ x,, x ].. Suppot vecto odnal egesson he task of odnal egesson s to compute a functon f : R {,, } such that f( x ) = y [0, ]. Moeove, SVOR ams at fndng paallel dscmnant hypeplanes wx b = 0 ( =,, ) that sepaate the data ponts of dffeent anks. So, the followng optmzaton poblem s defned [3,4] mn ww+ C ( ξ + ξ ) wb,, ξ, ξ = = st.. wξ b + ξ, ξ 0,, wξ ξ, ξ 0,, b b,, b Whee =, and =,,. Hee, two auxlay vaables b 0 = and b = + ae ntoduced. ote, SVM mplements the lage magn pncple [4]. So, SVOR also embodes the pncple snce t s deved fom SVM.. Kenel dscmnant leanng fo odnal egesson Fo the above gven tanng dataset, the wthn-class scatte matx SW s defned as [5, ] S = ( x u )( x u ) () W = x X Whee X = { x,,, y = = }, denotes vecto tanspose. Hee, optmzaton [] mn ws w C w W st +.. w ( u u ), =,,, u = x s the mean sample vecto of X, and x X s the cadnalty of () X. KDLOR defnes the followng (3) 3. Mnmum class vaance suppot vecto odnal egesson Followng the dea of SVOR, n the lnea case we defne the pmal optmzaton poblem of MCVSVOR as follows wξ b + ξ, ξ 0,, mn ws Ww+ C ( ξ + ξ ) st.. wξ b ξ, ξ 0,, (4) wb,, ξ, ξ = = b b,, Whee =,, =,,, and S W s the wthn-class scatte matx whch s defned as (). Smla to MCVSVM, by ths way, the dstbuton of the categoes s taken fully nto consdeaton. 895

3 Besdes, the poposed method embodes the lage magn pncple snce t s deved fom MCVSVM whch mplements lage magn pncple [5]. So, t s dffeent fom KDLOR although they both take the dstbuton of the categoes nto consdeaton. Smla to SVOR, the pmal optmzaton poblem of MCVSVOR can be effcently solved by ts dual optmzaton poblem. Obvously, (4) s a quadatc pogammng poblem. he pmal Lagangan (4) s L= wsww+ C ( ξ + ξ ) α ( + ξ wξ + β) = = = = (5) + α ( + ξ + wξ b ) βξ β ξ γ ( b b ) = = = = = = = α = [,, ], α = [,, ], β = [,, ], β = Whee the vectos α α α α β β [ β,, β ] and γ = [ γ,, γ ] ae the Lagangan multples fo the constants of (4). By dffeentatng wth espect to w, ξ, ξ and b and usng the Kaush-Kuhn-ucke (KK) condtons, the followng holds L = SWw ( α α ) ξ = 0 w = = = C α b = 0,, ξ (6) = C α 0,, b = ξ = ( α + γ ) + ( α + γ ) = 0, b = = If the matx S W s nonsngula o nvetble, we have W ( α α ) = = w = S x (7) As n MCVSVM and KDLOR, MCVSVOR may encounte the sngulaty poblem of S W snce ts nvese matx s necessay, whch often occus n the case whee the numbe of samples s smalle than the dmensonalty of the samples. o solve ths sngulaty poblem, smla to KDLOR, we can employ the egulazaton method [5, 6, 7] whch s to add a constant ρ > 0 to the dagonal elements of S W as SW = SW + ρi, whee I s an dentty matx. he optmum value of ρ can be estmated though a coss valdaton method. By eplacng (6) nto (5) and usng the KK condtons, the constant optmzaton poblem (4) s efomulated to the Wolf dual poblem ' ' mn ( α α )( α α )( x ) S x ( α + α ) α, α ' ' W, ' ',, st..0 α C,, + 0 α C,, α γ α γ γ = = + = +, 0, Whee uns ove,,. hs s a convex quadatc pogammng poblem and smla to the dual optmzaton poblem of SVOR. Suppose { α, α, γ } s the soluton of the above optmzaton poblem, w s obtaned fom (7), and so the dscmnant functon value fo a new nput vecto x s (8) 896

4 W W = = = = f ( x ) = wx = S ( α α ) x x = ( α α )( x ) S x (9) hus, the pedctve odnal decson functon s gven by mn ag{ : f( x ) < b} (0) 4. Expements 4. Synthetc dataset As s shown n Fg., the synthetc dataset ncludes thee odnal categoes and each categoy conssts n 00 samples. In ths expement, the kenel functon k( xy, ) = exp( γ x y ) s adopted. he expemental esult s llustated n Fg.. It s can be found that the samples can be aanged odely by the hypeplane geneated by MCVSVOR,.e., the samples wth the same ank s classfed n same bn by MCVSVOR. he expemental esult valdates the effectveness of the poposed method Illustaton of the decson hypeplane geneated by MCVSVOR 4. Benchmak datasets In ode to evaluate the pefomance of the poposed method, n ths secton the expemental esults on seveal benchmak datasets, whch wee used n [4] and [], wll be epoted. A summay of the chaactestcs of the selected datasets ae pesented n able. Fo each dataset, the taget values wee dscetzed nto ten odnal quanttes usng equal-fequency bnnng. Each dataset was andomly pattoned nto tanng/test splts as specfed n able. he pattonng was epeated 0 tmes ndependently. he nput vectos wee nomalzed to zeo mean and unt vaance, coodnate-wse. able Chaactestcs of the selected datasets. Datasets o. of Attbutes o. of anng Samples o. of est Samples Pydmnes Machne CPU Boston Housng Abalone Bank Compute Calfona Census

5 5. Concluson In ths pape, we popose a novel odnal egesson method called MCVSVOR. Dffeent fom tadtonal SVOR whch s obtaned by extendng SVM to tackle the odnal egesson poblems, the poposed method s deved fom MCVSVM and nhets ts mets such as good obustness and genealzaton ablty. he expemental esults ndcate the effectveness of MCVSVOR by compang t wth the tadtonal egesson methods SVOR and KDLOR. Refeences [] J. S. Cadoso, R. Sousa, Classfcaton models wth global constants fo odnal data, n: ICMLA, 00, pp [] W. Chu, Z. Ghahaman, Gaussan pocesses fo odnal egesson, Jounal of Machne Leanng Reseach 6 (005) [3] W. Chu, S. S. Keeth, ew appoaches to suppot vecto odnal egesson, n: Poceedng of Intenatonal Confeence on Machne Leanng (ICML-), 005, pp [4] W. Chu, S. S. Keeth, Suppot vecto odnal egesson, eual Computaton 9 (3) (007) [5] R. O. Duda, P. E. Hat, D. G. Stok, Patten Classfcaton (Second Edton), ew Yok: Wley, 00. [6] Y. Guo,. Haste, R. bshan, Regulazed lnea dscmnant analyss and ts applcaton n mcoaays, Bostatstcs 8 () (007) [7] R. Hebch,. Gaepel, K. Obemaye, Suppot vecto leanng fo odnal egesson, n: Intenatonal Confeence on Atfcal eual etwoks, 999, pp [8] M. Kadzńsk, S. Geco, R. Słowńsk, Robust odnal egesson fo domnance-based ough set appoach to multple ctea sotng, Infomaton Scences 83 () (04) -8. [9] Y. Lu, Y. Lu, K. C. Chan, Odnal egesson va manfold leanng, n: AAAI, 0, pp [0] A. Shashua, A. Levn. Rankng wth lage magn pncple: two appoaches, In: Advances n eual Infomaton Pocessng Systems 5, 003, pp [] S. K. Shevade, W. Chu, Mnmum enclosng sphees fomulatons fo suppot vecto odnal egesson, n: Sxth Intenatonal Confeence on Data Mnng, 006, pp: [] B. Y. Sun, J. L, D. D. Wu, X. M. Zhang, W. B. L, Kenel dscmnant leanng fo odnal egesson, IEEE ansactons on Knowledge and Data Engneeng (6) (00) [3] V. oa, J. Domngo-Fee, J. M. Mateo-Sanz, M. g, Regesson fo odnal vaables wthout undelyng contnuous vaables, Infomaton Scences 76 (4) (006) [4] V. Vapnk, he atue of Statstcal Leanng heoy. ew Yok: Spnge Velag, 995. [5] M. Wang, F. L. Chung, S.. Wang, On mnmum class localty pesevng vaance suppot vecto machne, Patten Recognton 43 (8) (00) [6] S. Zafeou, A. efas, I. Ptas, Mnmum class vaance suppot vecto machnes, IEEE ansanctons on Image Pocessng 6 (0) (007)