Interval Regression Analysis with Reduced Support Vector Machine

Transcription

1 Ieraoal DSI / Asa ad Pacfc DSI 007 Full Paper (July, 007) Ierval Regresso Aalyss wh Reduced Suppor Vecor Mache Cha-Hu Huag,), Ha-Yg ao ) ) Isue of Iforao Maagee, Naoal Chao Tug Uversy (leohkko@yahoo.co.w) ) Depare of Idusral ad Operaos Egeerg, Uversy of Mchga (eresak hk@yahoo.co.w) Absrac The suppor vecor ache (SVM) has bee wdely used paer recogo, regresso ad dsrbuo esao for crsp daa. However, whe dealg wh large-scale daa ses, he soluo by usg SVM wh olear kerels ay be dffcul o fd. Uder hs crcusace, o develop a effce ehod s ecessary. Recely he reduced suppor vecor ache (RSVM) was proposed as a alerave of he sadard SVM. I has bee proved ore effce ha he radoal SVM processg large-scaled daa. I hs paper we roduce he prcple of RSVM o evaluae erval regresso aalyss.. Iroduco Sce Taaka e al. [] roduced he fuzzy regresso odel wh syerc fuzzy paraeers, he properes of fuzzy regresso have bee suded exesvely by ay researchers. A colleco of rece sudes o fuzzy regresso aalyss ca be reached a []. Fuzzy regresso odel ca be splfed o erval regresso aalyss whch s cosdered as he sples verso of possblsc regresso aalyss wh erval coeffces. Soe coeffces erval lear regresso odel ed o becoe crsp because of he characersc of lear prograg (LP) []. To allevae he ssue of LP, Taaka ad Lee [3] propose a erval regresso aalyss wh a uadrac prograg (QP) approach whch gves ore dverse spread coeffces ha a LP oe. The suppor vecor ache (SVM) has bee wdely used paer recogo, regresso ad dsrbuo esao for crsp daa [4-]. Recely, usg SVM o solve he erval regresso odel becoes a alerave approach. Hog ad Hwag roduce SVM for ulvarae fuzzy regresso aalyss [] ad evaluae erval regresso odels wh uadrac loss SVM [3]. Jeg e al. [4] develop a suppor vecor erval regresso eworks (SVIRNs) based o boh SVM ad eural eworks. However, whe dealg wh large-scale daa ses, he soluo based o above ehods ay be dffcul o fd. Uder hs crcusace, o develop a effce ehod s ecessary. The reduced suppor vecor ache (RSVM) has bee proved ore effce ha he radoal SVM processg large-scaled daa [5]. The a dea of RSVM s o reduce he uber of suppor vecors by radoly selecg a subse of saples. I hs paper we roduce he prcple of RSVM o evaluae erval regresso aalyss. Ths paper s orgazed as follows. Seco revews erval regresso aalyss by QP approach ufyg he possbly ad ecessy odels. Seco 3 brefly revews he bass of he heory of RSVM. Seco 4 proposes a forulao of RSVM evaluag he erval regresso odels. Fally, seco 5 gves he cocludg rearks.. Ierval Regresso Aalyss wh QP Approach I hs seco we revew erval regresso aalyss wh uadrac prograg (QP) approach ufyg he possbly ad ecessy odels proposed by Taaka ad Lee [3].

2 A erval lear regresso odel s descrbed as Y (x ) = A + A x + + A x () 0 where Y ( x ), =,,,, s he esaed erval correspodg o he real pu vecor x = ( x,, x ). A erval coeffce A s defed as ( a, c ), where a s he ceer ad c s he radus. Hece, A ca also be represeed as A = [ a c, a + c ] = { a c a a + c} () The erval lear regresso odel () ca also be expressed as Y( x ) = A + Ax + + A x 0 = ( a, c ) + ( a, c ) x + + ( a, c ) x 0 0 = ( a + a x, c + c x ) 0 0 = = (3) For a daa se wh crsp pus ad erval oupus, wo erval regresso odels, he possbly ad ecessy odels, are cosdered. By assupo, he ceer coeffces of he possbly regresso odel ad he ecessy regresso odel are he sae [3]. For hs daa se he possbly ad ecessy esao odels are defed as Y ( x ) = A + Ax + + Ax (4) 0 Y ( x ) = A + A x + + A x (5) 0 where he erval coeffces A ad A are defed as A = ( a, c ) ad A = ( a, c ), respecvely. The erval Y ( x ) esaed by he possbly odel us clude he observed erval Y ad he erval Y ( ) x esaed by he ecessy odel us be cluded he observed erval Y. The followg cluso relaos exs Y ( x ) Y Y ( x ) (6) The erval coeffces A ad A ca be deoed as A = ( a, c + d ) (7) A ( a, c ) (8) whch sasfy he cluso relao A A where c ad d are assued o be posve. Thus, he possbly odel Y ( x ) ad he ecessy odel Y ( x ) ca also be expressed as Y ( x ) = ( a + a x, c + c x + d + d x ) (9) = = = Y ( x ) = ( a + a x, c + c x ) (0) 0 0 = = The erval regresso aalyss by QP approach ufyg he possbly ad ecessy odels subec o he cluso relaos ca he be represeed as follows ( d + ) ( ) 0 d x + ξ a + c = = = 0 s.. Y ( x ) Y Y ( x ), =,,, p c, d 0, = 0,,, ()

3 where ξ s a exreely sall posve uber ad akes he fluece of he er ξ = 0 ( a + c ) he obecve fuco eglgble. The cosras of he cluso relaos are euvale o y e ( a + a x ) ( c + c x ) 0 0 = = Y ( x ) Y ( a a x ) ( c c x ) y e = = () ( a + a x ) ( c + c x ) ( d + d x ) y e = = = Y Y ( x ) y e ( a a x ) ( c c x ) ( d d x ) = = = (3) 3. Reduced Suppor Vecor Mache We ow brefly roduce he bass of he heory of he reduced suppor vecor ache (RSVM) [5]. Suppose ha rag daa { x, y }, =,,,, are gve, where x R are he pu paers ad y {,} are he relaed arge values of wo-class paer classfcao case. The he sadard suppor vecor ache wh a lear kerel [] s wb,, ξ s.. + C ξ = y ( w x + b) ξ ξ 0, =,,, w (4) where b s he locao of hyperplae relave o he org. The regularzao cosa C > 0 s he pealy paraeer of he error er o deere he radeoff bewee he flaess of lear fucos ( wx + b) ad eprcal error. I Lee ad Magasara s approach [5], b / s added o he obecve fuco of (4). Ths s euvale o addg a cosa feaure o he rag daa ad fdg a separag hyperplae hrough he org ( w wb,, ξ b ) C s ξ = y ( w x + b) ξ ξ 0, =,,, (5) Is dual becoes he followg boud-cosraed proble I α ( Q+ + y y) α eα α C s.. 0 α, =,,, (6) where e s he vecor of all oes. Q s a posve se-defe arx, Q y y ( x, x ) ad ( x, x ) φ( x ) φ( x ) s he kerel fuco. I hs case, ( x, x ) x x s a lear kerel. I he opal soluo, w s a lear cobao of rag daa

4 w= yα x (7) = Subsug w o (5) by = α = α = ( α) = = ywx yy x x Q Q (8) w = y x w= Q = α α α (9) The he proble becoes ( α Qα + b ) + C ξ αβξ,, s.. Qα + by e ξ = (0) Dealg wh large-scale daa ses, he a dea of RSVM s o reduce he uber of suppor vecors by radoly selecg a subse of k saples for w w= yα x () where coas dces of he subse of k saples. The proble becoes () by subsug () wh he uber of aor varables reduced o k ( α Q α + b ) + C,,, ξ αβξ = s.. Q α + by e ξ, () where α s he colleco of all α,. Q represes he sub-arx of colus correspodg o., To splfy he er /( α Q α ) o /( α α ) followg he geeralzed suppor vecor ache (GSVM) by, Magasara [7], we oba he RSVM as follows ( α α + b ) + αβξ,, s.., C ξ = Q α + by e ξ (3) For specfc daa ses, a approprae olear appg x φ( x) ca be used o ebed he orgal R feaures o a Hlber feaure space F, φ : R F, wh a olear kerel ( x, x ) φ( x ) φ( x ). The followgs are well-kow olear kerels for regresso probles, where γ, r, h, ad θ are kerel paraeers h () ( γ x x + r) : Polyoal kerel, h, γ > 0 ad r > 0. [] x x () e γ : Gaussa (radal bass) kerel, γ > 0. [6] (3) ah( γ xx + θ ) : Hyperbolc age kerel, γ > 0. [8]

5 4. Ierval Regresso wh RSVM I hs seco we roduce a forulao of RSVM evaluag he erval lear ad olear regresso odels. Wh he prcple of RSVM, we ca forulae he erval lear regresso odel as he followg uadrac proble ( aa+ cc+ dd+ b) + C s.. acd,,, ξ Q a + by e ξ, ax c x y e ax + c x y + e ax c x d x y e ax + c x + d x y + e =,,, ξ = (4) where coas dces of he subse by radoly selecg k saples. a, c, ad d are he collecos of all a, c, ad d,, respecvely. Q represes he sub-arx of colus correspodg o., Gve (4), he correspodg Lagraga obecve fuco s L a a c c d d b C Q a by e : = ( ) + ξ ( ) α + + ξ, = = α ax c x y e α y e ax c x = = ( + ) ( + ) α y e ax c x d x 3 α ax c x d x y e 3 = = ( + + ) ( + + ) (5) Here L s Lagraga ad α, α, α, α, ad α are Lagrage ulplers. The dea o cosruc a Lagrage 3 3 fuco fro he obecve fuco ad he correspodg cosras s o roduce a dual se of varables. I ca be show ha he Lagraga fuco has a saddle po wh respec o he pral ad dual varables he soluo. The arush-uh-tucker (T) codos ha he paral dervaves of L wh respec o he pral varables ( a, c, d, ξ, b ) for opaly = 0 a = α Q ( α α ) ( α α ), + x 3 x 3 a = = = = 0 c = ( α ) ( ) + α x + α α 3 + x 3 c = = = 0 d = ( α α ) 3 + x 3 d = L α = 0 ξ = ξ C (6) = 0 b = α y b = Subsug (6) (5) gves he dual opzao proble

6 ax ( + ( )( ) + ( )( ) s.. αα Q Q α α α α α α α α,, xx xx , =, =, = α Q α α x α Q α α α α α α, x, 3 3 x x 3 3, =, =, = + ( ) ( ) ( )( ) ( α α )( α + α ) x x ( )( ) α + α α + α x x 3 3, =, = ( α + α )( α + α ) x x ) ( ) ( ) α + α α y α α e + (7), = 4C = = = ( α α ) y ( ) 3 3 α α e 3 3 = = + + α, α, α, α, α Slarly, we ca oba he erval olear regresso odel by appg x φ( x) o ebed he orgal R feaures o a Hlber feaure space F, φ : R F, wh a olear kerel ( x, x ) φ( x ) φ( x ) as dscussed Seco 3. Therefore, by replacg xx ad x x (7) wh ( x, x ) ad ( x, x ), respecvely, we oba he dual opzao proble as (8) ax ( αα Q Q + ( α α )( α α ) (, ) ( α α )( α α ) (, ),, x x + x x , =, =, = s.. α Q α α x α Q α α α α α α, x, 3 3 x x 3 3, =, =, = + ( ) ( ) ( )( ) (, ) ( α α α + α x x α + α α + α x x 3 3, =, = + + α α α α x x α α α y e α α, = = = = ( α α ) y ( ) 3 3 α α e 3 3 = = 3 3 )( ) (, ) ( )( ) (, ) + ( + )( + ) (, )) + ( ) ( + ) 4C + + α, α, α, α, α 0 (8) 5. Cocluso Ths paper proposes a RSVM approach evaluag erval regresso odels. The a dea of RSVM s o reduce he uber of suppor vecors by radoly selecg a subse of saples. I hs paper we esae he erval regresso odel wh crsp pus ad erval oupu. I fuure works, boh erval pus-erval oupu ad fuzzy pus-fuzzy oupu wll be cosdered. Ackowledge The auhors apprecae he coes fro he aoyous referees o ehace he readably of hs arcle. Ths sudy s suppored by Tawa Mer Scholarshps, Uversy-based Progra Regulao, No. 94B54 (Cha-Hu Huag) ad Tawa Mer Scholarshps, No. TMS-094--B-009 (Ha-Yg ao). Refereces [] H. Taaka, S. Uea, ad. Asa; Fuzzy Lear Regresso Model, IEEE Trasacos o Syses, Ma ad Cyberecs, Vol. 0 pp , 980. [] J. acprzyk, ad M. Fedrzz; Fuzzy Regresso Aalyss, Physca-Verlag, Hedelberg, 99. [3] H. Taaka, ad H. Lee; Ierval Regresso Aalyss by Quadrac Prograg Approach, IEEE Trasacos o Fuzzy Syses, Vol. 6, pp , 998.

7 [4] C. J. C. Burges; A Tuoral o Suppor Vecor Maches for Paer Recogo, Daa Mg ad owledge Dscovery, Vol., pp -67, 998. [5] C. Cores, ad V. N. Vapk; Suppor Vecor Neworks, Mache Learg, Vol. 0, pp 73-97, 995. [6] H. Drucker, C. J. C. Burges, L. aufa, A. Sola, ad V. N. Vapk; Suppor Vecor Regresso Maches, I: M. Mozer, M. Jorda, T. Pesche, (eds.): Advaces Neural Iforao Processg Syses, Vol. 9, Cabrdge, Massachuses, MIT Press, pp 55-6, 997. [7] O. L.Magasara; Geeralzed Suppor Vecor Maches, I: A. J. Sola, P. L. Barle, B. Schölkopf, D. Schuuras (eds.): Advaces Large Marg Classfers, Cabrdge, Massachuses, MIT Press, pp 35-46, 000. [8] B. Schölkopf, C. J. C. Burges, ad A. J. Sola (eds.); Advaces erel Mehods: Suppor Vecor Learg. Cabrdge, Massachuses, MIT Press, 999. [9] B. Schölkopf, ad A. J. Sola; Learg wh erels: Suppor Vecor Maches, Regularzao, Opzao, ad Beyod, Cabrdge, Massachuses, MIT Press, 00. [0] A. J. Sola, ad B. Schölkopf; A Tuoral o Suppor Vecor Regresso, NeuroCOLT Tech. Repor, NeuroCOLT, 998. Sascs ad Copug, Vol. 4, pp 99-, 004. [] V. N. Vapk; The Naure of Sascal Learg Theory, Sprger-Verlag, New York, 995. [] D. H. Hog, ad C. H. Hwag; Suppor Vecor Fuzzy Regresso Maches, Fuzzy Ses ad Syses, Vol. 38, pp 7-8, 003. [3] D. H. Hog, ad C. H. Hwag; Ierval Regresso Aalyss usg Quadrac Loss Suppor Vecor Mache, IEEE Trasacos o Fuzzy Syses, Vol. 3, pp 9-37, 005. [4] J. T. Jeg, C. C. Chuag, ad S. F. Su; Suppor Vecor Ierval Regresso Neworks for Ierval Regresso Aalyss, Fuzzy Ses ad Syses, Vol. 38, pp , 003. [5] Y. J. Lee, ad O. L. Magasara; RSVM: Reduced Suppor Vecor Maches. I: Proceedgs of s SIAM Ieraoal Coferece o Daa Mg, 00. [6] C. A. Mcchell; Ierpolao of Scaered Daa: Dsace Marces ad Codoally Posve Defe Fucos, Cosrucve Approxao, Vol., pp -, 986.