SMOOTH SUPPORT VECTOR REGRESSION BASED ON MODIFICATION SPLINE INTERPOLATION

Transcription

1 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: OOH SUPPOR VECOR REGRESSION BASED ON ODIFICAION SPLINE INERPOLAION BIN REN, HUIJIE LIU, LEI YANG, LIANGLUN CHENG School of Electroc Egeerg, Doggua Uversty of echology, Doggua, 5388, Cha School of Autoato, Guagdog Uversty of echology, Guagzhou,56, Cha ABSRAC Regresso aalyss s ofte forulated as a optzato proble wth squared loss fuctos. Facg the challege of the selecto of the proper fucto class wth polyoal sooth techques appled to Support Vector Regresso odels, ths study taes three terplato pots sple terpolato techology ad odfcato terpolato value to geerate a ew polyoal sooth fucto -sestve support vector regresso. he eperetal aalyss shows that ad -fucto s better tha p -fucto S -fucto propertes, ad the approato accuracy of the proposed sooth fucto s three order of hgher tha that of classcal p -fucto. Keywords: Support Vector Regresso, -sestve Loss Fucto, Sooth Polyoal Fucto, odfcato Iterpolato. INRODUCION Sooth fucto has bee wdely studed uercal odelg[-6], whch, especally the terest of the authors, has bee successfully appled for classfcato of ad regresso odel fttgs age processed ad patter recogto[, 3, 7, 8]. Applyg sooth fucto to regresso odels eas to deal wth square usoothed ssue -sestve loss fucto whle fttg the regresso odels [8]. Accordg to the basc cocept o how to solve classfcato proble, Lee et al, used p -fucto as to soothly approach the target fucto, ad brought forward the -sestve support vector regresso odel (-SSVR) 5 [8]. her results show that the effect of -SSVR s better tha both lght LIBSV[9] ad SV [] both regresso property ad effcecy. It s, however, stll a ope ad challegg ssue to fd a better sooth fucto [,,5,7,9]. Accordgly ths paper s otve to preset a study o usg three terpolato pots Cubc Sple Iterpolato polyoal ad odfcato terpolato value to prove ths d of sooth fucto fttg support vector regresso odel. he proposed -fucto s better tha p - fucto ad S -fucto property, ad the approato accuracy of the proposed sooth fucto s three order of hgher tha that of classcal that of classcal p -fucto ad oe order of hgher tha p -fucto S -fucto. he sulato case study shows that t proves the regresso effect. hs paper s orgazed as follows: secto troduces regresso probles ad dffcultes. secto 3 troduces -sestve loss fucto ad support vector regresso.i Secto 4, we frst troduce the prcple ad derve forula of Cubc Sple Iterpolato polyoal, the use odfcato Sple Iterpolato polyoal to sooth sgle varable postve fucto, ad we defe s polyoal approato fucto (, ). I Secto 5, we aalyze the perforace of polyoal sooth approato fucto (, ). It s the st-order sooth fucto, ad the approato accuracy s.8/. Secto 6, we ru two uercal sulato eperets by usg data sets fro artfcal database ad UCI database to verfy the valdty of the odel. Fally, we ae a cocluso ad foresee the future wor secto 7.. REGRESSION BASED DAA FIING Frst, we dscuss the splest regresso proble -desoal space: Let s suppose all values,,..., fro to, each s correspodg wth a observed value y. he 9

2 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: purpose s usg the desgated data set to geerate terdepedet fucto y = f ( ). We usually use ths way to solve the proble as below: frst, to restrct the fucto y = f ( ) a sple fucto class advace, the searchg for f ( ) that ca eet the followg codtos the fucto class as ush as possble: y = f ( ), =,, L, () I order to easy to deal wth, we always use lear regresso way,. e., restrctg f ( ) to be lear fucto f ( ) = w + b. he search for f () whch ca eet the equato (). ( y f ( )) = ( y w b) () Equato () s ofte used to easure the devato degree betwee y = f ( ) = w + b ad y = f ( ).he saller value s, the less error s ad hgher effcet t s. So ths process ca be traslated to the followg optzed forula. So that we ca defe w ad b the fucto f ( ) = w + b : w, b y w b (3) = ( ) Obvously, the regressve forula ad soluto above ca be eteded to a oral stuato. Frst, etedg data class () to data set S: S = {(, y ), L,(, y )} R R (4) Secodly, the fucto class whch restrct the fucto y=f () () above also ca be eteded to be a real fucto set F. Geerally, there s ot oly crtero to easure the devato of regresso fucto y = f ( ) fro y = f ( ). We call equato (3) above as quadratc loss fucto. Of course, other loss fucto also ca be used. If we ae loss fucto as c(, y, f ). he optzed forula (3) wll becoe zato forula wth eprcal rs. such as lear fuctos wll produce large regresso error a odel ature of olearty. O the other had, F caot be too large otherwse the regresso fucto wll be eagless. For eaple, we wll obta the followg equato based o data set S (4) whe F s the whole real fucto set. y, = f ( ) = {, =, L,. (6) Obvously, the regresso fucto s too llogcal. Accordgly the ey pot s how to choose the fucto class set F, ether too sple or too coplcated. Furtherore, t becoes dffcult to choose the rght oe for the regresso fucto. 3. SUPPOR VECOR REGRESSION For better aalyss, we defe the -sestve loss fucto of depedet varable X as, = a{, }, as show Fgure. Defto the square of -sestve loss fucto as, ad the postve fucto + as ( + ) = a{, }. Data set S = {(, y ), L,(, y )} R R, defe atr A=[,, ], s desoal vector, each s correspodg wth a observed value, obvously A R, that t s y S = {( A, y ) A R, y R, for =, L, }. he purpose s usg the desgated data set S to geerate a regresso fucto f ( ), let f ( ) predct y ore accurately accordg to the ew put of. he stadard we use s -sestve loss fucto: y f ( ) = a{, y f ( ) } (7) c(, y, f ( )) (5) f F = hus, the terdepedet fucto y = f ( ) ca be obtaed,. e., regresso fucto. Whe solvg the optzed forula(5), the frst ssue s how to choose the fucto class set F. For the desgated oral trag data set S (4), we ca ot restrct F to be too sall fucto class, w + Fgure : Ε-Isestve Loss Fucto For lear regresso case, f ( ) = w + b, where R s a deterate vector, b s a 93

3 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: deterate costat. -sestve lear regresso fucto s show Fgure, we select two hyperplaes of the arg a way ad we call the dstace betwee the two hyperplaes s zoe. Oly by there are o trag pots fallg to the arg, we ca have loss, ad the loss s y f ( ) -. f ( ) f ( ) = w + b + ( ) ( ) f = w + b f = w + b Fgure : Ε-Isestve Lear Regresso Fucto For olear regresso case f ( ) = ωϕ( ) + b, where ϕ( ) : olear fucto. I theory, we ca chage t to lear regresso oes to settle t accordg to erel techque. Stadard regresso proble s to solve the followg u proble []: * * Q( ω, b, ξ, ξ ) = ω ω + C ( ) ξ + ξ = s. t. y ω ϕ( ) b + ξ * ω ϕ( ) + b y + ξ * ξ, ξ, =,,, (8) Where * * * * ξ = ( ξ, ξ,, ξ ), ξ = ( ξ, ξ,, ξ ), ( ) s the au devato allowed durg the trag ad C(>) represets the assocated pealty for ecess devato durg the trag. he slac varables ξ ad ξ *, correspod to the sze of ths ecess devato for postve ad egatve devatos respectvely. he frst ter, ω ω s the regularzed paraeter; thus, t cotrols the fucto capacty; the secod ter * ( ξ + ξ ) =, s the eprcal error easured by the -sestve loss fucto. he coputato of Stadard support vector regresso s ore coplcated, because whe solvg the Optzato proble, you eed to solve quadratc prograg, especally whe the trag saple uber s creased. he soluto wll face curse of desoalty, result that we ca t tra t. Suyes J.A.K [] proposes least squares ethod-support vector aches (LS- SV) to ae the proble coes dow to lear equatos, ad solvg lear equatos s easer ad faster tha the quadratc prograg. Stadard regresso proble s to solve the followg proble: C Q( ω, ξ ) = ω ω + ξ s. t. y = ω ϕ( ) + bξ, =,,, = (9) I addto, Lee et al adds the paraeter b to the obectve fucto to duce strog covety ad to guaratee that the proble has a uque global optal soluto. he regresso ssue ca be epressed by below ucostraed optzed ssue forula [9]: C () ( w w + b ) + A. w + b y + ( w, b) R = Obvously, the forula () s ot dervatve, so ths target fucto s ot dervatve. 4. POLYNOIAL OOH APPROXIAION FUNCION Cubc sple fucto ay geerate sooth terpolato curve by cobg the dscotuous cubes ad the secod dervatve s cotuous at the ot pot, aely saplg pot. 4. atheatcal Descrpto Assupto a set of odes a < <... < b at [ a, b ], f the fucto s() eet below ter[], () s( ) C [ a, b] ; () s( ) s cubc polyoal at every rego [, + ] ( =,,..., ). If s() also eets the followg sple ter at ode, (3) S( ) = f, =,,.... he s( ) s called cubc sple terpolato fucto, the secod dervatve of s( ) at [ a, b ] s cotuous. I ths study, whe usg cubc spe terpolato polyoal approach postve fucto 94

4 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: , at the ed pot a=,b= of rego [a,b], usg followg boudary codtos: S ( ) = f, S ( ) = f, at rego[, + ], the forula of cubc sple fucto s ( ) ( ) h S( ) = + + ( f ) h 6h 6 h + h + ( f+ ) ( =,,, ) () 6 h o solve followg: we ca wrte t atr for as d µ λ d =... λ d d Where: d 6 f [,, ] () = (3) [ ] d = 6 f,, (4) + [ ] d = 6 f,, (5) = h µ h + h h λ = µ = h + h (6) (7) h = (8) 4. he Dervato Of Soothg Process We use the ethod of cubc sple terpolato polyoal to sooth sgle varato fucto at rego,. ae 3 terpolato data fro postve fucto + at rego < = ad >, pot =, + = ad correspodg f =, f + =, + = (>), f + =. Usg cubc sple terpolato polyoal to sooth postve fucto + at rego,, table s terpolato pot, correspodg fucto ad the frst dervatve value. able : Iterpolato Pot Ad Fucto Value -/ / h = h =, µ =, µ =, λ =, λ =, h f / d = 6 ( f [, ] f ) = [ ] d = 6 f,, = d = 6 ( f f [, ]) = h = f he have the aswer =, (9) =, =, so the cubc sple terpolato polyoal for the soothg of sgle varable fucto at rego, s as below: S( ), [, ], [,] = () S( ) +, = ± /, the dfferece value s the largest, a( + S( )) = / 7. I order to ae S( ) greater tha +, the S( ) as a whole oves up to / 7 []. he the approachg fucto of the postve fucto + s 95

5 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: S (, ) , < 7 3 = < <, 7 > () 4.3 he polyoal approachg fucto of loss fucto, fro forula () we have followg cocluso: Defe (, ) s the square of polyoal approachg fucto (, ) for -sestve ( ), + 3 ( ( ) + ( ) + ( ) + ), < < ( ( ) + ( ) + ( ) + ), + < < 7 (, ) =, + 3 ( ( + ) + ( + ) ( + ) + ), < < 7 3 ( ( + ) + ( + ) ( + ) + ), < < 7 ( ), 5. PROPERY ANALYSIS Lea [8] p -fucto p (, ) = ( p(, )) + ( p(, )), where p(, ) = + l( + e ), >, e s the base of atural logarth, t has the followg propertes: () () p -fucto s ay-order sooth w.r.t. ;; p (, ) ; (3) For R ad < ρ + : log ρ p (, ) ( ) + log. Lea [3] S -fucto () ( ), + ( ( ) + ( ) + ), + < < S (, ) =, + ( ( + ) ( + ) + ), < < 4 4 ( ), (3) has the followg propertes: () S s st-order sooth w.r.t.. hat s, at terpolato pots, S ( ± ( + ), ) = S ( ± ( + ), ) = () S (, ) S ( ± ( + ), ) = S ( ± ( + ), ) = ; 96

6 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: (3) S (, ) heore s defed (6), we have: () s st-order sooth w.r.t.. hat s, at terpolato pots, () (3) Proof: S S S ( ± ( + ), ) = (4) 9 4 ( ±, ) = (5) 79 ( ± ( + ), ) = (6) ( ± ( + ), ) = (7) (, ) S ± = (8) 7 S ( ± ( + ), ) = (9) S (, ) (3) (, ) (3) 4 () Accordg to the defto, we ca drectly prove t. () Verfy S (, ) For +, ad, Cocluso s obvously correct. For < < +, defe: g( ) = S (, ), the we have 3 g( ) = ( ( ) + ( ) + ( ) 3 + ) ( ) = ( ( ) ( ) + ( ) + ) 7 3 ( ( ) + ( ) ( ) + ) 7 Defe: 3 3 h ( ) = ( ) + ( ) + ( ) h ( ) = ( ) + ( ) ( ) he we get h ( ) =, for 79 < < h ( ) = ( ) + ( ) + 3 ( ( ) = ) + ( ) >. So h ( ) rego s strctly ootoc creasg at < < +. So for < < +, h ( ) > 4 h ( ) = >, so, 79 h ( ) > he we get h ( + ) = For < < + 3 h ( ) = ( ) + ( ) = ( ( ))( 3 ( )) < So h ( ) s strctly ootoc decreasg at rego < < +. So for h ( ) > h ( ) + =, so, h ( ) >. the < < +. < < +, S (, ) s correct for Slarly, for the case of < <, we have S (, ). Hece, S (, ) (3) Verfy (, ) 4 For +, ad +, the cocluso s obvously correct. For + < <,we have (, ) = (, ),due to s a strctly ootoe creasg fucto for + < < +, so 97

7 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: (, ) (, ) = < ;For 79 4 < <,we have agtude hgher tha that of the the sae K value. S -fucto at (, ) = (, ),due to s a strctly ootoe decreasg fucto for < <,so 4 (, ) (, ) = < ; For 79 4 < < + g( ) = (, ),we have varable substtuto for forula: 3 g( ) = ( ( ) + ( ) + ( ) ) ( ) +, a = ( ) (,),the 7 3 a a g( a) = ( a + + a + ) has 7 au value pot a=.75 at rego <a<,hece g( ) g(.75) <,therefore, 4 the cocluso s correct. Slarly, for the case of < <, we have g( ) g(.75) <. 4 Hece, (, ) EXPERIENAL RESUL I the case of =5,=.3, the sooth fucto approato coparso chart s as Fgure 3, the we ca see that, -fucto has hgher approato accuracy tha fucto wth the sae K value. Whe p -fucto ad p -fucto ad S - S -fucto as sooth fuctos, defe ρ=/, fro Lea ad Lea we have p (, ).3854 / ad S (, ).99 /, able lst the approato accuracy of three sooth fuctos, the we ca see that the approato accuracy of S -fucto s three order of agtude hgher tha that of the p -fucto ad oe order of Fgure 3: Sooth Fucto Approato Coparso Chart I he Case Of K=5 Ad Ε=.3 able: Approato Accuracy Of Sooth Fuctos sooth fucto p - fucto S - fucto - fucto approato.3854 /.99 /.8/ accuracy o further verfy the property of ths sooth fucto applyg to support vector regresso, wo sulated eperets were selected to deostrate the aalytcal results, whch were ru at atlab7. o a persoal coputer wth a AD X4 6 processor ad GB eory. Based o the frst order optalty codtos of ucostraed cove zato proble, our stoppg crtero was satsfed whe the -or of gradet of the obectve fucto s less tha 5.For a observato vector y ad the predcto vector ŷ, the -or relatve error of two vectors y ad ŷ was defed as follows: y yˆ y (3) hs relatve error used to easure the accuracy of regresso. I order to evaluate how well each ethod geeralzed to usee data, we splt the etre data set to two parts, the trag set ad testg set. he trag data was used to geerate the regresso fucto that s learg fro trag data; the testg set, whch s ot volved the trag procedure, was used to evaluate the predcto ablty of the resultg regresso fucto. We also used a stratfcato schee splttg the etre data set to eep the slarty betwee trag ad testg data sets. hat s, we tred to ae the trag set ad the testg set have the slar observato dstrbutos. A saller testg error dcates better predcto 98

8 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: ablty. We perfored tefold cross-valdato o each data set ad reported the average testg error our uercal results. o geerate a hghly olear fucto, a Gaussa erel was used for all olear uercal tests defed as below: µ A A K( A, A ) = e,, =,,3, L, (33) he paraeters µ ad C were detered by a tug procedure. Frst, we selected pots evely fro [-, ] as the put data of the artfcal data sets ad the observato was geerated fro a sple fucto as follows: 3 s( ) f ( ) =.5 π + ρ, (34) 3 π stadard devato σ =.4 ) were added. Slar to the frst eperet, we set =., µ = ad C =. Because of storg the fully dese erel atr requred olear -PSSVR, wll eceed the eory capacty ad the reduced erel techque was appled here. We radoly selected 3 pots whch are slghtly over percet of the etre trag data set to for a reduced set ad used the reduced erel forulato to geerate the olear regresso fucto. he resultg fucto of -PSSVR (a) ad the orgal fucto (b, wthout oses) were show Fgure 5. he dots that were show Fgure 5 for the reduced set. hs result was geerated.36 secods wth. relatve error. We also tested -SSVR o hs artfcal data set. However, they too a uch loger te to get the soluto wth the sae level of accuracy. We suarzed the uercal results of these two artfcal data sets able4. able 3: Nuercal Result For S Fucto ethods #SVs ra Error(%) est Error(%) CPU (s) -PSSVR -PSSVR Fgure 5: he Regresso Of 3D Artfcal Data Sets(A)(B) able4: Nuercal Result For 3D Artfcal Data Sets ra est ethods #SVs CPU(s) Error(%) Error(%) -PSSVR -PSSVR Fgure 4: Regresso Fucto Produced By Sooth Support Vector Regresso Where ρ s a addtve Gaussa ose wth ea= ad stadard devato σ =.4.We set =., whch s oe half stadard devato of the Gaussa ose. he rest of the paraeters, µ = 33 ad C=6, were detered by a tug procedure. he eperetal results show that the -PSSVR has the sallest relatve error. -PSSVR too.6 CPU secods, whle -PSSVR too. secods. We suarzed the results Fgure 4 ad able 3. he secod artfcal data set was obtaed by usg ALAB coad peas (7) to geerate 89 data pots R. Just le our frst eperet, the Gaussa oses (ea= ad 7. CONCLUSIONS I ths paper, we successfully obta the polyoal soothg fucto whch approaches the square of -sestve loss fucto by usg three terpolato pots cubc Sple terpolato ethod, that s -fucto, ad proved that ths fucto has better propertes, the approato accuracy s three order of hgher tha -fucto ad oe order of hgher tha P S -fucto. As a result, to apply -fucto support vector regresso, the uber of support vector s less, CPU te. herefore, we ca provde a ew, better polyoal sooth fucto sooth support 99

9 Joural of heoretcal ad Appled Iforato echology 5 th October. Vol. 44 No. 5 - JAI & LLS. All rghts reserved. ISSN: E-ISSN: vector regresso odel fttg ad other related felds. ACKNOWLEDGEENS hs wor was supported by the Guagdog Natural Scece Foudato Proect(No.S 44),he stry of Educato, Guagdog Provce, Producto ad Research Proects(No.B94457, No.B94 69, No.A98),Guagdog Provce Eterprse Laboratory Proect(No.A9 46), the Natural Scece Foudato of Dog Gua Uversty of echology Proect (No.ZQ4). REFERENCES: [] C. Che ad O. L. agasara, A class of soothg fuctos for olear ad ed copleetarty probles, Coputatoal Optzato ad Applcato, Vol.5, No.5, 996, pp [] Y. B. Yua, J. Ya ad C. X. Xu, Polyoal sooth support vector ache (PSSV), Chese Joural of Coputers, Vol. 8, No., 5, pp [3] C. Che ad O. L. agasara, Soothg ethods for cove equaltes ad lear copleetarty probles, atheatcal Prograg, Vol. 7, 995, pp [4] C. astroneto,. J. Yougseo, J. yog K. et al, AAD predcto usg support vector regresso wth data-depedet paraeters, Epert Systes wth Applcatos, Vol. 36, No. : PAR, 9, pp [5] C. Adreas ad V.. Arout, Boulgad dervatves ad robustess of support vector aches for regresso, Joural of ache Learg Research, Vol. 9, 8, pp [6] L. agasara ad D. R. uscat, Successve overrelaato for support vector aches, IEEE rasactos o Neural Networs, Vol., No. 8, 999, pp Y. J. Lee ad O. L. agasara, SSV: A sooth support vector ache for classfcato, Coputatoal Optzato ad Applcatos, Vol., No.,, pp. 5-. [7] Y. J. Lee, W. F. Hseh ad C.. Huag, - SSVR: A sooth support vector ache for - sestve regresso, IEEE rasactos o Kowledge ad Data Egeerg, Vol. 7, No. 5, 5, pp. 5-. [8] C. C. Chag ad C. J. L, LIBSV:A lbrary for support vector aches, software avalable at [9] J Platt. Sequetal al optzato: A fast algorth for trag support vector aches, Advaces Kerel ethods-support Vector Learg. I Press, Cabrdge, A, 999, PP [] N. Y. Deg ad Y. J. a. he ew ethod data g-support vector ache, Press of scece, 4, pp. 44. [] H. Q. Yua,W. G. u,j,z. Xog, et al, ew polyoal sooth support vector ache, Coputer Scece, Vol. 38, No. 3,, pp [] Re B, Cheg Laglu, Polyoal soothg support vector regresso, Joural of Cotrol heory ad Applcatos. Vol. 8, No.,, pp