On the Relationship between LS-SVM, MSA, and LSA

Transcription

1 IJCSS Iteatioal Joual of Compute Sciece ad etwo Secuity, VOL6 o, ovembe 006 O the Relatioship betwee LS-SVM, MSA, ad LSA Yatog Zhou, aiyi Zhag, ad Lieu Wag Dept of Ifomatio ad Commuicatio Egieeig, Xi a Jiaotog Uivesity, Xi a, P R Chia Summay A commo tas i sigal pocessig is to appoximate adequately a sigal It is cucial to udestad vaious methods to this tas I this pape we suvey thee diffeet appoximatio methods: least squaes suppot vecto machie LS-SVM), multiesolutio sigal appoximatio MSA), ad least squaes appoximatio LSA) to highlight thei mathematical elatioship Based o the theoetical aalysis, we pove that LS-SVM is idetical to the LSA with the miimum om solutio, ad MSA is idetical to the LSA with the least squaes solutio heefoe, both LS-SVM ad MSA ca be deived as specific istaces of LSA Key wods: Sigal pocessig, suppot vecto machie, multiesolutio, appoximatio, least squaes, elatioship Itoductio Sigal appoximatio is of pactical impotace as it is closely elated to the sigal compessio as well as may othe sigal pocessig techiques It has bee studied extesively ove the past decades aditioally, appoximatio methods such as low-ode polyomials appoximatio [], piecewise liea appoximatio [], splie appoximatio [3], eel o poectio based appoximatio [4], ad least squaes appoximatio LSA) [5], etc, have eceived cosideable attetio i the sigal pocessig commuity Howeve, may ew types of appoximatio methods, such as ough sets o eual etwo based appoximatio [6], suppot vecto machies SVM) [7], ad MSA [8] have emeged ecetly that claim to have moe flexibility ad bette appoximatio ability Amog these appoximatio methods developed, SVM ad MSA ae of the most successful methods he foudatio of SVM has gaied populaity due to its may attactive, aalytic ad computatioal featues, ad pomisig empiical pefomace [9] It has bee successfully exteded fom basic classificatio tass to hadle egessio, opeato ivesio, desity estimatio ad ovelty detectio SVM used fo appoximatio is itoduced by Vapi [7] ad futhe ivestigated by may othes LS-SVM is a least squae vesio fo SVM that ivolves equality istead of iequality costaits ad wos with a least squaes cost fuctio [0] his efomulatio geatly simplifies the poblem i such a way that the solutio is chaacteized by a set of liea equatios istead of a quadatic pogammig QP) he equatios ca be efficietly solved by iteative methods such as cougate gadiet method [] Ove the past few yeas, MSA has bee successfully applied to appoximate sigals i may fields It is fomulated based o the study of wavelet aalysis MSA employs a coase-to-fie stategy ad povides a simple hieachical appoximatio of the sigals At diffeet esolutios level, the appoximatio of the sigal chaacteizes diffeet physical stuctues [8] I this pape we loo at appoximatio as the poblem of ecoveig a sigal f fom give data set, beig closest to the udelyig sigal f o expess the cocept of two sigals beig close some measue citeia eed to be defied he most geeal citeio is the miimizatio of the mea squaes eo MSE) LSA is a taditioal ad widely applicable appoximatio method [5] It ca be fomulated i tems of miimizatio of the MSE that eflects the discepacy betwee f ad f Give so may appoximatio methods, it has become difficult fo a egiee to select the most appopiate method fo the poblems ude study So it would be vey meaigful if oe ca ow befoehad how much the solutio of the diffeet appoximatio methods is close to each othe Futhemoe, It is valuable to ivestigate the coectios betwee the taditioal ad cuetly developed appoximatio methods he pimay iteest of this pape lies i descibig the close mathematical elatioship of above-metioed thee appoximatio methods: LS-SVM, MSA, ad LSA Based o the theoetical aalysis, we pove that LS-SVM is idetical to the LSA with the miimum om solutio, ad MSA is equivalet to the LSA with the least squaes solutio heefoe, both LS-SVM ad MSA ca be deived as specific istaces of LSA he emaide of this pape is ogaized as follows Sec ad Sec3 ae bief oveviews of LS-SVM ad MSA Followig that, i Sec4 we eview LSA he elatioship betwee LS-SVM, MSA, ad LSA is poved i Sec5 Fially we give some coclusios i Sec6 Mauscipt eceived Octobe 5, 006 Mauscipt evised Octobe 5, 006

2 IJCSS Iteatioal Joual of Compute Sciece ad etwo Secuity, VOL6 o, ovembe 006 Review o LS-SVM Coside the followig appoximatio poblem see Fig): let t =,, L, ) be a set of samples of a sigal that cotais the udelyig sigal f x) ad oise Assume the sigal f x) satisfactoily appoximates f x sigal ) samples set D { x, t ), =, L }, we ae ased to ecove f x) based o the = t, t x x f x ) x t x t Fig he appoximatio poblem f x) f x) LS-SVM has bee developed fo solvig above poblem It allows the costuctio of a appoximatio by mappig the data set D implicitly ito some featue space F though some mappig ϕ Costuctig a simple liea appoximatio i F the coespods to a oliea appoximatio i sample space R All ca be doe implicitly i F by usig the eel tic K x x) = ϕ x ) ϕ x), LS-SVM aims at costuctig a appoximatio of the fom: f x) = ϕ x) + b x w, ) whee w is weight vecto ad b is bias tem o obtai w ad b oe solves the followig costaied optimizatio poblem mi w C + w, b, e = subect to the equality costaits x ) b ξ ξ w ϕ =,,, L t, ) =, 3) with ξ a eo vaiable, C a egulaizatio facto ad ξ the least squaes cost fuctio By costuctig a Lagage fuctio fom both the obective fuctio ad the coespodig costaits we yields the followig expessio mi G ) C α = w + = ξ = t x ) b ) + α w ϕ ξ whee α is Lagage multiplies he solutio of α ad b is give by 0 L b 0 = C L χ χ+ I α whee = t, t, L t ), = ϕ x), ϕ x), ϕ x )) α =,, L, ), ad = ) χ L, α α α L,, L Fially, we substitute w ito Eq ) ad aive at = ) α, ) f x = K x x + b 3 Multiesolutio Sigal Appoximatio MSA) Suppose thee is a multi-esolutio aalysis i L R ) such that the scale subspaces subspaces W satisfy V = V W, ) V = L R Z U, I V = 0, Z V ad the wavelet he symbol deotes diect sum ad Z is the set of iteges A othogoal compactly suppoted wavelet basis of W is fomed by the dilatio ad taslatio of a ψ, called the mothe wavelet fuctio ad is give by / ) ψ ) ψ x = x Similaly, the othogoal basis of V is give by ) = ) x x

3 IJCSS Iteatioal Joual of Compute Sciece ad etwo Secuity, VOL6 o, ovembe whee is called scalig fuctio he wavelet ad scalig fuctio satisfy the followig dilatio equatio fom ψ + x) = h0 x) + x) = h ψ x) he poectio of a sigal f x) ) Pf x) = c x) oto V has the It ca be see as a smooth appoximatio of f x) esolutio level Hece above equatio ca also be witte as he coefficiet ) f x P f x c x ) = ) = ) ) c is the poectio of f x) basis fuctio x) ; that is = ) ) at 4) o the ) c = f x) x) dx 5) f x x Δx whee Δx is sample ate I pactice Eq 5) has to be ewitte as sice f x ) ) = ) Δ, c t x x value t could be obtaied is coupted by oise thus oly coupted 4 Least Squaes Appoximatio LSA) Whe appoximatig a uow sigal f x), the sigal is usually epeseted by a weighted sum of the liealy idepedet basis fuctios he choice of the basis fuctios may vay depedig o the applicatio Examples of basis fuctios ae splies, complex expoetials, Gaussia adial basis fuctio, o wavelets that have the best time fequecy esolutio LSA is oe of appoximatio method that epesets a udelyig sigal f x x, that is ) o estimate that is usig the basis fuctios { )} q f q P = x) = u P x) = u oe ca miimize the mea squaes eo, mi e x) dx = mi f x) f x) ) dx Usually the set { x )} q P = is complete i L ) 6) R, which implies that Eq 6) ca be made abitaily small by selectig the ode of the appoximatio q popely Eq 6) taes a discete fom as followig mi u q = = u P x ) f x ) Δx, 7) sice we eed to be able to use it o the udelyig sigal f x), which is oly specified by the values it taes at the discete set of poits { x } ae ito accout that = desetig Δx has o ifluece o estimatig u, ad f x ) is coupted by oise thus oly coupted value t could be obtaied, Eq 7) ca be ewitte as q mi u P u x ) t = = 8) 5 Relatioship betwee LS-SVM, MSA, ad LSA Let A be a t, t, ), q matix with the elemet a = P x ), t = L t ad U = u, u, Lu ) q ae two vectos Aagig Eq 8) i matix fom we have the followig costaied optimizatio poblem subect to mi E E, 9) E AU t = E 0)

4 4 IJCSS Iteatioal Joual of Compute Sciece ad etwo Secuity, VOL6 o, ovembe 006 he above optimizatio pogam exists two ids of solutios by addig diffeet costais: i) he miimum om solutio U AA ) t = A, obtaied by addig the costai mi U AU = t is a ude-detemied system ii) he least squaes solutio A A) A t U =, obtaied by addig the costai mi E E, whe, whe AU = t is a ove-detemied system ow we fistly discuss the elatioship betwee LS-SVM ad LSA o elimiate bias tem b i Eq 3) we implemet coodiate tasfomatio i the featue space F [] Cosequetly, Eq 3) is efomulated as t whee ) w z = ξ,,, L =, ) z = z, z, Lzp is the tasfomatio vecto of ϕ x ) with the dimesio p Fo simplicity we assume C be, thus Eq ) ca be ewitte as mi w, ξ with ) ξ = ξ,, ξ, L ξ w ξ ) +, ) We suppose that Z = z z Lz ), A s Z, I) w ξ) =, U s =,, ad I is a uit matix with a p he Eq ) ad ) have the followig equivalet expessio subect to mi A s U, 3) s U s = t 4) It is otable that Eq 4) has the same fom as Eq 0) except the tem E Futhemoe, Eq 4) is a ude-detemied system sice A s is a + p) matix Motivated by Eq 9) ad 0), we ca obtai a miimum om solutio to Eq 4) by addig costai 3) heefoe, it is quite logical to coclude that LS-SVM is idetical to LSA with the miimum om solutio Similaly we ca discuss the elatioship betwee MSA ad LSA It has poited out that MSA is a optimal appoximatio i the sese of miimal mea squae eo [8] Hece the coefficiet c obtaied by Eq 5) maes the mea squae eo eachig a miimum, ie = ) ) c t ) * c ) t 5) = mi ) * c = Give the legth of sequece c be K, we have K < sice thee is a extactio pocess i Eq 5) whe the ) coditio > 0 is fulfilled ow let A be a K matix with the elemet a = ad ) ) ), ),, ) = c c c K ) ) U L be a K vecto, Eq 5) ca be aaged i the followig costaied optimizatio poblem subect to ) ) mi E E, 6) ) ) ) A U t = E A quic ispectio of Eq 6) shows it has the same fom as Eq 9) Futhemoe, thee exist a least squaes ) ) solutio to Eq 6) sice we ca ife that A U = t is a ove-detemied system fom the fact A ) is a K matix It is atue to coclude that MSA is idetical to LSA with the least squaes solutio 6 Coclusio I this pape we suvey two cuetly developed appoximatio method LS-SVM ad MSA, ad oe taditioal method LSA As descibed i above sectios, it is ot supisig that close mathematical elatioship betwee LS-SVM, MSA, ad LSA exist We pove that LS-SVM is idetical to the LSA with the miimum om solutio, ad MSA is idetical to the LSA with the least squaes solutio heefoe, both LS-SVM ad MSA ca be deived as specific istaces of LSA

5 IJCSS Iteatioal Joual of Compute Sciece ad etwo Secuity, VOL6 o, ovembe Acowledgmet he authos would lie to expess thei codial thas to D Xiaohe Li fo his valuable advice Refeeces [] D atale, G S Desoli, D D Giusto, ad G Veazza, Polyomial appoximatio ad vecto quatizatio: a egio-based itegatio, IEEE as Commuicatios, vol43, pp98-06, 995 [] B guye ad B J Oomme, Momet-pesevig piecewise liea appoximatios of sigals ad images, IEEE as PAMI, vol9, pp84-9, 997 [3] Y V Zahaov ad C oze Local splie appoximatio of time-vayig chael model Electoics Lettes, vol37, pp , 00 [4] D Dooho ad I Johstoe, Poectio-based appoximatio ad a duality with eel methods, A Statist, vol 7, pp 58-06, 989 [5] S Celebi ad J C Picipe, Paametic least squaes appoximatio usig Gamma bases, IEEE as Sigal Pocessig, vol43, pp78-784, 995 [6] D Wedge, D Igam, ad C Migham, O global local atificial eual etwos fo fuctio appoximatio, IEEE as eual etwos, vol7, pp94-95, 006 [7] V Vapi, S Golowich, ad A J Smola, Suppot vecto method fo fuctio appoximatio, egessio estimatio, ad sigal pocessig, I: M J Joda ad S A Solla eds): eual ifomatio pocessig system IPS), MI pess, Cambidge MA, pp3-37, 997 [8] S Mallat, A theoy fo multiesolutio sigal decompositio: he wavelet epesetatio, IEEE as PAMI, vol, pp , 989 [9] V Vapi, he atue of statistical leaig theoy, Spige Velag, ew Yo, 998 [0] J A K Suyes ad J Vadewalle, Recuet least squaes suppot vecto machies, IEEE as Cicuits ad System, vol 47, pp09-4, 000 [] J A K Suyes, L Luas, ad P Dooe, Least squaes suppot vecto machie classifies: a lage scale algoithm, I: S Golowich ad A J Smola eds): Euopea Cofeece o Cicuit heoy ad Desig ECCD), Stesa, Italy, pp839-84, 999 [] H Ya, X G Zhag, ad Y D Li, Relatio betwee a suppot vecto machie ad the least squae method, J sighua Uiv Sci & ech), vol4, pp77-80, 00 i Chiese) Yatog Zhou eceived the BE ad ME degees, fom Chegdu Uiv of echology i 995 ad 999, espectively He is cuetly pusuig the Ph D degee i the Dept of Ifomatio ad Ifomatio Egieeig, Xi a Jiaotog Uivesity, Chia He is autho o co-autho of 8 publicatios His cuet eseach iteests iclude machie leaig, patte ecogitio, ad adaptive sigal pocessig He is a studet membe of IEICE aiyi Zhag eceived the BE ad ME degees, fom Xi a Jiaotog Uivesity i 967 ad 98, espectively He wet o to wo at Xi a Optics ad Pecisio Mechaics, Academia Siica, whee he was a seio eseache o image pocessig ad commuicatio system util 998 He stated wo as a pofesso i mobile commuicatio ad WLA at Xi a Jiaotog Uivesity i the spig of 998 He is autho o co-autho of 06 publicatios His cuet eseach iteests iclude digital mobile commuicatios, patte ecogitio, ad image pocessig Lieu Wag eceived the BE ad ME degees, fom Xiiag Uiv i 994 ad 003, espectively He is cuetly pusuig the Ph D degee i the Dept of Ifomatio ad Ifomatio Egieeig, Xi a Jiaotog Uivesity, Chia He is autho o co-autho of 8 publicatios His cuet eseach iteests iclude digital wieless commuicatios, OFDM systems, ad sigal detectio