MULTICLASS LEAST SQUARES AUTO-CORRELATION WAVELET SUPPORT VECTOR MACHINES. Yongzhong Xing, Xiaobei Wu and Zhiliang Xu

Transcription

1 ICIC Express Letters ICIC Internatonal c 2008 ISSN Volume 2, Number 4, December 2008 pp MULTICLASS LEAST SQUARES AUTO-CORRELATION WAVELET SUPPORT VECTOR MACHINES Yongzhong ng, aobe Wu and Zhlang u School of Automaton Unversty of Nanjng Unversty of Scence and Technology Nanjng, , P. R. Chna xyz-1971@hotmal.com; wuxb@mal.njust.edu.cn Receved Aprl 2008; accepted July 2008 Abstract. In ths paper, combnng the auto-correlaton wavelet kernel wth multclass least squares support vector machne (MLS-SVM), a novel noton of multclass least squares support vector machne wth unversal auto-correlaton wavelet kernels (MLS- AWSVM) s proposed. The translaton nvarant property of the kernel functon enhances the generalzaton ablty of the LS-SVM method and the spral multclass classfcaton expermental results show some advantages of MLS-AWSVM over MLS-SVM on the classfcaton and the generalzaton performance. Keywords: Auto-correlaton wavelet kernel, LS-SVM, Multclass classfcaton 1. Introducton. Support vector machne (SVM) has been a standard tool n the machne learnng communty because t ncely deals wth hgh dmensonal data, provdes good generalzaton propertes and determnes the classfer archtecture once kernel functon and the parameters are chosen by user [1], [2]. SVM uses a kernel functon to project the nput data onto a hgh dmenson feature space, and then constructs an optmal separatng hyper-plane n that space. SVM s a kernel based method, whch allows the use of Gaussan, polynomal and wavelet kernels and so on that satsfy Mercer s condton [3]. Least squares support vector machnes (LS-SVM)saSVMversonwhchnvolves equalty nstead of nequalty constrants and works wth a least squares cost functon [4]. In the LS-SVM, Mercer s condton s stll applcable. A straghtforward extenson of LS-SVM to multclass problems (MLS-SVM) has been proposed n [5], where the Gaussan kernel functon s used. One ssue wth the kernel methods s fndng an approprate kernel for the gven problem because dfferent kernel functon and parameters can have wdely varyng performance. The wavelet theory and assocated multresoluton technques [6] has had tremendous mpact not only n sgnal and mage processng but also n scence and engneerng. Presently, there are several contrbutons to the theoretcal development of wavelet kernel reported n the lterature such as reproducng wavelet kernel[7],constructng translaton nvarant wavelet kernel[8]and auto-correlaton wavelet kernel[9]. Auto-correlaton of a compactly supported wavelet satsfes the translaton nvarant property. Ths property s very mportant n sgnal processng. The wavelet has a lmtaton on ths. The wavelet transform generates very dfferent wavelet coeffcents even f the nput sgnal s shfted a lttle bt. Ths lmtaton can be overcome by takng the auto-correlaton on the wavelet functon. Based on the property, any compactly supported wavelet functon can be chosen to construct auto-correlaton wavelet kernel, Daubeches-4 (D4) wavelet has been proven to perform the best for sgnal regresson [9]. In ths paper,combnng MLS-SVM wth the auto-correlaton D4 wavelet kernel functon, the multclass least squares autocorrelaton wavelet support vector machnes (MLS-AWSVM) s proposed. The goal of 345

2 346 Y. ING,. WU AND Z. U the MLS-AWSVM s to fnd the optmal classfcaton n the space spanned by multvarable wavelet kernel. The spral multclass classfcaton expermental results show some advantages of MLS-AWSVM over MLS-SVM on the classfcaton and the generalzaton performance. 2. Prepare Knowledge SVM for pattern recognton. Gven an ndependent dentcally dstrbuted (..d) tranng data set {(x 1,y 1 ),, (x k,y k )},where x R n,y { 1, 1}. For pattern recognton problem, by ntroducng Lagrange multpler technque, the resultng decson functon of the SVM takes the form k f(x) =sgn[ α y K(x, x )+β] (1) where α, are the Lagrange multplers and β s the bas, a kernel K(x, x j )=ϕ(x ) ϕ(x j ) s called a support vector (SV) kernel f t satsfes a certan condtons, ϕ( ) =R n R h,s a nonlnear projecton functon that dot products between projected vectors s computed by means of a kernel functon. Accordng to the dfferent classfcaton problems, the dfferent kernel functon can be selected to obtan the optmal classfcaton results.[1] 2.2. Wavelet theory and wavelet kernel. The dea of wavelet analyss s to approach a sgnal or functon usng a famly of functons whch are produced by dlaton and translaton of the mother wavelet ψ(x) ψ a,b (x) = a 1/2 ψ( x b a ) (2) where x, a, b R, a 6= 0 s dlaton factor,and b s a translaton factor. The wavelet translaton of any functon f(x) can be expressed as W a,b (f) =hf(x), ψ a,b (x),f(x) L 2 (R) (3) where the notaton h, refers the nner product n ÃL 2 (R). Equaton (3) means that any functon f(x) can be decomposed on wavelet bass ψ a,b (x) ftsatsfes the condton [8][11]. C f = Z+ 0 H(ω) 2 dω < (4) ω where H(ω) s Fourer transform of ψ a,b (x). The functon f(x) can be reconstructed as follows Z+ Z+ f(x) =(C f ) 1 W a,b (f)ψ a,b (x) da db (5) a2 To approxmate Equaton (5) [6],then the fnte can be wrtten as k ˆf(x) = W ψ a,b (x). (6) Here, f(x) s approxmated by ˆf(x). 0 For a common multdmensonal wavelet functon, the mother wavelet can be gven as the product of one-dmensonal (1-D) wavelet functons [11]: ψ n (x) = ψ(x ). (7)

3 ICIC EPRESS LETTERS, VOL.2, NO.4, where x =(x 1,,x n ) R n. So, every 1-D mother wavelet ψ(x) mustsatsfy(4). The wavelet kernel s defned as K(x, x )= ψ( x x ). (8) a 2.3. Auto-correlaton wavelet kernel. Wavelets satsfy a multresoluton analyss and they also obey the followng relatons: N 1 ϕ(x) =2 1/2 h m ϕ(2x m), (9) and m=0 N 1 ψ(x) =2 1/2 m=0 g m ϕ(2x m). (10) where g m =( 1) m h N m 1,m=0,,N 1. The auto-correlaton s defned by Φ(x) = Z + ϕ(t)ϕ(t x)dt, (11) Ψ(x) = Z + ψ(t)ψ(t x)dt. (12) It can be derved that Φ(x) =Φ(2x)+ 1 N/2 a 2n 1 (Φ(2x 2n +1)+Φ(2x +2n 1)), (13) 2 n=1 Ψ(x) =Φ(2x) 1 N/2 a 2n 1 (Φ(2x 2n +1)+Φ(2x +2n 1)). (14) 2 n=1 where {a m } are the auto-correlaton coeffcents of the flter {h 1,,h N 1 }, a m =2 N m 1 P h n h n+m,for m =1,,N 1and a 2m =0,for m =1,,N/2 1. n=0 It s not dffcult to fnd that both Φ and Ψ have support of [ N +1,N 1]. A translaton nvarant kernel K(x, x )=K(x x ) s an admssble SV kernel f and only f ts Fourer transformaton s non-negatve [8]. Ths can be satsfed by defnng the followng auto-correlaton wavelet kernel [9]: K(x, x )= ly (Ψ( x x a )). (15) where l sthedmensonofthenputfeaturevectorand a s the scale factor. It should be mentoned that we can choose any compactly supported wavelet functon to construct auto-correlaton wavelet kernel K(x, x ). The wavelet functon used here does not have an explct form. In order to generate t, we need to set one wavelet coeffcent to 1 and all the rest coeffcents to 0. An nverse wavelet transform generates the desred wavelet functon dependng on the selected nput wavelet flter. Snce the wavelet functon has an mplct form, we save t n memory as one dmensonal array wth a relatvely large number of sample ponts. Ths array needs to be generated only once and then saved for later use. It can be easly proved that ths kernel

4 348 Y. ING,. WU AND Z. U K(x, x ) s an admssble SV kernel. Fg.1 shows the D4 wavelet and ts auto-correlaton kernel. Fgure 1. Daubeches-4 wavelet and ts auto-correlaton kernel 3. Multclass Least Squares Auto-correlaton Wavelet Support Vector Machnes. Let {x,y (k) } =N, k=m k=1 be tranng data set, m be number of classfcaton, N be number of tranng data set, x = {x 1,,x n } nput ndex, y (k) output ndex, and y (k) =1meansthe th nput vector belongs to the kth class, y (k) = 1 not. The dervaton of the mult-class wavelet LS-SVM s based upon the formulaton [5] wth the equalty constrants mn W k,β k,ξ k J (m) (W k, β k, ξ k )= 1 m 2 ( k=1 W T k W k + C y (1) [W1 T ϕ 1 (x )+β 1 ]=1 ξ 1 y (2) [W2 T ϕ 2 (x )+β 2 ]=1 ξ 21 y (m), [Wmϕ T m (x )+β m ]=1 ξ m m k=1 w k N ξ 2 k) (16) where =1,,N, W R h s the weght vector, C>0 s the regularzaton factor [10], W k R s the weght of kth classfcaton error. ξ k R s the classfcaton eror and β k R s the bas. The correspondng Lagrange equaton s (17) L (m) (W k, β k, ξ k, α k )=J (m),k α k {y k [W T k ψ ( k)(x )+β k ] 1+ξ k } (18) The soluton concludes n a constraned optmzaton wth the equalty condtons, wth the expuncton of W k and ξ k, one can get the lnear system: 0 Y T M βm = (19) Y M Ω M α M 0 1 wth gven matrces 1 =[1,,N]; β M =[β 1,, β m ]; α M =[α 11,, α N1,, α 1m, α Nm ]; Ω M = blockdag{ω (1),, Ω (m) }; Ω (k) l = y (k) y (k) l ϕ T k (x )ϕ k (x l )+C 1 I;

5 ICIC EPRESS LETTERS, VOL.2, NO.4, y (1) 1 y (m) 1 Y M = blockdag{.,,. }. y (1) N y (m) N Let K k (x,x l )=ϕ T k (x )ϕ k (x l ) be Daubeches-4 auto-correlaton wavelet kernel functon, then K k (x,x l )= (Ψ( x t x lt )), k,,l =1,,N (20) a t=1 Decson functon of multclass auto-correlaton wavelet kernel support vector machnes (MLS-AWSVM) s f(x) =sgn[ N α k y k (Ψ( x t x lt )) + β a t k ],k=1,,m (21) t=1 Here, MLS-SVM can adopt the Daubeches-4 auto-correlaton wavelet kernel as ts kernel functon. It s dffcult to determne N n parameters, for the sake of smplcty, let a t = a k, the parameter a k of the auto-correlaton kernel can be obtaned by means of cross-valdaton. 4. Numercal Example. Now, we valdate the performance of MLS-AWSVM by a small llustratve example on a smple spral problem for whch four classes have been defned. Fgure.2 shows N = 60 tranng data ponts wth 4 classes that each contan 15 data ponts (outputs equal to [+1; +1], [+1; 1], [ 1; +1], [ 1; 1] ). The four classes have been encoded by takng m = 2. Note that the range of dsplay s lmted to [ 0.7, 0.7] [ 0.7, 0.7]. We let w k =1(k =1,,m)and C = 1 for all the followng examples. For comparson, we show the results obtaned by the auto-correlaton wavelet kernel and Gaussan kernel, respectvely. The Gaussan kernel s expresson s K k (x,x l ) = exp{ kx x l k 2 2/2σk 2},where σ k s kernel s parameter chosen by user. Fgure. 2 show the estmated separatng hyperplanes based on the Gaussan kernel (MLS-GSVM) usng parameters σ1 2 = σ2 2 =0.5. Fgure. 3 show the estmated separatng hyperplanes based on auto-correlaton wavelet kernel (MLS-AWSVM) usng parameters a 1 = a 2 =2. Table 1 shows the msclassfcaton rato n leave-one-out cross-valdaton of each condton. Smulaton results show that, compared wth MLS-GSVM, the recognton precson and the generalzaton ablty s mproved by our MLS-AWSVM. Fgure 2. The classfcaton result based on the MLS-GSVM for spral four class problem wth σ 2 1 = σ 2 2 =0.5

6 350 Y. ING,. WU AND Z. U Fgure 3. The classfcaton result based on the MLS-AWSVM for spral four class problem wth a 1 = a 2 =2 Table 1. Msclassfcaton rato n leave-one-out cross-valdaton Condton Msclassfcaton rato MLS-GSVM wth σ1 2 = σ2 2 = MLS-AWSVM wth a 1 = a 2 = Conclusons. In ths paper, we dscussed a practcal way to construct auto-correlaton wavelet kernel usng a compactly supported wavelet functon. The wavelet kernel s a knd of multdmensonal functon that can approxmate arbtrary functons. A new MLS-SVM verson s presented based on MLS-SVM and the auto-correlaton wavelet kernel, namely MLS-AWSVM. The MLS-AWSVM s appled to spral classfcaton problem. Smulaton shows that the wavelet kernel has better classfcaton than the Gaussan kernel. Notce that the wavelet kernel s orthonormal approxmately, whereas the Gaussan kernel s not. The Gaussan kernel s correlatve or even redundancy. Thus, for many real lfe applcatons MLS-AWSVM can offer a faster method for obtanng classfer wth better generalzaton performance than MLS-GSVM. REFERENCES [1] V. Vapnk, The Nature of Statstcal Learnng Theory, New York: Sprng - Verlag, [2] N. Y. Deng and Y. J Tan, The Method of Data Mnng-support Vector Machne, Scence Press, Bejng, [3] A. Smola, B. Schölkopf and K.-R. Müller, The connecton between regularzaton operators and support vector kernels, Neural Network, vol.11, pp , 1998d. [4] J. A. K. Suykens and J. Vandewalle, Least squares support vector machnes classfers,neural Processng Letters, vol.9, no.3, pp , [5] J. A. K. Suykens and J. Vandewalle, Multclass least squares support vector machnes, Proc.ofthe Internatonal Jont Conference on Neural Networks, Washngton DC, USA, pp , [6] G.Z.LuandY.Tan,Wavelet Analyss and Applcaton, an, Chna: dan Unv Press, [7] A. Rakotomamonjy and S. Canu, Frame, reproducng kernel,regularzaton, Journal of Machne Learnng Research, vol.6, pp , [8] L. Zhang, W. Zhou and L. Jao, Wavelet support vector machne, IEEE Transacton on Systems, Man, and Cybernetcs-Part B: Cybernetcs, vol.34. no.1, pp.34-39, [9] G. Y. Chen and G. Dudek, Auto-correlaton wavelet support vector machne and ts applcaton to regress, Proc. of the Second Conference on Computer and Robot Vson, Brtsh, Columba, [10] J. H. u,. G. Zhang and Y. D. L, Regularzed kernel forms of mnmum square error methods, Acta Automat. Sn., vol.30, no.1, pp.27-36, [11] Q. H. Zhang and A. Benvenste, Wavelet networks, IEEE Transacton on Neural Networks, vol.3, pp , 1992.