A Ridgelet Kernel Regression Model using Genetic Algorithm

Transcription

1 A Ridgeet Kerne Regression Mode using Genetic Agorithm Shuyuan Yang, Min Wang, Licheng Jiao * Institute of Inteigence Information Processing, Department of Eectrica Engineering Xidian University Xi an, China, 77 Nationa Lab of Radar Signa Processing, Department of Eectrica Engineering Xidian University Xi an, China, 77 Abstract. In this paper, a ridgeet kerne regression mode is proposed for approximation of high dimensiona functions. It is based on ridgeet theory, kerne and reguarization technoogy from which we can deduce a reguarized kerne regression form. Taking the objective function soved by quadratic programming to define the fitness function, we use genetic agorithm to search for the optima directiona vector of ridgeet. The resuts indicate that this method can effectivey dea with high dimensiona data, especiay those with certain kinds of spatia inhomogeneities. Some iustrative exampes are incuded to demonstrate its superiority. Introduction In Machine Learning(ML), many probems can be reduced to the tasks of mutivariate function approximation (MVFA). MVFA is an active subject that has attracted ots of researching interests in many science and engineering communities [,]. Depending on the community invoved, it goes by different names incuding noninear regression, function earning, system identification and others. The numeric methods for MVFA have been deepy studied in mathematics and computer science [3,4]. As we a know, approximation of a function from sparse sampes is i-posed, so one often assumes the function to be smooth to obtain a certain soution. However, in practica cases such as industria contro systems, faut detection, system identification and inteigent predicting, most systems are very compex MIMO(muti-input and muti-output) systems. They are equivaent to non-smooth mapping in high dimension, that is, they are MVFAs with spatia inhomogeneities. So the cassica mathematica methods cannot estimate or approximate them efficienty by sparse sampes. Reconstructing a function by a superposition of some basis functions is a very inspiring idea on which many regressor are based, such as Fourier Transform (FT) [5], Waveet Transform(WT) [6], Neura Network(NN) [7] and Projection Pursuit Regression (PPR) [8]. However, FT has a poor performance for singuar functions; WT can dea with one-dimension singuarity, but it can t extend to higher dimension. NN aows for non-smooth mapping in high dimension, but it inevitaby has some disadvantages such as overfitting, sow convergence and too much reiance on experience. PPR is a regression way for smooth functions in a greedy fashion. For PPR converges * This work was supported by the Nationa Science Foundation (grant no. 6375).

2 according to norm, it is of sow convergence. In 996, E.J.Candes deveoped a new system to represent arbitrary functions by a superposition of specific ridge functions in a more stabe way, the ridgeet [9]. Ridgeet proves to be good basis in high dimension, and it has optima property for functions with spatia inhomogeneities. To adaptivey estimate high dimensiona functions, ridgeets can be used in PPR or NN to construct a regressor. However, they ony minimize the experience risk, which eads to some drawbacks such as overfitting and bad generaization []. Recenty Kerne Machine(KM) has been a standard too in ML, which has stricter mathematica foundation than NN and PPR []. Based on ridgeet and KM, a ridgeet kerne regression mode for MVFA is proposed. The minimum squared error(mse) based on kernes and reguarization technoogy, or the reguarized kerne form of MSE, is adopted []. Empoyment of ridgeet can accompish a wider range of MVFA, and the reguarized items in objective function are used to improve the generaization of soutions. To get the directions of ridgeets, genetic agorithm (GA) is used. Ridgeet Kerne Regression based on Genetic Agorithm. Ridgeet Regression Given a pair of sampe set S={(x, y)}(perhaps pouted by noise) coming from the mode Y=f(X), a nonparametric regression is to estimate the unknown function f from the sparse data set S. Kerne smoothing, nearest-neighbor and spine smoothing are often used in regression. However, their performances decrease sharpy in high dimension due to the curse of dimensionaity, that is, if sampes are not dense enough, one wi get a bad mean squared error. Ridgeet is a new harmonic anaysis too, and it proves to be optima for estimating mutivariate regression surfaces especiay for those exhibiting specific sorts of spatia inhomogeneities, with a speed rapid than FT and WT. As an extension of waveet to higher dimension, ridgeet has attracted more and more attention of researches. It is defined as: ψξ ˆ( ) IfΨ:R d R satisfies the condition: Kψ = dξ < () ξ d / Then we ca the functions ψ ( x) a ( u x b τ = ψ ) as ridgeets. Parameterτ=(a,u,b) a d beongs to a space of neurons Γ= { τ = ( aub,, ), ab, Ra, >, u S, u = }.Denote the surface area of unit sphere S d- in d-dimension as σ d. Then for any function f L L (R d ), it can be expanded as a superposition of ridgeet functions: d d f = c < f, ψ > ψ µ ( dτ) = c ( π) K < f, ψ > ψ σ da / a + dudb ψ τ τ ψ ψ τ τ d () For function f (x):r d R m, it can be divided into m mappings of R d R. Seect the ridgeet as the basis, then the foowing approximation equation is obtained: N ) uj x bj yi = cijψ ( ) i =, L, m (3) a j= j x where ˆ d d Y = [ y, L, ym ]; uj =, u j R ; R ; c ij is the superposition coefficients of ridgeet. The ridgeet regressor based on PPR [3] and NN with ridgeet neurons are both based on the minimization of experience risk. In 988, Vapnik deveoped a new

3 earning machine-svm on the concept of VC dimension, which soves a the theoretic probems of NN [4]. Its strength ies on its minimum of structure risk experience and a consequent protruding optimization. Its good generaization and avoidance of oca minimum in earning is unachievabe for NN and PPR. SVM heps to buid up a famiy of KM based methods in ML. KM is a powerfu technoogy extending the standard inear methods to noninear cases. A foundationa idea behind KM is that the kerne function can be interpreted as an inner product in the feature space under certain conditions. This idea, commony known as the "kerne trick", has been used extensivey in generating noninear versions of conventiona inear supervised and unsupervised earning agorithms, such as SVM, kerne fisher decision (KFD) etc. To get better ridgeet regressor, in the foowing we discussed a kernebased regressor with ridgeet being the kerne function.. Ridgeet kerne regression mode X X Xd Consider training set S={(x, y ),, (x P, y P )} with L number P, where x i is d-dimension and y i is onedimension(i=,..,p). An unknown mapping with u u L u noise n can be described by Y=f (X)+n, and our goa is to reconstruct f from S. Denoting X=[x,., x P ] and Y=[y,.,y P ], ridgeet ψ r is a noninear mapping about the input sampes. After X going through the r r r directiona vector of ridgeets, we get R=[r,.., K( r, r ) K ( r, r ) L K( r, r ) r P ] T L =[r ij ] with r ij = u j x i (i=,..,p; j=,..,). Then the w w L w ridgeet regression of R d R becomes an R R waveet mapping. Considering the atter mapping, for the inear regressor in feather space, we construct such a inear function with weight w and Y threshodβ: fˆ ψ () r = w ψ() r + β (4) Fig. Ridgeet kerne regression According to the reproducing kerne Hibert space, the soution to this optimization probem must be in the space formed by the sampes, i.e.,w is a inear superposition of ψ a the sampes: w = αψ i ( ri)( αi R) (5) i= Denote the kerne function K(r i,r j )=Ψ(r i )Ψ(r j ), and it shoud satisfy strict aowance condition. Then the estimate function in the feather space is obtained: fˆ ( r ) = aψ ( r ) ψ ( r ) + β = a K ( r, r ) + β (6) i i i i i= i= The ridgeet kerne regression mode is shown in Fig..Ψ(r)=cos(.75r)exp(-r /) proves to be served as a good kerne function [5]. Then we get: K( r, r) = cos(.75 ( r r)/ a)exp( r r / a ) (7) i i i.3 MSE based reguarized kerne form Vapnik showed that the key to get an effective soution is to contro the compexity of the soution. In the context of statistica earning this eads to new techniques known

4 as reguarization networks or reguarized kerne methods. We consider the reguarized kerne methods, and define a generaized estimator for the approximation: P min E = V( y, fˆ ( r)) / P+ λ f ( λ > ) (8) i= where V is the oss function; H is the Hibert space of the hypotheses; λ is the reguarization parameter. Various kinds of penaty terms can act as oss function, and squared oss function known as the rue of minimum square error (MSE) is most often used. The second term in (8) is a reguarized term empoyed to get a specified soution and better generaization, and they are different in various function spaces. Then the soution is found by minimizing function consisting of the oss function and reguarized term in KM. We adopt such form: min E = y fˆ ( r) + λ( ww. ) (9) Let ek = yk w ψ( rk) β and convert the inequaity restriction in the standard regression to the equaity restriction; take the quadratic programming to represent the min Ew (, λ, β) = ek / + λ( ww. ) / MSE based reguarized kerne form: k = () s.t. yk = w ψ( rk) + β + ek, k =,.., The corresponding Lagrange function is L= E( w, λβ, ) µ k[ w ψλ( rk) + β yk + ek] () whereμis Lagrange factor. The optima soution of this probem is: H k= L/ w= w= µψ( r ), L/ β = µ =, k k k k= k= L/ e = µ = λ e, L/ µ = w ψ( r ) + β y + e = k k k k k k k ().4 Optimization of directiona vector based on GA From above we see that as ong as the directiona vector is determined, the above method can be used to get a certain soution. To obtain the direction of ridgeet, here genetic agorithm (GA) is empoyed. GA is a good optimization too which has the goba searching capacity, as we as inner paraeism and sef-earning [6]. Firsty define the reciproca of reguarized object function as the fitness function. Since the directiona vector of ridgeet u=[u,.., u d ] and u =, the directiona vector in u can be described using (d-) anges θ,.., θd : u = cos θ, u = sinθcos θ,.., ud = sinθsinθ...sinθ d.code the chromosome using (d-) anges. The agorithm is described as: Step : Init the iteration times t= and a popuation P with M chromosomes: Generate a set of anges randomy to form P={θ, θ M } where θ i =[θ i,.,θ i (d-) ] (i=,..,m); Step : Derive the directiona vectors U=[u, u M ] (u i R d ) from the popuation P; Step 3: The directiona vectors in U are corresponding to M ridgeet functions. Using them and the quadratic programming agorithm depicted in.3 to approximate the input sampes. Compute the fitness, that is, the reciproca of objective function /E; Step 4: Rouette seection is performed to form new popuation; Step 5: Judge the stop condition. When t is bigger than the given maximum iteration times T or the error is sma than ε, stop, ese go on;

5 Step 6: Repeat such operation M times on the popuation: Seect two individuas randomy and perform crossover with probabiity p c ; Step 7: Perform the mutation on each chromosome with probabiity p m :θ j i (t+)=θ j i (t) +η rand, where i=,., M, j=,,(d-) and rand means a random number in [,];η is a constant in [,8] which determine the mutation degree; Step 8: Get the new popuation P(t+), t=t+, go to Step. 3 Simuation Experiments Experiment : Radar Target Recognition Radar target recognition is a chaenging subject in radar signa processing. We appied our method to radar target recognition of three-cass panes(b5,j-6,j-7) using one-dimensiona image(or radar range profie). Our data are obtained in a microwave darkroom with imaging ange from to 79 degree and we get totay 84 images of 64 dimension. In this experiment three modes are considered-gaussian kerne LS- SVM(GSVM),Waveet kerne LS-SVM(WSVM) and our method. Kerne function in (7) is used in the atter two modes. The modes are under the same condition and in our method M=5,η=, T=, =6,λ=.5,a=,ε= -8,p c =.7, p m =.. We get the recognition resuts in tabe, from which we can see that our method has both high recognition rates and east training time than GSVM and WSVM. PLANE sampes GSVM(%) WSVM(%) our method(%) B-5 J-6 J-7 Time(s) train test train test train test train test Tabe : Recognition resuts of three panes. Experiment : Function approximation RMSE error GSVM WSVM our method expression f train e- 4.47e-9 f ( x, x) = test f train.55e e e- f (, xx) test f 3 3 train.686e e-5.75e- f (, x x) = test f 4 4 train 4.938e-5.865e e- test f 5 x+ x train.55e e e- f ( x, x) = test Tabe : Approximation resut of three methods x x x+ 4x <. [ 4sin cx (4 )] [ 3sin cx (3 )] xx - > = sin cx ( ) sin cx ( ) x+ x > x + ( x+.5) 3x+ x> f ( x, x ) =. x x x+ x <.5 e x x ( ) 5 Just as described above, our method can dea with not ony high dimensiona data, but aso data with spatia inhomogeneities. Consider approximations of some singuar functions f - f 5. In the experiment, the numbers of training and testing

6 sampes are 36 and. To estimate the approximation, the root mean squared error (RMSE) is empoyed. The approximation resuts are shown in tabe. 4 Concusion Ridgeet is a new geometrica muti-scae anaysis too deveoped recenty, which provides good basis for high dimensiona space. Starting from the probem of MVFA, we proposed a ridgeet kerne regression mode which can represent a wider range of high dimensiona functions more efficienty in a stabe way. The reguarized items are empoyed in the object function to improve the generaization of our method. The objective function soved by quadratic programming is used to define the fitness function, and GA is taken for optimizing the directions of ridgeets. Theoretica anaysis proves its accurate regression for high dimensiona data, especiay those with spatia inhomogeneities singuarities. Experiment resuts on pattern recognition and function approximation aso prove its efficiency. Reference [] Breiman, L. The II method for estimating mutivariate functions from noisy data (with discussion), Technometrics 33: 5-6, 99. [] G. G. Lorentz, Constructive Approximation, Advanced Probems. New York: Springer-Verag,996. [3] Wang ReHong, Approximation of mutivariate functions.beijing pubishing company of China,988. [4] Grimm, L.G., Yarnod, P.R.. Reading and understanding more mutivariate statistics. Washington, DC: American Psychoogica Association.. [5] Arfken, G. "Fourier Transforms--Inversion Theorem," 5.3 in Mathematica Methods for Physicists, 3rd ed. Orando, FL: Academic Press, pp.794-8, 985. [6] HÄarde, ect. Waveets, approximation,and statistica appications.new York: Springer-Verag,998 [7] Cybenko, G.Approximation by superpositions of a sigmoida function. Math. Contro Signas Systems,, 33-34, 989. [8] Friedman, J. H.,Stuetze, W. Projection pursuit regression. J. Amer. Statist. Assoc.,76, 87-83,98. [9] Candes, E. J. Ridgeets: theory and appications. dissertation, Stanford University,998. [] Vu, V. H. On the infeasibiity of training neura networks with sma mean-squared error. IEEE Transactions on Information Theory, vo 44, pp 89-9,998. [] Gasser, T. and Muer, H. G. Kerne Estimation of Regression Functions, New York: Springer- Verag, Vo. 757, pp. 3-68, 979. [] Xu JianHu, ect. Reguarized Kerne forms of Minimum square methods. ACTA AUTOMATIC SINICA, vo 3, No : 7-36, 4. [3] A. Rakotomamonjy, Ridgeet Pursuit : Appication to regression estimation, Technica Report, Perception Systèmes Information, ICANN. [4] V. Vapnik. Statistica Learning Theory. Wiey, New York, 988. [5] L. Zhang, W. D. Zhou, L. C. Jiao. Waveet Support Vector Machine. IEEE Trans. On Systems, Man, and Cybernetics. Part B: Cybernetics. Vo. 34, no., February 4. [6] Hoand J.H. Genetic agorithms and cassifier systems: foundations and their appications. Proceedings of the Second Internationa Conference on Genetic Agorithms, pp 8-89, 987.