Research Article Robust ε-support Vector Regression

Transcription

1 Matheatical Probles in Engineering, Article ID , 5 pages Research Article Robust ε-support Vector Regression Yuan Lv and Zhong Gan School of Mechanical Engineering, Northwestern Polytechnical University, Mailbox 301, No. 127 Youyi Road, Xi an, Shaanxi , China Correspondence should be addressed to Yuan Lv; lvyuan85@live.co Received 1 August 2013; Revised 25 Noveber 2013; Accepted 25 Noveber 2013; Published 20 February 2014 Acadeic Editor: Andrzej Swierniak Copyright 2014 Y. Lv and Z. Gan. This is an open access article distributed under the Creative Coons Attribution License, which perits unrestricted use, distribution, and reproduction in any ediu, provided the original work is properly cited. Spheroid disturbance of input data brings great challenges to support vector regression; thus it is essential to study the robust regression odel. This paper is dedicated to establish a robust regression odel which akes the regression function robust against disturbance of data and syste paraeter. Firstly, two theores have been given to show that the robust linear ε-support vector regression proble could be settled by solving the dual probles. Secondly, it has been focused on the developent of robust support vector regression algorith which is extended fro linear doain to nonlinear doain. Finally, the nuerical experients result deonstrates the effectiveness of the odels and algoriths proposed in this paper. 1. Introduction As a new achine learning algorith, support vector achine (SVM) is based on the statistical learning theory which was proposed by Vapnik [1]andWolfedualprograing theory. Copared with other learning algoriths, SVM possesses a rigorous foundation of atheatical theory, wellgeneralized ability, and global optiu, with the result that is widely used in pattern recognition and functional regression. In general, the input data is assued to be accurate. However, in practice, the statistical errors and easuring errors of observed data are inevitable; that is, the data is disturbed. The effect of perturbation of input data on solving the regression proble is often neglected. Although SVR is robust, while the paraeters are optial, these paraeters are usually reasonably sensitive to noise data and singular value. The robust optiization ethod, which focused on treatability of coputation in the case of data points disturbing in convex sets, was first proposed by Soyster [2]anddeveloped,respectively, by Ben-Tal and Neirovski [3, 4] and El-Ghaoui et al. [5, 6]. Since then, Melvyn [7] presented ore coprehensive collection of disturbance without increasing the coputing difficulty, which enabled the rapid developent of the robust optiization ethod. The research aied on solving the prograing probles with polyhedron disturbance or spheroid disturbance. Bertsias and Si [8] and Alizadeh and Goldfarb [9] put forward the robust linear prograing degenerated into a second-order cone prograing, secondorder cone prograing into a seidefinite prograing, and sei-definite prograing into the NP proble. Zhixia et al. [10] applied this ethod to classification proble and proposed the robust support vector classification. This paper ais to study the regression probles of spheroid disturbance by eans of the robust optiization ethod andtoestablisharobustregressionodelwhichakesthe regression functions highly robust for perturbed data and syste paraeters. 2. Theory 2.1. Training Set. Assue that the training set is where T = {(χ 1,y 1 ),...,(χ,y )}, (1) χ i ={x i +r i u i u i 1},,...,. (2) The input set x i can be viewed as a sphere of radius r i centering on the easured value x i, r i is closely related to easureent accuracy, and is saple size.

2 2 Matheatical Probles in Engineering 2.2. Original Proble. Referring to the original proble of linear ε-support vector regression, it is not difficult to derive the original proble of robust linear ε-support vector regression, which is a convex quadratic prograing, by replacing the original input with the input collection χ i : in w,b,ξ, ( 1 2 ) w 2 +C (ξ i + ξ i ), s.t. (w(x i +r i u i )+b) y i ε+ξ i, y i (w(x i +r i u i )+b) ε+ξ i,,...,. Theore 1. The necessary and sufficient conditions for the solution of the original optiization proble (3) are that w, b, ξ,andξ are the solutions for the following second-order cone prograing: in w,b,ξ,ξ,u,v,t ( 1 2 ) (u V) +C (ξ i + ξ i ), s.t. (w x i )+b y i +r i t ε+ξ i, y i ((w x i )+b)+r i t ε+ξ i, u+v =1, u ( t) L 3, V,...,, (3) (4) If variable t is introduced, satisfying w t, theabove proble could prove to be a second-order cone prograing: in w,b,ξ, ( 1 2 ) t2 +C (ξ i + ξ i ), s.t. (w x i )+b y i +r i t ε+ξ i, y i ((w x i )+b)+r i t ε+ξ i, w t.,...,, When variables u and V are brought in, u and V are supposed to satisfy the linear constraint u+v = 1 and the second-order cone constraint (t 2 + V 2 ) 1/2 u,andthe nonlinear ters t 2 in the objective function are replaced with u V; therefore, the proble (7) can be written as the proble (4). Fro the above, the proble (3) isequivalenttothe proble (4) Dual Proble. Introduce the Lagrange function: L= ( 1 2 ) (u V) +C (ξ i + ξ i ) + + α i ((w x i )+b y i +r i t ε ξ i ) β i (y i ((w x i )+b)+r i t ε ξ i ) θ i ξ i η i ξ i +φ (u+v 1) (7) (8) ( t w ) Ln+1, where n is the diensionality of the input data χ i. Proof. Since ax {r i (w u i ), u i 1}=r i w, in {r i (w u i ), u i 1}= r i w. The original proble (3)can be converted to in w,b,ξ, ( 1 2 ) w 2 +C (ξ i + ξ i ), s.t. (w x i )+b y i +r i w ε+ξ i, y i ((w x i )+b)+r i w ε+ξ i, (5) (6) z u u z V V δt z t t z T w w, where α, β, θ, η R,φ,z u,z V,δ,z t R,z w R n are the ultipliers vectors corresponding to the constraints. Eliinating variables θ, η, z t,andz w, the following theore can be obtained. Theore 2. The following second-order cone prograing is the dual proble of the second-order cone prograing (4). ax α,β,φ,δ,z u,z V α i (y i +ε)+ s.t. z u φ= 1 2, z V φ= 1 2, 0 α i C, 0 β i C, β i (y i ε) φ,,...,. z u (δ 2 +z 2 V )1/2,

3 Matheatical Probles in Engineering 3 α i β i =0,,...,, j=1,...,, δ r i (α i +β i ) ( (α i β i )(α j β j )(x i x j )) j=1 Proof. The feasible region of the second-order cone prograing proble (4)is 1/2. (9) Proceeding fro the lower bound of the estiate of p, the Lagrange function of the second-order cone prograing proble (4) is introduced and written as L(w, b, ξ, ξ, u, V,t,α,β,φ,δ,z u,z V ): w, b, ξ, ξ, u, V,t D,α,β,φ,δ,z u,z V E, L (w, b, ξ, ξ, u, V,t,α,β,φ,δ,z u,z V ) ( 1 2 ) (u V) +C (ξ i + ξ i ), inf w,b,ξ,ξ,u,v,t R n L (w, b, ξ, ξ, u, V,t,α,β,φ,δ,z u,z V ) inf L (w, b, ξ, ξ, u, V,t,α,β,φ,δ,z u,z V ) w,b,ξ,ξ,u,v,t D (13) D= { { w, b, ξ, ξ, u, V,t (w x i )+b y i { +r i t ε+ξ i,y i ((w x i )+b) +r i t ε+ξ i, (10) inf w,b,ξ,ξ,u,v,t D let g(α,β,φ,δ,z u,z V ) (( 1 2 ) (u V) +C (ξ i + ξ i )).,...,,u (t 2 + V 2 ) 1/2, t ( n j=1 w 2 j 1/2,j=1,...n; ) u+v =1 } }. } The feasible region of the dual proble (9)is E= {α,β,φ,δ,z u,z V 0 α i C, 0 β i C,z u (δ 2 +z 2 V )1/2, δ r i (α i +β i ) ( (α i β i )(α j β j )(x i x j )) j=1 z u φ= 1 2,z V φ= 1 2, α i 1/2 β i =0,,...,, j=1,...,}. Assuing that the optial value of the proble (4)isp, p, (11) =inf {( 1 2 ) (u V)+C (ξ i + ξ i ) w, b, ξ, ξ, u, V,t D}. (12) thus = inf w,b,ξ,ξ,u,v,t R n L (w, b, ξ, ξ, u, V,t,α,β,φ,δ,z u,z V ); (14) g(α,β,φ,δ,z u,z V ) p. (15) Forula (15) iplies w, b, ξ, ξ, u, V, t D, α, β, φ, δ, z u,z V E, g(α, β, φ, δ, z u,z V ) is the lower bound of p. To get the optial value, the axial value of g(α, β, φ, δ, z u,z V ), w,b,ξ,ξ, u, V,t D,α,β,φ,δ,z u,z V E is need to be obtained. So, the dual proble (9) is deduced Robust SVR Algorith Linear Robust SVR Algorith. (1) The given training data should be written as the way of forulae (1) (2). (2) By constructing and solving the secondorder cone prograing proble (9), the solution (α i,β i,φ,δ,z u,z V ) can be obtained. Besides, it is necessary to select an appropriate penalty paraeter C>0. (3) The solution (w,b ) of the original proble (4) is acquired in the following way: w = δ (α i β i )x i ( (α i +βi )r i δ ). (16) Select a coponent α j of α or a coponent β k of β,which is located in the open interval (0, C); if α j is chosen, it is got as follows: b = y j δ (α i β i )(x i x j ) ( (α i +β i )r i δ )+ε+r i δ. (17)

4 4 Matheatical Probles in Engineering If β k is chosen, it is got as follows: b = y k δ (α i β i )(x i x k ) ( (α i +β i )r i δ ) ε r i δ. (18) (4) Then regression function is established: f (x) = δ (α i β i )(x i x) ( (α i +β i )r i δ )+b. (19) Nonlinear Robust SVR Algorith. By aking the following odifications to the algorith shown in Section 2.4.1, the nonlinear robust SVR algorith can be constructed: (1) Kernel function K(x i,x j ) (coon kernel functions include Gaussian kernel function, polynoial kernel function, Cauchy kernel function, and Laplace kernel function) is introduced; replace the inner products (x i x j ) and (x i x)appearing in the linear robust SVR algorith shown in Section 2.4.1, withk(x i,x j ) and K(x i,x); (2) Substitute R i for r i appearing in the second-order cone prograing (9). To get R i, the apping should be written as X=Φ(x); obviously, χ i x i r i is equivalent to Φ(χ i) Φ(x i ) 2 So R i =F(r i ). = ((Φ (χ i ) Φ(x i )) (Φ (χ i ) Φ(x i ))) =K(χ i,χ i ) 2K(χ i,x i )+K(x i,x i ) =F( χ i x i ) R2 i. 3. Nuerical Experients 3.1. Linear Nuerical Experient (20) Exaple 1. Given that objective function is y = 2x (x [1, 10]), the training set could be written as the way of forulae (1) (2), where x i is 10 equidistant points distributed in the interval [1, 10]. r i can be regarded as a constant which equals 0.5. The disturbance of data u i is a rando nuber which is located in the interval [ 1, 1], wherei = 10.The ethods of standard SVR and robust SVR shown in Section are eployed to regress and fit the objective function. Then the errors of ten test points, which are randoly selected, are analysed and copared. The results are shown in Table Nonlinear Nuerical Experients Exaple 2. Given that objective function is y = x 2 (x [0.5, 10]), thetrainingsetcouldbewritteninthewayof forulae (1) (2), where x i is 20 equidistant points distributed in the interval [0.5, 10]. r i can be regarded as a constant which Table 1: Table of error analysis of linear function regression. Test points Algorith Linear standard SVR Linear robust SVR Point Point Point Point Point Point Point Point Point Point AVG Table 2: Table of error analysis of nonlinear function regression. Test points Algorith Nonlinear standard SVR Nonlinear robust SVR Point Point Point Point Point Point Point Point Point Point AVG equals The disturbance of data u i is a rando nuber which is located in the interval [ 1, 1], wherei = 20.The ethodsofstandardsvrandrobustsvrshowninsection areeployedtoregressandfittheobjectivefunction. Then the errors of ten test points, which are randoly selected, are analysed and copared. The results are given in Table Discussion. Obviously, the syste paraeters are very robust. In addition, it is shown that the forecast errors oftherobustethodarelowerthanthatofthestandard SVM ethod in Table 1. Asiilarconclusioncanalsobe presented in Table 2. In short, the robust SVR ethod has beenvalidatedtobeofadvantageinthefieldofregression analysis with perturbation proble. 4. Conclusions (1) In the paper, a robust ε-svr odel has been proposed, considering the input data as the convex set of spheroid disturbance, and two iportant theores arealsogiven:itisprovedbytheore1 that the

5 Matheatical Probles in Engineering 5 convex quadratic prograing proble can be written as the second-order cone prograing. Based on duality theory, the dual proble can be deduced by Theore 2.Bysolvingthedualproble,theoptial solution of original proble is obtained. (2) By virtue of kernel function, the robust linear ε-svr odel could be prooted to the nonlinear doains. Finally, the nuerical experients deonstrate the robustness and effectiveness of the proposed algorith. Conflict of Interests The authors declare that there is no conflict of interests regarding the publication of this paper. Acknowledgents The theores and algoriths proposed in this paper are the extension of robust support vector classification ethod [10], which provides a great assistance for the authors. The authors are here to express their sincere gratitude. References [1] V. N. Vapnik, The Nature of Statistical Learning Theory,Springer, New York, NY, USA, [2] A. L. Soyster, Inexact linear prograing with generalized resource sets, European Operational Research,vol.3, no. 4, pp , [3] A. Ben-Tal and A. Neirovski, Robust convex optiization, Matheatics of Operations Research,vol.23,no.4,pp , [4] A. Ben-Tal and A. Neirovski, Robust solutions of uncertain linear progras, Operations Research Letters,vol.25,no.1,pp. 1 13, [5] L.ElGhaouiandH.Lebret, Robustsolutionstoleast-squares probles with uncertain data, SIAM Journal on Matrix Analysis and Applications, vol. 18, no. 4, pp , [6] L. El Ghaoui, F. Oustry, and H. Lebret, Robust solutions to uncertain seidefinite progras, SIAM Journal on Optiization,vol.9,no.1,pp.33 52,1999. [7] S. Melvyn, Robust optiization [Ph.D. thesis],2004. [8] D. Bertsias and M. Si, Tractable approxiations to robust conic optiization probles, Matheatical Prograing, vol. 107,no.1-2,pp.5 36,2006. [9] F. Alizadeh and D. Goldfarb, Second-order cone prograing, Matheatical Prograing, Series B, vol. 95, no. 1, pp. 3 51, [10] Y. Zhixia, T. Yingjie, and D. Naiyang, Second order cone prograing forulations for robust support vector ordinal regression achine, in Proceedings of the 1st International Syposiu on Optiization and Systes Biology, pp , 2013.

6 Advances in Operations Research Advances in Decision Sciences Applied Matheatics Algebra Probability and Statistics The Scientific World Journal International Differential Equations Subit your anuscripts at International Advances in Cobinatorics Matheatical Physics Coplex Analysis International Matheatics and Matheatical Sciences Matheatical Probles in Engineering Matheatics Discrete Matheatics Discrete Dynaics in Nature and Society Function Spaces Abstract and Applied Analysis International Stochastic Analysis Optiization