Twin Support Vector Machine in Linear Programs

Transcription

1 Procedia Computer Science Volume 29, 204, Pages ICCS th International Conference on Computational Science Twin Support Vector Machine in Linear Programs Dewei Li Yingjie Tian 2 Information School, Ren University of China, Beijing 00872, China ruclidewei@26.com 2 Research Center on Fictitious Economy Data Science, Chinese Academy of Science, Beijing 0090, China tyj@ucas.ac.cn Abstract This paper propose a new algorithm, termed as, for binary classification problem by seeking two nonparallel hyperplanes which is an improved method for TWSVM. We improve the recently proposed ITSVM develop Generalized ITSVM. A linear function is chosen in the object function of Generalized ITSVM which leads to the primal problems of. Comparing with TWSVM, a -norm regularization term is introduced to the objective function to implement structural risk imization the quadratic programg problems are changed to linear programg problems which can be solved fast easily. Then we do not need to compute the large inverse matrices or use any optimization trick in solving our linear programs the dual problems are unnecessary in the paper. We can introduce kernel function directly into nonlinear case which overcome the serious drawback of TWSVM. The numerical experiments verify that our is very effective. Keywords: Twin support vector machine, Binary classification, Linear programs, Structural risk imization Introduction Support vector machines(svms), as machine learning methods which were constructed on the VC-dimension theory the principle of structural risk imization, were proposed by Corinna Cortes Vapnik in 995 [ 3]. With the evolution of SVMs, they have shown much advantages in classification with small samples, nonlinear classification high dimensional pattern recognition also they can be applied in solving other machine learning problems [4 0]. The stard support vector classification attempts to imize generalization error by maximizing the margin between two parallel hyperplanes, which results in dealing with an optimization task involving the imization of a convex quadratic function. But some Corresponding author 770 Selection peer-review under responsibility of the Scientific Programme Committee of ICCS 204 c The Authors. Published by Elsevier B.V. doi:0.06/j.procs

2 classifiers based on nonparallel hyperplanes were proposed recently. The generalized eigenvalue proximal support vector machine(gepsvm) twin support vector machine(twsvm) are two typical classification methods are also very popular. TWSVM seeks two nonparallel hyperplanes make each hyperplane close to one class far from the other as much as possible. However, the structural risk was not considered in TWSVM which may affect the computational efficiency accuracy. Based on TWSVM, TBSVM ITSVM was proposed in [, 2] which introduces a regularization term into the objective function their experiments perform better than TWSVM. The main contribution of the twin bounded support vector machine(tbsvm) is that the principle of structural risk imization is implemented by adding the regularization term in the primal problems. And the advantages of the improved twin support vector machine(itsvm) are that it can apply kernel trick directly in the nonlinear case it does not need to compute inverse matrices. TWSVM have been studied extensively [3 9]. In this paper, we propose a novel approach to classification problem which involves two nonparallel hyperplanes in two linear optimization problems, termed as, for binary classification. Since ITSVM is a successful method as an improved version of TWSVM, we develop Generalized ITSVM which follows the idea in [20]. replaces the abstract function in the objective function of Generalized ITSVM with -norm terms then we convert them to linear programs which can be solved easily quickly inherits the advantage of ITSVM. We can implement the principle of structural imization avoid computing inverse matrices. Also kernel function can be introduced to nonlinear case directly as the stard SVMs usually do. The paper is organized as follows: Section 2 briefly introduces two algorithms, the original TWSVM ITSVM; Section 3 proposes our new method ; Numerical experiments are implemented in Section 4 concluding remarks are summarized in Section 5. 2 Background In this section, we introduce the original TWSVM its improved algorithm ITSVM. 2. TWSVM Consider the binary classification problem with the training set T = {(x, +),, (x p, +), (x p+, ),, (x p+q, )}, (2.) where x i R n,i=,,p+q. LetA =(x,,x p ) T R p n, B =(x p+,,x p+q ) T R q n, l = p + q. TWSVM constructs the following primal problems w +,b +,ξ 2 (Aw + + e + b + ) T (Aw + + e + b + )+c e T ξ, (2.2) s.t. (Bw + + e b + )+ξ e,ξ 0, (2.3) w,b,ξ + 2 (Bw + e b ) T (Bw + e b )+c 2 e T +ξ +, (2.4) s.t. (Aw + e + b )+ξ + e +,ξ + 0, (2.5) 77

3 where c i, i=,2,3,4 are the penalty parameters, e + e are vectors of ones, ξ + ξ are slack vectors, e +,ξ + R p, e,ξ R q. The decision function is denoted by where is the absolute value. 2.2 ITSVM Class = arg k=,+ (w k x)+b k (2.6) ITSVM is the abbreviation of improved twin support vector machine which changes the form of TWSVM construct the following primal problems in linear case w +,b +,η +,ξ 2 c 3( w b 2 +)+ 2 ηt +η + + c e T ξ, (2.7) s.t. Aw + + e + b + = η +, (2.8) (Bw + + e b + )+ξ e,ξ 0, (2.9) w,b,η,ξ + 2 c 4( w 2 + b 2 )+ 2 ηt η + c 2 e T +ξ +, (2.0) s.t. Bw + e b = η, (2.) (Aw + e + b )+ξ + e +,ξ + 0, (2.2) where c i, i=,2,3,4 are the penalty parameters, e + e are vectors of ones, ξ + ξ are slack vectors, e +,ξ +,η + R p, e,ξ,η R q. The following dual problems are considered to be solved where max λ,α 2 (λt α T ) ˆQ(λ T α T ) T + c 3 e T α, (2.3) s.t. 0 α c e, (2.4) max θ,γ 2 (θt γ T ) Q(θ T γ T ) T + c 4 e T +γ, (2.5) s.t. 0 γ c 2 e +, (2.6) ( AA ˆQ = T + c 3 I + AB T AB T BB T ( BB Q = T + c 4 I BA T BA T AA T ) + E, (2.7) ) + E, (2.8) I + is the p p identity matrix, I is the q q identity matrix, E is the l l matrix with allentriesequaltoone.thusanewpointx R n is predicted to the class by (2.6) where w + = c 3 (A T λ + B T α),b + = c 3 (e T +λ + e T α), (2.9) w = c 4 (B T θ A T γ),b = c 4 (e T θ e T +γ), (2.20) 772

4 For the nonlinear case, after introducing the kernel function, the two corresponding problems in the Hilbert space H are w +,b +,η +,ξ 2 c 3( w b 2 +)+ 2 ηt +η + + c e T ξ, (2.2) s.t. Φ(A)w + + e + b + = η +, (2.22) (Φ(B)w + + e b + )+ξ e,ξ 0, (2.23) w,b,η,ξ + 2 c 4( w 2 + b 2 )+ 2 ηt η + c 2 e T +ξ +, (2.24) s.t. Φ(B)w + e b = η, (2.25) Their dual problems are constructed can be solved directly. (Φ(A)w + e + b )+ξ + e +,ξ + 0, (2.26) 3 In this section, based on ITSVM, we first develop Generalized ITSVM then introduce our which changes the 2-norm terms in the objective function of ITSVM to -norm terms then get a pair of linear programs. 3. Generalized ITSVM For the solution (2.9) (2.20), let λ + = c 3 λ, α + = c 3 α, λ = c 4 θ, α = c 4 γ, then w + = A T λ + B T α +,b + = e T +λ + e T α +, (3.) w = B T λ + A T α,b = e T λ + e T +α, (3.2) We introduce (3.) (3.2) into the primal problems (2.7)-(2.9) (2.0)-(2.2) thus get λ +,α +,η +,ξ 2 c 3(λ T + α+)q T + (λ T + α+) T T + 2 ηt +η + + c e T ξ, (3.3) s.t. (AA T + e + e T +)λ + (AB T + e + e T )α + = η +, (3.4) (BA T + e e T +)λ + +(BB T + e e T )α + + ξ e, (3.5) α + 0,ξ 0, (3.6) λ,α,η,ξ + 2 c 4(λ T α )Q T (λ T α ) T T + 2 ηt η + c 2 e T +ξ +, (3.7) s.t. (BB T + e e T )λ +(BA T + e e T +)α = η, (3.8) (AB T + e + e T )λ +(AA T + e + e T +)α + ξ + e +, (3.9) α 0,ξ + 0, (3.0) where ( AA Q + = T + e + e T + AB T e + e T BA T e e T + BB T + e e T ), (3.) 773

5 Q = ( BB T + e e T BA T e e T + AB T e + e T AA T + e + e T + ), (3.2) We state mathematical programs by replacing the first term in the object function with abstract function f,g as follows: f(λ +,α + )+ λ +,α +,η +,ξ 2 ηt +η + + c e T ξ, (3.3) s.t. (AA T + e + e T +)λ + (AB T + e + e T )α + = η +, (3.4) (BA T + e e T +)λ + +(BB T + e e T )α + + ξ e, (3.5) α + 0,ξ 0, (3.6) g(λ,α )+ λ,α,η,ξ + 2 ηt η + c 2 e T +ξ +, (3.7) s.t. (BB T + e e T )λ +(BA T + e e T +)α = η, (3.8) where f,g are some convex functions on R p R q. (AB T + e + e T )λ +(AA T + e + e T +)α + ξ + e +, (3.9) α 0,ξ + 0, (3.20) 3.2 Linear Programg ITSVM() In this section, we consider using linear function f,g in the objective function generated from the mathematical program (3.3)-(3.20) thus leading to linear programs. We chose -norm λ +,α +,λ,α for f,g change the η term to -norm form which leads to the following primal problems: λ +,α +,η +,ξ c 3 ( λ + + α + )+ η + + c e T ξ, (3.2) s.t. (AA T + e + e T +)λ + (AB T + e + e T )α + = η +, (3.22) (BA T + e e T +)λ + +(BB T + e e T )α + + ξ e, (3.23) α +,ξ 0, (3.24) c 4 ( λ + α )+ η + c 2 e T +ξ +, (3.25) λ,α,η,ξ + s.t. (BB T + e e T )λ +(BA T + e e T +)α = η, (3.26) (AB T + e + e T )λ +(AA T + e + e T +)α + ξ + e +, (3.27) α,ξ + 0, (3.28) where denote -norm, c i, i=,2,3,4 are the penalty parameters. We introduce the variables s + =(s +,,s +,2,,s +,p ) T,t + =(t +,,t +,2,,t +,p ) T,s = (s,,s,2,,s,q ) T,t =(t,,t,2,,t,q ) T, then we can convert the primal problems 774

6 (3.2)-(3.24) (3.25)-(3.28) to linear programg formulation: c 3 (e T +s + + e T α + )+e T +t + + c e T ξ, (3.29) λ +,α +,s +,t +,ξ s.t. t + (AA T + e + e T +)λ + (AB T + e + e T )α + t +, (3.30) (BA T + e e T +)λ + +(BB T + e e T )α + + ξ e, (3.3) s + λ + s +, (3.32) s +,t +,α +,ξ 0, (3.33) c 4 (e T s + e T +α )+e T t + c 2 e T +ξ +, (3.34) λ,α,s,t,ξ + s.t. t (BB T + e e T )λ +(BA T + e e T +)α t, (3.35) (AB T + e + e T )λ +(AA T + e + e T +)α + ξ + e +, (3.36) s λ s, (3.37) s,t,α,ξ + 0, (3.38) Then an unknown point is predicted to the class by (2.6) where w +,b +,w,b are same as (3.) (3.2). In nonlinear case, we introduce the transformation x = Φ(x) the corresponding kernel function K(x, x )=(Φ(x) Φ(x )) where x H, H is the Hilbert space. So the training set (2.) becomes T = {(x, +),, (x p, +), (x p+, ),, (x p+q, )}, (3.39) Then we construct the following primal problems c 3 (e T +s + + e T α + )+e T +t + + c e T ξ, (3.40) λ +,α +,s +,t +,ξ s.t. t + (K(A, A T )+e + e T +)λ + (K(A, B T )+e + e T )α + t +, (3.4) (K(B,A T )+e e T +)λ + +(K(B,B T )+e e T )α + + ξ e, (3.42) s + λ + s +, (3.43) s +,t +,α +,ξ 0, (3.44) c 4 (e T s + e T +α )+e T t + c 2 e T +ξ +, (3.45) λ,α,s,t,ξ + s.t. t (K(B,B T )+e e T )λ +(K(B,A T )+e e T +)α t, (3.46) (K(A, B T )+e + e T )λ +(K(A, A T )+e + e T +)α + ξ + e +, (3.47) s λ s, (3.48) s,t,α,ξ + 0, (3.49) Then an unknown point is predicted to the class by Class = arg k=,+ f k(x), (3.50) where f + (x) =K(x, A T )λ + K(x, B T )α + + e T +λ + e T α + (3.5) f (x) = K(x, B T )λ + K(x, A T )α e T λ + e T +α (3.52) 775

7 4 Numerical Experiments In this section, we have made numerical experiments to show the advantages of our algorithm. In order to get more objective results, some datasets were partially selected from the primary datasets to get balanced datasets. All the optimal parameters are selected through searching the set {0 2,, 0 2 }. The best accuracy of each dataset is obtained by three-fold cross validation method. We performed the experiments in linear case nonlinear case respectively. All experiments were implemented in MATLAB 202a on a PC with a Intel Core 2 processor with 2GB RAM all datasets were come from the UCI machine learning Repository. All samples were scaled to the interval [0,] before training to improve the computational efficiency. For all the methods, we applied the RBF kernel K(x, x )=exp( μ x x 2 ) to nonlinear case. The Accuracy used to evaluate methods is defined same as [2]. Accuracy=(TP+TN)/(TP+FP+TN+FN), where TP, FP, TN FN is the number of true positive, false positive, true negative false negative, respectively. The experiments results are listed in Table Table 2. The best accuracy is signed by bold-face. In Table, we compare our with SVC, TWSVM ITSVM in linear case, the classification accuracy the optimal parameters are listed. In fact, for ITSVM, the choice of c 3 c 4 affects the results significantly which shows that there are only two parameters to be tuned in practice [2]. For, there are four parameters need to be tuned every parameter can affect the results markedly. But it is observed that, in most cases, the best accuracy is got when c = c 2 c 3 = c 4 which can save much training time in practice. We have more space to adjust the parameters which leads to higher classification accuracy. We compare with SVC, TWSVM ITSVM in nonlinear case in Table 2. All methods can get better accuracy since the kernel function is introduced. Our have achieved the best accuracy in seven datasets. In summary, we can see that has got higher accuracy than the other methods on most datasets which demonstrate the effectiveness of our in binary classification. 5 Conclusion In this paper, based on TWSVM ITSVM, we have proposed another improved method for binary classification, terms as. Instead of solving a large sized programg problems in stard SVMS, we solve two linear optimization problems of a smaller size in similar as ITSVM. ITSVM has been developed to Generalized ITSVM we get our primal problems by using a linear function in the objective function of Generalized ITSVM. In contrast to the original TWSVM, our introduced -norm regularization term to the objective function which contribute to the structural risk imization. The primal problems of can be converted to linear programs easily the weights between the regularization term empirical risk can be adjusted freely. Furthermore, we do not need to calculate the inversion of matrices which can affect the computational accuracy. Numerical experiments have been made on ten datasets the results show that our performs better on most datasets, namely, we have higher generalization ability. The idea can be extended to regression machine, knowledge-based learning in future work. Acknowledgements This work has been partially supported by grants from National Natural Science Foundation of China (No.2736, No ), Major International (Ragional) Joint Research Project 776

8 Table : Average accuracy in linear case of binary classification SVM TWSVM ITSVM Dataset Accuracy Accuracy Accuracy Accuracy inst. attr. C c /c 2 c /c 2 /c 3 /c 4 c /c 2 /c 3 /c 4 WDBC (200 30) 00 / 0./0./0./0. //0.0/0. votes (300 6) /0. 0./0./0.0/0.0 /// hepatitis (55 9) 0. 0/ 0/00/00/00 //0./0. Heart-statlog (270 3) 0./0. 0.0/0.0// /0./0/0 Heart-c / (200 3) 0. 0./0. 0./0./0./0. 0/0/0/0 Ionosphere (200 34) 0 00/ //0.0/0.0 00//0.0/0.0 sonar (208 60) 0/ 0/0/0/0 0/00/0/ bupa (345 6) / /0.0/0.0/0.0 //0.0/0.0 hungarian (200 3) 0 00/00 00/00// 00/00/0.0/0.0 blood (300 4) / 0./0.// //0.0/0.0 Table 2: Average accuracy in nonlinear case of binary classification SVM TWSVM ITSVM Dataset Accuracy Accuracy Accuracy Accuracy inst. attr. C/g c /c 2 /μ c /c 2 /c 3 /c 4 /μ c /c 2 /c 3 /c 4 /μ WDBC (200 30) / 0./0./0.0 //0./0./0. 00/00/0./0./0. votes (300 6) 0/ 0./0./0. //// 00/00/0./0./0. hepatitis (55 9) /0. 0./0.0/0.0 //0./0./0.0 0/0/0/0/0.0 Heart-statlog (270 3) 00/ /0./0.0 0/0///0.0 ////0. Heart-c (200 3) 00/0. 0./0/0.0 0/0/0/0/0.0 //0./0./0. Ionosphere (200 34) 00/ 0.0/0./0.0 //0.0/0.0/ //0./0./0. sonar (208 60) 0/ // 0/0/0.0/0.0/ //0.0/0.0/0. bupa (345 6) 0/ //0. //0./0./ 0/0/0.0/0.0/0. hungarian (200 3) 00/0. 0.0/0./0.0 0/0/00/00/0. 00/00/0./0./0. blood (300 4) 0/ //0. //0.0/0.0/ 0/0./0/0./

9 (No ), the Ministry of water resources special funds for scientific research on public causes(no ). References [] C.Cortes V.N.Vapnik. Support-vector network. Machine Learning, 20(3): , 995. [2] V.N.Vapnik. The nature of statistical learning theory. Springer, New York, 996. [3] V.N.Vapnik. Statistical learning theory. Wiley-Interscience, New York, 998. [4] Naiyang Deng, Yingjie Tian, Chunhua Zhang. Support Vector Machine-theories, algorithms extensions. Science Press, Beijing, [5] Ji Zhu, Saharon Rosset, Trevor Hastie, Rob Tibshirani. -norm support vector machines. Neural Information Processing Systems, page 6, [6] O.L.Mangasarian. Exact -norm support vector machines via unconstrained convex differentiable imization. Machine Learning Research, 7:57 530, [7] Yingjie Tian, Yong Shi, Xiaohui Liu. Recent advances on support vector machines research. Technological Economic Development of Economy, 8():5 33, 202. [8] P.S.Bradley O.L.Mangasarian. Feature selection via concave imization support vector machines. Machine Learning Proceedings of the Fifteenth International Conference, 98:82 90, 998. [9] Chuanhua Zhang, Dewei Li, Junyan Tan. The support vector regression with adaptive norms. Procedia Computer Science, 8: , 203. [0] Chuanhua Zhang, Xiaojian Shao, Dewei Li. Knowledge-based support vector classification based on c-svc. Procedia Computer Science, 7: , 203. [] Yingjie Tian, Xuchan Ju, Zhiquan Qi, Yong Shi. Improved twin support vector machine. Science China Mathematics, 203. [2] Yanhai Shao, Chuanhua Zhang, Xiaobo Wang, Naiyang Deng. Improvements on twin support vector machines. IEEE Trans. neural netw., 22(6): , 20. [3] R.K.Jayadeva, J.R.Khemchani, S.Chra. Twin support vector machines for pattern classification. IEEE Trans. Pattern Anal. Mach. Intell., 29(5):905 90, [4] Zhiquan Qi, Yingjie Tian, Yongshi. Robust twin support vector machine for pattern classification. Pattern Recognition, 46():305 36, 203. [5] Zhiquan Qi, Yingjie Tian, Yongshi. Laplacian twin support vector machine for semi-supervised classification. Neural Networks, 35:46 53, 202. [6] Zhiquan Qi, Yingjie Tian, Yongshi. Twin support vector machine with universum data. Neural Networks, 36:2 9, 202. [7] M.A. Kumar M.Gopal. Application of smoothing technique on twin support vector machines. Pattern Recognition Letters, 29: , [8] M.A. Kumar M.Gopal. Least squares twin support vector machines for pattern classification. Expert Systems with Applications, 36: , [9] J.R.Khemchani, R.K.Jayadeva, S.Chra. Optimal kernel selection in twin support vector machines. Optimization Letters, 3:77 88, [20] O.L.Mangasarian. Generalized support vector machines. Advances in Large Margin Classifiers, pages 35 46,