Linear programming support vector machines

Pattern Recognition 35 (00) 97 936 www.elsevier.com/locate/patcog Linear programming support vector machines Weida Zhou, Li Zhang, Licheng Jiao Key Laboratory for Radar Signal Processing, Xidian University, Xi an, 710071, People s Republic of China Received 3 May 001; accepted 30 October 001 Abstract Based on the analysis of the conclusions in the statistical learning theory, especially the VC dimension of linear functions, linear programming support vector machines (or SVMs) are presented including linear programming linear and nonlinear SVMs. In linear programming SVMs, in order to improve the speed of the training time, the bound of the VC dimension is loosened properly. Simulation results for both articial and real data show that the generalization performance of our method is a good approximation of SVMs and the computation complex is largely reduced by our method.? 00 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Statistical learning theory; VC dimension; Support vector machines; Generalization performance; Linear programming 1. Introduction Since the 1970s, Vapnik et al. have applied themselves to the study of statistical learning theory [1 3]. Until the early of the 1990, a new kind of learning machines, support vector machine (SVM), was presented based on those theories [,4,5]. The main study of statistical learning theory is the model of learning from examples, which can be described as: there are l random independent identically distributed examples (x 1;y 1); (x ;y );:::;(x l ;y l ); ((x;y) (R n ;R)) drawn according to the uniform probability distribution P(x;y); (P(x;y) = P(x)P(y x)). Given a set of functions f(x;); (where is a parameter set) from which the goal of learning from examples is to select a function f(x; 0) that can express the relationship between x and y in the best possible way. In general, in order to obtain f(x; 0), one has to minimize the expected risk functional R()= L(y; f(x; ))P(x; y)dx dy; (1.1) where L(y; f(x;)) measures the loss between the response y to a given input x and the response f(x;a) provided Corresponding author. Fax: +86-9-83-6159. E-mail address: zhouwd@rsp.xidian.edu.cn (W. Zhou). by the learning machine. The learning problems such as the problems of pattern recognition, regression estimation and density estimation may be taken as the same learning models withdierent loss functions []. In this paper, we only deal withthe problem of pattern recognition. Consider the following loss function 0ify = f(x;); L(y; f(x;)) = (1.) 1ify f(x;): Due to the probability distribution P(x;y) in Eq. (1.1) is unknown, the expected risk functional is replaced by the empirical risk functional R 1 emp()= L(y i;f(x i;)): (1.3) l In order to know the quality of the empirical risk R emp() to approximate the expected risk, Vapnik presented the following bound theorem []. With probability at least 1 (0 6 6 1), the inequality ( R() 6 R emp()+ 1+ 1+ 4Remp() ) (1.4) holds true. Where =4(h(ln(l=h) +1) ln )=l and h is the VC dimension of the set of functions f(x;),. 0031-303/0/$.00? 00 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S0031-303(01)0010-

98 W. Zhou et al. / Pattern Recognition 35 (00) 97 936 From Eq. (1.4), we can see that the minimization of the expected risk R() is equal to the minimization of the two terms on the right-hand side of Eq. (1.4) at the same time. The rst term on the right of Eq. (1.4) R emp() is minimized by learning process. The second term varies with the VC dimension h and the number of examples l. The smaller the VC dimension h and the larger the number of examples l, the smaller the value of the second term. In fact, the number of examples is nite. So for the case of a small example set, the minimization of the expected risk is implemented by minimizing the empirical risk and the VC dimension. Generally speaking, a complex target function set or a large hypothesis space is required for minimizing the empirical risk. But a small hypothesis space is requested for minimizing the VC dimension of the target function set. Therefore, the minimization problem is in a dilemma, the best solution of the problem is to take a compromise between them. Now, restrict the target function to the linear function. Similar to the set of -margin separating hyperplanes de- ned in Ref. [3], we dene the set of m -margin separating hyperplanes. Let us denote the target functions set by f(x; w;b)=w T x + b. If these functions classify an example x as follows: 1; w T x + b ; y = 0 (1.5) 1; w T x + b 6 ; then the set f(x; w;b)=w T x + b is called the set of m -margin separating hyperplanes whose margin is m = w ; (1.6) where denotes l -norm, namely Euclidean distance. There is a conclusion about the VC dimension of the set of m -margin separating hyperplanes. Let vectors x X belong to a sphere of radius R. Then the set of m -margin separating hyperplanes has the VC dimension h bounded by the inequality: R ;n +1; (1.7) m where n is the dimension of input space. From Eq. (1.7), we can see that if the VC dimension of the target functions set h is n, then h varies inversely withthe margin m. In this way, f(x; w 0;b 0) can be approached by minimizing the empirical risk functional and maximizing the separating margin m, which is the structural risk in SVMs introduced by Vapnik: R 1 structure()=cr emp()+ ; (1.8) m where the constant C 0 is a parameter chosen by the users and is a parameter of the target function set. For l random independent identically distributed examples (x 1;y 1); (x ;y );:::;(x l ;y l ); ((x;y) (R n ;R)), the linear SVMs for pattern recognition have the following optimization problem [4]: min 1 w + C i; (1.9) s:t: y i((w x i)+b) 1 i; i 0; i =1;:::;l; where ( ; ) denotes the inner product. Minimizing the rst term in Eq. (1.9) 1 w plays the role of controlling the capacity of the learning machine and avoiding the overtting of the machine. While minimizing the second term is to minimize the empirical risk. The Wolfe dual programming of Eq. (1.9) is [4] max s:t: W ()= i 1 i jy iy j(x i x j); (1.10) i;j=1 iy i =0; (1.11) i [0;C]; i =1;:::;l: (1.1) The decision function has the following form: f(x)= and iy i(x i x)+b (1.13) y = sgn(f(x)): (1.14) The kernel functions are introduced in linear SVMs, which leads to nonlinear SVMs [4]: max s:t: W ()= i 1 i jy iy jk(x i x j) (1.16) i;j=1 iy i = 0 (1.17) i [0;C]; i =1;:::;l: (1.18) And then the decision function can be written as f(x)= iy ik(x i x)+b: (1.19) In a nutshell, the theory foundation of SVMs (statistical learning theory) is rather perfect. But training a SVM requires the solution of a quadratic programming (QP) optimization that is not easy to implement, in particular for large-scale problems. Based on the statistical learning theory, the linear programming SVMs that are extremely simple without explicitly solving QP problems are proposed.

W. Zhou et al. / Pattern Recognition 35 (00) 97 936 99. Linear programming support vector machines (LPSVMs).1. Linear LPSVMs Lemma.1. [6]: the equivalence of vector norm in n-dimension linear space V n. Let x and x be any vector norms in a nite dimension linear space V n. The norms include l p-norm; weight-norm and others. Then there exist two positive constants 0 c 1 + and 0 c + suchthat c 1 x 6 x 6 c x x V n holds true. Theorem.. Given the examples set ((x 1;y 1);:::;(x i;y i); :::;(x l ;y l )); (x;y) (R n ;R) and the vectors x belong to a sphere of radius R. If the set of m -margin separating hyperplanes f(x; w;b)=w T x + b classify vectors x as follows: 1; w T x + b ; y = 0: 1; w T x + b 6 ; Then there exist a constant 0 c + without depending on examples (x;y) and the parameters w and b suchthat the bound inequality c R w ;n + 1 (.1) holds true. Where w is any vector norm. The proof of Theorem. is shown in Appendix A. The theorem can lead to the following programming directly. Given the examples set ((x 1;y 1);:::;(x i;y i);:::;(x l ;y l )); (x;y) (R n ;R) and the set of linear target functions f(x; w;b)=w T x + b. We introduce the structural risk according to Theorem.. At rst, we will start with the linearly separable case, or without error. max L = r (.) c w s:t: y i(w T x i + b) r i =1;:::;l; r 0; where w is any vector norm and c is a constant. If w is -norm, then c =1. Now consider l -norm x. Certainly there exists a positive constant c without the set of examples and the parameters of the target functions set such that c R w ;n +1 holds on. And so Eq. (.) can be rewritten as r L = : (.3) c w The denominator in Eq. (.) is translated into a constraint. Since the constant c has no eect on solving of linear programming, it can be ignored. In doing so, we have the following programming: LP1: max L = r (.4) s:t: y i(w T x i + b) r i =1;:::;l; (.5) r 0 1 6 w j 6 1 j =1;:::;n: The decision function takes the form f(x; w;b)=w T x + b (.6) and the output of pattern recognition is y = sgn(f(x; w;b)); (.7) where sgn( ) is the sign function. For the nonseparable case, we have the following linear programming by introducing positive slack variable i: LP: min L = r + C i (.8) s:t: y i(w T x i + b) r i i =1;:::;l; (.9) r 0 1 6 w j 6 1 j =1;:::;n; i 0 i =1;:::;l; where C 0 is a constant. The decision function of the nonseparable case is the same as the separable case... Nonlinear LPSVMs Similar to the nonlinear SVMs [1 4], here we get the nonlinear LPSVMs by adopting the kernel functions K(x i; x j) [7]. In Eqs. (.5) and (.9), the relationship between weights w and vectors x is expressed by the form of dot products, (w x). Now suppose we rst mapped the examples from n-dimensional space into some high-dimensional Euclidean space H: : R n H: Then of course the training algorithm would only depend on the examples through dot products in H, i.e. on functions of the form ((x i) (x j)). If there were a kernel function K suchthat K(x i; x j)=(x i) (x j), we would only need to use K in the training algorithm and would need to know what the mapping function is. Theorem.3. Given the set of examples (((x 1);y 1);:::; ((x i);y i);:::;((x l );y l )); ((x);y) (H; R). The vectors x belong to a sphere of radius R. Let the set of target

930 W. Zhou et al. / Pattern Recognition 35 (00) 97 936 functions be f(x)= iy i(x i) (x)+b = iy ik(x i; x)+b: (.10) And the decision on examples x takes the form 1; f(x) ; y = 0 1; f(x) 6 ; then there exist a constant 0 c + without depending the examples (x;y) and the parameter of the target functions set and b suchthat c R ;n + 1 (.11) holds on. Where =( 1 l ) T and is any vector norm. The proof of Theorem.3 is shown in Appendix B. Similar to linear LPSVMs, consider. We can have the following programming: LP3: max L = r (.1) ( ) s:t: y if(x i)=y i jy jk(x j; x i)+b r j=1 i =1;:::;l; (.13) r 0 1 6 k 6 1 k =1;:::;l: The decision function is f(x)= iy ik(x i; x)+b (.14) and the output of pattern recognition is y = sgn(f(x)): (.15) For the nonseparable case, we introduced the slack variables i to get the following linear programming: LP4: min L = r + C i (.16) s:t: y if(x i)=y i ( j=1 jy jk(x j; x i)+b ) r i i =1;:::;l; (.17) r 0 1 6 k 6 1 k =1;:::;l; i 0 i =1;:::;l; where C 0 is a constant. The decision function is identical to Eqs. (.14) and (.15). In Section.1, the linear LPSVMs are obtained under the hypothesis of linear target functions set f(x; w;b)=w T x + b. For high-dimensional examples (namely the number of examples is less than or equal to the dimension of example), we can use the target functions f(x)= l iyi(xi x)+b, and then get the following linear LPSVMs: LP5: min L = r + C i (.18) s:t: y if(x i)=y i ( j=1 jy j(x j x i)+b ) r i i =1;:::;l; (.19) r 0 1 6 k 6 1 k =1;:::;l; i 0 i =1;:::;l: 3. Simulation experiments Four experiments have been done. For linear LPSVMs, a set of articial data was used in experiments. And for nonlinear LPSVMs, the dual-spiral data, the handwritten digital data and the DS-CDMA multiuser detection data were adopted. For the sake of comparison, we used the programs in Matlab Optimization toolboxes (Version 5.3) for bothquadratic programming support vector machines (or QPSVMs) and LPSVMs. 3.1. The experiment on articial data Let us consider two-class articial linear separable data. 100 training examples and 00 test ones are generated randomly according to the uniform distribution in rectangles ((0; 0); (1= ; 1= ); (0; ); ( 1= ; 1= )) and ((1= ; 1= ); ( ; ); (1= ; 3= ); (0; )). The results are shown in Table 1 and Fig. 1. The results lead us to conclude that the training speed of LPSVMs is faster than that of QPSVMs without loss in the classication performance. 3.. The experiment on the dual-spiral data The dual-spiral problem is referred to as touchstone to test the ability of learning algorithm, which requires us to classify the points on one spiral from ones on another spiral in two-dimensional Cartesian coordinates [8]. In this experiment, we used Gaussian kernel K(x i; x j)= exp( x i x j =p ) with p = 8. The results are shown in Table and Fig.. From the results in Table it can be concluded that the training speed of LPSVMs has been improved remarkably. The VC dimension of LPSVMs is larger than that of QPSVMs, so in theory the generalization performance of LPSVMs is worse than that of QPSVMs,

W. Zhou et al. / Pattern Recognition 35 (00) 97 936 931 Table 1 The results of articial data Learning No. of training No. of testing Training time Recognition R =m model examples examples (s) error (%) QPSVMs 100 00 16. 0.5 780. LPSVMs 100 00 0.8 0.5 780. Fig. 1. The results of articial data obtained by LPSVMs. The real line is f(x) = 0, the dotted lines are f(x)=±r. which is not be conrmed by the recognition errors in Table because of the examples without noise. From Fig. we can see that the generalization performance of LPSVMs is worse than that of QPSVMs. At the cost of generalization performance the training speed of LPSVMs is improved. Fortunately the loss in generalization performance is tolerable because the recognition errors is invisible. 3.3. The experiment on the handwritten digits data The experiment data is the MNIST database of 60,000 training and 10,000 testing handwritten digits from AT& T ResearchLabs 1, which has been taken from the experiment data in Refs. [9 11]. Since the database is large, we only had two-class examples belonging to the classes 6 and 9, respectively, and normalized these examples. The kernel function is Gaussian kernel with p = 30. From the results shown in Table 3, we can conclude that the training time of QPSVMs increases exponentially as the increase of the number of examples, while that of LPSVMs increases linearly. Therefore, LPSVMs at least make an order of magnitude improvement in training speed. 3.4. Multiuser detection in DS-CDMA system Code division multiple access (or CDMA) is characterized by its soft capacity, dynamic sharing of channel 1 URL: www.research.all.com./ vann/ocr/moist. resources, high anti-jamming capability and others [1]. The introduction of Multiuser detection technique is to solve the multiple access interference problems (mainly caused by the nonorthogonality of the signature waveform, the asynchronization and the multipath propagation) in wireless communication, which improves the quality and the quantity of communication. Simultaneously, the high complex requirement of signal processing in CDMA systems leads to a wide attention in the eld of signal processing. Multiuser detection is a new direction in CDMA systems [13,14]. Many methods of signal processing have been proposed to solve multiuser detection. But from the view of the - nal goal of multiuser detection, it can be taken as a pattern recognition problem. The goal of multiuser detection is to obtain the minimal bit-error-rate, so the optimal measure of detectors performance is the very principle of the minimum probability of error, not others such as minimum mean-square error, maximum signal-to-noise rate and so on. There exists close relationship or even equivalence between those signal processing principles and the principle of the minimum probability of error, but dierence in many cases. At present, the methods for pattern recognition have received considerable attention in multiuser detection. Here, we take the multiuser detection problem as a fourth experiment. The baseband signal of multiuser detection in a synchronous DS-CDMA system can be expressed as x(t)= K A ib i(j)s i(t jt )+n(t); t [jt; jt + T ]; where T is the symbol interval, b i(j) 1; 1}} is the bit transmitted by the ithuser, A i is the received amplitude 1 of the ithuser, A i is referred to as the energy of the ith user, s i(t); t [0;T] is the deterministic signature waveform assigned to the ithuser and n(t) is white Gaussian noise withunit power spectral density. It is assumed that s i(t) is supported only on the interval [0;T] and has unit energy and that b i(j)} is a collection of independent equiprobable ±1 random variables. We sample the input signal at a rate of 1=T s.iflett s = T=N, where N is the spreading gain, then the sampling rate is equal to the chip rate. On the interval t [jt; jt + T ], we have a vector form x(j)= K A ib i(j)s i + n(j); x(j); s i; n(j) R N 1 :

93 W. Zhou et al. / Pattern Recognition 35 (00) 97 936 Table The results of the dual-spiral data Learning No. of training No. of testing Training time Recognition R =m model examples examples (s) error (%) QPSVMs 5 68 4103.5 0 19.7 LPSVMs 5 68 199.7 0 679.1 Fig.. The result of the dual-spiral data obtained by LPSVMs is shown in (a) and QPSVMs in (b). The real lines are f(x) = 0 and the dotted lines are f(x)= ±r. Table 3 The results of the handwritten digital data No. of No. of LPSVMs QPSVMs training testing examples examples Training Recognition R =m Training Recognition R =m time (s) error (%) time (s) error (%) 100 + 100 1009+95 117.0 1.4 175.4 187.0 1.0 47.3 8 00 + 00 1009 + 95 369.3 1. 311.9 117.4 1.1 60.4 8 300 + 300 1009 + 95 546.1 0.7 33.4 5978.1 1 73. 8 400 + 400 1009 + 95 83.3 1.0 355.8 18509.5 0.875 87.7 8 500 + 500 1009 + 95 1076.0 0.8 40.0 100878.7 0.7 101.8 8 The baseband signal of multiuser detection in an asynchronous DS-CDMA system can be expressed as x(t)= K A ib i(j)s i(t jt i)+n(t); t [jt; jt +T ]; where i is a relative oset, given 1 6 6 6 i 6 6 K. Synchronizing the signal with the expected kth user and sampling at the chip rate, we get K x(j)=a k b k (j)s k + A ib i(j)s i + n(j); i k x(j); s k ; n(j) R N 1 ; where b i and s i denote the equivalent bit stream and the signature waveform of the ithuser, respectively. In general, an

W. Zhou et al. / Pattern Recognition 35 (00) 97 936 933 asynchronous user can be transformed into two synchronous users. Consider a DS-CDMA system. Signature waveform is Golden code with 31 bits. Let the power of the expected user be denoted by P EU. A relative power P rui is dened by the power ratio of the ithuser to the expected user, namely P rui = P IUi =P EU where P IUi = 1 A i is the power of the ith user and P EU = 1 A k the expected user. Similarly Gaussian noise power is expressed as P N = NP rnp EU, where N is spreading gain and P rn is the relative power of noise. We performed synchronous and asynchronous multiuser detection by using SVM techniques. The results are shown in Tables 4 and 5. The baseband signal of multiuser detection was sampled to generate the examples. Approximately the examples have normal distribution with dierent mean vectors and a common covariance matrix. According to Raudys [15], we randomly took the training examples N = 100 3p, where p denotes the dimension of examples and the test ones 10,000 in each run. Table 4 shows the results of synchronous multiuser detection. The number of users is 10, the power of expected user is 1, the relative power of other users is P ru= =P ru5=10(db) and P ru6= =P ru10=0(db) and the relative power of Gaussian noise is shown in Table 4. The results of asynchronous multiuser detection are given in Table 5. The number of asynchronous users is 6 and each of asynchronous users (including the expected one) has three propagation paths. Let the power of the expected user be 1. The relative power of other users is P ru = P ru3 = 10(db) and P ru4 = P ru5 = P ru6 = 0(db). The power of eachpathis identical for the same user. The relative power of Gaussian noise is indicated in Table 5. In this experiment, the recognition errors of training examples are identical for LPSVMs and QPSVMs. The parameters of RBF are obtained by using leave-one-out methods of QPSVMs [16], so they are optimal for QPSVMs. Experimental results support LPSVMs presented and witness the validity of LPSVMs. The bound of VC dimension in LPSVMs is loosened to some extent, which leads to the loss in generalization performance that is tolerable. Thus, LPSVMs have linear structure risk by relaxing the VC bound and improve the speed of training. 4. Conclusion and discussion Linear programming SVMs are proposed based on the statistical learning theory, in particular the theory of capacity control, VC dimension and structural risk. The bound of VC dimension in LPSVMs is loosened to some extent, which leads to the loss in generalization performance that is tolerable. Training LPSVMs is simpler than QPSVMs, especially for large-scale problems. LPSVMs at least make an order magnitude improvement in the training speed, so it is worthwhile losing some generalization performance. We note that the VC dimension of LPSVMs is larger than that of QPSVMs in Section 3.4, but the generalization error Table 4 Synchronous multiuser detection Gauss Linear LPSVMs Linear QPSVMs RBF LPSVMs RBF QPSVMs noise C =10 C =10 p1 = 406; C =10; 000 p1 = 406; C =10; 000 Prn Training Recognition R =m Training Recognition R =m Training Recognition R =m Training Recognition R =m (db) time (s) error (%) time (s) error (%) time (s) error (%) time (s) error (%) 8 3.9 1.41 898.54 10.0.83 1919.4 6.7 7.09 998.70 13.9 4.0 9168.76 10 3.8 0. 187.1 1.9 0.45 148.43 5.7 1. 1455.81 10.3 0.93 10356.89 1 3.5 0.04 1167.98 10.9 0.04 105.14 6.3 0.58 1193.65 10.4 0.15 9364.60 14 3.4 0.01 1035.83 11.8 0 95.64 13.1 0.09 10354.11 6.1 0.0 885.64 16 3.1 0 933.1 14.0 0 866.61 13.8 0.01 9694.48 65.0 0 8785.74

934 W. Zhou et al. / Pattern Recognition 35 (00) 97 936 Table 5 Asynchronous multiuser detection Gauss Linear LPSVMs Linear QPSVMs RBF LPSVMs RBF QPSVMs noise C = 100 C = 100 p1 = 490; C =10; 000 p1 = 490; C =10; 000 Prn Training Recognition R =m Training Recognition R =m Training Recognition R =m Training Recognition R =m time (s) error (%) time (s) error (%) time (s) error (%) time (s) error (%) 8 5.8 17.55 43.1 11.8 18.35 1536.4 3.8 4.89 0771.73 11..1 1600.10 10 5.4 16.8 1577.35 13.4 16.9 941.44 3.7 1.11 15857.43 10.0 0.86 1609.39 1 4.4 16.77 74.98 11.5 16.77 654.44 4.0 0.58 115.86 8.5 0.15 18451.13 14 4.9 16.77 75.97 11.6 16.77 599.53 3.6 0. 10166.39 9.1 0 993.04 16 4.6 16.77 74.05 1.4 16.77 58.80 3.6 0 9160.6 11.3 0 8948.6 of LPSVMs is smaller than that of QPSVMs in linear case, which is not strange. As we know, SVMs dene a structure risk by the empirical risk and the VC dimension and give a bound to the expected risk. Although SVM algorithm can guarantee the structure risk converge to the expected risk withincreasing number of examples. The structure risk is not the expected one exactly. Especially the bound of the expected risk is not very accuracy for the nite number of examples. It is possible that other learning machines get better generalization performance than SVMs, for example, Euclidean distance classier is the optimal one for Gaussian model [15]. Roughly we agree to the viewpoint in Ref. [17]: SVMs are the general and very ecient statistical learning model, but not the optimal. Priori information is helpful for us to select proper learning machine. In fact it is the problem concerned by statistical theory researchers that how to introduce the priori information into SVMs to obtain good generalization performance. Presently in QPSVMs many algorithms for large-scale problems have been developed, such as Chunking algorithm [18], Osuna algorithm [19] and SMO algorithm [0]. Further studies will aim to nd the algorithm for large-scale problems in LPSVMs. In addition the property (say geometry) of LPSVMs deserves further research. Acknowledgements Thanks to all those who contributed to the data sets used in this paper. This work was supported in part by grants of the Nature Science fund (No. 697709) and the National 863 project number 863-306-06-06-1. Appendix A. Proof of Theorem. Proof. Given the examples set ((x 1;y 1);:::;(x i;y i);:::; (x l ;y l )); (x;y) (R n ;R). The examples belong to a sphere of radius R. The set of m -margin separating hyperplanes f(x; w;b)=w T x + b classies examples x as follows: 1; w T x + b ; y = 0; 1; w T x + b 6 ; according to Eqs. (1.6) and (1.7); then the upper bounds on the VC dimension of the set of m -margin separating hyperplanes R w ;n +1: (A.1) In terms of Lemma.1; there exist 0 c + withno dependence withthe vector w suchthat w 6 c w (A.)

W. Zhou et al. / Pattern Recognition 35 (00) 97 936 935 holds true. Where w is any vector norm. Hence; we have the upper bounds on the VC dimension R w ;n +1 c R w 6 min ;n +1: Thus Theorem. is proved. Appendix B. Proof of Theorem.3 Proof. Given the examples set (((x 1);y 1);:::;((x i); y i);:::;((x l );y l )); ((x);y) (H; R). The examples belong to a sphere of radius R. The set of the target functions f(x)= = iy i(x i) (x)+b iy ik(x i; x)+b classies examples x as follows: 1; f(x) ; y = 1; f(x + b) 6 ; 0: (B.1) The set of the target functions is identical to a hyperplane in the space H: In terms of Eqs. (1.6) and (1.7); the bounds is R w ;n +1 R T K = min ;n +1: (B.6) In the following we introduce the concept of weight-norm. Denition.3 (Weight-norm [6]). Let A be any nthpositive denite matrix and x R n be row vectors. Then a function x A =(x T Ax) 1= is a kind of vector norm called weight-norm; or ellipse-norm. By the denition of weight-norm, Eq. (5:7) is a weight-norm. We have w = T K = a K: (B.7) According to Lemma.1, there exits a constant 0 c + without depending on and K suchthat K 6 c (B.8) holds true. Where is any vector norm. Combining Eqs. (5:8), (5:9) and (5:10), we have the bounds on the VC dimension: R T K ;n +1 ([ R K = min ] ) ;n +1 c R 6 min ;n +1: Thus, Theorem is proved. f(x)=w (x)+b; (B.) References where w = iy i(x i): In H; according to Eq. (5:5) we have w = w w = = i jy iy j(x i) (x j) j=1 i jy iy jk(x i; x j): j=1 (B.3) (B.4) Let K denote a l l matrix K(i; j) =y iy jk(x i; x j) and =( 1;:::; l ) T. Then Eq. (5:6) can be rewritten as w = a T Ka: (B.5) [1] V. Vapnik, Statistical Learning Theory, Wiley-Interscience Publication, New York, 1998. [] V. Vapnik, The Nature of Statistical Learning Theory, Springer, NY, 1995. [3] V. Vapnik, An overview of statistical learning theory, IEEE Transac. Neural Networks 10 (5) (1999) 988 999. [4] C.J.C. Burges, A tutorial on support vector machines for pattern recognition, Data Min. Knowledge Discovery (1998) 11 167. [5] A.J. Smola, B. Scholkopf, A tutorial on support vector regression, NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London, UK, 1998. [6] W.P. Cheng, K.Y. Zhang, Z. Xu et al. (Eds.), Matrix Theory. Xi an: North-West Industry University Press, Xi an, China, 1989. [7] G. Wahba, An introduction to model building with reproducing kernel hilbert spaces, Technical Report 100, University of Wisconsin-Madison, Statistics Dept. 000. [8] K.J. Lang, M.J. Witbrock, Learning to tell two spirals apart, Proceedings of the 1989 Connectionist Models Summer School, 1989, pp. 5 61.

936 W. Zhou et al. / Pattern Recognition 35 (00) 97 936 [9] B. Scholkopf, C. Burges, V. Vapnik, Extracting support data for a given task, in: M. Fayyad, R. Uthurusamy (Eds.), Proceedings, First International Conference on Knowledge Discovery & Data Mining, Menlo Park, AAAI Press, http://www.kernel-machines.org/ 1995. [10] B. Scholkopf, C. Burges, V. Vapnik, Incorporating invariances in support vector learning machines, in: C. von der Malsburg, W. von Seelen, J.C. Vorbrggen, B. Sendho (Eds.), Articial Neural Networks ICANN 96, Berlin, Lecture Notes in Computer Science, Vol. 111, Springer, Berlin, 1996, pp. 47 5. [11] C.J.C. Burges, B. Scholkopf, Improving the accuracy and speed of support vector learning machines, in: M. Mozer, M. Jordan, T. Petsche (Eds.), Advances in Neural Information Processing Systems 9, MIT Press, Cambridge, MA, 1997, pp. 375 381. [1] S. Verdu, Multiuser Detection, Cambridge University Press, Cambridge, 1996. [13] U. Madhow, Blind adaptive interference suppression for timing acquisition and demodulation in direct-sequence CDMA systems, IEEE Trans. Commun. 46 (1998) 1065 1075. [14] X. Wang, H.V. Poor, Blind multiuser detection: a subspace approach, IEEE Trans. Inform. Theory 44 (1998) 677 691. [15] S. Raudys, Evolution and generalization of a size neurone, II. Complexity of statistical classiers and sample size consideration, Neural Networks 11 (1998) 97 313. [16] T. Joachims. Estimating the generalization performance of a SVM eciently, LS VIII-Report 5, Universitat Dortmund, Germany. Available in URL: (www.kernel-machine.com/.) [17] S. Raudys, How good are support vector machines, Neural Networks 13 (000) 17 19. [18] V. Vapnik, Estimation of Dependencies Based on Empirical Data, Nauka, Moscow, 1979 (in Russian) (Englishtranslation: Springer, New York, 198). [19] E. Osuna, R. Freund, G. Girosi, Improved training algorithm for support vector machines, Proceedings of the IEEE NNSP 97, Amelia Island, 1997. [0] J. Platt, Fast training of support vector machines using sequential minimal optimization, in: B. Scholkopf, C.J.C. Burges, A.J. Smola (Eds.), Advances in Kernel Methods Support Vector Learning, MIT Press, Cambridge, MA, 1999, pp. 185 08. About the Author WEIDA ZHOU received his B.S. in Engineering from Xidian University, Xi an, China, in 1996. Since 1998 he has been working towards the M.S. degree and the Ph.D. degree at Xidian University. His research interests include machine learning, learning theory and data mining. About the Author LI ZHANG received her B.S. degree in Electronic Engineering from Xidian University, Xi an, China. Since 1997 she has been working towards the M.S. degree and the Ph.D. degree at Xidian University. Her research interests have been in the areas of pattern recognition, machine learning and data mining. About the Author LICHENG JIAO received his B.S. from Shanghai Jiaotong University, Shanghai, China, in 198, the M.S. degree and Ph.D. degree from Xi an Jiaotong University, Xi an, China, in 1984 and 1990, respectively. He is currently Professor and Subdecanal of Graduate School. His research interests include neural network, data mining, nonlinear intelligence signal processing and communication.