Nearest Neighbors Methods for Support Vector Machines A. J. Quiroz, Dpto. de Matemáticas. Universidad de Los Andes joint work with María González-Lima, Universidad Simón Boĺıvar and Sergio A. Camelo, Universidad de Los Andes September 15, 2015
Background on Pattern Recognition Supervised Learning An important part of Pattern Recognition deals with the problem of classification of data into a finite number of categories. In the usual setting of supervised learning, examples are given that consists of pairs, (X i, Y i ), i n, where X i is the d-dimensional covariate vector and y i is the corresponding category in some finite set C. In the examples, y i is known! Based on these examples ( training data ) an algorithm must be produced that will be able to predict the category, given a new value of the covariate.
Background on Pattern Recognition Supervised Learning: applications Examples of applications of Supervised Learning algorithms cover a wide area: 1 Credit Scoring: Deciding if a bank client will be a good payer of the loan he/she receives. 2 Support in medical diagnostics (for instance classification of tumor cells) 3 Face identification in pictures. 4 Text classification algorithms. 5 Music genre identification. 6 many others...
Background on Pattern Recognition interdisciplinarity Pattern Recognition is an interdisciplinary field, drawing ideas and methods from Probability, Statistics, Optimization and Computer Science. Probabilistic methods (in particular related to VC dimension and Uniform Laws of Large Numbers) are used to prove convergence of the classifiers to the best possible classifier in its class (not necessarily the Bayes Classifier) as the size of the training sample grows. The problem of finding the best classifier within a class very often reduces to solving a quadratic optimization problem. Computer Science has provided an important input to the field in the form of efficient algorithms.
Background on Pattern Recognition Linear/Non-linear methods Methods can be divided into two broad categories: linear and non-linear. Among the linear procedures, should mention Linear Discriminant Analysis, Rosenblatt s Perceptron, Logistic Regression. In these methods the covariate space is divided into halves by a classification hyperplane.
Background on Pattern Recognition Non-linear methods In the last decades there have been lots of developments in non-linear procedures: Neural Networks, CART (Classification and Regression Trees) and its general version, Random Forests, Nearest Neighbors Classifiers, Bagging, Boosting and Support Vector Machines. The superiority (in complex problems) of the non-linear methods has been reported in the literature. In particular, SVM has a great performance in many important problems.
Support Vector Machines Separating Hyperplane in the original covariate space As usual, will assume the context of binary classification, in which y i = ±1 and the problem of finding an optimal classification hyperplane can be stated as 1 minimize w,b 2 w 2 subject to y i (w t X i + b) 1 i = 1,..., n, (1) This problem can be solved when the data are linearly separable (ls), that is, there exists a hyperplane that leaves in one side all the covariate vectors of examples in one category and the covariate vectors for the other category in the other half space.
Support Vector Machines Support Vectors In a linearly separable problem, the support vectors are the covariate points from the training dataset that are closer to the separating hyperplane. This is, the points X i where the constraints are satisfied with equality, i.e. (w ) t X i + b = 1, with w and b the solution to the optimization problem. The solution of problem (1) is the same if only the support vectors are provided as training data. This points out to the importance of these vectors.
Support Vector Machines Example of non linearly separable data
Support Vector Machines Use of kernels and slack variables In order to deal with non-ls data two changes are introduced in the problem: (i) A transformation of the d-dimensional covariates φ(x) = (φ 1 (x), φ 2 (x),...) t is used, where {φ m } m r are some real functions and r > d may be. Thus, the covariate vectors X i are substituted with the new feature vectors φ(x i ). (ii) The dot product in the image r-dimensional space defines a kernel function, K, in the original feature space, given by K(x, x ) = φ(x), φ(x ). (iii) Slack variables are introduced to allow for a margin of error in the classification.
Support Vector Machines The primal problem with the kernel and slack With the changes introduced, problem (1) becomes 1 minimize w,b,ξ 2 w 2 + C n i=1 ξ i subject to y i (w t φ(x i ) + b) 1 ξ i, i = 1,..., n, (2) where C is a positive constant that expresses the cost of loosing separation margin or misclassification. The ξ i are the slack variables and ξ = (ξ 1,..., ξ n ).
Support Vector Machines In the dual problem only K appears The dual problem corresponding to (2) can be written as maximize n α i=1 α i 1 n 2 i,j=1 y iy j α i α j K(X i, X j ) subject to i y iα i = 0, 0 α i C for i = 1,..., n. (3) The corresponding classifier is ( ) class(x) = sgn y i αi K(X i, x) + b i
Support Vector Machines Kernel functions The fact that the transformation φ to a higher dimensional space, appears only through the kernel function K, in the solution of the problem, allows shifting the attention to 1 Conditions for a symmetric function to be a valid kernel, that is, to correspond to the inner product in some higher dimensional space (Mercer s Theorem). 2 Properties of kernels (how to build more sophisticated kernels from simple ones) 3 Which kernels are preferable for a certain application problem. A fairly successful one is the Gaussian (or radial ) kernel, given by K(x, x ) = exp ( γ x x 2), for γ > 0.
Support Vector Machines The support vectors in the 1-norm soft margin problem - From the dual problem solution we have that only those covariate vectors with α i > 0, matter for the solution of the 1-norm soft margin problem. These are the support vectors for the problem. - From the Karush-Kuhn-Tucker conditions, the support vectors will be those for which the margin in problem (2) is 1 or less, that is, those points closer to the separating surface. - The number of support vectors gives a rough upper estimate of the classification error by the SVM solution, namely, the bound is #support vectors n
Support Vector Machines The dual problem is usually very large The dual problem obtained for (2), although easier than the primal, can still be quite large since the number of variables subject to optimization is O(n) and the matrix K(X i, X j ) is an n n matrix, not necessarily sparse. In the optimization literature, decomposition methods that divide the dual problem into a sequence of smaller problems of the same type, have been developed in order to make the calculation feasible for problems in the order of tens of thousand data points. Many of the existing procedures are variations of the ideas proposed by Osuna, Girosi, and Freund (1997).
A sampling and nearest neighbors approach to SVM Sampling and k-nearest-neighbors A fundamental principle in statistics is that a large enough sample will be, with very high probability, representative of the behavior of the data population from which it is sampled. On the other hand, k-nearest-neighbors methods have found many applications in different statistical contexts, from the two-sample-problem, to clustering and even in dimensionality reduction. Given a data set S = {X 1,..., X n } and a vector x, all in R d, the k-nearest-neighbors of x in S is the set N N k (x), formed by those k points that attain the k smallest values of X i x among the data set. When x is one of the X i, N N k (X i ) is formed by the k points from the data set, different from X i and closest to it.
A sampling and nearest neighbors approach to SVM basic idea of proposal Procedure Nearest Neighbors SVM (i) Select a large enough random subsample, T, of the data set. (ii) For the subsample, solve the SVM problem. (iii) Identify the support vector set, V T, corresponding to the subsample T. (iv) For a previously chosen value of k (k << n), find the k-nearest-neighbors of each support vector in V T in the (complete) sample S. Denote N N (V) this set of nearest neighbors.
A sampling and nearest neighbors approach to SVM basic idea of proposal Procedure Nearest Neighbors SVM, continued (v) Redefine the subsample T as T = V T N N (V). (vi) Solve the SVM problem for the new subsample T. If there is not significant improvement in terms of classification error with respect to the previous definition of T, or if the maximum number of iterations has been reached, end the procedure and return the current support vector set V T, the weights vector w and the estimated classification error. Else, go to step (iii).
A sampling and nearest neighbors approach to SVM rationale of proposal The rationale of this procedure is as follows: Let V S be the (unknown) support vectors for the SVM solution corresponding to the whole sample, S. If the subsample T is large enough, we are guaranteed to have, with large probability, a sub-sample point in a small neighborhood of radio ρ n for every vector in V S (as we will see below). These neighbors are likely to be support vectors for the solution of the SVM on problem T. Then, if k is large enough, we will probably find a large percentage of the support vectors in V S by looking at neighbors of points in V T.
A sampling and nearest neighbors approach to SVM Finding k-nearest-neighbors The univariate projection method of Friedman, Basquett and Shustek (1975) continues to be a very good compromise in terms of ease of implementation and low complexity when looking for the k-nearest-neighbors of a point in a sample. The number of operations performed to find the k-nearest neighbors of a set of m points with respect to a sample of size n in dimension d is O(mdk 1/d n 1 1/d ). We only need to look for neighbors in the original feature space R d.
Some covering properties of k-nearest-neighbors of subsamples A sample will have points near a distinguished subset We prove a couple of results that serve as motivation for the methods proposed here. If a sample of size n is taken from a distribution with compact support S in R d, we have a distinguished subset of the sample and a random subsample of size δn for some positive δ. Then, very likely, the subsample will contain points at distance less than ρ from every point in the distinguished set, for any ρ such that ρ d is significantly larger than ln n/n.
Some covering properties of k-nearest-neighbors of subsamples A sample will have points near a distinguished subset Proposition Suppose the covariates X i, for i n form an i.i.d. sample from a probability distribution P on R d with totally bounded support set S. Furthermore, let us suppose that the probability law P is bounded away from zero, in the sense that there exists a positive constant a 1, such that, for every x S, and r > 0 small enough, if X P, then Pr( X x r) a 1 ν d r d, where ν d is the volume of the unit ball in R d. Let us also assume that the kernel K is a Lipschitz function, with Lipschitz constant C. Then, for every φ(x) φ(s), X P, and r > 0 small enough, we have ( ) r 2 d Pr( φ(x ) φ(x) r) p := a 1 ν d. 2C
Some covering properties of k-nearest-neighbors of subsamples Theorem Under the assumptions of Proposition 1, let A = {x 1, x 2,..., x m } be a subset of the covariate sample {X 1,..., X n } and A = {x 1, x 2,..., x m} a subset of the transformed covariate sample, {φ(x 1 ),..., φ(x n )} where m = αn, for some α (0, 1). Suppose we take a random subsample M of the covariate sample of size n = δn, where δ is a small positive number. Let M = φ(m). If the radii ρ = ρ(n) and r = r(n) satisfy ρ d >> ln(αn) δ n a 1 ν d and ( r 2 2C ) d >> ln(αn) δ n a 1 ν d (4) then, with high probability, for every x i A and every x j A there are a X i M and a φ(x j ) φ(m), such that x i X i ρ and x j φ(x j ) r. (5)
Some covering properties of k-nearest-neighbors of subsamples Covering by nearest neighbors To connect Theorem 1 with the procedures we propose here, we would like to know how large must be the number of neighbors, k, in order for all the k-nearest-neighbors balls of the points in the subsample M to have radius larger or equal than the ρ of inequality (4). To give a result in that direction, we will add some assumptions to those of the previous Theorem. For a positive integer k, (k < n) and x S, let N N k (x) be the set of k-nearest-neighbors of x in the covariate sample S. Let τ k (x) denote the largest distance from x to an element of N N k (x).
Some covering properties of k-nearest-neighbors of subsamples Covering by nearest neighbors Theorem In the same setting of Theorem 1, add the following assumption: Suppose the distribution P of the training covariate data {X 1,..., X n } is bounded above, in the sense that there exists a positive constant a 2, such that, for every x S, and ɛ > 0 small enough, if X P, then Pr( X x ɛ) a 2 ν d ɛ d. Then, for ρ d = ln2 (α n) δna 1 ν d and k = 2a 2 ln 2 (α n) δa 1, (6) with high probability, τ k (x) > ρ for every x in the subsample M.
Implementation issues and performance evaluation Using small values of k iteratively The value of k in the statement of Theorem 2 could in principle be approximated: The constants a 1 and a 2 are estimable by non-parametric density estimation methods, for instance, while δ is given by our choice of subsample size and α, although unknown, could be guessed from solutions obtained previously for similar problems, or estimated from solutions for subsamples or some other method. Still, the value of k thus obtained, being an upper bound, could be too large in terms of computation time savings. It will be preferable to use relatively small values of k and work iteratively, augmenting in relatively small steps the size of the subsample considered, until the classification error on the test data set ceases to improve significantly.
Implementation issues and performance evaluation Modification of the basic algorithm Procedure Nearest Neighbors SVM, modified (i) Select a large enough random subsample, T, of the data set. (ii) For the subsample, solve the SVM problem. (iii) Identify the support vector set, V T, corresponding to the subsample T. (iv) For a previously chosen value of k (k << n), find the k-nearest-neighbors of each support vector in V T in the (complete) sample S. Denote N N (V) this set of nearest neighbors.
Implementation issues and performance evaluation Modification of the basic algorithm Procedure Nearest Neighbors SVM, continued (v) Redefine the subsample T as T = V T N N (V) R (j), where R (j) is a new random subsample of size ɛn. (vi) Solve the SVM problem for the new subsample T. If there is not significant improvement in terms of classification error with respect to the previous definition of T, or if the maximum number of iterations has been reached, end the procedure and return the current support vector set V T, the weights vector w and the estimated classification error. Else, go to step (iii).
Implementation issues and performance evaluation Parameter choice In preliminary evaluations, we found that δ = 0.01 (that is, subsamples of 1% of the training data set) is a small initial subsample size for most data sets considered in our experiments, while δ = 0.1 (subsamples of 10% of the training data) is a medium-to-large value in the sense that with this choice of δ, the procedures are able to find a significant part of the support vectors of the full training data and achieve a better testing classification error. In the evaluations presented below, we used these two values, 1% and 10% for the parameter δ. When δ = 0.01 we used k = 5 as the number of neighbors, while a smaller value, k = 3 was used when δ was 10%. For the parameter ɛ in Algorithm 2, we used always 10%. Thus, in what follows, the performance of four procedures is considered: Algorithms 1 and 2 with small sized (δ = 1%) and medium sized (δ = 10%) starting subsamples.
Implementation issues and performance evaluation Kernels considered Kernels used in our evaluation were the linear, radial and polynomial of degree two. These are given by the formulas K l (x, y) = x t y K r (x, y) = exp( γ x y 2 ) K p (x, y) = (x t y + a) 2 Parameters corresponding to the kernels and other algorithm parameters, such as the constant C, were chosen by cross-validation in subsamples.
Implementation issues and performance evaluation Information on the datasets used We test the performance of Algorithms 1 and 2 on six datasets from the LIBSVM library using the LIBSVM code 1. We compile with R using the e1071 package 2 on a personal computer, MacBook 2.4 GHz and Intel Core 2 Duo processor with 4 GB 1067 MHz DDR3 memory. In the selection of the datasets for our evaluation, we have tried to include problems with widely varying numbers of features, to appreciate the effect of dimensionality on the methods studied and also to consider datasets with different fractions of support vectors. 1 For more information on the LIBSVM library and the datasets go to www.csie.ntu.edu.tw/ cjlin/libsvm/ 2 For more information on the e1071 package go to cran.r-project.org/web/packages/e1071/.
Implementation issues and performance evaluation Information on the datasets used DATASET TrainSize TestSize Features A9A 32561 10000 123 COD-RNA 59535 20000 8 IJCNN1 49990 15000 22 SVMGUIDE1 3089 4000 21 W7A 24692 8000 300 COVTYPE 50000 10000 54 Table : Datasets Description
Implementation issues and performance evaluation Some results COD-RNA - Radial IJCNN1 - Polynomial Iter Error Rate Supports RSupports Time (s) Iter Error Rate Supports RSupports Time (s) 0 3.81% (0.12%) 9.18% 9.18% 2.108 0 3.23% (0.19%) 6.75% 6.75% 1.592 1 3.75% (0.05%) 31.78% 22.49% 12.286 1 2.88% (0.19%) 22.09% 16.50% 8.494 2 3.65% (0.11%) 50.04% 35.20% 29.778 2 2.41% (0.2%) 35.82% 27.28% 18.614 3 3.67% (0.07%) 64.74% 48.03% 54.194 3 2.48% (0.22%) 47.97% 38.58% 31.516 4 3.58% (0.07%) 76.34% 60.15% 84.524 4 2.3% (0.21%) 57.93% 49.82% 48.518 Total 3.48% 215.11 Total 2.07% 466.63 W7A - Linear SVMGUIDE1 - Polynomial Iter Error Rate Supports RSupports Time (s) Iter Error Rate Supports RSupports Time (s) 0 2.22% (0.38%) 5.71% 5.71% 0.84 0 4.99% (0.91%) 7.59% 7.59% 0.018 1 2.07% (0.23%) 17.15% 12.62% 3.654 1 3.92% (0.49%) 26.72% 19.34% 0.046 2 1.79% (0.15%) 27.94% 19.56% 8.01 2 4.06% (0.48%) 43.43% 29.78% 0.072 3 1.66% (0.11%) 35.50% 28.41% 13.772 3 3.73% (0.29%) 57.37% 43.21% 0.124 4 1.48% (0.04%) 41.09% 37.18% 20.692 4 3.81% (0.13%) 68.10% 54.38% 0.194 Total 0.98% 211.36 Total 3.60% 0.33 COVTYPE - Radial A9A - Linear Iter Error Rate Supports RSupports Time (s) Iter Error Rate Supports RSupports Time (s) 0 35.34% (1.27%) 8.74% 8.74% 8.092 0 15.98% (0.17%) 8.99% 8.99% 4.988 1 35.47% (1.45%) 35.33% 29.11% 90.632 1 15.45% (0.19%) 34.01% 26.99% 56.064 2 34.58% (0.55%) 57.69% 47.96% 261.66 2 15.41% (0.18%) 54.65% 44.23% 187.068 Total 33.43% 908.97 Total 15.33% 1647.33