A New SVM Model for Classifying Genetic Data: Method, Analysis and Results

Transcription

1 A New SVM Model for Classifying Genetic Data: Method, Analysis and Results Rosemary A Renaut Wang-Juh Chen Hongbin Guo Abstract In this paper we present a new formulation of the Support Vector Machine for classifying data. It is based on development of ideas from methods of total least squares, in which error in measured data is incorporated in the model design. The new formulation studied is similar to the soft margin SVM, but has to be solved using nonlinear optimization rather than quadratic programming. Initial results are discussed demonstrating its robustness for classification of data with a large feature space. 1 Introduction There has been recently increased interest in algorithms based on the Support Vector Machine (SVM), [20], as providing novel data mining tools for feature selection (classification) [4, 22, 10]. It has been found to be useful in many applications including pattern recognition, image processing, and text categorization etc. [5, 21]. In the analysis of genome function, microarray data, which are obtained as a gene expression matrix, may be analysed using the SVM, [8, 3, 7] and [13]. Brazma and Vilo, [2], and Brown et al [3] performed a minireview of techniques for microarray data analysis and deduced the efficacy of SVM which among algorithms tested gave the best prediction accuracy for the functional classes that are expected to be co-regulated. In this proposal we present a brief overview of, what are by now, standard, formulations for the SVM and modifications proposed by different authors. Many references exist which provide more details and overview of recent developments, [9, 6, 16, 21, 11, 1]. Sufficient overview is provided so as to relate the proposed inequality constrained data least squares (DLS) SVM with the hard margin SVM. In Section 2 we present the existing formulations. Our new formulation is developed in Section 3. Section 4 contains the experimental set-up and initial re- Supported by NSF grant DMS Arizona State University, Department of Mathematics and Statistics, Tempe, AZ Tel: , Fax: Maricopa County, Assessor Department, Suite 130B, 301 W. Jefferson Rd, Phoenix, AZ, sults and in Section 5 we discuss future work. The basic premise of the new model is to recognize that feature data presented to the SVM for classification, is prone to error in the measurements. The resulting SVM model can be seen as an errors in variables model [19], and can thus be treated accordingly. Dependent on whether the constraint equations, that define the classification of each data pair (feature space and its class), are imposed as equality or inequality, different models result. Here we focus on the constraints imposed as inequality, which yields a dual problem that is solved using a standard bound constrained nonlinear minimization, and assume that required background on optimization as standard, can be found for example in and standard text such as [12]. 2 Methods Given a training set {x m, t m } M with input data x m R N and class labels t m { 1, +1}, the purpose of the SVM is to find a decision function (classifier) (2.1) f(x) = w T φ(x) + b, which separates the data into two classes, according to the sign of f. The hyperplane given by f(x) = 0, separates the two classes of data, assuming that they are separable, and is determined by maximizing the margin 2/ w between the two sets of points lying on the planes given by f(x) = ±1, [20]. Function φ(x) can be a suitably chosen nonlinear function which maps the input space into a higher-dimensional space, although here we will primarily focus on the linear SVM in which φ(x) = x. In this case, the classification of the training data provides the linear constraint equations t m [w T φ(x m ) + b] = t m [w T x m + b] 1, m = 1,, M. (2.2) Dependent on how the constraints for the training data are imposed, whether the data is separable or not, different SVMs can be derived. 2.1 Hard Margin SVM The optimal separating hyperplane is found by solving the following optimization problem {w, b } = argmin J H (w)

2 = argmin { 1 2 w 2 2, subject to (2.3) t m [w T φ(x m ) + b] 1 m = 1,, M.} We introduce the matrices (2.4) A(m, :) = T (m, m)(φ(x m )) T where A(m, :)is the m th row of A R M N, (2.5) T = diag(t 1, t 2,..., t M ), and the vectors of length M may be classified using (2.1), although this classification is typically written in terms of the dual variables (2.15) f(x) = M α m t m (x m ) T x + b. Without loss of generality this sum is over positive values for the support values only. Let, for any object o, vector or matrix, õ be the object with rows m removed when α m = 0. Then the decision function for data classification is given by e = [1,, 1] T, α = [α 1,, α M ] T. (2.16) class(x) = sign{ α T Ã x + b}. The Lagrangian for (2.3) is (2.6) L PH (w, b, α) = 1 2 w 2 The dual problem is given by (2.7) α T [Aw + bt e], max min L P H (w, b, α). α 0 w,b The optimal solution {w, b, α } with respect to (2.7) is given by (2.8) w L PH = w A T α = 0, L PH (2.9) = α T t = 0 b (2.10) α m [Aw + bt e] m = 0, m = 1,... M (2.11) α m 0, m = 1,... M. Substituting in (2.6) yields the solvable convex quadratic programming problem for the Hard Margin SVM (2.12) α = argmin L DH (α)} = argmin { 1 2 AT α 2 e T α subject to α T t = 0 and α 0}. The positive entries of α correspond to constraints (2.10) which are active and are chosen as the support values for the classification. Zero Lagrange multipliers, α m = 0, correspond to the inactive constraints, those which are only satisfied as inequalities. The primal variables are obtained from (2.8) (2.13) (2.14) w = A T α, b = av(t m w T x m ), where av(x) for vector x is the average of its components taken only over values for which α m > 0. Future data 2.2 Soft Margin SVM In general the data are not linearly separable and there is no feasible solution to the Hard Margin SVM. Slack variables ξ introduced in the constraints (2.2) measure the classification error and lead to the Soft Margin SVM. (2.17) {w, b, ξ } = argmin J S (w, b, ξ) = argmin { 1 M 2 w (ξ m ) p subject to Aw + bt e ξ and ξ 0}. Here is a real positive number which trades off between the size of the margin and the total classification error. When p = 1(2) we obtain the L1(2) Soft Margin SVM, resp, either of which lead to a convex QP problem [1, 6]. The Lagrangian for (2.17) is (2.18) L PS (w, b, α, β) = 1 2 w 2 + e T ξ = α T [Aw + bt + ξ e] β T ξ. The KKT conditions ( ) are augmented by (2.19) α + β = e, β 0, ξ 0, β m ξ m = 0, m = 1,... M, and (2.10) is modified to (2.20) α m [Aw + bt + ξ e] m = 0, m = 1,... M. The Soft Margin SVM is obtained by solving the convex QP (2.12) with additional upper bound constraints imposed on the support values (2.21) 0 α. Data are classified again by (2.16) using (2.13) and (2.14) for calculating w and the bias, but with the average for the bias taken only over all support values which are not at the upper bound.

3 3 Development of Errors in Variables Model The constraint equations on the training data for the soft and LS SVMs described in Section 2.2 introduce a classification error in the data space. Here, recognizing that feature space data is typically obtained from error prone measurements of the features, we propose accounting for the error in the formulation of the problem. Suppose that training datum x m is contaminated by error ɛ m, the m th row of error matrix E is (ɛ m ) T and that matrix E = T E. Notice that as compared to the introduced slack variables in the soft SVM, errors in feature space are not required to be positive. The new constraint equations, without introduction of slack variables for measuring distance from the margin, presented as inequalities are (3.22) (A + E)w + bt e. The objective function to be minimized with respect to the constraint equations (3.22) is (3.23) min J DLS(w, b, E) = 1 w,e 2 w E 2 F, where F denotes the Frobenious norm. Noting the relationship with the data least squares problem, details in [18, 19], we call the classifier obtained after solving this minimization the data least squares (DLS) SVM. The Lagrangian for (3.23) with constraints (3.22) is given by (3.24) L PDLS (w, b, E, α) = 1 2 w E 2 F α T [(A + T E)w e + bt], 0, α 0. The dual problem is given by (3.25) max min L P DLS (w, b, E, α). α 0 w,b,e Minimizing with respect to the primal variables yields the conditions (3.26) (3.27) (3.28) E L = E T αw T = 0, w L = w (A + T E) T α = 0, L b = α T t = 0. Hence E = 1 T αw T and w = (A+T E) T α, from which we obtain, for α 2, (3.29) (3.30) E = w = 1 α 2 T ααt A α 2 AT α. This yields the dual problem (3.31)α = argmin L DDLS argmin { 2( α 2 ) AT α 2 α T e subject to (3.32)α T t = 0 and α 0, α 2.} We now consider the impact of α 2 = on the derivation of the dual problem. In this case, from (3.26) and (3.27) we obtain w = A T α + (T E) T, from which A T α = 0, so that w = (T E) T α. This implies E = T αw T and E 2 F = w 2. Substituting in (3.25) the first terms of the objective cancel and we obtain the new objective to be minimized, L = α T e, subject to α 0 and α 2 =. The latter condition defines one component of α in terms of the others, say without loss of generality, the last component α M so that the problem is to minimize L with respect to components of α other than the last. Rewriting L as α m + αm, 2 differentiating with respect to α j, and setting to zero yields the conditions Hence 1 = α 2 = α j, j = 1... M 1. α2 m ( j=1 i=1 = M( α 2 + α 2 m), αi 2 ) + j=1 and we obtain Mα 2 m =. We conclude that L is minimized with respect to α in the case with α 2 = when for some m, α m = /M. We therefore impose the constraint α m /M as upper bound on each component of α. Naturally, assuming that some of the support values are zero, r are zero, increases the bound proportionally to /(M r). In particular the support values for the DLS-SVM are given as the solution of the the nonlinear constrained minimization for objective (3.31) with the constraints (3.32) replaced by (3.33)α T t = 0 and 0 α m /M, m = 1,... M. We contrast this with the Soft Margin SVM (2.17) in which 2( α 2 ) is replaced by the identity and the upper bound on the support values is. α 2 j

4 4 Initial Experiments We collected lymphoma, ovarian, and myeloma microarray data from different public web sites, for which the dimension of the feature space is far greater than the sample size. To assess the new algorithm DLSSVM, we compare its accuracy with that of SVM in [9]. Thus in all cases when referring to results using SVM, this implies the algorithm in [9]. All data are normalized with respect to the array mean prior to analysis. Robustness with respect to data error is considered through addition of normally distributed random perturbations of size ɛ. To reduce the data dimension, a feature selection technique is introduced for selecting significant features (genes). During the solution of the optimization problem a small amount of zero order regularization is needed in order to prevent problems when the Hessian occurring in the SQP is badly conditioned. One advantage of the SVM is that it provides an appropriate set of support vectors which represent the whole data for future analysis. The choice of the threshold used in identifying the support vectors influences the accuracy during the testing phase. In our algorithm, we calculate the threshold based on the distribution of all the candidate support vectors, as compared to the standard SVM approach of using a constant threshold value. To reduce the computational time, we utilize the distributed MATLAB engine (MDCE). 4.1 Initial Results The error rates using DLSSVM and SVM are the same for the ovarian data set, except when no gene selection is performed. In this case, DLSSVM yields an error rate of 29.00% as compared to an error rate of 64.50% with SVM. When 54.84% of the data set belongs to class 1, it is very easy to get an rate less than 54.84% if all the predictions belong to one class. Due to the nature of this data, for which the samples are not linearly-separable, the best error rate using the standard linear SVM is 26% [8], which represents for this data set just one more sample correctly classified as compared to DLSSVM. This raises the question of why the actual SVM error rate is larger than 54.84%? The reason is in the process of choosing the support values. The candidate support values for both methods are similar, but the selected support values in each case are totally different, because of the threshold used for their selection. The range of candidate support values is from to but 4 are actually less than When the threshold is determined dependent on the data, as in our implementation of the DLSSVM algorithm dependent on 1% of the maximum of the candidate support values, the threshold is set at , and none are excluded. The difference between these two sets of support values impacts their predictive ability in the testing phase. We also consider an additional data set obtained from the ionosphere [23]. This data set has 35 features, with 225 instances belonging to class I and 126 instances belonging to class II. The error rates of DLSSVM and SVM are 31.9% and 33.0% resp., for which the 1.1% difference represents 4 instances. This difference occurs due to the ill-conditioning of the Hessian matrix which is dealt with for the DLSSVM using a Hessian dependent weight on the regularization, instead of a fixed weight for the SVM regularization. For the lymphoma data set, without additional error added, the error rates of DLSSVM and SVM are 2.80% and 9.70% resp.. But when ɛ is of the order of 5%, the error rate of SVM increases to 11.10%, representing one additional misclassification, but that for DLSSVM is unchanged. For ɛ increased to an unrealistically high value of 50%, the DLSSVM error rate increases to 8.30%, but that of SVM increases to 22.2%, representing additional 4 and 8 misclassifications, resp. This result suggests the potential to use DLSSVM when samples are error contaminated. Initial results indicate advantages of DLSSVM implementation Choosing the support values based on the distribution of all candidate support values, rather than due to a constant set threshold, leads to an improvement in performance, even when the support values are very small. The inclusion of a Hessian dependent, as opposed to fixed, regularization, is very helpful, particularly when the Hessian is very ill-conditioned. Through assuming error in the data (feature space), rather than in the classification of the samples, the method is successful where other techniques give poor clasification predictions. These initial results suggest that this particular adaptation of the SVM is worthy of future analysis and efficient implementation, and should be investigated for different kinds of data sets. 5 Future Work There are a number of aspects of our future work, first and foremost is to fully understand and design a computationally efficient and stable algorithm for the solution of (3.33). Currently we have only used the standard constrained nonlinear solver with Matlab, without any particular consideration of the impact of provision of gradient and/or Hessian functions, or on

5 the ability to find a suitable minimum of the ojective. Also, at the moment, we are assuming that the data provided is such that the feature space is much larger than the sample size, so that it makes sense to solve the dual problem. For a larger sample space, it may make sense to solve the primal problem directly, which again then impacts the implementation and stability of the algorithm. Looking at the prior history in the development of the SVM, we notice that one direction of research imposes constraint constraints as equalities, as with the Least Squares (LS) SVM [17]. A similar development for the case with feature contamination, not presented here, generates a SVM very similar to the LS-SVM, but with more complicated eigenvalue structure, very similar also to the methods used for the regularized total least squares problems [15]. Including nonlinear mapping of the feature space, also significantly modifies the formulation, and is a topic of future study. References [1] S. Abe, Support Vector Machines for Pattern Classification, Springer Verlag, London, (2005). [2] A. Brazma and J. Vilo, Minireview: Gene Expression Data Analysis, Federation of European Biochemical Societies, 480, (2000), pp [3] M. P. Brown, W. N. Gumdy, D. Lin, N. Cristiani, C. W. Sugnet, T. S. Furey, M. Ares and D. Haussler, Knowledge-based analysis of microarray gene expression data by using support vector machines, Proceedings of National Academy of Sciences, 97(1), (2000), pp [4] P. S. Bradley and O. L. Mangasarian, Feature selection via concave minimization and support vector machines. In J. Shavlik, editor, Machine Learning Proceedings of the Fifteenth International Converence (ICML 98). ftp://ftp.cs.wisc.edu/math-prog/ tech-reports/98-03.ps. [5] J. Brank, M. Grobelink, N. Milic-Frayling, and D. Mladenic, Training text classifiers with SVM on very few positive examples, 2003, tech. report, MSR- TR [6] C. J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowlege Discovery, 2, (1998), pp [7] M. L. Chow, E. J. Moler and I. S. Mian, Identifying marker genes in transcription profiling data using a mixture of feature relevance experts, Physio Genomics, 5(2), (2001), pp [8] T. S. Furey, N. Duffy, N. Cristianini, D. Bednarski, M. Schummer and D. Haussler, Support Vector Machine Classification and Validation of Cancer Tissue Samples Using Microarray Expression Data, Bioinformatics, 16(10), (2000), pp [9] S. R. Gunn, Support Vector Machines for Classification and Regression, tech. report ISIS-1-98, Department of Electronics and Computer Science, University of Southampton, [10] T. Jebara and T. Jaakkola, Feature selection and dualities in maximum entropy discrimination, In Proceedings, UAI 00: Proceedings of the 16th Conference in Uncertainty in Artificial Intelligence, Stanford University, Stanford, California, USA, June 30 - July 3, 2000, eds. Craig Boutilier and Moisés Goldszmidt, Morgan Kaufmann, , [11] V. Kecman, Support Vector Machines: An Introduction, StuddFuzz., 177, (2005), pp [12] M. Minoux, Mathematical Programming: Theory and Algorithms, John Wiley, [13] S. Mukherjee, Classifying Microarray Data Using Support Vector Machines, in A Practical Approach to Microarray Data Analysis, edited by D. P. Berrar, W. Dubitzky and M. Granzow, [14] J. C. Platt, Using Analytic QP and Sparseness to Speed Training of Support Vector Machines, Advances in Neural Information Processing Systems 11, (1999), pp [15] R. A. Renaut and H. Guo, Efficient Algorithms for Solution of Regularized Total Least Squares, SIAM J.Matrix Analysis, 26, 2, (2005), pp [16] B. Schölkopf, Statistical Learning and Kernel Methods, tech. report MSR-TR , 2000, // [17] J. A. K. Suykens, L. Lukas, P. Van Dooren, B. De Moor and J. Vandewalle, Least Squares Support Vector Machine Classifiers: a Large Scale Algorithm, Proceedings of the European Conference on Circuit Theory and Design, (1999), pp [18] S. Van Huffel, Editor, Recent Advances in Total Least Squares Techniques and Errors in Variables Modeling: Proceedings of the Second International Workshop on Total Least Squares and Errors-in-Variable Modeling, SIAM [19] S. Van Huffel and J. Vandewalle, The Total Least Squares Problem: Computational Aspects and Analysis, SIAM Frontiers in Applied Mathematics, [20] V. Vapnik, The Nature of Statistical Learning Theory, Springer, N.Y., [21] L. Wang, Support Vector Machines: Theory and Applications, StuddFuzz., Editor, 177, [22] J. Weston, S. Mukherjee, O. Chapelle, M. Pontil, T. Poggio and V. Vapnik, Feature Selection for SVMs, In Advances in Neural Information Processing Systems 13, editors, Sara A Solla, Todd K Leen, and Klaus- Robert Mulle, MIT Press, [23] UC Irvine Machine Learning Repository. ics.uci.edu/~mlearn/mlrepository.html