Learning Kernel Parameters by using Class Separability Measure

Transcription

1 Learning Kernel Parameters by using Class Separability Measure Lei Wang, Kap Luk Chan School of Electrical and Electronic Engineering Nanyang Technological University Singapore, P 3733@ntu.edu.sg,eklchan@ntu.edu.sg Abstract Learning kernel parameters is important for kernel based methods because these parameters have significant impact on the generalization abilities of these methods. Besides the methods of Cross-Validation and Leave-One-Out, minimizing some upper bounds on the generalization error, such as the radius-margin bound, was also proposed to more efficiently learn the optimal kernel parameters. In this paper, a class separability criterion is proposed for learning kernel parameters. The optimal kernel parameters are regarded as those that can maximize the class separability in the induced feature space. With this criterion, learning the kernel parameters in SVM can avoid solving the quadratic programming problem. The relationship between this criterion and the radius-margin bound is also explored. Both theoretical analysis and experimental results show that the class separability criterion is effective in learning kernel parameters for SVM. 1 Introduction Recently, learning kernels has become an active research area because the performance of a kernel based method, e.g., Support Vector Machines (SVMs [], heavily depends on the kernel function and its parameters. In particularly, for a given kernel function, learning the kernel parameters is the main task. The methods based on Cross-Validation and Leave-One-Out (LOO have been used to learn optimal kernel parameters, however, the computation load is prohibitive in kernels with multiple parameters (An improved LOO estimation method has been proposed in [9] recently. In recent years, minimizing some upper bounds on the generalization error, e.g., the radius-margin bound, was proposed to speed up the learning process for SVMs [,, 5, 1, 3, 7, 13]. In this paper, the class separability measure, a classical concept in pattern recognition, is proposed for learning the kernel parameters. Learning the kernel parameters with this criterion does not require solving the quadratic programming problems when it is applied to SVMs classifiers. It is an easily understood and effective criterion, and it has potential to be used as a general measure for evaluating the goodness of kernels for classification tasks. It is known that a kernel can be interpreted as a function delivering the prior knowledge of classification. It implicitly determines the distributions and the similarity of the patterns in the feature space induced

2 by the selected kernel. Intuitively, a good kernel should maximize the class discriminant in the feature space. It is well-known that the discriminability among the classes in a space can be measured by using the class separability []. Hence, by inspecting the class separability in the feature space, the goodness of a kernel can be estimated, and the optimal kernel parameters can be regarded as those which can maximize the class separability. In this paper, the relationship between the class separability criterion and the radius-margin bound is also explored. It is found that, in pair-wise classification, this criterion is the same as the radius-margin bound when the training samples in the same class equally contribute to the classification (In SVM, only the support vectors are considered contributing to the classification. Both theoretical analysis and experimental results show the effectiveness of the proposed criterion for learning kernel parameters. The criterion of class separability measure Scatter matrix based class separability measure is commonly used due to its simplicity []. The scatter matrices include Within-class scatter matrix (S W, Between-class scatter matrix (S B, and Total scatter matrix (S T. They are defined as follows. S W = c i=1 [ ni j=1 (x i,j m i (x i,j m i ] S B = c i=1 n i(m i m(m i m (1 S T = S W + S B where c is the number of classes, and c is in the case of pair-wise SVM classification. n i (i = 1,, c denotes the number of samples in the i-th class, and x i,j (x i,j R d denotes the j-th sample in the i-th class. m i denotes the mean vectors of the samples in the i-th class and m denotes the mean vector of all the samples. A large separability means that these classes have small within-class scatter and large between-class scatter, and the class separability measure, J, can be defined as J = tr(s B tr(s W or J = tr(s B tr(s T where tr(a denotes the trace of the matrix A. Based on the above definition, the corresponding class separability measure in the feature space can be given as follows. In the following, K denotes a kernel and K θ (x i, x j = Φ(x i, Φ(x j, where Φ( is the nonlinear mapping from the input space, R d, to the feature space, F,, denotes a dot product, and θ denotes the set of kernel parameters. K denotes the kernel matrix (or Gram matrix and {K} i,j = K θ (x i, x j. Let K A,B denote the kernel matrix where {K A,B } i,j = K θ (x i, x j with the constraint of x i A and x j B. Let Sum(K A,B denote the sum of all elements in the matrix K A,B. In the following, the superscript Φ is used to distinguish the variables in the feature space from those in the input space. Let S Φ B and SΦ W denote the between-class and within-class scatter matrices in F, respectively. tr(s Φ B and tr(sφ W can be calculated as follows. tr(s Φ B = c i=1 n i m Φ i m Φ = [ ] c i=1 n Sum(KDi,D i i Sum(K D i,d n n i i n + Sum(K D,D (3 n where D i denotes the set of the training samples of the i-th class, and D is the set of all training samples, n is the total number of training samples, and n = c i=1 n i. Similarly, it can be obtained that tr(s Φ W = c = c i=1 ni i=1 [ ni j=1 Φ(x i,j m Φ i j=1 K θ(x i,j, x i,j Sum(K D i,d i n i ] ( (

3 Then the optimal kernel parameter set, θ, is [ ] tr(s θ Φ = arg max B θ Θ tr(s Φ W = arg max θ Θ [ ] tr(s Φ B tr(s Φ T where Θ denotes the parameter space. This maximization problem can be tackled by using the optimization tools, and the optimal kernel parameter set can be obtained. 3 The relationship to the radius-margin bound The class separability criterion can be used to learn kernel parameters for SVM by setting the number of classes, c, to. It is found that this criterion has a surprising relationship with the radius-margin bound. Let w be the normal vector of the optimal separating hyperplane in an SVM classifier. There is w = n i=1 α0 i y iφ(x i, where n denotes the total number of training samples, and y i is the label of the training sample x i. The coefficient αi 0 is obtained by solving the well-known convex quadratic optimization problem in SVM []. Let γ denote the margin and it is known that γ = w 1. The radius-margin bound, T, proposed by Vapnik [1] can be written as T = 1 R n γ = 1 n R w ( where R is the radius of the smallest sphere in the feature space which can enclose all the training samples, and R can be obtained by solving another convex quadratic optimization problem [1, ]. The optimal kernel parameters are obtained by minimizing the value of R w. Intuitively, there should be a relationship between the class separability criterion and the radius-margin bound. From the definition of the between-class scatter matrix, it is known that tr(s Φ B measures the distance between the mean vectors of two classes. Hence, tr(s Φ B and the margin γ reflect the similar property of the data separability. From the definition of the total-class scatter matrix, it is known that tr(s Φ T measures the scatter of all the samples with respect to the total mean vector. Hence, it positively correlates with the magnitude of R. When learning the kernel parameters with the radius-margin bound, γ is maximized while R is minimized. Correspondingly, when the class separability criterion is used, tr(s Φ B is maximized while tr(sφ T is minimized. Hence, w and R correspond to tr(s Φ B and tr(sφ T, respectively. It can be expected that there is resemblance, to some extent, between these two criteria. In [], Chapelle et al. give the derivatives of w and R for the t-th parameter, θ t (t = 1,, θ as follows. and R = w n i=1 β 0 i = n i,j=1 K θ (x i, x i (5 α 0 i α 0 jy i y j K θ (x i, x j (7 n βi 0 βj 0 i,j=1 K θ (x i, x j ( where βi 0 is obtained by solving the convex quadratic optimization problem of minimizing R. To handle the training error, the L- norm based soft margin is used and the kernel is modified as { Kθ (x K θ (x i, x j = i, x j when i j, K θ (x i, x j + 1 (9 C when i = j.

4 where C is the regularization parameter, and the parameter set, θ, is redefined to be θ = {θ, C}. Let αi 0 (i = 1,, n in equation (7 be { n 1 α i = +1 when y i = +1, n 1 ( 1 when y i = 1. where n +1 and n 1 denote the number of training samples belonging to +1 and 1 classes, respectively. It is easy to find that equation ( satisfies the constraints in SVM that n i=1 α iy i = 0 and α i 0. It can be proven that w α 0 i =eα i = ( n+1+n 1 n +1n 1 tr(s Φ B = w α 0 i =eα i = ( n+1+n 1 n +1 n 1 tr(s Φ B + Z 0 where Z 0 is a constant and Z0 = 0. This equation shows that when αi 0 is set as α i, the derivative of w to θ t has the( opposite sign but the same magnitude to the derivative of tr(s Φ B to θ t if the coefficient n+1 +n 1 n +1 n 1 is ignored. It implies that, in this case, w reaches the minima while tr(s Φ B reaches the maxima, and vice versa. It is known that α0 i represents the contribution of the training sample x i to the optimal separating hyperplane, and the training samples corresponding to non-zero α 0 values are called the support vectors. By solving the convex quadratic optimization problem in SVM, the values of α 0 are obtained and the support vectors can be identified. For the case of tr(s Φ B, it can be found that it corresponds to a special case of w in which the training samples in each class equally contribute to the optimal separating hyperplane. This is because all the training samples are treated equally in calculating the Between-class scatter matrix. A similar relationship exists between R and tr(s Φ T. Let β0 i (i = 1,, n in equation ( be β i = n 1. Recall that n is the total number of training samples. Also, this setting satisfies the constraints that n β i=1 i = 1 and β i 0. In this case, equation ( becomes R β = 0 i =e 1 tr(s Φ T n βi = R β 0 i =e βi = 1 n tr(sφ T + Z 1 where Z 1 is a constant and Z 1 = 0. It is known that R is the radius of the smallest sphere enclosing all the training samples, and the center of the sphere, c 0, is c 0 = n i=1 β0 i Φ(x i. Hence, βi 0 represents the contribution of the training sample x i to c 0. It can be seen that tr(s Φ T corresponds to a special case of R when the center of the sphere is just the mean vector of all the training samples. Also, this result is expected because in the calculation of the Total-class scatter matrix all training samples are treated equally. As mentioned above, in the case of L- norm based soft margin, the regularization parameter C is also included as a special kernel parameter. It is found that [ ] tr(s Φ B C tr(s Φ T = 0 = C (13 This means that the class separability criterion cannot be used to predict the optimal regularization parameter for SVM. However, because this criterion can predict the other kernel parameters in general, the optimal kernel parameter set can still be learnt by performing an extra Cross-Validation against C. In summary, there is really a relationship between these two criteria. Roughly speaking, maximizing tr(s Φ B is analogous to minimizing w while minimizing tr(s Φ T is analogous to minimize R. (11 (1

5 Preliminary experimental results The following experiments verify the relationship between the class separability criterion and the radius-margin bound, and evaluate the effectiveness of the proposed criterion in learning the kernel parameters for SVM. The benchmark data sets including Thyroid, Heart, Titanic, Banana, Breast Cancer, Diabetes, and Germen are used in the experiments []. Each data set includes 0 instantiations of training and test sets for +1 and 1 classes, respectively. The Gaussian RBF kernel, K(x, y = exp( x y σ is used and the learnt parameter is the width σ. The value of ( 1.0*trace(S B The value of Wsq The log value of sigma (Thyroid The value of trace(s T The log value of sigma (Thyroid The value of ( 1.0*trace(S B /trace(s T The log value of sigma (Thyroid (a-1 ( tr(s Φ B (a- (tr(sφ T (a-3 ( tr(sφ B tr(s Φ T The value of Rsq The value of Rsq*Wsq The log value of sigma (Thyroid The log value of sigma (Thyroid 0 The log value of sigma (Thyroid (b-1 ( w (b- (R (b-3 (R w Figure 1: The comparison on the Thyroid data set In the first experiment, under different values of σ and C (Recall that C is the regularization parameter, tr(s Φ B and tr(sφ T are compared with w and R, respectively. Figure 1 shows the comparison on the Thyroid data set, and these results are averaged over the first ten training sets. The horizontal axes of these six sub-figures show the log values of σ. From sub-figures (a-1 and (b-1, it can be seen that, under different values of C, the curves of tr(s Φ B are similar to the ones of w, respectively. For the same value of C, the two curves give the minima at about similar values of σ. The similarity between tr(s Φ T and R can also be seen from sub-figures (a- and (b-. Sub-figures (a-3 and (b-3 show the curves of the class separability criterion and the radius-margin bound. It can be seen that these two criteria show surprising similar appearance. In this case, given a C, the optimal σ obtained by the radius-margin bound and that by the class separability criterion is close. Similar observation can be obtained from Figures and 3 for the Heart and Titanic data sets, respectively. In the second experiment, the class separability criterion and the radius-margin bound are used to learn the width σ for the seven benchmark data sets. The test errors obtained by the SVM classifiers using the Gaussian RBF kernels with the two widths, respectively, are compared. The class separability criterion and the radius-margin bound are optimized, respectively, on the first five training sets of each data set, and the average values of the five optimized widths are selected. Note that because the class separability criterion cannot predict the optimal value of the regularization parameter, C, the value predicted by

6 The value of ( 1.0*trace(S B The log value of sigma (Heart The value of trace(s T The log value of sigma (Heart The value of ( 1.0*trace(S B /trace(s T The log value of sigma (Heart (a-1 ( tr(s Φ B (a- (tr(sφ T (a-3 ( tr(sφ B tr(s Φ T The value of Wsq The value of Rsq 1 The value of Rsq*Wsq The log value of sigma (Heart The log value of sigma (Heart 0 The log value of sigma (Heart (b-1 ( w (b- (R (b-3 (R w Figure : The comparison on the Heart data set 1 The value of ( 1.0*trace(S B 15 0 The value of trace(s T The value of ( 1.0*trace(S B /trace(s T The log value of sigma (Titanic The log value of sigma (Titanic The log value of sigma (Titanic (a-1 ( tr(s Φ B (a- (tr(sφ T (a-3 ( tr(sφ B tr(s Φ T The value of Wsq The value of Rsq 1 The value of Rsq*Wsq The log value of sigma (Titanic The log value of sigma (Titanic 1 The log value of sigma (Titanic (b-1 ( w (b- (R (b-3 (R w Figure 3: The comparison on the Titanic data set the radius-margin bound is used instead 1. With the selected width, the SVM classifier is applied to the total 0 pairs of training and test sets in each data set, and the test errors are averaged. The test errors obtained from the same data sets by other state-of-art classifiers (RBF classifier, AdaBoost, Regularized AdaBoost, SVM, and Kernel Fisher Discriminant are extracted from [11] and listed against our method. In these classifiers, 1 The values of C and the part of σ values learnt by the radius-margin bound are taken from []. The width σ in this classifier is selected by using a 5-fold-cross validation.

7 Data set C σ C SVM C σ R SVM R RBF AB AB R SVM KFD Banana (±0.5 (±0.5 (±0. (±0.7 (±0. (±0.7 (±0.5 B. Cancer (±.3 (±.3 (±.7 (±.7 (±.5 (±.7 (±. Diabetes (±1. (±1. (±1.9 (±.3 (±1. (±1.7 (±1. German (±. (±.5 (±. (±.5 (±.1 (±.1 (±. Heart (±3.1 (±3. (±3.3 (±3. (±3.5 (±3.3 (±3. Thyroid (±.3 (±.0 (±.1 (±. (±. (±. (±.1 Titanic (±1. (±1. (±1.3 (±1. (±1. (±1.0 (±.0 Table 1: Regularized parameter, learnt widths, and averaged test error on the benchmark data sets (Number in the bracket is the standard deviation the selection of model parameters (including kernel parameters are based on the first five pairs of training and test sets in each data set. A 5-fold-cross validation is performed to estimate the good model parameters, and the median of the five estimations is selected. Also, the test errors are averaged on the total 0 test sets. Table 1 shows the regularization parameter, the two learnt widths, and the values of test errors. The column of C lists the optimal values of C given by the radius-margin bound. They are used by the class separability criterion when optimizing the width σ. σc and σ R are the optimal widths obtained by using the class separability criterion and the radius-margin bound, respectively. SVM C and SVM R denote the SVM classifiers in which the width of the Gaussian RBF kernels are σc and σ R, respectively. RBF denotes a single RBF classifier, and AB and AB R denote the classifiers based on AdaBoost and the regularized AdaBoost, respectively. SVM denotes the SVM classifier in which the width is learnt by using the 5-fold-cross validation. KFD denotes the kernel fisher discriminant in which the Gaussian RBF kernel is used. It can be seen, from Table 1, that the widths predicted by the two criteria are similar and the performance of SVM C is comparable to those of SVM R and the other classifiers. These results indicate that the class separability measure is effective in learning the kernel parameters. In summary, the preliminary experimental results verified the relationship between the class separability criterion and the radius-margin bound. Also, the effectiveness of the proposed criterion in learning the kernel parameters for SVM is demonstrated. The class separability criterion can be easily extended to multi-class classification, e.g. Kernel based Multiple Discriminant Analysis (KMDA, for which the radius-margin bound cannot be applied directly. The class separability criterion also has the potential to be a general measure to evaluate and compare different kernels, even if they have different forms. 5 Conclusion and future work This paper proposes a novel criterion to learn the optimal kernel parameters in the kernel methods for classification. This criterion is based on the class separability measure, and the optimal kernel parameters are regarded as those that maximize the separability of the classes in the induced feature space. The relationship between this criterion and the radius-margin bound is analyzed and verified, and the experimental results demonstrate the effectiveness of this criterion in learning the kernel parameters for SVM. By applying

8 the class separability criterion to learn the kernel parameters for SVM, the quadratic optimization problem can be avoided. In the future work, the use of this criterion in handling multi-class classification and as a general measure to evaluate the goodness of kernels will be explored. References [1] O. Chapelle and V. Vapnik. Model selection for Support Vector Machines. Advances in Neural Information Processing Systems 1, MIT Press, 000. [] O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for Support Vector Machines. Machine Learning, (1-3: , 00. [3] N. Cristianini, C. Campbell, and J. Shawe-Taylor. Dymamically Adapting Kernels in Support Vector Machines. Technical Report, NC-TR , May, 199. [] N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines. Cambridge University Press, 000. [5] K. Duan, S. S. Keerthi, and A. N. Poo. Evaluation of simple performance measures for tunning SVM hyperparameters. Technical Report CD-01-11, National University of Singapore, Singapore. mpessk/svm.shtml, 001. [] R. O. Duda, D. G. Stork, and P. E. Hart. Pattern classification (second edition. John Wiley and Sons, page 115, 001. [7] T. Joachims. Estimating the generalization performance of an SVM efficiently. Proceedings of the 17th International Conference on Machine Learning (ICML, pages 31 3, 000. [] S. S. Keerthi. Efficient tunning of SVM hyperparameters using radius/margin bound and iterative algorithms. Technical Report CD-01-0, National University of Singapore, Singapore. mpessk/svm.shtml, 001. [9] J.-H. Lee and C.-J. Lin. Automatic Model Selection for Support Vector Machines. cjlin/papers.html, November, 000. [] G. Rätsch. raetsch/data/benchmarks.htm. [11] G. Rätsch, T. Onoda, and K.-R. Müller. Soft margins for AdaBoost. Machine Learning, (3:7 30, 001. [1] V. Vapnik. Statistical Learning Theory. John Wiley and Sons Inc., New York, 199. [13] V. Vapnik and O. Chapelle. Bounds on error expectation for SVM. Advances in Large Margin Classifiers, Cambridge, MA, MIT Press, pages 1 0, 000.