Multi-instance Support Vector Machine Based on Convex Combination

Transcription

1 The Eighth International Symosium on Oerations Research and Its Alications (ISORA 09) Zhangjiajie, China, Setember 20 22, 2009 Coyright 2009 ORSC & APORC, Multi-instance Suort Vector Machine Based on Convex Combination Zhi-Xia Yang,2 Naiyang Deng 3, College of Mathematics and System Science, Xinjiang University, Urumuchi Academy of Mathematics and Systems Science, CAS, Beijing College of Science, China Agricultural University, 00083, Beijing, China Abstract This aer resents a new formulation of multi-instance learning as maximum margin roblem, which is an extension of the standard C-suort vector classification. For linear classification, this extension leads to, instead of a mixed integer quadratic rogramg, a continuous otimization roblem, where the objective function is convex quadratic and the constraints are either linear or bilinear. This otimization roblem is solved by an iterative strategy solving a convex quadratic rogramg and a linear rogramg alternatively. For non-linear classification, the corresonding iterative strategy is also established, where the kernel is introduced and the related dual roblems are solved. The reliary numerical exeriments show that our aroach is cometitive with the others. Keywords Multi-instance classification; Suort vector machine; Convex combination Introduction Multi-instance learning (MIL) is a growing field of research in machine learning. The term multi-instance learning was coined by Dietterich et al.[] when they were investigating the roblem of drug activity rediction. In the MIL roblem, the training set is comosed of many bags each involves in many instances. A bag is ositively labeled if it contains at least one ositive instance; otherwise it is labeled as a negative bag. The task is to find some decision function from the training set for correctly labeling unseen bag. So far, the MIL has been studied in many fields, e.g. in [2, 3, 4]. An increasing number of methods have been develoed to solve MIL roblems, e.g. [5, 6, 7]. In this aer, insired by [8, 0], we roose a new formulation of MIL as maximum margin roblem, which is an extension of the standard C-suort vector classification (C-SVC). For linear classification, this extension leads to, instead of a mixed integer quadratic rogramg, a continuous otimization roblem, where the objective function is convex quadratic and the constraints are either linear or bilinear. This otimization roblem is solved by an This work is suorted by the National Natural Science Foundation of China (No.0802), the Key Project of the National Natural Science Foundation of China (No ), the China Postdoctoral Science Foundation funded Project (No ) and the Ph.D Graduate Start Research Foundation of Xinjiang University Funded Project (No.BS0800). Corresonding author. dengnaiyang@vi.63.com

2 482 The 8th International Symosium on Oerations Research and Its Alications iterative strategy solving a convex quadratic rogramg and a linear rogramg alternatively. For non-linear classification, the corresonding iterative strategy is also established, where the kernel is introduced and the related dual roblems are solved. The reliary numerical exeriments show that our aroach is cometitive with other formulations. 2 Multi-instance SVM Muliti-instance learning roblem: Suose there is a training set {(X,y ),,(X l,y l )}, () where, when y =, X i = {x i,,x il } is call as ositive bag and (X i,y i ) imlies that there exists at least one instance with ositive label in X i ; when y i =, X i = {x i,,x ili } is called as negative bag and there exists no any instance with ositive label in X i. The task is to find a function g(x) such that the label of any instance in R n can be deduced by the decision function 2. Linear Multi-instance SVM The training set () is reresented as f (x) = sgn(g(x)). (2) T = {(X,y ),,(X,y ),(x r+,y r+ ),,(x,y )}, (3) where y = = y =, y r+ = = y r+q =, (X i,y i ) = (X i,) imlies that there exists at least one instance with ositive label in X i = {x j j I(i)} and (x i,y i ) = (x i, ) imlies that the label of the instance x i is negative. Suose the searating suerlane is given by Note that it has been ointed out by [0] that the constraint (w x) + b = 0. (4) max (w x j) + b (5) is equivalent to the fact that there exist convex combination coefficients v i j 0, v i j = j (w v i jx j ) + b. (6) Thus, introducing slack variable ξ = (ξ,,ξ,ξ r+,,ξ ) T and the enalty aram-

3 Multi-instance Suort Vector Machine Based on Convex Combination 483 eters C and C 2, maximum margin rincile leads to w,b,v,ξ 2 w 2 +C ξ i +C 2 ξ i, (7) (w v i jx j ) + b ξ i, i =,,, (8) (w x i ) + b + ξ i, i = r +,,r + s, (9) ξ i 0, i =,,,r +,,r + s, (0) v i j 0, j I(i), i =,,, () v i j =, i =,,. (2) For solving the roblem (7) (2), we get the following algorithm: Algorithm. (Linear Multi-instance Suort Vector Classification) () Given a training set (3); (2) Select roer enalty arameters C,C 2 > 0; (3) Select initial values {v i j }, and denote it as vi j (), e.g select vi j () = I(i), j I(i), j =,,, where I(i) stands for the number of the element in the set I(i), i.e. the number of the instances in the ositive bag X i. Set k = ; (4) Comute w(k) from {v i j (k)}: First, construct the series x,, x, x r+,, x by x i = v i jx j, i =,,, (3) x i = x i, i = r +,,r + s, (4) where v i j is substituted by vi j (k). Then solve the convex quadratic rogramg α j=y i y j ( x i x j )α i α j + 2 α j y i α i + j=y i y j ( x i x j )α i α j + 2 j=r+ y i y j ( x i x j )α i α j j=r+ y i y j ( x i x j )α i α j α j, (5) y i α i = 0, (6) 0 α i C,i =,,, (7) 0 α i C 2,i = r +,,r + s, (8) and get the solution ᾱ = (ᾱ,,ᾱ,ᾱ r+,,ᾱ ) T. At last, comute w by w = ᾱ i y i x i + ᾱ i y i x i, (9)

4 484 The 8th International Symosium on Oerations Research and Its Alications and take it as w(k); (5) Comute {v i j (k + )} from w(k): Solve the linear rogramg with the variable {v i j }, b and ξ obtained from roblem (7) (2) by taking w = w(k). Take its solution {v i j } with resect to v as {vi j (k + )}; (6) Comare {v i j (k + )} to {vi j (k)}. When their difference is small enough, construct the decision function f (x) = sgn((w x) + b ), (20) where w = w(k) and b is the just obtained solution b with resect to b to the linear rogramg in ste (5). And sto; Otherwise, set k = k +, go to ste (4). 2.2 Nonlinear Multi-instance SVM Using a kernel function K(x,x ), the general multi-instance suort vector classification is obtained by extending the linear multi-instance suort vector classification. In fact, in order to arrive nonlinear searation, introduce the ma Φ : R n H, x x = Φ(x). (2) Let the searating suerlane in the Hilbert sace H be (w x) + b = 0. (22) For a ositive bag X i, the convex combination v i j Φ(x j) of the images {Φ(x j ) j I(i)} is considered. Thus, corresonding to the roblem (7) (2), we should solve the otimization roblem w,b,v,ξ 2 w 2 +C ξ i +C 2 ξ i, (23) (w v i jφ(x j )) + b ξ i, i =,,, (24) (w Φ(x i )) + b + ξ i, i = r +,,r + s, (25) ξ i 0, i =,,,r +,,r + s, (26) v i j 0, j I(i), i =,,, (27) v i j =, i =,,. (28) For solving the above roblem, we arrive the following algorithm. Algorithm 2. (Multi-instance Suort Vector Classification) () Given a training set (3); (2) Select roer kernel function K(x,x ) and anelty arameter C,C 2 > 0; (3) Select initial values {v i j }, and denote it as vi j, e.g select vi j = I(i), j I(i), j =,,, where I(i) stands for the number of the elements in the set I(i), i.e. the number of the instances in the ositive bag X i ;

5 Multi-instance Suort Vector Machine Based on Convex Combination 485 (4) Comute ᾱ from {v i j }: Solve the convex quadratic rogramg α j=y i y j α i α j K(x i,x j ) + 2 α i y i α i + j= j=r+ j=r+ y i y j α i α j ( v i k K(x k,x j )) k I(i) y i y j α i α j ( k I(i) v i k l I( j) y i y j α i α j ( v j k K(x i,x k )) k I( j) v j l K(x k,x l )) α i, (29) y i α i = 0, (30) 0 α i C, i =,,, (3) 0 α i C 2, i = r +,,r + s, (32) and get the solution ᾱ = (ᾱ,,ᾱ,ᾱ r+,,ᾱ ) T, For all i, j, set {ṽ i j } = {vi j }; (5) Comute { v i j } from ᾱ,ṽ: Solve the linear rogramg obtained from roblem (23) (28) with substituting w by w = ᾱ i y i Φ(x i ) + ᾱ i y i ( ṽ i jφ(x j )), (33) and get its solution { v i j } with resect to {vi j }; (6) Comare { v i j } to {ṽi j }. When their difference is small enough, construct the decision function f (x) = sgn(g(x)), (34) where g(x) is obtained by and either or b = y j g(x) = b = y j ᾱ i y i K(x i,x) + y i ᾱ i ( k I( j) y i ᾱ i K(x i,x j ) v j k K(x i,x k )) ᾱ i y i ( v i jk(x j,x)) + b. (35) y i ᾱ i ( v i l K(x l,x j )); (36) l I(i) y i ᾱ i ( l I(i) v i l k I( j) v j k K(x l,x k )). (37) Here ᾱ and {v i j } are the latest corresonding values; Otherwise set {vi j } = { vi j }, go to ste (4).

6 486 The 8th International Symosium on Oerations Research and Its Alications 3 Numerical Exeriment To estimate the caabilities of our roosed methods, we reort the results on four ublic benchmark datasets, one from the UCI machine learning reository [], and three from [8]. The datasets from [8] are used to evaluate our linear multi-instance suort vector classification, where two datasets, Elehant" and Tiger", are from an image annotation task in which the goal is to detere whether or not a given animal is resent in an image; the rest one dataset, TST", is from the OHSUMED data and the task is to learn binary concets associated with the Medical Subject Headings of MEDLINE documents. The Musk" dataset from the UCI machine learning reository [] is used to test our nonlinear multi-instance suort vector classification, which involves bags of molecules and their activity levels and is commonly used in multi-instance classification. Detailed information about these datasets can be found in [0]. We use linrog function in MATLAB to solve the linear rogramg in the two algorithms. For the quadratic rogramg, we use SVMlight[2] and quadrog function with MATLAB in Algorithm and Algorithm 2 resectively. The testing accuracies for our method are calculated using standard 0-fold cross validation method[3]. The arameters, the regularization arameter C and the RBF kernel arameter γ, are selected from the set {2 i i = 8,,8} by 0-fold cross validation on the tuning set comrising of random 0% of the training data. Once the arameters are selected, the tuning set was returned to the training set to learn the final classifier. In Algorithm, the regularization arameters are set u C = C 2 = C. In Algorithm 2, C = C n N instance n P bag + n N instance n P bag and C 2 = C, where n N instance is the number of instances in all negative bags, n P bag is the number of all ositive bags. Our algorithms are stoed if the n P bag + n N instance difference between the convex combination coefficients v is less than 0 ( 4) or if the iterations k > 80. We comare our results with MICA[0], SVM[0], mi-svm[8],mi-svm[8],r-svm [9] and EM-DD[5]. Our methods are denoted as SVM-CC since they are based on convex combination. The results of 0-fold cross validation accuracy are listed in Table. Correctness results for mi-svm, MI-SVM and EM-DD are taken from [8] and for R-SVM from [9]. We see from Table that our method has the best correctness on the Musk dataset. Although our method does not achieve the best accuracy results in the rest three datasets, they are comarable second bests in Table. Table : Results for four models Dataset MICA mi-svm MI-SVM R-SVM EM-DD SVM-CC Elehant 80.5% 82.2% 8.4% 84.3% 78.3% 8.5% Tiger 82.6% 78.4% 84% 83.7% 72.% 83% TST 94.5% 93.6% 93.9% 95.% 85.8% 95% Musk 84.4% 87.4% 77.9% 85.8% 84.8% 88.9%

7 Multi-instance Suort Vector Machine Based on Convex Combination Conclusion In this aer, we develo a novel aroach to multi-instance learning roblem based on convex combination for the instances in each ositive bag and establish two algorithms for linear multi-instance SVC and nonlinear multi-instance SVC resectively. The key oint is that the multi-instance learning is formulated, as an extension of the standard C-suort vector classification, as a nonlinear otimization roblem with continuous variables. This roblem, in rincile, is easy to be solved since it does not involved in integer variables like MI-SVM. It is interesting to solve this roblem by a more reasonable and efficient way. References [] T.G. Dietterich, R.H.Lathro, and T. Lozano-Pérez. Solving the multile-instance roblem with axis-arallel rectangles. Artificial Intelligence, 89( 2):3 7,997. [2] T. Brow, B. Settles and M. Craven. Classifying biomedical articles by making localized decisions. In Proc. of the Fourteenth Text Retrieval Conference (TREC),2005. [3] G. Carneiro, A. B. Chan, P. J. Moreno and N. Vas concelos. Suervised on attern analysis and machine intelligence, 29(3):394 40,2007. [4] D. R. Dooly, Q. Zhang, A. Goldman and R. A. Amar. Multile instance learning of realvalued data. Journal of Machine Learning Research, 3, , [5] Q. Zhang and S.Goldman. EM-DD: an imroved multile instance learning technique. In Neural Information Processing Systems,4,200. [6] J. Wang and J. D. Zucher. Solving the multile-instance roblem: a lazy learning aroach. In Proceedings of the 7th International conference on Machine Learning, San Francisco, CA, 9 25, [7] G. Ruffo. Learning single and multile instance decision tree for comuter security alications. PhD dissertation, Deartment of Comuter Science, University of Turin, Torino, Italy, Feb [8] S. Andrews, I. Tsochantaridis, and T Hofmann. Suort vector machines for multileinstance learning. Advances in Neural Information Processing Systems 5, MIT Press, , [9] O. Seref and O.E. Kundakcioglu. Multile instance classification with relaxed suort vector machines. Proceedings of the 3rd INFORMS Worksho on Data Mining and Health Informatics (DM-HI), [0] O. L. Mangasarian, and E. W. Wild. Multile instance classification via successive linear rogramg. Journal of Otimization Theory and Alication, 37(): , [] P. M. Murhy and D. W. Aha. UCI machine learning reository. mlearn/mlreository.html,992. [2] T. Joachims. Making large-scale suort vector machine learning ractical. Advances in Kernel Methods Suort Vector Learning, B. Schölkof,C.J.C. Burges, and A.J.Smola,eds,.69 84, Cambridge,Mass:MIT Press,999. [3] T.M. Mitchell. Machine learning. McGraw-Hill International, Singaore,997.