Pairwise Multi-classification Support Vector Machines: Quadratic Programming (QP-P A MSVM) formulations

Transcription

1 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) Parwse Mult-classfcaton Support Vector Machnes: Quadratc Programmng (QP-P A MSVM) formulatons HEODORE B. RAFALIS and OLUAYO OLADUNNI School of Industral Engneerng Unversty of Oklahoma 0 West Boyd, Room 4 Norman, OK 7309 USA Abstract: - he bnary support vector machnes (SVMs) have been extensvely nvestgated. However ther extenson to a mult-classfcaton model s stll an on-gong research. In ths paper we present an extenson of the bnary support vector machnes (SVMs) for the k > class problems. he SVM model as orgnally proposed requres the constructon of several bnary SVM classfers to solve the mult-class problem. We propose a sngle quadratc optmzaton problem called a parwse mult-classfcaton support vector machnes (P A MSVMs) for constructng a parwse lnear and nonlnear classfcaton decson functons. A kernel approach s also dscussed for nonlnear classfcaton problems. Computatonal results are presented for two real data sets. Key Words: - parwse, SVM, MSVM, mult-class, kernel, classfcaton, quadratc programmng Introducton Support Vector Machnes (SVMs) developed by Vapnk [] are based on statstcal learnng theory and have been successfully appled to a wde range of problems. hey provde tools n modelng and smulaton of decson makng processes for classfcaton and regresson problems. SVMs for classfcaton problems perform a mappng of the nput varables nto a hgh dmensonal (possbly nfnte) feature space. Classfcaton s done n ths feature space by the use of a hyperplane. he resultng dscrmnant functon n the nput space s generally a nonlnear functon. In order to map the varables nto a hgher dmensonal feature space we use mplctly the concept of a kernel functon. Mult-classfcaton SVM s an extenson of support vector machnes (SVMs), nvolvng three or more classes. here s actve research n ths area, amng at the constructon of sngle optmzaton models for the reducton of the computatonal effort needed to solve the resultng large scale optmzaton problems and subproblems. Earler attempts nvolved solvng k SVM models, where k s the number of classes and k(k-)/ s the number of SVM classfers [, 3]. Other attempts nvolved the soluton of a sngle optmzaton problem usng all data at once [3, 4, 5]. he latter are arguably the most well constructed multclass formulatons most closely algned wth Vapnk s structural mnmzaton prncple []. Our man contrbuton wll be a multclassfcaton formulaton whch wll be expressed as a sngle optmzaton problem. We wll look at the development of a parwse mult-classfcaton support vector machne (QP-P A MSVM) expressed as a quadratc optmzaton problem. Smlar to the two-class problem we wll formulate the optmal parwse separator. hs paper s organzed as follows. In secton we revew the SVM for the bnary case. In secton 3 we present a quadratc programmng parwse mult-classfcaton support vector machne (QP-P A MSVM). In secton 4 we gve computatonal results for

2 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) two real data sets, and fnally, secton 5 concludes the paper. Support Vector Machnes In ths secton we consder the two-class classfcaton problem. he SVM avods overfttng by maxmzng the margn between two classes of tranng data,.e., maxmzng the dstance between the separatng hyperplane and the tranng data on ether sde of t. Fg : A Support Vector Machne classfcaton problem; the optmal hyperplane s orthogonal to the shortest lne connectng the two classes, and ntersects t halfway. he formulaton can be wrtten n ts prmal form [, 5, 6, 7, 8] as follows: l mn w + C,, ξ wbξ = () st.. y ( w x γ ) + ξ ξ 0 =, K, l d where x R are the nput tranng vectors, y { +, } are the correspondng labels, w = w w s the square of the -norm of the weght vector defnng the separatng hyperplane, and ξ s a non-negatve slack (penalty term) that measures the degree of volaton of the constrants. he parameter C s a constant, called the regularzaton parameter, whch controls the trade-off between mnmzng tranng errors and mnmzng the norm of the weght vector (generalzaton ablty). 3 Parwse Mult-classfcaton Support Vector Machnes In parwse classfcaton, we tran a classfer for each possble par of classes. For k classes, ths results to k(k-)/ SVM classfers. For the mult-classfcaton case we express all k classes as a sngle optmzaton problem that wll produce k(k- )/ SVM classfers. n Gven that the data sets n R are represented by a matrx m n A R, where =,.., k (k classes). Let A be an m nmatrx whose rows are ponts n the th class. Let A be a m nmatrx whose rows are n ponts n the th class. hen f x R can be classfed as follows: xw γ > 0, x A () x w γ < 0, x A, < In the separable case, the parwse lnear dscrmnant functon between two classes must satsfy the followng set of nequaltes: Fnd w R n and γ R, such that Aw > γ e, γ e> A w <, (3) where e s a vector of ones of approprate dmenson. If such a w and γ exst, we say that the sets are parwse lnearly separable. o classfy a new pont x, we employ the Max Wns strategy. hs s a votng approach [, 3]. For example, f the sgn of [ xw γ ] gves that x s n the th class, then the vote for the th class s ncreased by one. Otherwse, the th s ncreased by one. Hence we predct x as beng n the class wth the largest vote. In the case that those two classes have dentcal votes, we select the one wth the smallest ndex. 3. P A MSVM lnear separablty formulaton We propose the constructon of a parwse lnear and parwse nonlnear SVM usng a sngle quadratc program (QP). Lke n the dchotomous case we formulate the optmal

3 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) parwse lnear separator for the separable case. For the parwse separable case there n exsts a w R and γ R, such that Aw > γ e γ e> A w <, (4) Snce nfntely many w and γ exst, the optmal soluton would provde the largest margn of classfcaton. he margn of separaton between classes and s w. herefore, one would mnmze w for <. A Let A = and y = ± for classes and A respectvely. For the parwse lnearly separable problem we formulate the constraned optmzaton problem as below: k mn w w, γ (5) < s.. t y ( A w eγ ) e, < Here s a 3 classes problem (k = 3) rewrtten n matrx notaton Let I 0 0 C = 0 I 0, 0 0 I A 0 0 e 0 0 A 0 0 e A 0 0 e 0 A= E = A 0 0 e A 0 0 e A 0 e n n I R, m n m n A R, A R m <, e R where the dentty matrx and m e R < are the vectors of ones. So, for k > problem (5) can be expressed n the followng form: mn Cw w, γ s.. t Aw+ Eγ e 0, < (6) where w = w, w,.., w and γ = γ, γ,.., γ 3 ( k ) k 3 ( k ) k he constraned optmzaton s solved by ntroducng Lagrange multplers α 0 and a Lagrangan L( α, w, γ) Cw α ( Aw Eγ e) = + (7) Dfferentatng the Lagrangan wth respect to wandγ leads to dl = ( CCw ) Aα = 0 dw (8) dl E α 0 dγ (9) o elmnate varables w and γ from the Lagrangan, matrx ( CC ) needs to be an nvertble matrx. Snce ( CC ) s nvertble, we have ( CC) A = A (0) So from equatons (8) & (0) and (9) we can obtan the relatons w= ( C C) A α = A α () 0 = E α Usng the relatonshps n () we elmnate ( w, γ ) from the Lagrangan and we obtan the Wolfe dual quadratc optmzaton problem below: L( α, w, γ) = Cw α ( Aw+ Eγ e) = Cw, Cw α ( Aw + Eγ e) = wccw α ( Aw+ Eγ e) L( α) = α AA α α AA α 0+ α e L( α) = α e α AA α () In matrx notaton the Wolfe dual becomes max α e α AA α α st.. E α = 0 α 0 (3) ( ) ( ) where k k k k α = α, α,.., α, α, < Solvng for w n matrx notaton w = A α A α

4 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) Usng a summaton notaton the Wolfe dual of problem (5) s gven as k m k m max αc αcαdyyaa c d c d α < < c, d= m αc yc = αc 0, < st.. 0 (4) Where m s the number of tranng ponts for parwse comparson between classes and. Solvng for w : w m = αc yc Ac 3. P A MSVM nonlnear separablty formulaton For the parwse nonlnearly separablty, we employ the kernel trck to map the nput data nto a hgher dmenson feature space usng a kernel functon [9]. Replacng AA wth a kernel K( A, A ), the dual problem n (4) becomes k m k m max αc αcαdyyka c d ( c, Ad ) α < < c, d= m (5) st.. α y = 0 c c α 0, < c In matrx notaton the dual problem (4) becomes max α e α K( A, A ) α α st.. E α = 0 (6) α 0, < 3.3 P A MSVM nseparablty formulaton o construct a parwse nseparable classfer, we ntroduce a parameter λ. hs parameter s a constant called the regularzaton parameter, whch controls the trade-off between mnmzng tranng errors and mnmzng the norm of the weght vector (generalzaton ablty). Addng the error crteron ξ 0 and weghtng t wth the regularzaton parameter λ, then the prmal problem (6) becomes mn Cw + λe ξ w, γ st.. Aw+ Eγ e+ ξ 0 (7) ξ 0, < and the dual of problem (7) becomes max α e α AA α α st.. E α = 0 0 α λ, < (8) Replacng AA wth a kernel K( A, A ) and usng the summaton notaton the dual problem (8) can be expressed as k m k m αc αcαdyyka c d c Ad < < c, d= m αc yc = 0 αc λ, < max (, ) α st.. 0 (9) 4 Numercal estng In ths secton we present the computatonal results that utlze mult-class classfcaton formulaton of problem (8) for dscrmnatng between k classes. Experments were carred out on a vertcal nch two-phase flow dataset [0] and the admsson data for graduate school of busness []. Descrpton of datasets s as follows: Vertcal wo-phase Flow Dataset: he twophase flow dataset [0] uses a par of flow rates (superfcal gas and lqud velocty) to delneate the flow regme. here are 09 nstances (ponts) and attrbutes (features). he dstrbuton of nstances wth respect to ther class s as follows: 44 nstances n class (bubble flow), 0 nstances n class (ntermttent flow), and 63 nstances n class 3 (annular flow).

5 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) Admsson Data for Graduate School of Busness: he admsson data dataset [] uses the undergraduate grade pont average (GPA) and graduate management apttude test (GMA) scores to help determne whch applcants should be admtted to the school s graduate program. here are 85 nstances (ponts) and attrbutes (features). he dstrbuton of nstances wth respect to ther class s as follows: 8 nstances n class (not admtted), 6 nstances n class (borderlne), and 3 nstances n class 3 (admtted). he QP formulaton was mplemented usng the optmzaton and matrx decomposton routnes n the MALAB [] software. he two-phase flow dataset were scaled by takng the natural logarthm of each nstance, whle the admsson data were scaled usng the unt vector normalzaton for the QP formulaton. he methods were traned on 50% of the dataset, and tested on the whole dataset (50% tranng, 50% testng data), all randomly drawn from the dataset to obtan 3 tranng sample data. We report the results of the quadratc programmng (QP) formulaton of P A MSVM (λ = ). able : Performance of P A MSVM able contans the results for the QP- P A MSVM on the wo-phase flow and Admsson dataset. he errors rates are low enough to demonstrate the capablty of the model. he lnear kernel employed was adequate enough to gve a low msclassfcaton error, however further studes could nvolve the use of nonlnear kernels n problem (9). Note that the qualty of the soluton s dependent on the choce of λ. For ths problem λ = was suffcent enough to present good results. Further computatonal studes could be of nterest such as varyng choces of λ to determne the effect on the soluton of the QP MSVM model. 5 Concluson and Future Work In ths paper we presented an extenson of the bnary SVMs to the mult-class SVMs. We have used a quadratc programmng formulaton. he proposed method presents an accurate and good alternatve to exstng earler methods based on solvng k SVM models whch can become computatonally ntensve due to the number of SVM models one would have to solve n order to dscrmnate between k classes. Formulaton (8) was appled to the wo- Phase flow and Admsson dataset and the results are very encouragng consderng that the kernel used s a lnear kernel. he lnear kernel ndcates that the both datasets are lnearly separable but wth a tolerable msclassfcaton rate (see able ). Future work wll nvolve the nvestgaton of nonlnear kernels to solve nonlnear classfcaton problems. Mercer s condton s applcable, so kernel functons can be ncorporated nto the MSVM methods. Further computatonal studes would nclude the mplementaton of several preprocessng schemes to normalze the data and varyng the choce of meta-parameter λ to help determne the effect on the model performance. Acknowledgements: he present work has been partally supported by the NSF grant EIA References [] Vapnk, V. Statstcal Learnng heory. John Wley & Sons, Inc., 998. [] Santosa, B.; Conway,.; rafals,. B. Knowledge Based-Clusterng and Applcaton of Mult-Class SVM for Genes Expresson Analyss, Intellgent Engneerng Systems through Artfcal Neural Networks 00,,

6 Proceedngs of the 6th WSEAS Int. Conf. on NEURAL NEWORKS, Lsbon, Portugal, June 6-8, 005 (pp05-0) [3] Hsu, C-W.; Ln, C-J. A Comparson of Methods for Mult-class Support Vector Machnes, IEEE ransactons on Neural Networks 00 3, [4] Bredenstener E. J.; Bennet, K. P. Multcategory Classfcaton by Support Vector Machnes, Computatonal Optmzaton and Applcatons 999,, [5] Crstann, N.; Shawe-aylor, J. Support Vector Machnes and other kernel-based learnng methods. Cambrdge Unversty Press, Cambrdge, UK, 000. [6] Burges, C.J.C. A tutoral on support vector machnes for pattern classfcaton. Data Mnng and Knowledge Dscovery, (): 998, -67. [7] Chang, C-C.; Ln, C-J. LIBSVM: A Lbrary for Support Vector Machnes 00, [8] Hsu, C-W.; Chang, C-C.; Ln, C-J. A Practcal Gude to Support Vector Classfcaton, echncal Report Department of Computer Scence and Informaton Engneerng, Natonal awan Unversty, ape 06, awan, 003. [9] Scholokopf, B. Statstcal Learnng and Kernel Methods. echncal Report MSR- R-000-3, Mcrosoft Research Ltd., Mcrosoft Corporaton, 000 [0] rafals,. B.; Oladunn, O.; Papavasslou, D. V. wo-phase Flow Regme Identfcaton wth a Mult- Classfcaton SVM Model, School of Industral Engneerng, College of Engneerng, Unversty of Oklahoma. (accepted) to appear n the Industral & Engneerng Chemstry Research. [] Johnson, R. A.; Wchern, D. W. Appled Multvarate Statstcs Analyss; Prentce Hall: New Jersey, 00. [] MALAB User s Gude. he Math- Works, Inc., Natwck, MA 0760, [3] Bazaraa, M.S.; Sheral, H.D.; Shetty, C.M.; Nonlnear Programmng heory and Algorthms; John Wley & Sons, Inc., 993 [4] Reklats, G.V.; Ravndran, A.; Ragsdell, K.M. B.; Engneerng Optmzaton: Methods and Applcatons; John Wley & Sons, Inc., 983