Performance Evaluation of Kernels in Multiclass Support Vector Machines

Transcription

1 Internatonal Journal of Soft Computng and Engneerng (IJSCE) ISSN: 3-307, Volume-, Issue-5, November 0 Performance Evaluaton of n Multclass Support Vector Machnes R. Sangeetha, B. Kalpana Abstract In recent ears, Kernel based learnng algorthm has been recevng ncreasng attenton n the research doman. Kernel based learnng algorthms are related nternall wth the kernel functons as a ke factor. Support Vector Machnes are ganng populart because of ther promsng performance n classfcaton and predcton. he success of SVM les n sutable kernel desgn and selecton of ts parameters. SVM s theoretcall well-defned and exhbts good generalzaton result for man real world problems. SVM s extended from bnar classfcaton to multclass classfcaton snce man real-lfe datasets nvolve multclass data. In ths paper, we propose an optmal kernel for one-versus-one (OAO) and one-versus-all (OAA) multclass support vector machnes. he performance of the OAO and OAA are evaluated usng the metrcs lke accurac, support vectors, support vector percentage, classfcaton error, and speed. he emprcal results demonstrate the ablt to use more generalzed kernel functons and t goes to prove that the polnomal kernel s performance s consstentl better than other kernels n SVM for these datasets. Index erms Support Vector Machne, Multclass Classfcaton, Kernel functon,,. I. INRODUCION Improvng effcac of classfers have been an extensve research area n machne learnng over the past two decades, whch led to state-of-the-art classfers lke support vector machnes,neural networks and man more. Support Vector Machne s a robust classfcaton tool, effectvel overcomes man tradtonal classfcaton problems lke local optmum and curse of dmensonalt. hree maor ssues of SVM are Kernel Mappng, Quadratc Optmzaton and Maxmum Margn Classfers. hs paper focuses n the frst ssue. Multclass SVM decomposes multclass labels nto several two class labels and t trans a svm classfer to solve the problems and then reconstruct the soluton of the multclass problem from outputs of the classfers [9], such as OAO-SVM and OAA-SVM. he paper s organzed as follows. Secton and 3 descrbe SVM and Multclass SVM. Secton 4 explans the kernels and ts parameters. Secton 5 elucdates the expermental results. Lastl, Secton 6 concludes wth future work. Sangeetha.R, Department of Computer Scence, Avnashlngam Insttute for Home Scence and Hgher Educaton for Women, Combatore, Inda. (e-mal:sangeethad@gmal.com). Dr.B.Kalpan, Department of Computer Scence, Avnashlngam Insttute for Home Scence and Hgher Educaton for Women, Combatore, Inda.(e-mal:kalpanabsekar@gmal.com) II. SUPPOR VECOR MACHINES [, 3] Support Vector Machne has been a new and mportant tool for classfcaton and regresson. In dealng wth large data classfcaton, tradtonal optmzaton algorthms such as Newton Method or Quas-Newton Method cannot work an more due to the memor problem. SVMs belong to a faml of generalzed lnear classfcaton. A specal propert of SVM [3-6] s t smultaneousl mnmzes the emprcal classfcaton error and maxmzes the geometrc margn. So SVM s called as Maxmum Margn Classfers. SVM maps nput vector to a hgher dmensonal space where a maxmal separatng hperplane s constructed. wo parallel hperplanes are constructed on each sde of the hperplane that separates the data. he separatng hperplane s a hperplane that maxmze the dstance between the two parallel hperplanes. An assumpton s made that the larger the margn or dstance between these parallel hperplanes then better the generalzaton error of the classfer. Consder the problem of separatng the set of tranng vectors belongng to bnar classes or dchotomzaton (x, ),,. l, x R n, {+, }, where the R n s the nput space, x s the feature vector and s the class label of x. he separatng hperplanes are lnear dscrmnatng functons as follows, f ( x) w x + b, () where w s a weght vector and b s called the bas value. One of the hperplanes that maxmzes the margn s named as the optmal separatng. he optmal separatng hperplane [4] can be found b solvng the followng optmzaton problem: subect to mn ω, b, ξ ω + C l ξ w, () ( ω x ) b ξ, ξ 0 (3) + or ts dual problem subect to mn α Q α e α, (4) α 0 α,,..., l, α 0 C, (5) 38

2 Performance Evaluaton of n Multclass Support Vector Machnes where e s the vector of all ones, C s the penalt of error whch s postve; Q s ξ x, x and s the relaxaton parameter. hus f we obtan α and b then we can classf the decson functon as follows l f ( x ) α x. x + b (6) Most optmzaton problems nvolve terms that are unknown and are usuall not drectl obtanable from the tranng data and the are not eas to guess, e.g., ξ n above equaton. hus, t becomes convenent to formulate an equvalent optmzaton problem that has the same soluton as the orgnal one, but does not nvolve an other nformaton than what s provded b the tranng samples. hs nvolves the use of Karush-Kuhn-ucker condtons. he former problem s then called the Prmal problem, and the latter s called as Dual. Bnar Classfcaton Dataset Selecton Data Preprocessng Classfcaton usng SVM Feasble Kernel Selecton Creaton of Hbrd Kernel and ts model Classfcaton Result Multclass Classfcaton Fg. Flow of Proposed work. III. MULICLASS SUPPOR VECOR MACHINES [5] Support Vector Machnes are based on varatonal-calculus whch constraned to have structural rsk mnmzaton (SRM) prncple and t uses convex optmzaton wth unque optmum soluton. In SVM, hperplanes are derved to separate the class labels n feature space. One of the hperplanes that maxmzes the margn s an optmal separatng hperplane. Bnar classfcaton s explcated n [, 3]. Fgure represents the flow of the proposed work. Multclass SVM can be solved b combnng the bnar classfcaton decson functons. Multclass SVM s of two tpes namel, decomposton and One versus All decomposton. he OAA decomposton [0] transforms the multclass problem nto a seres of c bnar subtasks that can be traned b the bnar SVM. Let the tranng set {( x, ),..., ( x, ) contan the XY modfed hdden states defned as for for he dscrmnant functons f l l}, (7) ( x) α K ( x) + b, Y, (8) s are traned b the bnar SVM solver from the set XY, Y he OAO decomposton [0] transforms the mult-class problem nto a seres of g c(c )/ bnar subtasks that can be traned b the bnar SVM. Let the tranng set {( x, ),..., ( x, ) contan the tranng XY l vectors x Є I {: V } and the modfed the hdden states defned as for, I (9) for, he tranng set XY,,,... g s constructed for all gc(c )/ combnatons of classes & \ { Y Y } he bnar SVM rules. q,,..., g are traned on the data. l XY IV. KERNELS IN MULICLASS SUPPOR VECOR MACHINES Kernel functons establsh the characterstcs of SVM model and level of non lneart. A necessar and suffcent condton for a smple nner product kernel to be vald s that t must satsf Mercer s theorem []. In general, kernels are of two tpes namel Local and Global kernels. Data that are close to each other n local kernels nfluence on the kernel ponts and data that are far awa from each other n global kernels nfluence on the kernel ponts. Commonl used kernels lke polnomal,, lnear are used n ths paper. Few other kernels are shown n able. In exstng statstcal learnng theor, when kernels are postve defnte, there s one approach to obtan the mappng from orgnal data set to feature space.e. the kernels are demanded to satsf Mercer s condton [6] and as a result the can be seen as dot product n some Hlbert space. Mercer s condtons serousl confne the wder applcaton of SVM. Almost all the current revew on kernel methods n machne learnng focuses on kernels whch are postve defnte. 39

3 A. heorem. (Mercer s) Suppose that K : χ X R s smmetrc and satsfes sup x, K(x, ) <,and defne f x) K ( x, ) f ( ) d K ( (0) x suppose that K : L ( χ ) L ( X ) s postve sem-defnte; thus, χ χ K ( x, ) f ( x ) f ( ) dxd 0 () for an, f L ( χ ). Let λ, ψ be the Egen functons and Egen vectors of K, wth K ( x, ) ψ ( ) d λ ψ ( x ) () χ hen. < λ. sup ( x ) < x ψ K ( x, ) λ ψ ( x ) ψ ( ) 3. where the convergence s unform n x,. Such a kernel defnes a Mercer Kernel accordng to Mercer theorem gven n [6]. hs gves the mappng n to feature space as x ),...) a φ ( x) ( λψ ( x), λ ψ ( (3) B. Reproducng Kernel Hlbert Spaces [6] Let us consder an nner product. A usual dot product: u, v as u, v v w v w u, v k( v, w) ψ( v) ψ( w. A kernel product: ) where ψ (u) ma have nfnte dmenson. However, an nner product.,. followng condtons must satsf the. Smmetr u, v v, u u, v χ α u + β v, w α u, w + β v, w. Blneart u, v, w χ, α, β R u, u 0, u χ 3. Postve defnteness u, u 0 u 0 Defnton A Hlbert Space s an nner product space that s complete and separable wth respect to the norm defned b the nner product. Defnton K (,.) s a reproducng kernel Hlbert spaces H f f H, f ( x ) k ( x,.), f (.).A Reproducng Kernel Hlbert Space (RKHS) s a Hlbert space H wth a reproducng Internatonal Journal of Soft Computng and Engneerng (IJSCE) ISSN: 3-307, Volume-, Issue-5, November 0 kernel whose span s dense n H. We could equvalentl defne an RKHS as a Hlbert space of functon wth all evaluaton functonals bounded and lnear. From the above defnton and theorem, kernel functon K must be contnuous, smmetrc, and have a postve defnte gram matrx. Such a K means that there exsts a mappng to a reproducng kernel Hlbert space such that the dot product there gves the same value as the functon K. If a kernel does not satsf Mercers condton, then the correspondng Quadratc Problem has no soluton. Hence, f an new kernel s proposed t should be checked wth mercer kernel. Laplacan Ratonal Quadratc Multquadratc Log Bessel Cauch Wavelet able. pes of K Functon K ( x, ) exp K( x, ) x σ x x + c k ( x, ) x + c K ( x, ) log x + ) σ J V + ( x ) ( x, ) n( v+ ) K ( x, ) K( x, ) N x + d d x x c c h h a m a able. Data Sets Used Datasets Sze Features Class Pentagon 99 5 Irs Wne V. RESULS AND DISCUSSIONS In ths secton, OAO and OAA SVM s kernel functons are evaluated usng the metrcs lke accurac, support vectors, support vector percentage, tranng error, classfcaton error and tme taken.for expermentaton, two benchmark datasets (Irs, Wne) are taken from the UCI machne learnng repostor and one snthetc dataset from [0].Bref sketch of the datasets s gven n table. In multclass SVM, the optmal regularzaton parameter C and the kernel parameters are estmated b repeatng classfcatons. d 40

4 Performance Evaluaton of n Multclass Support Vector Machnes Lnear kernel K ( x, x ) + x x s a smple kernel functon based on the penalt parameter C, snce parameter C controls the trade-off between frequenc of error c and complext of decson rule [7]. Also, t reduces the support vectors, tranng error and classfcaton error b ncrementng the parameter C.But t s not sutable for large datasets. Polnomal kernel p ( x x also known K ( x, x ) + ) as global kernel, s non-stochastc kernel estmate wth two parameters.e. C and polnomal degree p. Each data from the set x has an nfluence on the kernel pont of the test value x, rrespectve of ts the actual dstance from x [4], It gves good classfcaton accurac wth mnmum number of support vectors and low classfcaton error. Radal bass functon K( x, x ) exp( γ x x ) also known as local kernel, s equvalent to transformng the data nto an nfnte dmensonal Hlbert space.hus, t can easl solve the non-lnear classfcaton problem. It has an effect on the data ponts n the neghborhood of the test value [4]. gves smlar result as polnomal wth mnmum tranng error but for some cases the number of support vector and classfcaton error ncreases. Exponental radal bass functon x x K( x, x ) exp( ) gves pecewse lnear soluton. σ Gaussan radal bass functon x x K( x, x ) exp( ) σ deals wth data that has condtonal probablt dstrbuton approachng gaussan functon. kernels perform better than the lnear and polnomal kernel. However, t s dffcult to fnd an optmum parameters σ and equvalent C that gves better result for a gven problem. Sgmod kernel K ( x, x ) tanh( kx x δ ) s not effcent as other kernel functon, because t lacks the necessar condton of a vald kernel. s κ and δ must be chosen properl to obtan hgh classfcaton accurac. he performance metrcs of several kernels are compared to fnd an optmal and effcent kernel and t s carred out usng MALAB and C++. he tables 3., 3., 3.3 show tranng error, classfcaton error and tme taken for dfferent kernels n OAO and OAA SVM on three datasets. And, the are graphcall depcted n fgures, 3, 4 for OAO and fgures 5, 6, 7 for OAA usng kernel parameters n X axs and range of values n Y axs. Smlarl support vectors, support vector percentage, accurac are llustrated n tables 4., 4., 4.3. Also, the are vsuall portraed n fgures 8, 9, 0 for OAO and fgures,, 3 for OAA. In table 3. () Exponental kernel s tranng error, classfcaton error rate and tme are lesser than the other kernels for OAO SVM, () Polnomal and Exponental tranng tme, error rate and tme are lesser than the other kernels for OAA SVM. In table 3., Polnomal, and kernels tranng error, classfcaton error and tme are better compared to other kernels for OAO and OAA. In table 3.3,Polnomal, and kernels tranng error, classfcaton error and tme are better compared to other kernels for OAO and OAA.Smlarl, from tables [ ] Polnomal kernel and kernels gve better result. In kernel functon, number of support vector ncreases then the classfcaton accurac dmnshes. After analzng all the features of the kernel functon, approprate and optmal kernels for our datasets are polnomal kernel and kernels. he have mnmum number of support vectors, mnmum value as classfcaton error and good classfcaton accurac whch s shown n Fgure -3. VI. CONCLUSION Classfcaton tme and Computatonal complext for the multclass SVM classfer depend on the number of support vectors requred. In SVM classfcaton, the requred memor to store the support vectors s drectl proportonal to the number of support vectors. Hence, support vectors must be reduced to speed up the classfcaton and to mnmze the computatonal and hardware resources requred for classfcaton. Here, performance metrcs of dfferent kernels n multclass SVM on three datasets are compared. As a result, the effcent kernel for multclass SVM classfer s polnomal kernel for these datasets. Hbrd kernels can be created usng sstematc methodolog and optmzaton technque. herefore, the best method to combne the optmal feasble kernels would be our research work n future. 4

5 Internatonal Journal of Soft Computng and Engneerng (IJSCE) ISSN: 3-307, Volume-, Issue-5, November 0 APPENDIX able 3. ranng and est Error Rate for Irs Dataset Lnear Polnomal Sgmod E CE me(s) E CE me(s) C C C C C, p C0,p C,p C00,p C, γ C0, γ C, γ C0, γ C,σ C, σ C0, σ C00, σ C0, σ C0, σ C0, σ C00, σ C, k,δ C000, k, δ C000, k 5, δ C000,k, δ able 3. ranng and est Error Rate for Pentagon Dataset Lnear Polnomal Sgmod E CE me (s) E CE me (s) C C C C C000,p C00,p C000,p C000,p C0, γ C00, γ C00, γ C00, γ C00, σ C00, σ C000, σ Cnf, σ C00, σ C0, σ C000, σ Cnf, σ C0, k, δ C00,k,δ C00, k0.5, δ Cnf, k, δ Lnear Polnomal Sgmod able 3.3 ranng and est Error Rate for Wne Dataset E CE me(s) E CE me(s) C C C C C0,p C0,p C00,p C00,p C00, γ C00, γ C00, γ C000, γ C00, σ C00, σ C00, σ C00, σ C000, σ C000, σ C000, σ C000, σ C00, k, δ C00, σ,δ C00, k,δ C00 k,δ

6 Performance Evaluaton of n Multclass Support Vector Machnes able 4. Accurac, Support Vector and Support Vector % for Irs Dataset Lnear Polnomal Sgmod SV SV% Accurac% SV SV% Accurac % C C C C C, p C0,p C,p C00,p C, γ C0, γ C, γ C0, γ C,σ C, σ C0, σ C00, σ C0, σ C0, σ C0, σ C00, σ C, k, δ C000, k, δ C000, k 5, δ C000,k, δ able 4. Accurac, Support Vector and Support Vector % for Pentagon Dataset Lnear Polnomal Sgmod SV SV% Accurac% SV SV% Accurac% C C C C C000,p C00,p C000,p C000,p C0, γ C00, γ C00, γ C00, γ C00, σ C00, σ C000, σ Cnf, σ C00, σ C0, σ C000, σ Cnf, σ C0, k, δ C00,k,δ C00, k0.5, δ Cnf, k, δ able 4.3 Accurac, Support Vector and Support Vector % for Wne Dataset Lnear Polnomal Sgmod SV SV% Accurac% SV SV% Accurac% C C C C C0,p C0,p C00,p C00,p C00, γ C00, γ C00, γ C000, γ C00, σ C00, σ C00, σ C00, σ C000, σ C000, σ C000, σ C000, σ C00, k, δ C00, σ,δ C00, k,δ C00 k,δ

7 Internatonal Journal of Soft Computng and Engneerng (IJSCE) ISSN: 3-307, Volume-, Issue-5, November 0 Fg. OA O- Error Rate for Irs dataset. Fg.3 OAO- Error Rate for Pentagon dataset Fg.4 OAO- Error Rate for Wne dataset. Fg.5 OAA- Error Rate for Irs dataset. Fg.6 OAA- Error Rate for Pentagon dataset. Fg.7 OAA- Error Rate for Wne dataset Fgure (-4) represent OAO multclass SVM Error Rate for Irs,Pentagon and Wne.Fgure (5-7) represent OAA multclass SVM Error Rate for Irs,Pentagon and Wne. Fg.8 OAO- Accurac for Irs dataset. Fg.9 OAO- Accurac for Pentagon dataset 44

8 Performance Evaluaton of n Multclass Support Vector Machnes Fg.0 OAO- Accurac for Wne dataset. Fg. OAA- Accurac for Irs dataset. Fg. OAA- Accurac for Pentagon dataset. Fg.3 OAA- Accurac for Wne dataset Fgure (8-0) represent OAO multclass SVM Accurac for Irs,Pentagon and Wne.Fgure (-3) represent OAA multclass SVM Accurac for Irs,Pentagon and Wne. REFERENCES [] [] [3] [4] [5] [6] [7] [8] [9] [0] [] [] [3] J. Han and M. Kamber, Data Mnng Concepts and echnque, nd ed. San Mateo, CA: Morgan Kaufmann, 006. P.-N. an, M. Stenbach, and V. Kumar, Introducton to Data Mnng. Readng, MA: Addson-Wesle, 005. V. Vapnk, An overvew of statstcal learnng theor, IEEE rans. on Neural Networks, 999. N. Crstann and J. Shawe-alor, Introducton to Support Vector Machnes, Cambrdge Unverst Press, 000. B. Schölkopf and A. Smola, Leanng wth, MI Press, 00. C. J. C. Burges, A tutoral on support vector machnes for pattern recognton. Data Mnng and Knowledge Dscover, 998, pp Cornna Cortes and V. Vapnk, Support-Vector Networks, Machne Learnng, 995. J. Mankandan, B.Venkataraman,Stud and evaluaton of a mult-class SVM classfer usng dmnshng learnng technque, Neurocomputng, 00. Anna Wang, Wenng Yuan, Junfang Lu, Zhguo Yu, Hua L, A novel pattern recognton algorthm: Combnng AR network wth SVM to reconstruct a mult-class classfer, Computers and Mathematcs wth Applcatons, 009. Votech Franc, Václav Hlavá, Statstcal Pattern Recognton oolbox for Matlab, 009. Ralf Herbrch, Learnng kernel classfers: theor and algorthms, MI Press, Cambrdge, Mass, ISBN X, 00. Sangeetha, R., Kalpana, B, A comparatve stud and choce of an approprate kernel for support vector machnes, In: Das, V.V., Vakumar, R. (eds.) IC 00. CCIS, vol. 0,pp Sprnger, Hedelberg (00) Sangeetha, R., Kalpana, B, Optmzng the Kernel Selecton for Support Vector Machnes usng Performance Measures, In: ACWC 00, ISBN: ,00 [4] G.F. Smts, E.M Jordaan,Improved SVM Regresson usng Mxtures of, IJCNN 0. Proceedngs of the Internatonal Jont Conference on Neural Networks, 00. [5] J Weston, C Watkns, Mult class support vector machnes, echncal Report. [6] XIA Guo-en and SHAO Pe-. Factor Analss Algorthm wth Mercer Kernel, IEEE Second Internatonal Smposum on Intellgent Informaton echnolog and Securt Informatcs, Ms. R. Sangeetha completed her M.C.A from DJ Academ for Manageral Excellence, Combatore. Her area of nterest s Data Mnng. She s pursung her Ph.D Full me and workng as a Research Assstant n Avnashlngam Unverst, Combatore. Dr. B. Kalpana receved her Ph.D n Computer Scence from Avnashlngam Unverst, Combatore. She specalzes n Data mnng. She has around 0 ears of teachng experence at the post graduate and under graduate level. She has publshed and presented papers n several refereed nternatonal ournals and conferences. She s a member of the Internatonal Assocaton of Engneers and Computer Scentsts, Hongkong, Indan Assocaton for Research n Computng Scences (IARCS) and the Computer Socet of Inda.