Exact Simplification of Support Vector Solutions

Transcription

1 Jounal of Machne Leanng Reseach 2 (200) Submtted 3/0; Publshed 2/0 Exact Smplfcaton of Suppot Vecto Solutons Tom Downs TD@ITEE.UQ.EDU.AU School of Infomaton Technology and Electcal Engneeng Unvesty of Queensland, Bsbane, Q. 4072, Austala Kevn E Gates KEG@MATHS.UQ.EDU.AU School of Infomaton Technology and Electcal Engneeng and Depatment of Mathematcs Unvesty of Queensland, Bsbane, Q. 4072, Austala Annette Mastes AMASTERS@MATHS.UQ.EDU.AU Depatment of Mathematcs Unvesty of Queensland Bsbane, Q. 4072, Austala Edto: Nello Cstann, John Shaw-Taylo, and Robet C. Wllamson Abstact Ths pape demonstates that standad algothms fo tanng suppot vecto machnes geneally poduce solutons wth a geate numbe of suppot vectos than ae stctly necessay. An algothm s pesented that allows unnecessay suppot vectos to be ecognzed and elmnated whle leavng the soluton othewse unchanged. The algothm s appled to a vaety of benchmak data sets (fo both classfcaton and egesson) and n most cases the pocedue leads to a educton n the numbe of suppot vectos. In some cases the educton s substantal. Keywods: Suppot vecto machnes, kenel methods.. Intoducton The tme taken fo a suppot vecto classfe to compute the class of a new patten s popotonal to the numbe of suppot vectos, so f that numbe s lage, classfcaton speed s slow. Ths can be a poblem fo some applcatons and Buges (996) descbed a method of speedng up the classfcaton pocess by appoxmatng the soluton usng a smalle numbe of vectos. The educed set of vectos detemned by Buges method ae geneally not suppot vectos. When appled to the NIST data set usng second and thd degee polynomal kenels, speed mpovements of an ode of magntude wee acheved wth vey small effects on genealzaton pefomance. The method was somewhat efned n (Buges and Schoelkopf, 997) and agan appled to the NIST data set, ths tme usng ffth degee polynomal kenels, and 20-fold gans n speed wee acheved, agan wthout sgnfcant mpament to genealzaton pefomance. The educed set of vectos used n ths method s computed fom the ognal suppot vecto set n such a way that t povdes the best appoxmaton to the ognal decson suface. Unfotunately, detemnaton of ths educed set poved to be computatonally expensve and the appoach seems not to have been pusued futhe. 200 T Downs, K E Gates, A Mastes

2 DOWNS, GATES AND MASTERS Moe ecently t has been shown (Syed et al., 999) that the dscadng of even a small popoton of the suppot vectos can lead to a sevee educton n genealzaton pefomance. Syed et al. (999) stated that ths mples that the suppot vecto set chosen by the SVM s a mnmal set. But Buges (996) and Buges and Schoelkopf (997) had aleady ponted out that thee exst non-tval cases whee the educed set appoxmaton s exact, showng that the suppot vecto set delveed by the SVM s not always mnmal. Buges and Schoelkopf offeed no explanaton fo ths phenomenon. In ths pape we explan ccumstances n whch a educed set can be computed that avods any appoxmaton. The equed computaton s essentally tval and bascally nvolves dscadng unnecessay suppot vectos and modfyng the Lagange multples of the emanng suppot vectos so that the soluton emans unchanged. 2. Reducng the Numbe of Suppot Vectos 2. Patten Classfcaton Suppose we tan an SVM classfe wth patten vectos x, and that of these ae detemned to be suppot vectos. Denote them by x, =,2,...,. The decson suface fo patten classfcaton then takes the fom = α y K( = whee α s the Lagange multple assocated wth patten x and K(, ) s a kenel functon that mplctly maps the patten vectos nto a sutable featue space. Now suppose that suppot vecto x k s lnealy dependent on the othe suppot vectos n featue space,.e. K( x, x ) = c K( x ) (2) k = k whee the c ae scala constants. Then the decson suface () can be wtten () = α y K( x) + α k yk ck ( x) + b = = k k Now defne α k y k c = α y γ so that (3) can be wtten (3) = α ( + γ ) y K( x ) = k = α = k + b y K( x, x ) b (4) + whee α = α( + γ) (5) Compang (4) and () we see that the lnealy dependent suppot vecto s not equed n the epesentaton of the decson suface. Note, howeve, that the Lagange multples must be 294

3 EXACT SIMPLIFLIFICATION OF SUPPORT VECTOR SOLUTIONS modfed accodng to (5) n ode to obtan the smplfed epesentaton. But ths s a vey smple modfcaton that can be appled to any lnealy dependent suppot vecto that s dentfed. 2.2 Regesson Fo egesson poblems the suppot vecto soluton takes the fom whch can be wtten = (α = α* ) K( f ( x ) = β K( (6) = whee β = α o β = α *, dependng on whch constant s actve. Now suppose suppot vecto x k s lnealy dependent on the othe suppot vectos n featue space accodng to (2). We then fnd, as fo the classfcaton case, that x k can be elmnated fom (6) and ths gves whee β = β + β k c. 3. Results f ( x ) = β K( (7) = In ode to detemne the smplfcatons that ae possble usng ths method, we need to be able to dentfy suppot vectos that ae lnealy dependent n featue space. Ths can be done usng technques fom elementay lnea algeba and we employ the ow educed echelon fom (Noble and Danel, 988). The esults below ndcate the degee of smplfcaton acheved on solutons obtaned usng standad SVM pocedues on seveal benchmak data sets. We do not gve any ndcaton of genealzaton pefomance because the smplfcatons leave pefomance unchanged. 3. Classfcaton Results fo classfcaton wee obtaned usng fou data sets abtaly selected fom the UCI database. We employed the SMO algothm (Platt, 999) to geneate an ntal suppot vecto set and then appled the method descbed n Secton 2. to elmnate suppot vectos that ae lnealy dependent n featue space. We used lnea, polynomal and RBF kenels and the esults ae detaled n Tables 3. Lnea Kenel (x x j ) Contaceptve Method Choce Dabetes Habeman Wsconsn Beast Cance SMO # SVs Modfed set # SVs Reducton n SVs 75.0% 92.0% 8.25% 57.4% Table : Results usng the lnea kenel 295

4 DOWNS, GATES AND MASTERS Polynomal Kenel (x x j + ) d Contaceptve Method Choce Dabetes Habeman Wsconsn Beast Cance SMO # SVs d= Modfed set # SVs Reducton n SVs 22.22% 88.6% 90.45% 52.38% SMO # SVs d=2 Modfed set # SVs 42 0 Reducton n SVs 8.33% 2.5% 88.5% 8.33% SMO # SVs d=3 Modfed set # SVs Reducton n SVs 0.0% 7.54% 80.22% 0.0% SMO # SVs d=4 Modfed set # SVs Reducton n SVs 4.2% 0.0% 63.29% 3.45% SMO # SVs d=5 Modfed set # SVs Reducton n SVs 3.3%.62% 30.44% 5.39% SMO # SVs d=6 Modfed set # SVs Reducton n SVs 0.0% 38.66% 34.92% 7.4% SMO # SVs d=7 Modfed set # SVs Reducton n SVs 8.33% 50.2% 9.52% 0.0% Table 2 : Results usng the polynomal kenel RBF Kenel Contaceptve Method Choce Dabetes Habeman Wsconsn Beast Cance σ =.5 SMO # SVs Modfed set # SVs Reducton n SVs 0.0% 2.08% 0.0%.85% σ =2.0 SMO # SVs Modfed set # SVs Reducton n SVs 0.0% 2.0% 4.35% 5.36% σ =2.5 SMO # SVs Modfed set # SVs Reducton n SVs 3.83% 2.04% 33.3% 8.4% σ =3.0 SMO # SVs Modfed set # SVs Reducton n SVs 0.0% 6.94% 45.45% 4.95% σ =3.5 SMO # SVs Modfed set # SVs Reducton n SVs 29.59% 67.88% 3.43% 2.3% Table 3 : Results usng RBF kenels 296

5 EXACT SIMPLIFLIFICATION OF SUPPORT VECTOR SOLUTIONS 3.2 Regesson Fo the egesson case, the ntal suppot vectos wee geneated usng SVMToch (Collobet and Bengo, 200) on a data set of 4000 ponts also povded n Collobet and Bengo (200). Equaton (7) was then used to elmnate those suppot vectos that wee dentfed as lnealy dependent. The smplfcatons obtaned ae detaled n Table 4. RBF Kenel SVMToch - # SVs Modfed set - #SVs Reducton n SVs σ = % σ = % σ = % σ = % σ = % 4. Dscusson Table 4 : Results usng RBF kenels fo egesson Tables -4 demonstate that n most cases ou pocedue leads to a educton n the numbe of suppot vectos and n some cases a substantal educton. The amount of educton achevable s both kenel and poblem dependent and does not appea to be pedctable a po. Howeve, the examples ndcate that lage eductons tend to occu wth lowe degee polynomal kenels and wth RBF kenels havng lage σ values. Note that the solutons we obtan ae not geneally unque. Lnea dependence s a collectve popety of the suppot vectos and the choce of whch suppot vectos to elmnate s not a unque one. Ths ndcates that those suppot vectos that Vapnk tems essental (Vapnk, 998) ae the ones that ae lnealy ndependent befoe the mplementaton of ou pocedue. We ae cuently developng an SVM tanng algothm that wll geneate lnealy ndependent suppot vectos only. Acknowledgement Ths wok was suppoted by the Austalan Reseach Councl, Gant Numbe A Refeences C. J. C. Buges. Smplfed suppot vecto decson ules. Poceedngs 3 th Intenatonal Confeence on Machne Leanng, Ba, Italy, 996, pp C. J. C. Buges and B. Schoelkopf. Impovng speed and accuacy of suppot vecto leanng machnes. Advances n Neual Infomaton Pocessng Systems, 9, MIT Pess, 997, R. Collobet and S. Bengo. SVMToch: Suppot Vecto Machnes fo Lage-Scale Regesson Poblems, Jounal of Machne Leanng Reseach, :43-60, 200. B. Noble and J. W. Danel. Appled Lnea Algeba. 3 d Edton, Pentce-Hall, 988. J. Platt. Fast Tanng of suppot vecto machnes usng sequental mnmal optmzaton. In B. Schölkopf, C. J. C. Buges, and A. J. Smola, edtos, Advances n Kenel Methods Suppot Vecto Leanng, pages , 999, MIT Pess. N. A. Syed, H. Lu and K. K. Sung. Incemental Leanng wth Suppot Vecto Machnes. Poceedngs of Wokshop on Suppot Vecto Machnes at Intenatonal Jont Confeence on Atfcal Intellgence, Stockholm, 999. V. N. Vapnk. Statstcal Leanng Theoy. Wley, New Yok,