Robust Feature Induction for Support Vector Machines

Transcription

1 Robust Featue Inducton fo Suppot Vecto Machnes Rong Jn Depatment of Compute Scence and Engneeng, Mchgan State Unvesty, East Lansng, MI4884 Huan Lu Depatment of Compute Scence and Engneeng, Azona State Unvesty, Tempe, AZ Abstact The goal of featue nducton s to automatcally ceate nonlnea combnatons of exstng featues as addtonal nput featues to mpove classfcaton accuacy. Typcally, nonlnea featues ae ntoduced nto a suppot vecto machne (SVM) though a nonlnea kenel functon. One dsadvantage of such an appoach s that the featue space nduced by a kenel functon s usually of hgh dmenson and theefoe wll substantally ncease the chance of ove-fttng the tanng data. Anothe dsadvantage s that nonlnea featues ae nduced mplctly and theefoe ae dffcult fo people to undestand whch nduced featues ae ctcal to the classfcaton pefomance. In ths pape, we popose a boostng-style algothm that can explctly nduces mpotant nonlnea featues fo SVMs. We pesent empcal studes wth dscusson to show that ths appoach s effectve n mpovng classfcaton accuacy fo SVMs. The compason wth an SVM model usng nonlnea kenels also ndcates that ths appoach s effectve and obust, patculaly when the numbe of tanng data s small.. Intoducton Fo the last couple of yeas, suppot vecto machnes (SVMs) have been successfully appled to many applcatons, such as text classfcaton (Joachms, 00), mage classfcaton (Teytaud & Saut, 00), and face ecognton (Phllps, 998). Compaed to othe leanng machnes, SVMs have two advantages: ) contollng model complexty though a egulazaton tem, and ) nducng nonlnea featues Appeang n Poceedngs of the st Intenatonal Confeence on Machne Leanng, Banff, Canada, 004. Copyght 004 by the authos. though a nonlnea kenel functon. The fst advantage makes SVMs moe obust to noses and less lkely to ove-ft the tanng data. The second advantage nceases the expessve powe of SVMs and makes SVMs sutable fo complcated classfcaton tasks. Many empcal studes have shown that appopate choce of nonlnea kenels can sgnfcantly mpove the classfcaton accuacy (Teytaud & Saut, 00). Howeve, thee ae also some shotcomngs of usng kenel-nduced featues fo an SVM model: ) on-spase weghts. The featue space nduced by a nonlnea kenel functon s usually of lage dmensonalty and could be nfnte n some cases. When the data can be sepaated wth a few nonlnea featues, a kenel-nduced featue scheme may have to assgn non-zeo weghts to many tanng examples n ode to concentate on those mpotant nduced featues. Ths can esult n spaseness n the nduced featue space that can lead to a non-spase weght dstbuton of the tanng examples. ) Inappopate egulazaton. In pactce, people found that data nomalzaton (.e., subtacted by means and dvded by standad vaance) usually mpoves the pefomance of an SVM. Ths s because an SVM uses the l nom of vecto w (.e. w ) as ts penalty tem, n whch weghts of dffeent featues ae added togethe usng the same scale (.e., ). Thus, the penalty tem n an SVM eques the values of dffeent nput featues compaable. Howeve, t s athe dffcult to nomalze kenel-nduced nonlnea featues because they usually do not have explct expessons. As a esult, hgh ode nonlnea featues may suffe moe fom the penalty tem w snce the values ae athe small and need lage weghts to have notceable mpact on decson boundaes. 3) Implct featues. It s usually dffcult to deve explct expessons fo kenel-nduced featues. Ths makes t had to nfe ctcal nonlnea featues fom leaned SVM models usng nonlnea kenel functons. Ths nceases the dffculty fo uses to compehend the leaned esults.

2 In ths pape, we popose a featue nducton algothm that explctly ntoduces nonlnea featues nto an SVM to boost ts classfcaton pefomance. Patculaly, ths new algothm addesses the above thee shotcomngs wth the kenel-nduced featues. The explct ntoducton of nonlnea featues allows the poposed algothm to concentate on the subspace of mpotant featues. Data nomalzaton can be pefomed on the nduced featues n the same fashon as on the exstng featues. It s ease to ntepet an SVM model usng the explctly nduced featues than an SVM model wth the featues mplctly nduced by kenel functons. The poposed algothm woks as follows: fo each teaton, the algothm examnes the chaactestcs of the msclassfed tanng examples and geneates a new featue amed to coect mstakes. The algothm s smla to AdaBoost (Schape & Snge, 999) n that both boost the classfcaton pefomance by utlzng the msclassfed tanng examples. The most mpotant dstncton between them s that the poposed algothm ceates new featues whle the AdaBoost algothm geneates new nstances of weak classfes. If we vew each nstance of the weak classfe as a featue geneaton functon, the AdaBoost algothm may also be teated as a featue nducton method. Howeve, the poposed algothm dffes fom AdaBoost n the followng thee mpotant aspects: ) The poposed algothm only needs to dentfy the mpotant nonlnea featues whle the AdaBoost algothm has to ceate new weak classfes as well as compute the optmal weghts that combne multple weak classfes. ) The poposed algothm can take advantage of the egulazaton mechansm wthn an SVM model to avod the ove-fttng poblem. In contast, AdaBoost s a geedy algothm. Empcal studes have shown that the AdaBoost algothm tends to ove-ft tanng examples when the data s nosy (Optz & Maclne, 999; Ratsch et al., 000; Gove & Schuumans,998; Dettech, 000; Jn et al., 003). 3) Unlke AdaBoost whee the nduced weak classfes ae lnealy combned, the new featues nduced by the poposed algothm can be combned nonlnealy though kenel functons. The est of the pape s aanged as follows: Secton dscusses the elated wok; Secton 3 descbes the detals of the poposed featue nducton algothm fo SVMs; Secton 4 pesents an empcal study fo the poposed algothm; and Secton 5 concludes ths wok.. Related Wok We wll fst dscuss the pevous wok on featue nducton, next evew an SVM model and ts elated kenel method because ths wok tagets on the featue nducton poblem fo SVMs, and then descbe the coe dea of the AdaBoost algothm snce the poposed algothm utlzes the dea of boostng algothms. Pevous wok on featue nducton. Della Peta et al. (997) poposed an effcent algothm to seach fo featues that can effectvely ncease the lkelhood of tanng data (.e. log py ( x) ). Ths dea Tanng was appled to the whole sentence language model (Rosenfeld et al., 999), n whch a sentence s summazed nto a set of featues and descbed by a condtonal exponental model. It s late extended to the condtonal andom feld (MacCallum, 003) to allow fo moe tunng of the algothm n ode to futhe mpove the leanng effcency. Smlaly, the poposed featue nducton algothm s to seach fo featues that can effectvely decease the objectve functon n an SVM. The dffculty of nducng effectve nonlnea featues fo SVMs ases due to the fact that SVMs attempt to solve a constaned optmzaton poblem n whch featues only appea n the constants. As a esult, changes n featues cannot dectly nfluence the objectve functon of an SVM model. Ths s n contast to the featue nducton fo a condtonal exponental model n whch the objectve functon s expessed explctly n tems of nput featues. Renne & Jaakkola s wok (003) also nvolved featue nducton fo classfyng spam emals. They popose a compesson algothm fo text categozaton and ntoduce effectve featues to acheve the maxmum compesson ate. It s dffeent fom afoementoned woks on featue nducton n that t does not have an analytc objectve functon that s elated to featues. Theefoe, the seach of new featues s athe heustc and ad-hoc. Suppot Vecto Machnes. Hee we only evew the SVM model fo bnay classfcaton (Buges, 998). Let D denote the tanng data, D = { < x, } y > =. Each pa n the tanng data conssts of an nput vecto x and output label y (ethe + o -). An SVM model wth the lnea kenel s wtten as the followng optmzaton poblem: mn wb,, ξ,..., ξ w + c ξ = subject to [... ], y ( x w b) ξ, ξ 0 () whee c s a constant that contols the amount of admssble eos. The dual fom of the above optmzaton poblem s:

3 max α αα yy x x ( ),..., j j j α α = = j= subject to α 0, [... ], 0 α = y = c By eplacng the dot poduct x x j n the above equaton wth a kenel functon K( x, xj), we can mplctly ntoduce nonlnea featues nto the SVM model. As ndcated fom Equaton (), the nput featues do not appea n the objectve functon. Instead, they show up n the constants y( x w b) ξ and can only nfluence the objectve functon though the constants. Ths ndectness makes the nducton of effectve featues athe dffcult. To dectly elate nput featues to the objectve functon, we can ewte the optmzaton poblem n Equaton () n the followng fom: mn w + c max(0, y ( x w b) + ) wb, = (3) We can obseve n the above Equaton that constants ae emoved by eplacng each slack vaable ξ wth a tem max(0, y( x w b) + ) n the objectve functon. Optmzng Equaton (3) s stll dffcult because t nvolves dscontnuous functon `max. To solve ths poblem, we follow the dea n (Zhang et al., 003) to appoxmate the functon max(0, y( x w b) + ) wth a log-lnea functon log { + exp( [ y ( ) ] ) x w b + }. It s poved by Zhang et al. (003) that ths log-lnea functon foms an uppe bound fo functon max(0, y( x w b) + ) and wll unfomly convege to the max functon when +. The poposed featue nducton algothm s based on ths patcula appoxmaton fom of an SVM model. AdaBoost. As we mentoned eale, the poposed featue nducton algothm s smla to AdaBoost n that both ae teatve algothms and ceate a new entty at each teaton to coect the mstakes made by the cuent model. The man dea of AdaBoost s to ntoduce a new nstance of a weak classfe at each step to geedly educe the classfcaton eo. Specfcally, let H 0 ( x ) be the classfcaton functon of cuent teaton and ou goal s to geneate a new weak classfe hx ( ) such that the combnaton of H 0 ( x ) wth hx ( ),.e., H ( x) = H0 ( x) + αh( x), wll have a smalle classfcaton eo. In AdaBoost, a weak classfe s taned ove weghted examples () wth a weght exp( H0 ( x) y) fo each example < x, y >. The fnal classfe H f ( x ) s the lnea combnaton of all weak classfes geneated fom evey teaton,.e., H ( x ) = α h ( x ). f t= t t If each weak classfe ht ( x ) s vewed as a featue functon, the fnal classfe H f ( x ) s smply a lnea model fo the nduced featues { ht ( x )}. In ths sense, we can teat AdaBoost as a specal featue nducton algothm. Howeve, unlke othe featue nducton algothms that ae only esponsble fo geneatng new featues, AdaBoost needs to not only geneate new featues (.e., weak classfes) but also lean the lnea model (.e., combnaton weghts α t ). Thus, the featue nducton algothms have advantages ove AdaBoost n that they can ely on the classfcaton model to appopately combne the nduced featues wth the exstng featues. Patculaly, a featue nducton algothm fo SVMs can beneft geatly fom the nce popetes of SVMs: ) Unlke AdaBoost that may ove-ft nosy data, the featue nducton algothm fo SVM can avod the ove-fttng poblem because of the egulazaton mechansm nsde the SVM model; ) Unlke AdaBoost whee the nduced weak classfes ae lnealy combned, the featues nduced fo the SVM model ae able to nteact wth each othe nonlnealy though a kenel functon. 3. Featue Inducton Algothm fo SVMs As dscussed n Secton, n ode to fnd effectve featues fo an SVM, we use the specal fom of SVMs n Equaton (3) and appoxmate functon max(0, y( x w b) + ) wth a log-lnea functon log { + exp( [ y ( ) ] ) x w b + }. Puttng them togethe, the objectve functon n the ognal SVM model s appoxmated as follows: lsvm = c w + max(0, ( ) ) y x w b + = lapp = log + exp( y ( x w b) + ) = { } ote that the egulazaton tem cw s emoved fom the appoxmated objectve functon lapp. Ths s because log { + exp( [ y ( ) ] ) } max(0, ( ) ) x w b + y x w b + whee the equalty s taken only when +. Theefoe, we can always choose constant > 0 to (5)

4 make the appoxmated objectve functon lapp equal to the tue one lsvm. In effect, constant plays the smla ole as the egulazaton tem. An appopate choce of wll lead to less genealzaton eo. Ths pont wll be futhe dscussed late. Let x be the cuent featue vecto and g( x ) be the new featue that s ntoduced to the cuent SVM model. Let lapp be the appoxmated objectve functon that only uses the old featues x, and ' lapp ({ w, α}, b) be the appoxmated objectve functon that uses both old featues x and the new featue g( x ). Theefoe ' lapp ({ w, α}, b) can be expessed as follows: ' lapp ({ w, α}, b) = (6) log { + exp( y ( x w+ αg ( x ) b )) } whee α stands fo the weght of the new featue and b stands fo the new theshold value. In the above expesson fo the new objectve functon, we assume both the weghts fo the old featues,.e., w, and the theshold value b ae almost unchanged wth the ncluson of the new featue g( x ). Ths s a easonable assumpton when the classfcaton eo of an SVM usng the old featues x s low and the nduced featue wll not esult n a lage change n the SVM model. To fnd an effectve new featue g( x ), we consde the dffeence between the two objectve functons: ' lapp ({ w, α}, b) lapp + exp( y ( ( ) )) log x w+ αg x b = + exp( y( x w b)) (7) exp( y ( ))) log αg x = + + exp( H( x )) whee H ( x ) s defned as H( x) = y( x w b). Usng the nequalty log( + x) x, we have an uppe bound fo the dffeence of two objectve functons as: ' lapp ({ w, α}, b') lapp exp( α yg ( x))) (8) exp( H( x )) + + exp( H( x )) [ ] To emove the nteacton between the nduced featue g( x ) and the weght α, we futhe assume the outputs of g( x ) fall nto the nteval [-,]. Based on the λx + x λ x λ nequalty e e + e fo x [,], the uppe bound n Equaton (8) can be futhe smplfed as: l ({ w, α}, b') l ' app app α α e + e + exp( H( x )) [ + exp( H( x ))] α α e e yg ( x) + exp( H( x )) (8 ) The fst two tems n the above equaton ae not elated to the nduce featue and theefoe can be gnoed. The thd tem compses of a summaton ove all the tanng examples wth each example contbutng yg ( x). Theefoe, a good featue functon + exp( H ( x )) g( x ) that mnmzes the quantty n Equaton (8) can be obtaned by leanng a classfcaton model ove the tanng examples that ae weghted by the facto. Ths facto gves hgh weghts to + exp( H ( x )) msclassfed examples n whch H ( x ) ae negatve, and low weghts to coectly classfed examples whee H ( x ) ae postve. Ths s smla to the weght H ( x) dstbuton e used n AdaBoost but wth two mpotant dstnctons: ) The weghts used by the poposed featue nducton algothm s bounded,.e., <, whle + exp( H( x )) the weghts used by AdaBoost s unbounded. Ths bounded natue allows the poposed algothm to avod ove-emphass on the msclassfed examples. ) In the poposed weghts fo the tanng examples, constant contols the tadeoff between the tanng eo and the model complexty. Wth a lage value of, moe weghts ae assgned to the msclassfed examples and theefoe tanng eos can be educed moe effcently. On the othe hand, a small wll esult n less weght assgned to the msclassfed examples and less chance to ove-ft tanng data. Theefoe, constant has the smla mpact on the genealzaton eo as the egulazaton tem n an SVM model. The emanng queston s how to detemne the appopate value fo constant, whch has been justfed as an mpotant facto n the above dscusson. Snce the ntoducton of s to appoxmate the ognal objectve functon of an SVM model n Equaton (3), the optmal can be found by mnmzng

5 the dffeence between the two objectve functons app ( l w, b ) and l SVM,.e., * R ( l ) app w b lsvm = ag mn (, ) It can be seen that that functon ( l l ) app SVM s a convex functon wth egad to, whch can be easly justfed by ts second ode devatve. Theefoe, thee s a global mnmum soluton fo. Standad optmzaton algothms such as conjugate gadent and the ewton method can be appled to solve the optmzaton poblem n Equaton (9). Fgue summazes the poposed featue nducton algothm fo SVMs, whch altenatvely apples the pocedue of computng weghts and the pocedue of tanng a new classfcaton model. 4. Expements Featue Inducton fo SVM F: a set of featues F = { f, f,..., f } W: weghts fo tanng examples W = { W, W,..., W } D: tanng examples D = { < x, } y > = Intalzaton: W ={,,,} F = {all the ognal featues} Repeat untl the desed pefomance Tan a SVM model usng featue set F Compute W usng W = / [ + exp( H( x ))] Sample examples accodng the dstbuton j W / W j Tan a classfcaton model gx ( ) : X [...] usng the sampled examples F { F, g( x)} Fgue : Descpton of the featue nducton algothm fo SVMs In ths expement, we wll examne the effectveness of the poposed featue nducton algothm fo SVM. Patculaly, we wll addess the followng thee questons: ) How effectve and obust s the poposed featue nducton algothm fo mpovng classfcaton D (9) accuacy of SVMs? 50 dffeent nonlnea featues ae geneated fo each expement and the change of testng eos usng lnea SVM model wth espect to the numbe of nduced featues ae examned. ) How effectve s the poposed featue nducton algothm compaed to SVMs usng nonlnea kenels? SVMs wth polynomal kenels and a RBF kenel ae expemented and the esults ae compaed wth those usng the featue nducton algothm. 3) How effectve s the poposed featue nducton algothm compaed to AdaBoost? The AdaBoost algothm usng a lnea SVM as ts bass classfe s un ove the same set of datasets and ts esults ae compaed to the featue nducton algothm. Data Set # Examples # Featues onosphee beast_cance 68 9 spam cmc people outdoo Table : Statstcs of Expemental Datasets Sx dffeent datasets ae used n expements: fou of them ae fom the UCI machne leanng epostoy and the othe two ae fom eal wold applcatons. The statstcs of all eghts datasets ae lsted n Table. People and Outdoos ae the two datasets fom mage classfcaton tasks: dataset People s used fo the task of dentfyng scenes that have moe than two people, and dataset Outdoo s used fo the task of dentfyng outdoo scenes. Both ae data samples used fo TREC vdeo eteval evaluaton (TRECVID, 003). In the expements, we choose the decson tee algothm (C4.5, Qunlan, 993) to be the classfcaton model fo the poposed featue nducton algothm. Ths s because the decson tee algothm can ntoduce nonlnea nteactons between multple featues though conjunctons of Boolean lteals. A ewton method s used to fnd that optmzes Equaton (9). To futhe pevent ove-fttng the tanng data, we set the uppe bound fo to be 50 n the expements. The WEKA SVM mplementaton (Wtten & Fank, 999) s used fo expements. The kenel functons used n compason ae a RBF kenel and polynomal kenels wth degees angng fom to 5. The maxmum numbe of featues nduced by the poposed algothm s 50. Unless specfed, all expements ae conducted usng the 0-fold coss valdaton, wth 90% of the data used fo tanng and the emanng 0% fo testng.

6 0.035 beast_cance onosphee E E E E cmc people #Induced Featues E E spam outdoo #Induced Featues Fgue : Tanng and testng eos of a lnea SVM model usng the poposed featue nducton algothm fo sx dffeent datasets. Each sold lne epesents the change of testng eos wth espect to the numbe of nduced featues and each dash-dot lne epesents the coespondng change n tanng eos. 4. Expement I: Effectveness and Robustness of The Featue Inducton Algothm Fgue dsplays both the tanng eo and the testng eos of a lnea SVM model that uses the poposed featue nducton algothm fo sx dffeent datasets. Each sold lne epesents the change of tanng eos wth espect to the numbe of nduced featues. Each dash-dot lne epesents the change of the coespondng testng eos. Fst, fo all datasets, the nduced featues ae able to sgnfcantly educe the tanng eo of a lnea SVM model. The most notceable case s dataset onosphee, n whch the tanng eo has been educed fom 5.6% to zeo. Ths fact ndcates that the appoxmaton used n the pevous analyss s vald and easonably accuate. Second, despte of the mno fluctuaton n testng eos, fo all datasets, the oveall tend s that the moe the nduced featues ae, the smalle the classfcaton eo s. otce that although datasets such as people, cmc, and beast_cance have no moe than 0 featues, by ntoducng 50 new nonlnea featues, we ae stll able to educe the testng eo of a lnea SVM model wthout sgnfcantly ovefttng tanng data. The most notceable case s dataset beast_cance, n whch a sgnfcant dop n the testng eo s obseved when the numbe of nduced featues s between 7 and 9. Howeve, wth moe nduced featues, we see that the testng eo fo dataset beast_cance keeps deceasng. Ths tend s consstent wth the analyss pesented n the ntoducton secton. Explctly nduced new nonlnea featues can be nomalzed to the same scale as the ognal featues. As a esult, the Poly. Degee beast_cance onosphee cmc spam people outdoo 3.97% 9.7% 33.40% 6.67% 4.98% 7.3%.06% 0.9% 30.48% 8.07% 45.35%.% 3.79% 6.57% 30.75% 8.6% 45.54%.% 4.65% 8.57% 30.54% 36.43% 45.5%.5% 5 3.8% 8.9% 33.3% 37.7% 45.5%.7% Table : Testng eos fo SVM model usng polynomal kenels wth degee angng fom to 5.

7 egulazaton mechansm n the SVM model s effectve n peventng the ovefttng of tanng data that could potentally be caused by the lage numbe of nduced featues. In contast, applyng nonlnea kenels to nduce nonlnea featues can actually lead to a sgnfcant ovefttng poblem. Table pesents the testng eos of the SVM model usng polynomal kenels of dffeent degees. otce that, fo many datasets, a hgh degee polynomal kenel can sgnfcantly degade the SVM pefomance. Fo example, compaed to a lnea SVM model (poly. degee equals ), usng a polynomal kenel of degee 5 esults n a substantally wose classfcaton eo fo almost all datasets except outdoo as shown n Table. The most notceable case s dataset spam, whose testng eo s only 6.67% fo a lnea SVM model. Howeve, the testng eo shoots up to 37.7% when a polynomal kenel of degee 5 s used. As dscussed n the ntoducton secton, ths phenomenon can be attbuted to the fact that nonlnea featues ae ntoduced mplctly though kenel functons and theefoe ae dffcult to be nomalzed to the same scale as the ognal featues. As a esult, the egulazaton mechansm n SVM may not be able to wok appopately to pevent ovefttng poblems. The tend seen n Fgue shows that the nceased numbe of nduced featues can often help mpove testng accuacy. Howeve, the nceased numbe of nduced featues s assocated wth nceased computng costs. Its dmnshng gan n accuacy as the numbe of nduced featues nceases suggests a heustc stoppng cteon: stop when the gan between the two subsequent uns s nsgnfcant. 4. Expement II: Compason wth onlnea Kenels We apply both polynomal kenels and the RBF kenel to ntoduce nonlnea featues fo SVMs. Both the degee of polynomal kenels and vaance fo RBF kenel ae detemned though an ntenal coss valdaton pocedue usng 0% of tanng data that ae andomly selected fom the ognal tanng pool. The same coss valdaton pocedue s appled to detemne the vaance fo the RBF kenel. The esults of testng eos fo polynomal kenels and RBF kenels ae 50.00% 40.00% 30.00% 0.00% 0.00% 0.00% beast_cance Baselne AdaBoost onosphee cmc spam people outdoo Fgue 3: Tanng eos of AbaBoost and a lnea SVM model (efeed as baselne). pesented n Table 3, togethe wth the esults fo the poposed featue nducton algothm. Fo the pupose of compason, we also nclude the testng eos fo an SVM model usng the lnea kenel, whch s efeed as baselne model n Table 3. A boldfaced numbe ndcates that ts testng eo s substantally bette than the baselne model, and an talc font s appled when t s substantally wose than the baselne model. Fst, between the two kenel functons, the RBF kenel appeas to pefom sgnfcantly bette than the polynomal kenels fo almost all datasets. It manages to mpove the pefomance substantally fo thee out of sx datasets and acheves the same pefomance as the baselne model fo the est datasets. In contast, the polynomal kenels mpove classfcaton accuacy fo only two out of sx datasets whle degadng the pefomance fo the othe thee datasets. Second, the poposed featue nducton algothm consstently pefoms bette than the baselne model fo all datasets. Patculaly, t acheves substantal educton n testng eos fo fve out of sx datasets. Ths s notceably bette than both the RBF and polynomal kenels. Based on the above obsevaton, we conclude that the poposed algothm s moe obust and effectve than the nonlnea kenels n tems of nducng nonlnea featues fo the SVM model. 4.3 Expement III: Compason wth AdaBoost Methods beast_cance onosphee cmc spam people outdoo baselne 3.97% 9.7% 33.40% 6.67% 4.98% 7.3% polynomal.69% 3.43% 34.90% 9.50% 45.40%.03% RBF.79% 8.97% 30.30% 6.8% 4.4% 0.50% Featue Inducton.75% 5.4% 9.80% 6.35% 39.3% 4.33% AdaBoost 3.0%.8% 33.00% 9.37% 4.40% 6.5% Table 3: Testng eos fo dffeent methods. The baselne model efes to a lnea SVM model. A bold font s appled when a testng eo s substantally bette than the baselne model, and an talc font s appled when t s substantally wose than the baselne model

8 We compae the poposed featue nducton algothm fo SVMs to the AdaBoost algothm that uses SVM model as ts base classfe. Fo both algothms, a lnea kenel s used fo the SVM model. The maxmum numbe of teatons fo AdaBoost s set to be 0. The testng eos fo AdaBoost ae pesented n Table 3. Compaed to the poposed featue nducton algothm, AdaBoost appeas to be less effectve n mpovng the pefomance of an SVM model. Fo most datasets, t only acheves the same pefomance as the ognal SVM model. A futhe examnaton of tanng eos fo AdaBoost n Fgue 3 ndcates that the AdaBoost algothm cannot even educe the tanng eos sgnfcantly fo almost all datasets except people. Ths can be explaned by the fact that both a lnea SVM model and the AdaBoost algothm usng a lnea combnaton model. As a esult, the ntoducton of AdaBoost does not change the lnea natue of a lnea SVM model and theefoe has lttle mpact on the classfcaton accuacy. 5. Conclusons Induced featues can educe classfcaton eos. In ths pape, we popose a new featue nducton algothm fo SVMs. Unlke the kenel method whee the numbe of nduced featues s usually lage (even nfnte), the poposed algothm only ceates nonlnea featues that can effectvely educe tanng eos. Specfcally, new featues ae nduced teatvely. At each step, t weghts tanng examples dffeently accodng to the outputs of the cuent SVM model. The weghted tanng examples ae then used to tan a classfcaton model that becomes a new featue fo the SVM model. Empcal studes ove both UCI datasets and datasets of eal applcatons ndcate that the poposed featue nducton algothm s not only effectve n mpovng the pefomance of a lnea SVM model but also obust even when the numbe of nduced featues s substantally lage than the numbe of ognal featues. Refeences Joachms, T. (00). A Statstcal Leanng Model of Text Classfcaton fo SVMs. In Poceedng of 4 th ACM Intenatonal Confeence on Reseach and Development n Infomaton Reteval. Teytaud, O. and Saut, O. (00). Kenel-based Image Classfcaton, Lectue otes n Compute Scence, Phllps, P. J. (998). Suppot Vecto Machnes Appled to Face Recognton. In Advances n eual Infomaton Pocessng Systems, page 803. Optz, D., & Maclne, R. (999). Popula Ensemble methods: An Empcal Study. Jounal of AI Reseach pp Jn, R., Lu, Y., S, L., James, C., & A.G. Hauptmann (003). A ew Boostng Algothm Usng Input- Dependent Regulaze. In Poceedngs of 0 th Intenatonal Confeence on Machne Leanng. Dettech, T. G. (000). An Expemental Compason of Thee Methods fo constuctng Ensembles of Decson Tees: Baggng, Boostng, and Randomzaton. Machne Leanng, 40, Gove, A. J., & Schuumans, D. (998). Boostng n the Lmt: Maxmzng the Magn of Leaned Ensembles. In Poceedngs of the Ffteenth atonal Confeence on Atfcal Intellgence pp Ratsch, G., Onoda, T., & Mulle, K. (000). Soft Magns fo Adaboost. Machne Leanng, 4, Renne, J D. M. & Jaakkola, T. (00). Automatc Featue Inducton fo Text Classfcaton, MIT Atfcal Intellgence Laboatoy Abstact Book. Della Peta, S., Della Peta, V.J., & Laffety, J.D. (997). Inducng Featues of Random Felds. IEEE Tansactons on Patten Analyss and Machne Intellgence, 9, Rosenfeld, R., Wasseman, L., Ca, C., & Zhu, X.J. (999). Inteactve Featue Inducton and Logstc Regesson fo Whole Sentence Exponental Language Models. In Poc. IEEE wokshop on Automatc Speech Recognton and Undestandng, Keystone, Coloado. Buges, C.J.C. (998). A Tutoal on Suppot Vecto Machne fo Patten Recognton, Knowledge Dscovey and Data Mnng, (). Wtten, I.H. & Fank, E. (999). Data Mnng: Pactcal Machne Leanng Tools and Technques wth Java Implementatons. Mogan Kaufmann. Schape, R.E. & Snge, Y. (999). Impoved Boostng Algothms usng Confdence-ated Pedctons, Machne Leanng 37 (3): Qunlan, R. (993). C4.5: Pogams fo Machne Leanng, Mogan Kaufmann Publshes, San Mateo, CA. TRECVID (003). Zhang, J., Jn, R., Yang, Y. & Hauptmann, A.G. (003). Modfed Logstc Regesson: An Appoxmaton to SVM and ts Applcatons n Lage-Scale Text Categozaton, In Poc. of Intenatonal Confeence on Machne Leanng (ICML 003). McCallum, A. (003). Effcently Inducng Featues of Condtonal Random Felds. In Poc. Of 9 th Uncetanty n Atcfcal Intellgence (UAI 003).