Convergence of Large Margin Separable Linear Classification

Transcription

1 Covergece of Large Marg Separable Lear Classfcato Tog Zhag Mathematcal Sceces Departmet IBM T.J. Watso Research Ceter Yorktow Heghts, NY 0598 Abstract Large marg lear classfcato methods have bee successfully appled to may applcatos. For a learly separable problem, t s kow that uder approprate assumptos, the expected msclassfcato error of the computed optmal hyperplae approaches zero at a rate proportoal to the verse trag sample sze. Ths rate s usually characterzed by the marg ad the maxmum orm of the put data. I ths paper, we argue that aother quatty, amely the robustess of the put data dstrbuto, also plays a mportat role characterzg the covergece behavor of expected msclassfcato error. Based o ths cocept of robustess, we show that for a large marg separable lear classfcato problem, the expected msclassfcato error may coverge expoetally the umber of trag sample sze. Itroducto We cosder the bary classfcato problem: to determe a label y 2 f?; g assocated wth a put vector x. A useful method for solvg ths problem s by usg lear dscrmat fuctos. Specfcally, we seek a weght vector w ad a threshold such that w T x < f ts label y =? ad w T x f ts label y =. I ths paper, we are maly terested problems that are learly separable by a postve marg (although, as we shall see later, our aalyss s sutable for o-separable problems). That s, there exsts a hyperplae that perfectly separates the -class data from the out-ofclass data. We shall also assume = 0 throughout the rest of the paper for smplcty. Ths restrcto usually does ot cause problems practce sce oe ca always apped a costat feature to the put data x, whch offset the effect of. For learly separable problems, gve a trag set of labeled data (x ; y ); : : : ; (x ; y ), Vapk recetly proposed a method that optmzes a hard marg boud whch he calls the optmal hyperplae method (see []). The optmal hyperplae w s the soluto to the followg quadratc programmg problem: m w 2 w2 s.t. w T x y for = ; : : : ; : ()

2 For learly o-separable problems, a geeralzato of the optmal hyperplae method has appeared [2], where a slack varable s troduced for each data pot (x ; y ) for = ; : : : ;. We compute a hyperplae w that solves m w; 2 wt w + C s.t. w T x y? ; 0 for = ; : : : ; : (2) Where C > 0 s a gve parameter (also see []). I ths paper, we are terested the qualty of the computed weght w for the purpose of predctg the label y of a usee data pot x. We study ths predctve power of w the stadard batch learg framework. That s, we assume that the trag data (x ; y ) for = ; : : : are depedetly draw from the same uderlyg data dstrbuto D whch s ukow. The predctve power of the computed parameter w the correspods to the classfcato performace of w wth respect to the true dstrbuto D. We orgaze the paper as follows. I Secto 2, we brefly revew a umber of exstg techques for aalyzg separable lear classfcato problems. We the derve a expoetal covergece rate of msclassfcato error Secto 3 for certa large marg lear classfcato. Secto 4 compares the ewly derved boud wth kow results from the tradtoal marg aalyss. We expla that the expoetal boud reles o a ew quatty (the robustess of the dstrbuto) whch s ot explored a tradtoal marg boud. Note that for certa batch learg problems, expoetal learg curves have already bee observed [0]. It s thus ot surprsg that a expoetal rate of covergece ca be acheved by large marg lear classfcato. 2 Some kow results o geeralzato aalyss There are a umber of ways to obta bouds o the geeralzato error of a lear classfer. A geeral framework s to use techques from emprcal processes (aka VC aalyss). May such results that are related to large marg classfcato have bee descrbed chapter 4 of [3]. The ma advatage of ths framework s ts geeralty. The aalyss does ot requre the estmated parameter to coverge to the true parameter, whch s deal for combatoral problems. However, for problems that are umercal atural, the potetal parameter space ca be sgfcatly reduced by usg the frst order codto of the optmal soluto. I ths case, the VC aalyss may become suboptmal sce t assumes a larger search space tha what a typcal umercal procedure uses. Geerally speakg, for a problem that s learly separable wth a large marg, the expected classfcato error of the computed hyperplae resulted from ths aalyss s of the order O( log ). Smlar geeralzato bouds ca also be obtaed for o-separable problems. I chapter 0 of [], Vapk descrbed a leave-oe-out cross-valdato aalyss for learly separable problems. Ths aalyss takes to accout the frst order KKT codto of the optmal hyperplae w. The expected geeralzato performace from ths aalyss s O( ), whch s better tha the correspodg bouds from the VC aalyss. Ufortuately, ths techque s oly sutable for dervg a expected geeralzato boud (for example, t s ot useful for obtag a PAC style probablty boud). Aother well-kow techque for aalyzg learly separable problems s the mstake boud framework ole learg. It s possble to obta a algorthm wth a small geeralzato error the batch learg settg from a algorthm wth a small ole mstake Bouds descrbed [3] would mply a expected classfcato error of log2 O( ), whch ca be slghtly mproved (by a log factor) f we adopt a slghtly better coverg umber estmate such as the bouds [2, 4].

3 boud. The readers are referred to [6] ad refereces there for ths type of aalyss. The techque may lead to a boud wth a expected geeralzato performace of O( ). Besdes the above metoed approaches, geeralzato ablty ca also be studed the statstcal mechacal learg framework. It was show that for learly separable problems, expoetal decrease of msclassfcato error s possble uder ths framework [, 5, 7, 8]. Ufortuately, t s uclear how to relate the statstcal mechacal learg framework to the batch learg framework cosdered ths paper. Ther aalyss, employg approxmato techques, does ot seem to mply small sample bouds whch we are terested. The statstcal mechacal learg result suggests that t may be possble to obta a smlar expoetal decay of msclassfcato error the batch learg settg, whch we prove the ext secto. Furthermore, we show that the expoetal rate depeds o a quatty that s dfferet tha the tradtoal marg cocept. Our aalyss reles o a PAC style probablty estmate o the covergece rate of the estmated parameter from (2) to the true parameter. Cosequetly, t s sutable for o-separable problems. A drect aalyss o the covergece rate of the estmated parameter to the true parameter s mportat for problems that are umercal ature such as (2). However, a dsadvatage of our aalyss s that we are uable to drectly deal wth the learly separable formulato (). 3 Expoetal covergece We ca rewrte the SVM formulato (2) by elmatg as: where = =(C) ad w () = arg m w f(w T x y? ) + 2 wt w; (3) f(z) =?z z 0; 0 z > 0: Deote by D the true uderlyg data dstrbuto of (x; y), ad let w () be the optmal soluto wth respect to the true dstrbuto as: w () = arg f Df(w T xy? ) + w 2 wt w: (4) Let w be the soluto to w = arg f w 2 w2 s.t. E D f(w T xy? ) = 0; (5) whch s the fte-sample verso of the optmal hyperplae method. Throughout ths secto, we assume kw k 2 <, ad E D kxk 2 <. The latter codto esures that E D f(w T xy? ) kwk 2 E D kxk 2 + exsts for all w. 3. Cotuty of soluto uder regularzato I ths secto, we show that kw ()? w k 2! 0 as! 0. Ths cotuty result allows us to approxmate (5) by usg (4) ad (3) wth a small postve regularzato parameter. We oly eed to show that wth ay sequece of that coverges to zero, there exsts a subsequece! 0 such that w ( ) coverges to w strogly. We frst cosder the followg equalty whch follows from the defto of w (): E D f(w () T xy? ) + 2 w () 2 2 w2 : (6)

4 Therefore kw ()k 2 kw k 2. It s well-kow that every bouded sequece a Hlbert space cotas a weakly coverget subsequece (cf. Proposto 66.4 [4]). Therefore wth ay sequece of that coverges to zero, there exsts a subsequece! 0 such that w ( ) coverges weakly. We deote the lmt by ~w. Sce f(w () T xy? ) s domated by kw k 2 kxk 2 + whch has a fte tegral wth respect to D, therefore from (6) ad the Lebesgue domated covergece theorem, we obta 0 = lm E D f(w ( ) T xy? ) = E D lm f(w ( ) T xy? ) = E D f( ~w T xy? ): (7) Also ote that k ~wk 2 lm kw ( )k 2 kw k 2, therefore by the defto of w, we must have ~w = w. Sce w s the weak lmt of w ( ), we obta kw k 2 lm kw ( )k 2. Also sce kw ( )k 2 kw k 2, therefore lm kw ( )k 2 = kw k 2. Ths equalty mples that w ( ) coverges to w strogly sce lm (w ( )? w ) 2 = lm w ( ) 2 + w 2? 2 lm w ( ) T w = 0: 3.2 Accuracy of estmated hyperplae wth o-zero regularzato parameter Our goal s to show that for the estmato method (3) wth a ozero regularzato parameter > 0, the estmated parameter w () coverges to the true parameter w () probablty whe the sample sze!. Furthermore, we gve a large devato boud o the rate of covergece. From (4), we obta the followg frst order codto: E D (; x; y)xy + w () = 0; (8) where (; x; y) = f 0 (w () T xy? ) ad f 0 (z) 2 [?; 0] deotes a member of the subgradet of f at z [9]. 2 I the fte sample case, we ca also terpret (; x; y) (8) as a scaled dual varable : =?=C, where appears the dual (or Kerel) formulato of a SVM (for example, see chapter 0 of []). The covexty of f mples that f(z ) + (z 2? z )f 0 (z ) f(z 2 ) for ay subgradet f 0 of f. Ths mples the followg equalty: whch s equvalet to: f(w () T x y? ) + (w ()? w ()) T f(w () T x y ); f(w () T x y? ) + 2 w () 2 + (w ()? w ()) T [ f(w () T x y? ) + 2 w () 2 : (; x ; y )x y (; x ; y )x y + w ()] + 2 (w ()? w ()) 2 2 For readers ot famlar wth the subgradet cocept covex aalyss, our aalyss requres lttle modfcato f we replace f wth a smoother covex fucto such as f 2, whch avods the dscotuty the frst order dervatve.

5 Also ote that by the defto of w (), we have: f(w () T x y? ) + 2 w () 2 Therefore by comparg the above two equaltes, we obta: kw ()? w ()k 2 k 2 (w ()? w ()) 2 (w ()? w ()) T [ Therefore we have kw ()? w ()k 2 2 k = 2 k f(w () T x y? ) + 2 w () 2 : (; x ; y )x y + w ()] (; x ; y )x y + w ()k 2 : (; x ; y )x y + w ()k 2 (; x ; y )x y? E D (; x; y)xyk 2 : (9) Note that (9), we have already bouded the covergece of w () to w () terms of the covergece of the emprcal expectato of a radom vector (; x; y)xy to ts mea. I order to obta a large devato boud o the covergece rate, we eed the followg result whch ca be foud [3], page 95: P Theorem 3. Let be zero-mea depedet radom vectors a Hlbert space. If there exsts M > 0 such that for all atural umbers l 2: Ek = k l bl!m l 2. The for 2 all > 0: P (k P k 2 ) 2 exp(? 2 2 =(bm 2 + M )). Usg the fact that (; x; y) 2 [?; 0], t s easy to verfy the followg corollary by usg Theorem 3. ad (9), where we also boud the l-th momet of the rght had sde of (9) usg the followg form of Jese s equalty: ja + bj l 2 l? (jaj l + jbj l ) for l 2. Corollary 3. If there exsts M > 0 such that for all atural umbers l 2: E D kxk l 2 b 2 l!m l. The for all > 0: P (kw ()? w ()k 2 ) 2 exp(? =(4bM 2 + M )): Let P D () deote the probablty wth respect to dstrbuto D, the the followg boud o the expected msclassfcato error of the computed hyperplae w () s a straghtforward cosequece of Corollary 3.: Corollary 3.2 Uder the assumptos of Corollary 3., the for ay o-radom values ; ; K > 0, we have: E P D (w () T xy 0) P D (w () T xy ) + P D (kxk 2 K) + 2 exp(? =(4bK 2 M 2 + KM )); where the expectato E s take over radom samples from D wth w () estmated from the samples. We ow cosder learly separable classfcato problems where the soluto w of (5) s fte. Throughout the rest of ths secto, we mpose a addtoal assumpto that the

6 dstrbuto D s ftely supported: kxk 2 M almost everywhere wth respect to the measure D. From Secto 3., we kow that for ay suffcetly small postve umber, kw? w ()k 2 < =M, whch meas that w () also separates the -class data from the outof-class data wth a marg of at least 2(? Mkw? w ()k 2 ). Therefore for suffcetly small, we ca defe: () = supfb : P D (w () T xy b) = 0g? Mkw? w ()k 2 > 0: By Corollary 3.2, we obta the followg upper-boud o the msclassfcato error f we compute a lear separator from (3) wth a o-zero small regularzato parameter : E P D (w () T xy 0) 2 exp(? 8 2 () 2 =(4M 4 + ()M 2 )): Ths dcates that the expected msclassfcato error of a approprately computed hyperplae for a learly separable problem s expoetal. However, the rate of covergece depeds o ()=M 2. Ths quatty s dfferet tha the marg cocept whch has bee wdely used the lterature to characterze the geeralzato behavor of a lear classfcato problem. The ew quatty measures the covergece rate of w () to w as! 0. The faster the covergece, the more robust the lear classfcato problem s, ad hece the faster the expoetal decay of msclassfcato error s. As we shall see the ext secto, ths robustess s related to the degree of outlers the problem. 4 Example We gve a example to llustrate the robustess cocept that characterzes the expoetal decay of msclassfcato error. It s kow from Vapk s cross-valdato boud [] (Theorem 0.7) that by usg the large marg dea aloe, oe ca derve a expected msclassfcato error boud that s of the order O(=), where the costat s marg depedet. We show that ths boud s tght by usg the followg example. Example 4. Cosder a two-dmesoal problem. Assume that wth probablty of?, we observe a data pot x wth label y such that xy = [; 0]; ad wth probablty of, we observe a data pot x wth label y such that xy = [?; ]. Ths problem s obvously learly separable wth a large marg that s depedet. Now, for radom trag data, wth probablty at most + (? ), we observe ether x y = [; 0] for all = ; : : : ;, or x y = [?; ] for all = ; : : : ;. For all other cases, the computed optmal hyperplae w = w. Ths meas that the msclassfcato error s (? )(? + (? )? ). Ths error coverges to zero expoetally as!. However the covergece rate depeds o the fracto of outlers the dstrbuto characterzed by. I partcular, for ay, f we let = =, the we have a expected msclassfcato error that s at least (? =) =(e). 2 The above tghtess costructo of the lear decay rate of the expected geeralzato error (usg the marg cocept aloe) requres the scearo that a small fracto (whch shall be the order of verse sample sze) of data are very dfferet from other data. Ths small porto of data ca be cosdered as outlers, whch ca be measured by the robustess of the dstrbuto. I geeral, w () coverges to w slowly whe there exst such a small porto of data (outlers) that caot be correctly classfed from the observato of the remag data. It ca be see that the optmal hyperplae () s qute sestve to eve a sgle outler. Itutvely, ths stablty s qute udesrable. However, the prevous large marg learg bouds seemed to have dsmssed ths cocer. Ths

7 paper dcates that such a cocer s stll vald. I the worst case, eve f the problem s separable by a large marg, outlers ca stll cause a slow dow of the expoetal covergece rate. 5 Cocluso I ths paper, we derved ew geeralzato bouds for large marg learly separable classfcato. Eve though we have oly dscussed the cosequece of ths aalyss for separable problems, the techque ca be easly appled to o separable problems (see Corollary 3.2). For large marg separable problems, we show that expoetal decay of geeralzato error may be acheved wth a approprately chose regularzato parameter. However, the boud depeds o a quatty whch characterzes the robustess of the dstrbuto. A mportat dfferece of the robustess cocept ad the marg cocept s that outlers may ot be observable wth large probablty from data whle marg geerally wll. Ths mples that wthout ay pror kowledge, t could be dffcult to drectly apply our boud usg oly the observed data. Refereces [] J.K. Alauf ad M. Behl. The AdaTro: a adaptve perceptro algorthm. Europhys. Lett., 0(7): , 989. [2] C. Cortes ad V.N. Vapk. Support vector etworks. Mache Learg, 20: , 995. [3] Nello Crsta ad Joh Shawe-Taylor. A Itroducto to Support Vector Maches ad other Kerel-based Learg Methods. Cambrdge Uversty Press, [4] Harro G. Heuser. Fuctoal aalyss. Joh Wley & Sos Ltd., Chchester, 982. Traslated from the Germa by Joh Horváth, A Wley-Iterscece Publcato. [5] W. Kzel. Statstcal mechacs of the perceptro wth maxmal stablty. I Lecture Notes Physcs, volume 368, pages Sprger-Verlag, 990. [6] J. Kve ad M.K. Warmuth. Addtve versus expoetated gradet updates for lear predcto. Joural of Iformato ad Computato, 32: 64, 997. [7] M. Opper. Learg tmes of eural etworks: Exact soluto for a perceptro algorthm. Phys. Rev. A, 38(7): , 988. [8] M. Opper. Learg eural etworks: Solvable dyamcs. Europhyscs Letters, 8(4): , 989. [9] R. Tyrrell Rockafellar. Covex aalyss. Prceto Uversty Press, Prceto, NJ, 970. [0] Dale Schuurmas. Characterzg ratoal versus expoetal learg curves. J. Comput. Syst. Sc., 55:40 60, 997. [] V.N. Vapk. Statstcal learg theory. Joh Wley & Sos, New York, 998. [2] Robert C. Wllamso, Alexader J. Smola, ad Berhard Schölkopf. Etropy umbers of lear fucto classes. I COLT 00, pages , [3] Vadm Yursky. Sums ad Gaussa vectors. Sprger-Verlag, Berl, 995. [4] Tog Zhag. Aalyss of regularzed lear fuctos for classfcato problems. Techcal Report RC-2572, IBM, 999. Abstract NIPS 99, pp