Adaptive Regularization of Weight Vectors

Size: px

Start display at page:

Download "Adaptive Regularization of Weight Vectors"

Ruth Fields
6 years ago
Views:

1 Adapive Regulaizaion of Weigh Vecos Koby Camme Depamen of Elecical Engineing he echnion Haifa, Isael Alex Kulesza Depamen of Compue and Infomaion Science Univesiy of Pennsylvania Philadelphia, PA Mak Dedze Human Language ech. Cene of Excellence Johns Hopkins Univesiy Balimoe, MD Absac We pesen, a new online leaning algoihm ha combines seveal useful popeies: lage magin aining, confidence weighing, and he capaciy o handle non-sepaable daa. pefoms adapive egulaizaion of he pedicion funcion upon seeing each new insance, allowing i o pefom especially well in he pesence of label noise. We deive a misake bound, simila in fom o he second ode pecepon bound, ha does no assume sepaabiliy. We also elae ou algoihm o ecen confidence-weighed online leaning echniques and show empiically ha achieves sae-of-he-a pefomance and noable obusness in he case of non-sepaable daa. 1 Inoducion Online leaning algoihms ae fas, simple, make few saisical assumpions, and pefom well in a wide vaiey of seings. Recen wok has shown ha paamee confidence infomaion can be effecively used o guide online leaning [2]. Confidence weighed ) leaning, fo example, mainains a Gaussian disibuion ove linea classifie hypoheses and uses i o conol he diecion and scale of paamee updaes [6]. In addiion o fomal guaanees in he misake-bound model [11], leaning has achieved sae-of-he-a pefomance on many asks. Howeve, he sic updae cieion used by leaning is vey aggessive and can ove-fi [5]. Appoximae soluions can be used o egulaize he updae and impove esuls; howeve, cuen analyses of leaning sill assume ha he daa ae sepaable. I is no immediaely clea how o elax his assumpion. In his pape we pesen a new online leaning algoihm fo binay classificaion ha combines seveal aacive popeies: lage magin aining, confidence weighing, and he capaciy o handle non-sepaable daa. he key o ou appoach is he adapive egulaizaion of he pedicion funcion upon seeing each new insance, so we call his algoihm Adapive Regulaizaion of Weighs ). Because i adjuss is egulaizaion fo each example, is obus o sudden changes in he classificaion funcion due o label noise. We deive a misake bound, simila in fom o he second ode pecepon bound, ha does no assume sepaabiliy. We also povide empiical esuls demonsaing ha is compeiive wih sae-of-he-a mehods and impoves upon hem significanly in he pesence of label noise. 2 Confidence Weighed Online Leaning of Linea Classifies Online algoihms opeae in ounds. In ound he algoihm eceives an insance x R d and applies is cuen pedicion ule o make a pedicion ŷ Y. I hen eceives he ue 1

2 label y Y and suffes a loss ly, ŷ ). Fo binay classificaion we have Y = { 1, +1} and use he zeo-one loss l 01 y, ŷ ) = 0 if y = ŷ and 1 ohewise. Finally, he algoihm updaes is pedicion ule using x, y ) and poceeds o he nex ound. In his wok we conside linea pedicion ules paameeized by a weigh veco w: ŷ = h w x) = signw x). Recenly Dedze, Camme and Peeia [6, 5] poposed an algoihmic famewok fo online leaning of binay classificaion asks called confidence weighed ) leaning. leaning capues he noion of confidence in a linea classifie by mainaining a Gaussian disibuion ove he weighs wih mean µ R d and covaiance maix Σ R d d. he values µ p and Σ p,p, especively, encode he leane s knowledge of and confidence in he weigh fo feaue p: he smalle Σ p,p, he moe confidence he leane has in he mean weigh value µ p. Covaiance ems Σ p,q capue ineacions beween weighs. Concepually, o classify an insance x, a classifie daws a paamee veco w N µ, Σ) and pedics he label accoding o signw x). In pacice, howeve, i can be easie o simply use he aveage weigh veco E [w] = µ o make pedicions. his is simila o he appoach aken by Bayes poin machines [9], whee a single weigh veco is used o appoximae a disibuion. Fuhemoe, fo binay classificaion, he pedicion given by he mean weigh veco uns ou o be Bayes opimal. classifies ae ained accoding o a passive-aggessive ule [3] ha adjuss he disibuion a each ound o ensue ha he pobabiliy of a coec pedicion is a leas η 0.5, 1]. his yields he updae consain P [y w x ) 0] η. Subjec o his consain, he algoihm makes he smalles possible change o he hypohesis weigh disibuion as measued using he KL divegence. his implies he following opimizaion poblem fo each ound : µ, Σ ) = min D )) KL N µ, Σ) N µ 1, Σ 1 µ,σ s.. P w N µ,σ) [y w x ) 0] η Confidence-weighed algoihms have been shown o pefom well in pacice [5, 6], bu hey suffe fom seveal poblems. Fis, he updae is quie aggessive, focing he pobabiliy of pedicing each example coecly o be a leas η > 1/2 egadless of he cos o he objecive. his may cause sevee ove-fiing when labels ae noisy; indeed, cuen analyses of he algoihm [5] assume ha he daa ae linealy sepaable. Second, hey ae designed fo classificaion, and i is no clea how o exend hem o alenaive seings such as egession. his is in pa because he consain is wien in discee ems whee he pedicion is eihe coec o no. We deal wih boh of hese issues, coping moe effecively wih label noise and genealizing he advanages of leaning in an exensible way. 3 Adapive Regulaizaion Of Weighs We idenify wo impoan popeies of he updae ule ha conibue o is good pefomance bu also make i sensiive o label noise. Fis, he mean paamees µ ae guaaneed o coecly classify he cuen aining example wih magin following each updae. his is because he pobabiliy consain P [y w x ) 0] η can be wien explicily as y µ x ) φ x Σx, whee φ > 0 is a posiive consan elaed o η. his aggessiveness yields apid leaning, bu given an incoecly labeled example, i can also foce he leane o make a dasic and incoec change o is paamees. Second, confidence, as measued by he invese eigenvalues of Σ, inceases monoonically wih evey updae. While i is inuiive ha ou confidence should gow as we see moe daa, his also means ha even incoecly labeled examples causing wild paamee swings esul in aificially inceased confidence. In ode o mainain he posiives bu educe he negaives of hese wo popeies, we isolae and sofen hem. As in leaning, we mainain a Gaussian disibuion ove weigh vecos wih mean µ and covaiance Σ; howeve, we ecas he above chaaceisics of he consain as egulaizes, minimizing he following unconsained objecive on 2

3 each ound: C µ, Σ) = D KL N µ, Σ) N µ 1, Σ 1 )) + λ1 l h 2 y, µ x ) + λ 2 x Σx, 1) whee l h 2 y, µ x ) = max{0, 1 y µ x )}) 2 is he squaed-hinge loss suffeed using he weigh veco µ o pedic he oupu fo inpu x when he ue oupu is y. λ 1, λ 2 0 ae wo adeoff hypepaamees. Fo simpliciy and compacness of noaion, in he following we will assume ha λ 1 = λ 2 = 1/2) fo some > 0. he objecive balances hee desies. Fis, he paamees should no change adically on each ound, since he cuen paamees conain infomaion abou pevious examples fis em). Second, he new mean paamees should pedic he cuen example wih low loss second em). Finally, as we see moe examples, ou confidence in he paamees should geneally gow hid em). Noe ha his objecive is no simply he dualizaion of he consain, bu a new fomulaion inspied by he popeies discussed above. Since he loss em depends on µ only via he inne-poduc µ x, we ae able o pove a epesene heoem Sec. 4). While we use he squaed-hinge loss fo classificaion, diffeen loss funcions, as long as hey ae convex and diffeeniable in µ, yield algoihms fo diffeen seings. 1 o solve he opimizaion in 1), we begin by wiing he KL explicily: C µ, Σ) = 1 ) de 2 log Σ de Σ 2 Σ 1 1 Σ) + 1 µ 1 µ ) Σ 1 1 µ 1 µ ) d l h 2 y, µ x ) x Σx 2) We can decompose he esul ino wo ems: C 1 µ), depending only on µ, and C 2 Σ), depending only on Σ. he updaes o µ and Σ can heefoe be pefomed independenly. he squaed-hinge loss yields a consevaive o passive) updae fo µ in which he mean paamees change only when he magin is oo small, and we follow leaning by enfocing a coespondingly consevaive updae fo he confidence paamee Σ, updaing i only when µ changes. his esuls in fewe updaes and is easie o analyze. Ou updae hus poceeds in wo sages. 1. Updae he mean paamees: µ = ag min µ C 1 µ) 3) 2. If µ µ 1, updae he confidence paamees: Σ = ag min Σ C 2 Σ) 4) We now develop he updae equaions fo 3) and 4) explicily, saing wih he fome. aking he deivaive of C µ, Σ) wih espec o µ and seing i o zeo, we ge µ = µ 1 1 [ ] d 2 dz l h 2 y, z) z=µ x Σ 1 x, 5) assuming Σ 1 is non-singula. Subsiuing he deivaive of he squaed-hinge loss in 5) and assuming 1 y µ x ) 0, we ge µ = µ 1 + y 1 y µ x )) Σ 1 x. 6) We solve fo µ by aking he do poduc of each side of he equaliy wih x and subsiuing back in 6) o obain he ule µ = µ 1 + max ) 0, 1 y x µ 1 x Σ 1 y x. 7) Σ 1 x + I can be easily veified ha 7) saisfies ou assumpion ha 1 y µ x ) 0. 1 I can be shown ha he well known ecusive leas squaes RLS) egession algoihm [7] is a special case of wih he squaed loss. 3

4 Inpu paamees Iniialize µ 0 = 0, Σ 0 = I, Fo = 1,..., Receive a aining example x R d Compue magin and confidence m = µ 1 x v = x Σ 1x Receive ue label y, and suffe loss l = 1 if sign m ) y If m y < 1, updae using eqs. 7) & 9): µ = µ 1 + α Σ 1y x Σ = Σ 1 β Σ 1x x Σ 1 1 β = α x = max 0, 1 y x µ Σ 1x + 1 β Oupu: Weigh veco µ and confidence Σ. Figue 1: he algoihm fo online binay classificaion. he updae fo he confidence paamees is made only if µ µ 1, ha is, if 1 > y x µ 1. In his case, we compue he updae of he confidence paamees by seing he deivaive of C µ, Σ) wih espec o Σ o zeo: Σ 1 = Σ x x Using he Woodbuy ideniy we can also ewie he updae fo Σ in non-inveed fom: Σ = Σ 1 Σ 1x x Σ 1 + x Σ 1 x 9) Noe ha i follows diecly fom 8) and 9) ha he eigenvalues of he confidence paamees ae monoonically deceasing: Σ Σ 1 ; Σ 1 Σ 1 1. Pseudocode fo appeas in Fig Analysis We fis show ha can be kenelized by saing he following epesene heoem. Lemma 1 Repesene heoem) Assume ha Σ 0 = I and µ 0 = 0. he mean paamees µ and confidence paamees Σ poduced by updaing via 7) and 9) can be wien as linea combinaions of he inpu vecos esp. oue poducs of he inpu vecos wih hemselves) wih coefficiens depending only on inne-poducs of inpu vecos. Poof skech: By inducion. he base case follows fom he definiions of µ 0 and Σ 0, and he inducion sep follows algebaically fom he updae ules 7) and 9). We now pove a misake bound fo. Denoe by M M) = M ) he se of example indices fo which he algoihm makes a misake, y µ 1 x 0, and by U U = U ) he se of example indices fo which hee is an updae bu no a misake, 0 < y µ x ) 1. Ohe examples do no affec he behavio of he algoihm and can be ignoed. Le X M = M x ix i, X U = U x ix i and X A = X M + X U. heoem 2 Fo any efeence weigh veco u R d, he numbe of misakes made by Fig. 1) is uppe bounded by M u 2 + u X A u log de I + 1 )) X A + U + g U, 10) whee g = max 0, 1 y u x ). he poof depends on wo lemmas; we omi he poof of he fis fo lack of space. 8) 4

5 Lemma 3 Le l = max 0, 1 y µ 1x ) and χ = x Σ 1 x. hen, fo evey M U, u Σ 1 µ = u Σ 1 1 µ 1 + y u x µ Σ 1 µ = µ 1Σ 1 1 µ 1 + χ + l 2 χ + ) Lemma 4 Le be he numbe of ounds. hen χ χ + ) log de Σ 1 +1)). Poof: We compue he following quaniy: x Σ x = x Using Lemma D.1 fom [2] we have ha Combining, we ge χ χ + ) = We now pove heoem 2. Poof: Σ 1 β Σ 1 x x ) Σ 1 x = χ χ2 χ + = χ χ +. 1 x Σ x 1 de Σ 1 )) 1 de Σ 1 ) We ieae he fis equaliy of Lemma 3 o ge u Σ 1 µ = y u x We ieae he second equaliy o ge µ Σ 1 µ = χ + l 2 χ + ) = 1 de ) Σ 1 1 de Σ 1 ). 11) = )) de Σ 1 1 log de Σ 1 ) 1 g = M + U χ χ + ) + 1 log de Σ 1 +1)). g. 12) 1 l 2 χ +. 13) Using Lemma 4 we have ha he fis em of 13) is uppe bounded by 1 log de Σ 1 )). Fo he second em in 13) we conside ) wo cases. Fis, if a misake occued on example, hen we have ha y x µ 1 0 and l 1, so 1 l 2 0. Second, if ) an he algoihm made an updae bu no misake) on example, hen 0 < y x µ 1 1 and l 0, hus 1 l 2 1. We heefoe have 1 l 2 χ + 0 χ χ + = 1 χ +. 14) M U U Combining and plugging ino he Cauchy-Schwaz inequaliy u Σ 1 µ u Σ 1 u µ Σ 1 µ, we ge M + U 1 1 g u Σ 1 u log de Σ 1 )) 1 + χ +. 15) Reaanging he ems and using he fac ha χ 0 yields M u Σ 1 u log de Σ 1 )) + U + U g U. 5

6 By definiion, Σ 1 =I + 1 x i x i =I + 1 X A, so subsiuing and simplifying complees he poof: M u I + 1 ) X A u log de I + 1 )) X A + U + = u 2 + u X A u log de I + 1 )) X A + U + g U g U. A few commens ae in ode. Fis, he wo squae-oo ems of he bound depend on in opposie ways: he fis is monoonically inceasing, while he second is monoonically deceasing. One could expec o opimize he bound by minimizing ove. Howeve, he bound also depends on indiecly via ohe quaniies e.g. X A ), so hee is no diec way o do so. Second, if all he updaes ae associaed wih eos, ha is, U =, hen he bound educes o he bound of he second-ode pecepon [2]. In geneal, howeve, he bounds ae no compaable since each depends on he acual unime behavio of is algoihm. 5 Empiical Evaluaion We evaluae on boh synheic and eal daa, including seveal popula daases fo documen classificaion and opical chaace ecogniion OCR). We compae wih hee baselines: Passive-Aggessive PA), Second Ode Pecepon SOP) 2 and Confidence- Weighed ) leaning 3. Ou synheic daa ae as in [5], bu we inve he labels on 10% of he aining examples. Noe ha evaluaion is sill done agains he ue labels.) Fig. 2a) shows he online leaning cuves fo boh full and diagonalized vesions of he algoihms on hese noisy daa. impoves ove all compeios, and he full vesion oupefoms he diagonal vesion. Noe ha -full pefoms wose han -diagonal, as has been obseved peviously fo noisy daa. We seleced a vaiey of documen classificaion daases popula in he NLP communiy, summaized as follows. Amazon: Poduc eviews o be classified ino domains e.g., books o music) [6]. We ceaed binay daases by aking all pais of he six domains 15 daases). Feaue exacion follows [1] bigam couns). 20 Newsgoups: Appoximaely 20,000 newsgoup messages paiioned acoss 20 diffeen newsgoups 4. We binaized he copus following [6] and used binay bag-of-wods feaues 3 daases). Each daase has beween 1850 and 1971 insances. Reues RCV1-v2/LYRL2004): Ove 800,000 manually caegoized newswie soies. We ceaed binay classificaion asks using pais of labels following [6] 3 daases). Deails on documen pepaaion and feaue exacion ae given by [10]. Senimen: Poduc eviews o be classified as posiive o negaive. We used each Amazon poduc eview domain as a senimen classificaion ask 6 daases). Spam: We seleced hee ask A uses fom he ECML/PKDD Challenge 5, using bag-ofwods o classify each as spam o ham 3 daases). Fo OCR daa we binaized wo well known digi ecogniion daases, MNIS 6 and USPS, ino 45 all-pais poblems. We also ceaed en one vs. all daases fom he MNIS daa 100 daases oal). Each esul fo he ex daases was aveaged ove 10-fold coss-validaion. he OCR expeimens used he sandad spli ino aining and es ses. Hypepaamees including 2 Fo he eal wold high dimensional) daases, we mus dop coss-feaue confidence ems by pojecing ono he se of diagonal maices, following he appoach of [6]. While his may educe pefomance, we make he same appoximaion fo all evaluaed algoihms. 3 We use he vaiance vesion developed in [6]. 4 hp://people.csail.mi.edu/jennie/20newsgoups/ 5 hp://ecmlpkdd2006.og/challenge.hml 6 hp://yann.lecun.com/exdb/mnis/index.hml 6

7 Misakes Pecepon PA SOP full diag full diag Insances a) synheic daa Misakes PA SOP Insances Misakes b) MNIS daa PA SOP Insances Figue 2: Leaning cuves fo full/diagonal) and baseline mehods. a) 5k synheic aining examples and 10k es examples 10% noise, 100 uns). b) MNIS 3 vs. 5 binay classificaion ask fo diffeen amouns of label noise lef: 0 noise, igh: 10%). fo ) and he numbe of online ieaions up o 10) wee opimized using a single andomized un. We used 2000 insances fom each daase unless ohewise noed above. In ode o obseve each algoihm s abiliy o handle non-sepaable daa, we pefomed each expeimen using vaious levels of aifical label noise, geneaed by independenly flipping each binay label wih fixed pobabiliy. 5.1 Resuls and Discussion Ou expeimenal esuls ae summaized in able 1. oupefoms he baselines a all noise levels, bu does especially well as noise inceases. Moe deailed esuls fo and, he oveall bes pefoming baseline, ae compaed in Fig. 3. and ae compaable when hee is no added noise, wih winning he majoiy of he ime. Noise level Algoihm PA SOP able 1: Mean ank ou of 4, ove all daases) a diffeen noise levels. A ank of 1 indicaes ha an algoihm oupefomed all he ohes. As label noise inceases moving acoss he ows in Fig. 3) holds up emakably well. In almos evey high noise evaluaion, impoves ove as well as he ohe baselines, no shown). Fig. 2b) shows he oal numbe of misakes w... noise-fee labels) made by each algoihm duing aining on he MNIS daase fo 0% and 10% noise. hough absolue pefomance suffes wih noise, he gap beween and he baselines inceases. o help inepe he esuls, we classify he algoihms evaluaed hee accoding o fou chaaceisics: he use of lage magin updaes, confidence weighing, a design ha accomodaes non-sepaable daa, and adapive pe-insance magin able 2). While all of hese popeies can be desiable in diffeen siuaions, we would like o undesand how hey ineac and achieve high pefomance while avoiding sensiiviy o noise. Based on he esuls in able 1, i is clea ha he combinaion of confidence infoma- Algoihm Magin idence Sepaable Magin Lage Conf- Non- Adapive ion and lage magin leaning PA Yes No Yes No is poweful when label noise is SOP No Yes Yes No low. easily oupefoms Yes Yes No Yes he ohe baselines in such siuaions, as i has been shown o Yes Yes Yes No do in pevious wok. Howeve, able 2: Online algoihm popeies oveview. as noise inceases, he sepaabiliy assumpion inheen in appeas o educe is pefomance consideably. 7

8 news amazon eues senimen spam news amazon eues senimen spam news amazon eues senimen spam USPS 1 vs. All USPS All Pais MNIS 1 vs. All USPS 1 vs. All USPS All Pais MNIS 1 vs. All USPS 1 vs. All USPS All Pais MNIS 1 vs. All Figue 3: Accuacy on ex op) and OCR boom) binay classificaion. Plos compae pefomance beween and, he bes pefoming baseline able 1). Makes above he line indicae supeio pefomance and below he line supeio pefomance. Label noise inceases fom lef o igh: 0%, 10% and 30%. impoves elaive o as noise inceases., by combining he lage magin and confidence weighing of wih a sof updae ule ha accomodaes non-sepaable daa, maches s pefomance in geneal while avoiding degadaion unde noise. lacks he adapive magin of, suggesing ha his chaaceisic is no cucial o achieving song pefomance. Howeve, we leave open fo fuue wok he possibiliy ha an algoihm wih all fou popeies migh have unique advanages. 6 Relaed and Fuue Wok is mos simila o he second ode pecepon [2]. he SOP pefoms he same ype of updae as, bu only when i makes an eo., on he ohe hand, updaes even when is pedicion is coec if hee is insufficien magin. Confidence weighed ) [6, 5] algoihms, by which was inspied, updae he mean and confidence paamees simulaneously, while makes a decoupled updae and sofens he had consain of. he algoihm can be seen as a vaian of he PA-II algoihm fom [3] whee he egulaizaion is modified accoding o he daa. Hazan [8] descibes a famewok fo gadien descen algoihms wih logaihmic ege in which a quaniy simila o Σ plays an impoan ole. Ou algoihm diffes in seveal ways. Fis, Hazan [8] consides gadien algoihms, while we deive and analyze algoihms ha diecly solve an opimizaion poblem. Second, we bound he loss diecly, no he cumulaive sum of egulaizaion and loss. hid, he gadien algoihms pefom a pojecion afe making an updae no befoe) since he nom of he weigh veco is kep bounded. Ongoing wok includes he developmen and analysis of syle algoihms fo ohe seings, including a muli-class vesion following he ecen exension of o muli-class poblems [4]. Ou misake bound can be exended o his case. Applying he ideas behind o egession poblems uns ou o yield he well known ecusive leas squaes RLS) algoihm, fo which offes new bounds omied). Finally, while we used he confidence em x Σx in 1), we can eplace his em wih any diffeeniable, monoonically inceasing funcion fx Σx ). his genealizaion may yield addiional algoihms. 8

9 Refeences [1] John Blize, Mak Dedze, and Fenando Peeia. Biogaphies, bollywood, boom-boxes and blendes: Domain adapaion fo senimen classificaion. In ACL, [2] Nicoló Cesa-Bianchi, Alex Conconi, and Claudio Genile. A second-ode pecepon algoihm. Siam J. of Comm., 34, [3] Koby Camme, Ofe Dekel, Joseph Keshe, Shai Shalev-Shwaz, and Yoam Singe. Online passive-aggessive algoihms. Jounal of Machine Leaning Reseach, 7: , [4] Koby Camme, Mak Dedze, and Alex Kulesza. Muli-class confidence weighed algoihms. In Empiical Mehods in Naual Language Pocessing EMNLP), [5] Koby Camme, Mak Dedze, and Fenando Peeia. Exac convex confidence-weighed leaning. In Neual Infomaion Pocessing Sysems NIPS), [6] Mak Dedze, Koby Camme, and Fenando Peeia. Confidence-weighed linea classificaion. In Inenaional Confeence on Machine Leaning, [7] Simon Haykin. Adapive File heoy [8] Elad Hazan. Efficien algoihms fo online convex opimizaion and hei applicaions. PhD hesis, Pinceon Univesiy, [9] Ralf Hebich, hoe Gaepel, and Colin Campbell. Bayes poin machines. Jounal of Machine Leaning Reseach JMLR), 1: , [10] David D. Lewis, Yiming Yang, ony G. Rose, and Fan Li. Rcv1: A new benchmak collecion fo ex caegoizaion eseach. JMLR, 5: , [11] Nick Lilesone. Leaning when ielevan aibues abound: A new linea-heshold algoihm. Machine Leaning, 2: ,

Online Ranking by Projecting

Online Ranking by Projecting LETTER Communicaed by Edwad Haingon Online Ranking by Pojecing Koby Camme kobics@cs.huji.ac.il Yoam Singe singe@cs.huji.ac.il School of Compue Science and Engineeing, Hebew Univesiy, Jeusalem 91904, Isael