Recap. Online Learning RWM. Avrim Blum. [ACFS02]: applying RWM to bandits

Transcription

1 Algorihm Online Learning Recap Noregre algorihms for repeaed decisions: Algorihm has N opions World chooses cos vecor Can view as marix like his (maybe infinie # cols) World life fae Your guide: Avrim Blum Carnegie Mellon Universiy A each ime sep, algorihm picks row, life picks column Alg pays cos (or ges benefi) for acion chosen Alg ges column as feedback (or jus is own cos/benefi in he bandi model) Goal: do nearly as well as bes fixed row in hindsigh [Machine Learning Summer School 2012] RWM [ACFS02]: applying RWM o bandis Wha if only ge your own cos/benefi as feedback? (1ec 12 )(1ec 11 ) 1 (1ec 22 )(1ec 21 ) 1 (1ec 32 )(1ec 31 ) (1ec n2 )(1ec n1 ) 1 World life fae c 1 c 2 scaling so coss in [0,1] Use of RWM as subrouine o ge algorihm wih cumulaive regre O( (TN log N) 1/2 ) [average regre O( ((N log N)/T) 1/2 )] Will do a somewha weaker version of heir analysis (same algorihm bu no as igh a bound) Guaranee: E[cos] + 2( log n) 1/2 Since T, his is a mos + 2(Tlog n) 1/2 So, regre/ime sep 2(Tlog n) 1/2 /T! 0 For fun, alk abou i in he conex of online pricing Online pricing Say you are selling lemonade (or a cool new sofware ool, or boles of waer a he world cup) View each possible For =1,2, T price as a differen Seller ses price p row/exper Buyer arrives wih valuaion v If v p, buyer purchases and pays p, else doesn Repea $2 Assume all valuaions h Goal: do nearly as well as bes fixed price in hindsigh Muliarmed bandi problem Exponenial Weighs for Exploraion and Exploiaion (exp 3 ) [Auer,CesaBianchi,Freund,Schapire] Exper i ~ q Gain g i q Exp3 q = (1 )p + unif ĝ = (0,,0, g i /q i,0,,0) Disrib p Gain vecor ĝ 1 RWM believes gain is: p ĝ = p i (g i /q i ) g RWM 2 g RWM /(1+²) O(² 1 nh/ log n) 3 Acual gain is: g i = g RWM (q i /p i ) g RWM(1 ) nh/ RWM n = #expers 4 E[ ] Because E[ĝ j ] = (1 q j )0 + q j (g j /q j ) = g j, so E[max j [ ĝ j ]] max j [ E[ ĝ j ] ] = 1

2 Muliarmed bandi problem Exponenial Weighs for Exploraion and Exploiaion (exp 3 ) [Auer,CesaBianchi,Freund,Schapire] Exper i ~ q Gain g i q Exp3 q = (1 )p + unif ĝ = (0,,0, g i /q i,0,,0) Disrib p Gain vecor ĝ Conclusion ( = ²): E[Exp3] /(1+²) 2 O(² 2 nh log(n)) nh/ RWM n = #expers Balancing would give O(( nh log n) 2/3 ) in bound because of ² 2 Bu can reduce o ² 1 and O(( nh log n) 1/2 ) more care in analysis A naural generalizaion (Going back o fullinfo seing, hinking abou pahs ) A naural generalizaion of our regre goal is: wha if we also wan ha on rainy days, we do nearly as well as he bes roue for rainy days And on Mondays, do nearly as well as bes roue for Mondays More generally, have N rules (on Monday, use pah P) Goal: simulaneously, for each rule i, guaranee o do nearly as well as i on he ime seps in which i fires For all i, wan E[cos i (alg)] (1+e)cos i (i) + O(e 1 log N) (cos i (X) = cos of X on ime seps where rule i fires) Can we ge his? A naural generalizaion This generalizaion is esp naural in machine learning for combining muliple ifhen rules Eg, documen classificaion Rule: if <wordx> appears hen predic <Y> Eg, if has fooball hen classify as spors So, if 90% of documens wih fooball are abou spors, we should have error 11% on hem Specialiss or sleeping expers problem Assume we have N rules, explicily given For all i, wan E[cos i (alg)] (1+e)cos i (i) + O(e 1 log N) (cos i (X) = cos of X on ime seps where rule i fires) A simple algorihm and analysis (all on one slide) Sar wih all rules a weigh 1 A each ime sep, of he rules i ha fire, selec one wih probabiliy p i / wi Updae weighs: If didn fire, leave weigh alone If did fire, raise or lower depending on performance compared o weighed average: r i = [ j p j cos(j)]/(1+e) cos(i) w i à w i (1+e) ri So, if rule i does exacly as well as weighed average, is weigh drops a lile Weigh increases if does beer han weighed average by more han a (1+e) facor This ensures sum of weighs doesn increase Final w i = (1+e) E[cos i(alg)]/(1+e)cosi(i) So, exponen e 1 log N So, E[cos i (alg)] (1+e)cos i (i) + O(e 1 log N) Los of uses Can combine muliple ifhen rules Can combine muliple learning algorihms: Back o driving, say we are given N condiions o pay aenion o (is i raining?, is i a Monday?, ) Creae N rules: if day saisfies condiion i, hen use oupu of Alg i, where Alg i is an insaniaion of an expers algorihm you run on jus he days saisfying ha condiion Adaping o change Wha if we wan o adap o change do nearly as well as bes recen exper? For each exper, insaniae copy who wakes up on day for each 0 T1 Our cos in previous days is a mos (1+²)(bes exper in las days) + O(² 1 log(nt)) (no bes possible bound since exra log(t) bu no bad) Simulaneously, for each condiion i, do nearly as well as Alg i which iself does nearly as well as bes pah for condiion i 2

3 Summary Algorihms for online decisionmaking wih srong guaranees on performance compared o bes fixed choice Applicaion: play repeaed game agains adversary Perform nearly as well as fixed sraegy in hindsigh Can apply even wih very limied feedback Applicaion: which way o drive o work, wih only feedback abou your own pahs; online pricing, even if only have buy/no buy feedback More general forms of regre 1 bes exper or exernal regre: Given n sraegies Compee wih bes of hem in hindsigh 2 sleeping exper or regre wih imeinervals : Given n sraegies, k properies Le S i be se of days saisfying propery i (migh overlap) Wan o simulaneously achieve low regre over each S i 3 inernal or swap regre: like (2), excep ha S i = se of days in which we chose sraegy i Inernal/swapregre Eg, each day we pick one sock o buy shares in Don wan o have regre of he form every ime I bough IBM, I should have bough Microsof insead Formally, regre is wr opimal funcion f:{1,,n}!{1,,n} such ha every ime you played acion j, i plays f(j) Weird why care? Correlaed equilibrium Disribuion over enries in marix, such ha if a rused pary chooses one a random and ells you your par, you have no incenive o deviae Eg, Shapley game R P S R P S 1,1 1,1 1,1 1,1 1,1 1,1 1,1 1,1 1,1 In generalsum games, if all players have low swapregre, hen empirical disribuion of play is apx correlaed equilibrium Inernal/swapregre, cond Algorihms for achieving low regre of his form: Foser & Vohra, Har & MasColell, Fudenberg & Levine Will presen mehod of [BM05] showing how o conver any bes exper algorihm ino one achieving low swap regre Can conver any bes exper algorihm A ino one achieving low swap regre Idea: Insaniae one copy A j responsible for expeced regre over imes we play j Play p = pq Alg Cos vecor c q 2 Allows us o view p j as prob we play acion j, or as prob we play alg A j Give A j feedback of p j c Q A j guaranees (p j c ) q j min i p j c i + [regre erm] Wrie as: p j (q j c ) min i p j c i + [regre erm] p 2 c A 1 A 2 A n 3

4 Can conver any bes exper algorihm A ino one achieving low swap regre Idea: Insaniae one copy A j responsible for expeced regre over imes we play j Play p = pq Sum over j, ge: Q Alg Cos vecor c q 2 p 2 c p Q c j min i p j c i + n[regre erm] Our oal cos Wrie as: p j (q j c ) min i p j c i + [regre erm] A 1 A 2 A n For each j, can move our prob o is own i=f(j) Iinerary Sop 1: Minimizing regre and combining advice Randomized Wd Majoriy / Muliplicaive Weighs alg Connecions o game heory Sop 2: Exensions Online learning from limied feedback (bandi algs) Algorihms for large acion spaces, sleeping expers Sop 3: Powerful online LTF algorihms Winnow, Percepron Sop 4: Powerful ools for using hese algorihms Kernels and Similariy funcions Sop 5: Somehing compleely differen Disribued machine learning Transiion So far, we have been examining problems of selecing among choices/algorihms/expers given o us from ouside Now, urn o design of online algorihms for learning over daa described by feaures A ypical ML seing Say you wan a compuer program o help you decide which messages are urgen and which can be deal wih laer Migh represen each message by n feaures (eg, reurn address, keywords, header info, ec) On each message received, you make a classificaion and hen laer find ou if you messed up Goal: if here exiss a simple rule ha works (is perfec? low error?) hen our alg does well Simple example: disjuncions Suppose feaures are boolean: X = {0,1} n Targe is an OR funcion, like x 3 v x 9 v x 12 Can we find an online sraegy ha makes a mos n misakes? (assume perfec arge) Sure Sar wih h(x) = x 1 v x 2 v v x n Invarian: {vars in h} {vars in f } Misake on negaive: hrow ou vars in h se o 1 in x Mainains invarian and decreases h by 1 No misakes on posiives So a mos n misakes oal Simple example: disjuncions Suppose feaures are boolean: X = {0,1} n Targe is an OR funcion, like x 3 v x 9 v x 12 Can we find an online sraegy ha makes a mos n misakes? (assume perfec arge) Compare o expers seing: Could define 2 n expers, one for each OR fn #misakes log(# expers) This way is much more efficien bu, requires some exper o be perfec 4

5 Simple example: disjuncions Bu wha if we believe only r ou of he n variables are relevan? Ie, in principle, should be able o ge only O(log n r ) = O(r log n) misakes Can we do i efficienly? Winnow algorihm Winnow algorihm for learning a disjuncion of r ou of n variables eg f(x)= x 3 v x 9 v x 12 h(x): predic pos iff w 1 x w n x n n Iniialize w i = 1 for all i Misake on pos: w i à 2w i for all x i =1 Misake on neg: w i à 0 for all x i =1 Winnow algorihm Winnow algorihm for learning a disjuncion of r ou of n variables eg f(x)= x 3 v x 9 v x 12 h(x): predic pos iff w 1 x w n x n n Iniialize w i = 1 for all i Misake on pos: w i à 2w i for all x i =1 Misake on neg: w i à 0 for all x i =1 Thm: Winnow makes a mos O(r log n) misakes Proof: Each Mop doubles a leas one relevan weigh (and noe ha rel ws never se o 0) A mos r(1+log n) of hese Each Mop adds < n o oal weigh Each Mon removes a leas n from oal weigh So #(Mon) 1+ #(Mop) Tha s i! A generalizaion Winnow algorihm for learning a linear separaor wih nonneg ineger weighs: eg, 2x 3 + 4x 9 + x x 12 5 h(x): predic pos iff w 1 x w n x n n Iniialize w i = 1 for all i Misake on pos: w i à w i (1+²) for all x i =1 Misake on neg: w i à w i /(1+²) for all x i =1 Use ² = O(1/W), W = sum of ws in arge Thm: Winnow makes a mos O(W 2 log n) misakes Winnow for general LTFs More generally, can show he following: Suppose 9 w* s: w* x c on posiive x, w* x c on negaive x Then misake bound is O((L 1 (w*)/ ) 2 log n) Muliply by L 1 (X) if feaures no {0,1} Percepron algorihm An even older and simpler algorihm, wih a bound of a differen form Suppose 9 w* s: w* x on posiive x, w* x on negaive x Then misake bound is O((L 2 (w*)l 2 (x)/ ) 2 ) L 2 margin of examples 5

6 Percepron algorihm Thm: Suppose daa is consisen wih some LTF w* x > 0, where we scale so L 2 (w*)=1, L 2 (x) 1, = min x w* x Then # misakes 1/ Algorihm: Iniialize w=0 Use w x > 0 Misake on pos: w à w+x Misake on neg: w à wx + + w* + + Example: Percepron algorihm (0,1) (1,1) + (1,0) + + Algorihm: Iniialize w=0 Use w x > 0 Misake on pos: w à w+x Misake on neg: w à wx + Analysis Thm: Suppose daa is consisen wih some LTF w* x > 0, where w* =1 and = min x w* x (afer scaling so all x 1) Then # misakes 1/ 2 Proof: consider w w* and w Each misake increases w w* by a leas (w + x) w* = w w* + x w* w w* + Each misake increases w w by a mos 1 (w + x) (w + x) = w w + 2(w x) + x x w w + 1 So, in M misakes, M w w* w M 1/2 So, M 1/ 2 Wha if no perfec separaor? In his case, a misake could cause w w* o drop Impac: magniude of x w* in unis of Misakes(percepron) 1/ 2 + O(how much, in unis of, you would have o move he poins o all be correc by ) Proof: consider w w* and w Each misake increases w w* by a leas (w + x) w* = w w* + x w* w w* + Each misake increases w w by a mos 1 (w + x) (w + x) = w w + 2(w x) + x x w w + 1 So, in M misakes, M w w* w M 1/2 So, M 1/ 2 Wha if no perfec separaor? In his case, a misake could cause w w* o drop Impac: magniude of x w* in unis of Misakes(percepron) 1/ 2 + O(how much, in unis of, you would have o move he poins o all be correc by ) Noe ha was no par of he algorihm So, misakebound of Percepron min (above) Equivalenly, misake bound min w* w* 2 + O(hinge loss(w*)) 6