Announcements EWA with ɛ-exploration (recap) Lecture 20: EXP3 Algorithm. EECS598: Prediction and Learning: It s Only a Game Fall 2013.

Transcription

1 Lecture 0: EXP3 Algorthm 1 EECS598: Predcton and Learnng: It s Only a Game Fall 013 Prof. Jacob Abernethy Lecture 0: EXP3 Algorthm Scrbe: Zhhao Chen Announcements None 0.1 EWA wth ɛ-exploraton (recap) Ths algorthm generates an unbased estmator for l t. Strategy Wth probablty 1 ɛ Choose p t = EW A( l 1,..., l t 1 ) and let l t = 0 Wth probablty ɛ Choose p t = 1 n,..., 1 n, sample I t p t, and let l t = 0,...,0, n ɛ lt I t,0...,0 = n ɛ lt I t e It (e It s the I t th unt vector). Proof that ths algorthm works E[ l t ] = (1 ɛ) 0 + ɛ p t ( n ɛ lt e ) = l t e = lt Nave expected regret bound O ( ɛt + n T η ɛ + logn ) = O ( T 3 1 ) 4 n(logn) 4 η (wth tunng) Better expected regret bound ( O ɛt + nt η ɛ + logn ) η = O ( T ) 3 n (wth tunng) Queston from last lecture: Is there some algorthm that s better than EWA wth ɛ-exploraton? In partcular, s t possble to reduce the power on T to 1 n the expected regret?

2 Lecture 0: EXP3 Algorthm 0. EXP3 We clam that the EXP3 algorthm s a better algorthm than EWA wth ɛ-exploraton, and, n fact, has an expected regret bound of T nlogn. Let us begn by statng the algorthm Algorthm Let L t be the cumulatve losses up to perod t. for,...,t do Sample I t p t Observe li t t Set l t = Set L t = L t 1 + l t for,...,n do Set p t+1 = end for end for 0,...,0, lt I t,0...,0 e η Lt j=1 e η L t 0.. Comments on EXP3 The EXP3 algorthm looks very smlar to that of EWA wth ɛ-exploraton. Indeed n both cases, the chosen loss vectors are dvded by the probablty of obtanng that vector. The key dfference between the algorthms s that EXP3 does not drop observatons n any round (as opposed to EWA wth ɛ-exploraton droppng observatons wth a probablty of 1 ɛ). Intutvely, t seems that EXP3 mght be a pretty bad algorthm, gven that p t s could get exponentally small, meanng that we could be dvdng by a very small number n the algorthm. However, ths works out n the end, as we wll see n the analyss of the expected regret Analyss of the expected regret for EXP3 We analyze the regret of EXP3 by lookng at the potental functon Φ t = 1 η log e η L t 1 and takng the expected ncrease n potental n every perod. The ncrease n potental from perod t to t + 1 s Φ t+1 Φ t = 1 n η log e η L t n e η L = 1 n t 1 η log e η L t 1 n e η L t 1 To proceed, we need the followng lemma. η l t = 1 η log( E p t [ ]) e η l t

3 Lecture 0: EXP3 Algorthm 3 Lemma 0.1. For all x 0, e x 1 x + 1 x You can see ths by plottng the two graphs e x and 1 x + 1 x. The blue lne s e x and the red lne s 1 x + 1 x n the plot below Usng the lemma, we get Φ t+1 Φ t 1 η log (E p t = 1 η log (1 E p t 1 η E p t [ η l t + 1 η ( l t ) ] [1 η l t + 1 η ( l t ) ]) [η l t + 1 η ( l t ) ]) (because log(1 x) x) = p t l t η p t ( l t ) Takng the expectaton on both sdes, E[Φ t+1 Φ t ] E p t l t η p t ( l t ) = p t lt η E pi t l t t I t = p t l t η E (li t t ) = p t l t η p t l t ηn (l t )

4 Lecture 0: EXP3 Algorthm 4 Now, we sum the dfferences n potental to get E[Φ T +1 Φ 1 ] = E (Φ t+1 Φ t ) p t l t T ηn Furthermore, E[Φ T +1 Φ 1 ] E Combnng the two nequaltes, we get [ L T ( 1η )] logn = L T + 1 η logn Theorem 0.. E-regret T (EXP 3) = p t l t L T 1 η logn + T ηn E-regret T (EXP 3) T nlogn ( ) logn Proof. Tune η = T n n ( ). 0.3 Bandt problem n stochastc shortest path settng (n-class slde presentaton) Problem Want to fnd the shortest path from source to snk n a network wth stochastc costs. Only flow costs on the selected path are known at the end of a perod (bandt settng). Flows paths The number of paths s exponental, so we would want to work wth flow solutons nstead of arc solutons. It s easly seen that a path soluton can be reduced to a flow soluton. Flow polytope Path solutons Flow soluton Conversely, assumng that the decson maker can choose paths wth some dstrbuton, t can be shown that a flow soluton s equvalent to some convex combnaton of path solutons.

5 Lecture 0: EXP3 Algorthm 5 FTRL n the bandt settng In every round, x t = argmn x K Can we use estmated loss functons nstead?.e. x t = argmn x K t 1 f s x + λr(x) s=1 t 1 f s x + λr(x) As t turns out, yes! Due of the convexty of regret n f t s, s=1 E[Regret( f 1,..., f T )] Regret(E[ f 1 ],...,E[ f T ]) = Regret(f 1,...,f T ) It s suffcent to compete wth an unbased estmate of the loss functons (nstead of the actual loss functons).