Online Learning Applications

Transcription

1 Online Learning Applicaions Sepember 19, 2016 In he las lecure we saw he following guaranee for minimizing misakes wih Randomized Weighed Majoriy (RWM). Theorem 1 Le M be misakes of RWM and M i he misakes of exper i. Then, RWM saisfies E[M] (1 + ɛ)m i + log n/ɛ. Recall ha o prove his, we kep weighs w (i) on each exper i, and updaed hem as w +1 (i) = (1 m (i)η)w (i). Here m was he misake vecor (oucome). RWM played i wih probabiliy p (i) = w (i)/w, where W = i w (i). This gives W +1 = W (1 ηm p ). We hen used ha any exper ha makes k misakes mus have weigh w 0 (1 η) k. 1 The General Seing There is nohing special abou m being 0-1. So one can consider a more general game. Alernae Game: Online declares disribuion p a ime. Then, naure gives a loss vecor l a ime. Then Online incurs cos p l. We now wan o compare wih he loss of he bes exper in hindsigh. We ge his heorem. Theorem 2 General maser heorem (hm 2.1 in AHK). If losses l [ 1, 1] n and L = p l. For each exper i, L L i + log n + ɛ l (i) ɛ Le us noe a few quick corollaries. Corollary 3 If losses l [0, 1] n, can wrie above as relaive guaranee. L (1 + ɛ)l i + (log n)/ɛ If have gains g [0, 1] n, and wan o maximize i, we ge When losses or gains are in [0, ρ] n, ge G (1 ɛ)g i log n/ɛ G (1 ɛ)g i ρ log n/ɛ This follows by jus feeding g /ρ o he RWM algorihm (gain version). Of course, an analogous guaranee holds for loss. 1

2 Remark: Bw, we noe ha when we run RWM for he gain version, we updae weigh as w +1 (i) = (1 + g (i)η)w (i) Tha is, you wan o increase probabiliy of he expers geing a high gain. If losses or gains have arbirary signs, he following Corollary 4 If l [ 1, 1] n. By balancing he erms, ge average per ime sep regre of 1 log n (L min L i ) O( T i T ). So, o ge ime average regre +ɛ, need log n/ɛ 2 ime seps. If l [ ρ, ρ] n, we need O(ρ 2 log n/ɛ 2 ) ime o ge average regre of ɛ. Analogous bounds holds for gain. 2 Zero sum games Payoff marix M. Row player wans o maximize his gain. Column player wans o minimize. Who should go firs? If Row player plays row x firs, Column player picks y (in second sep) o minimize payoff M(x, y). Le us hink of x and y as indicaor (or more generally disribuions) on rows and columns. Then payoff can be wrien as x T My. Le C(x) = min(x T M)y = min(x T M)e j y denoe he column sraegy (in second sep) given row plays x firs. Here j indexs a column. Similarly, if Row player has o go firs, he mus jus play x ha achieves max x C(x). If column player goes firs and chooses y, row player will pick row i wih maximum e T i My. Call his sraegy for row player R(y) = max x T My = e T x i My The following says (as we would expec) ha he row player never wans o go firs. Fac 1 For any y and x. R(y) = max x x T My x T My min x T My = C(x). y In fac, no maer wha y column player plays in is firs sep, R(y) C(x ), where x is he bes hing row player could have played if we had o go firs. Now, he amazing his is ha if you allow mixed sraegies (randomized over heir choices), he gap disappears. Row player does no care wheher he has o go firs or no. Moreover, he can even publish his sraegy and he column player can opimize all he wans, bu i sill canno gain any advanage! Theorem 5 If you allow mixed (randomized) sraegies, and he payoff marix is finie. I does no maer who goes firs. In oher words, here exis x and ỹ, so ha R(ỹ) = C( x) j 2

3 Proof: By he fac above, R(ỹ) C( x) for any ỹ and x. So, we jus need o prove he oher direcion. Le us scale all enries of payoff marix o be in [ 1, 1]. Will show ha for every ɛ > 0, can find some x and ỹ saisfying he claim of he heorem. Leing ɛ go o 0 will imply he resul. Le us use muliplicaive updae. Row and Column will play a game over rounds. In round, Row player chooses disribuion x, hen Column chooses y o penalize Row mos. So he gain g of row player a ime is g = x T My = min x T y My If row player does MWU, by he corollary for gain version on average we ge ha in T = O((log n)/ɛ 2 ) seps For every i, 1 T x My 1 T max e T i ( i My ) ɛ (1) Le x := (1/T ) x and ỹ = My. To finish proof, noe ha he rhs in (1) is R(ỹ) ɛ. Moreover, if we fix any column j for he column player, we also have ha (1/T ) x My xme j because he column player hurs Row he mos when i choose y. So he lhs in (1) is a mos min xme j = C( x) j This gives C( x) R(ỹ) ɛ, and we are done. 3 Solving LPs approximaely We discuss a MWU based framework for solving LPs approximaely, which can somes gives surprisingly good resuls. We firs discuss he Plokin, Shmoys, Tardos framework for Packing/covering LPs, and laer skech wha is needed for general LP. Consider a packing LP (wih non-negaive A), and where P is some simple polyope. A j x 1 max cx j = 1,..., m x P A useful example o keep in mind is s- maxflow. Insead of he flow conservaion consrains based LP, le us consider a differen equivalen one (jus for ease of descripion). Each edge has a capaciy of 1. We have variable f p for each s- pah p. So he polyope P is jus f p 0. The marix A has a consrains for each edge e, ha f e = (Af) e 1, where f e denoe he flow on edge e. Le us also assume (by binary search) ha he opimum objecive value is known. The resul is he following. 3

4 Theorem 6 Suppose ha given any probabiliy disribuion on rows, you have a fas oracle o find a feasible soluion o cx >= F, p T Ax 1 x P Then one can find a soluion ha saisfies all consrains o wihin 1+ɛ in ime O(ρ log m/ɛ 2 ). Here ρ is he widh, defined as maximum violaion in any consrain for any soluion can be reurned by he oracle. We also remark ha he above resul also holds (easily) wih an approximae oracle p T Ax 1+ɛ. Exercise 7 Show ha he oracle for max-flow example is he shores pah problem. Also, show ha he widh is m. For a general packing LP, he widh can also depend on he coefficiens, bu one can use some ricks o ge rid of his (we will see such a rick laer). Proof: We will use MWU. A key idea is ha he row player will mainain a disribuion on rows, bu use he oracle o learn he righ flow over ime. A ime, he row player gives a disribuion p. The oracle reurns a soluion x () wih p T Ax () 1. Noe ha if no such soluion exiss, his gives an infeasibiliy cerificae. So le us assume his is always possible. Given x (), we feed he gain vecor g = Ax (). By definiion of widh g [0, ρ] m. By he corollary abou posiive gains, we know ha for each j [m], which is p g (1 ɛ) p Ax () (1 ɛ) e j g ρ log m ɛ (Ax () ) j ρ log m ɛ Noe ha he lef hand side is a mos T by he propery of x (). Consider he soluion x = ( x )/T. Then, choosing T = (ρ log m)/ɛ 2 and dividing above by T gives 1 (1 ɛ)(ax () ) j ɛ for every j. So he heorem is proved. The same proof works for covering LPs. For general LPs, he gain/loss vecors can be boh negaive or posiive. So we acually ge a dependence on ρ in he running ime insead of ρ above. Exercise 8 I is insrucive o open he black box and see wha MWU is acually doing. If a consrain/row is violaed oo much, i will ge penalized a lo. So i canno be doing consisenly much worse han he average. 4

5 Mea principle: The following is an ofen appicable mea-principle o remember for applicaions of MWU. If here are several requiremens o be saisfied, bu you can do well for any averaging of he requiremens hen MWU can be used well o find a soluion o almos saisfies all requiremens. 4 Approx Max flow in m 4/3 The above approach gives a roughly (ignoring polylog facors) O(m 2 ) ime algorihm, as he widh is O(m). We will show how o reduce he widh o m 1/2, using elecrical neworks. Laer we will use some simple properies of flows and elecrical neworks o reduce his furher o m 1/3. We remark ha now almos linear ime algorihms are known for his, bu he above is sill a good applicaion of MWU. As previously, we consider he case when c e = 1. I suffices o solve his version, via sandard preprocessing argumens. max F f e 1 Le maxflow be F, which we assume is known. I will be useful o work wih weighs w e direcly insead of probabiliies p e (slighly opening he MWU black box). In his above framework his means ha given w e, we need he oracle o reurn a flow wih value F saisfying w e f e w e. e Recall ha i also suffices o have w e f e (1 + ɛ) e w e. 4.1 Basic elerical neworks Given a graph wih resiances r e on edges, here are wo hings we need o know abou elecrical flows. Fac 2 If we pu volage Φ s = 1 and Φ = 0, hen naure s law is o choose Φ(v) s on remaining nodes o minimize he energy E(Φ) = (Φ(u) Φ(v) 2 )/r e e:e=(u,v) If E if he energy, we call R eff = E he effecive resisance of (s, ). Exercise 9 Take derivaive of energy wr Φ(v) and see ha you ge flow (curren) conservaion. Fac 3 Anoher useful (in fac dual) view is ha if we injec one uni of curren ino s and remove i from, hen naure s law is o choose currens so ha fe 2 r e e:e=(u,v) In fac, his induces Φ s so ha f u,v = (Φ(u) Φ(v))/r e. 5

6 4.2 The Widh m 1/2 oracle The oracle is implemened as follows. Given weighs w e, se resisance r e = w e + ɛw/m, where W = e w e. Consider he leas energy flow, obained by injeced F unis of curren from s o. Le f e be he flow on he edges. Claim 10 Le E be he opimum energy, E (1 + ɛ)w. Proof: Consider a feasible flow of value F saisfying he capaciies. This has energy fe 2 r e r e (1 + ɛ)w e e Claim 11 max e f e (m/ɛ) 1/2. Proof: As he oal energy is a mos W (1 + ɛ), he energy on any edge saisfies. fe 2 r e = fe 2 (w e + ɛw/m) (1 + ɛ)w. So, fe 2 m(1 + ɛ)/ɛ. As previously, we will feed f e as he gain vecors o MWU rouine. So, we wan o bounded he average gain e f ew e Lemma 12 For f e above, we have f e w e (1 + ɛ)w Proof: By Cauchy Schwarz, and he firs claim above ( e f e w e ) 2 ( e f 2 e w e )( e w e ) (1 + ɛ)w 2 Tha s i. This gives good oracle. However, m 3/2 running ime is no oo impressive, as even exac algorihms are known in his ime. 4.3 Improving he widh Though experimen: Widh as defined is a raher brile noion. Wha if only a couple of edges in he flow reurned by he oracle have congesion m 1/2. If we could hrow away hese edges from he graph, he max flow could only decrease by a mos 2. In fac we can afford o hrow away ɛf fracion of he edges. 4.4 Algorihm Le ρ be a parameer ha we will opimize laer. 1. Sar wih w 1 e = 1 iniially. 2. Feed resisances r e = w e + ɛw /m. The oracle reurns flow f e. If f e > ρ, delee e and repea his sep. 3. Updae rule he weighs now as w +1 e = w e(1 + f e/ρ) 6

7 We will ensure ha ρ is chosen in such a way ha he number of edges deleed, call i k, is never more han ɛf. Observaion 1 The above ensures ha here is a sill a maxflow of value of (1 ɛ)f in he graph. So one can sill injec F unis of flow, so ha each edge has congesion a mos 1/(1 ɛ) 1 + ɛ. Running Time of algorihm will be roughly ρm log m/ɛ 2 + km where k is he number edges deleed, because you need o call oracle abou (ρ log m/ɛ 2 ) + k imes. 4.5 Analysis We will choose k m 1/3, as we shall see. Observaion 2 If F k log m, hen we can jus solve he problem using Ford Fulkerson. So le us assume F k. By observaion 1 we can afford o delee k m 1/3 edges and ensure ha e r ef 2 e (1 + 3ɛ)W. By argumen in lemma 12, his ensures Claim 13 w e f e W (1 + 4ɛ). As he weigh increases as W ( + 1) = e w e ()(1 + f e ) W ()(1 + 1/ρ) ρ wih muliplicaive updae, he above implies ha afer l ieraions of weigh updaes, he oal weigh is sill m(1 + 1/ρ) l. If he widh was always less han ρ, we need abou l = (ρ log m)/ɛ 2 ieraions. Undersanding edge deleions: Bu wha happens when we delee an edge. We will find anoher parameer, effecive resisance, ha rises oo fas. Evenually, since he oal weigh (which is he sum of resisances of all he edges) is growing prey slowly, we will ge a bound on he number of deleions by arguing ha he resisance could no have risen so much. If f e ρ, his means ha f 2 e (w e + ɛw/m) ρ 2 R e ρ 2 ɛw/m On he oher hand, he energy used overall is a mos W (1 + 3ɛ). So he energy on e is a leas abou β = ρ 2 ɛ/m fracion of oal energy. The following lemma is key. Lemma If we increase resisance on edges, effecive resisance can only go up. 2. If we delee an edge e wih β fracion of energy consumed, he effecive resisance can go up by a leas 1/(1 β) facor. 7

8 Proof: Pu one vol on s, hen 1/R eff = min v (i,j) e Φ(i) Φ(j) 2 If we increase denominaor and keep he same volages, he expression can only decrease. Now he opimum choice of volages wih hese new resisances can only lower his furher. For he second par, we have As e conribues a leas β fracion of energy, r(ij) Φ(i) Φ(j) 2 1/R eff = min v r(ij) (i,j) (1 β)/r eff (i,j) E\{e} v(i) v(j) 2 Bu he rhs is he energy in he nework using old volages and his edge removed. Opimum choice of volages can only reduce he rhs furher, which he new inverse effecive resisance 1/R eff. So, 1/R eff (1 β)/r eff r(ij) which implies ha R eff R eff /(1 β). So, afer k edge deleions R eff (T ) (1 + β) k R eff0 (1 + β) k (1/m) Effecive resisance iniially mus be a leas 1/m as each edge has 1 resisance. Bu by he rae of rise of weighs, and claim?? above, he oal energy is a mos W 0 (1 + ɛ/ρ) ρ log m/ɛ2 exp(log m/ɛ) As injeced curren F is a leas 1 (infac a leas m 4/3 ), (1 + β) k (1/m) R eff W (T ) exp(log m/ɛ) To opimize running ime se k = ρ. Then we ge (ignoring log facors) ha So, ρ m 1/3 as he resul follows. exp(ρ 3 /m) = exp(log m/ɛ). 8