Plan for Today & Next time (B) Machine Learning Theory. 2-Player Zero-Sum games. Game Theory terminolgy. Minimax-optimal strategies

Transcription

1 5-859(B) Machine Learning Theory Learning and Game Theory Avrim Blum Plan for Today & Ne ime 2-player zero-sum games 2-player general-sum games Nash equilibria Correlaed equilibria Inernal/swap regre and connecion o correlaed equilibria Many-player games wih srucure: congesion games / eac poenial games Bes-response dynamics Price of anarchy, Price of sabiliy 2-Player Zero-Sum games Two players R and C. Zero-sum means ha wha s good for one is bad for he oher. Game defined by mari wih a row for each of R s opions and a column for each of C s opions. Mari ells who wins how much. an enry (,y) means: = payoff o row player, y = payoff o column player. Zero sum means ha y = -. E.g., penaly sho: goalie Game Theory erminolgy Rows and columns are called pure sraegies. Randomized algs called mied sraegies. Zero sum means ha game is purely compeiive. (,y) saisfies +y=0. (Game doesn have o be fair). goalie shooer shooer (,-) (0,0) (,-) (0,0) Minima-opimal sraegies Minima opimal sraegy is a (randomized) sraegy ha has he bes guaranee on is epeced gain, over choices of he opponen. [maimizes he minimum] I.e., he hing o play if your opponen knows you well. goalie Minima-opimal sraegies Can solve for minima-opimal sraegies using Linear programming No-regre sraegies will do nearly as well or beer. I.e., he hing o play if your opponen knows you well. goalie shooer shooer (,-) (0,0) (,-) (0,0)

2 Minima Theorem (von Neumann 928) Every 2-player zero-sum game has a unique value V. Minima opimal sraegy for R guaranees R s epeced gain a leas V. Minima opimal sraegy for C guaranees C s epeced loss a mos V. Eisence of no-regre sraegies gives one way of proving he heorem. Ineresing game o hink abou Graph G, source s, sink. Player A chooses pah P from s o. Player B chooses edge e in G. If e is in P, B wins. Else A wins. Wha is minima opimal sraegy for B, A? Noe ha can run RWM for B, and besresponse for A (shores pah alg on B s weighs) o ge ap-minima-opimal. General-sum games Now, o General-Sum games In general-sum games, can ge win-win and lose-lose siuaions. E.g., wha side of sidewalk o walk on? : you (,) (-,-) person walking owards you (-,-) (,) General-sum games In general-sum games, can ge win-win and lose-lose siuaions. E.g., which movie should we go o? : Bully Hunger games Nash Equilibrium A Nash Equilibrium is a sable pair of sraegies (could be randomized). Sable means ha neiher player has incenive o deviae on heir own. E.g., wha side of sidewalk o walk on : Bully (8,2) (0,0) (,) (-,-) Hunger games (0,0) (2,8) (-,-) (,) No longer a unique value o he game. NE are: boh lef, boh righ, or boh 50/50. 2

3 Uses Economiss use games and equilibria as models of ineracion. E.g., polluion / prisoner s dilemma: (imagine polluion conrols cos $4 bu improve everyone s environmen by $3) don pollue don pollue pollue pollue (2,2) (-,3) (3,-) (0,0) Need o add era incenives o ge good overall behavior. NE can do srange hings Braess parado: Road nework, raffic going from s o. ravel ime as funcion of fracion of raffic on a given edge. ravel ime =, indep of raffic s Fine. NE is 50/50. Travel ime =.5 ravel ime ()=. NE can do srange hings Braess parado: Road nework, raffic going from s o. ravel ime as funcion of fracion of raffic on a given edge. ravel ime =, indep of raffic s 0 Add new superhighway. NE: everyone uses zig-zag pah. Travel ime = 2. ravel ime ()=. Eisence of NE Nash (950) proved: any general-sum game mus have a leas one such equilibrium. Migh require mied sraegies. This also yields minima hm as a corollary. Pick some NE and le V = value o row player in ha equilibrium. Since i s a NE, neiher player can do beer even knowing he (randomized) sraegy heir opponen is playing. So, hey re each playing minima opimal. Eisence of NE in 2-player games Proof will be non-consrucive. Unlike case of zero-sum games, we do no know any polynomial-ime algorihm for finding Nash Equilibria in n n general-sum games. [known o be PPAD-hard ] Noaion: Assume an nn mari. Use (p,...,p n ) o denoe mied sraegy for row player, and (q,...,q n ) o denoe mied sraegy for column player. Proof We ll sar wih Brouwer s fied poin heorem. Le S be a compac conve region in R n and le f:s! S be a coninuous funcion. Then here mus eis 2 S such ha f()=. is called a fied poin of f. Simple case: S is he inerval [0,]. We will care abou: S = {(p,q): p,q are legal probabiliy disribuions on,...,n}. I.e., S = simple n simple n 3

4 Proof (con) S = {(p,q): p,q are mied sraegies}. Wan o define f(p,q) = (p,q ) such ha: f is coninuous. This means ha changing p or q a lile bi shouldn cause p or q o change a lo. Any fied poin of f is a Nash Equilibrium. Then Brouwer will imply eisence of NE. Try # Wha abou f(p,q) = (p,q ) where p is bes response o q, and q is bes response o p? Problem: no necessarily well-defined: E.g., penaly sho: if p = (0.5,0.5) hen q could be anyhing. (,-) (0,0) Try # Wha abou f(p,q) = (p,q ) where p is bes response o q, and q is bes response o p? Problem: also no coninuous: E.g., if p = (0.5, 0.49) hen q = (,0). If p = (0.49,0.5) hen q = (0,). Insead we will use... f(p,q) = (p,q ) such ha: q maimizes [(epeced gain wr p) - q-q 2 ] p maimizes [(epeced gain wr q) - p-p 2 ] (,-) (0,0) p p Noe: quadraic + linear = quadraic. Insead we will use... f(p,q) = (p,q ) such ha: q maimizes [(epeced gain wr p) - q-q 2 ] p maimizes [(epeced gain wr q) - p-p 2 ] p p Noe: quadraic + linear = quadraic. Insead we will use... f(p,q) = (p,q ) such ha: q maimizes [(epeced gain wr p) - q-q 2 ] p maimizes [(epeced gain wr q) - p-p 2 ] f is well-defined and coninuous since quadraic has unique maimum and small change o p,q only moves his a lile. Also fied poin = NE. (even if iny incenive o move, will move lile bi). So, ha s i! 4

5 Wha if all players minimize regre? Inernal regre and correlaed equilibria In zero-sum games, empirical frequencies quickly approaches minima opimal. In general-sum games, does behavior quickly (or a all) approach a Nash equilibrium? (afer all, a Nash Eq is eacly a se of disribuions ha are no-regre wr each oher). Well, unforunaely, no. A bad eample for general-sum games Augmened Shapley game from [Z04]: RPSF Firs 3 rows/cols are Shapley game (rock / paper / scissors bu if boh do same acion hen boh lose). 4 h acion play foosball has sligh negaive if oher player is sill doing r/p/s bu posiive if oher player does 4 h acion oo. NR algs will cycle among firs 3 and have no regre, bu do worse han only Nash Equilibrium of boh playing foosball. We didn really epec his o work given how hard NE can be o find Wha can we say? If algorihms minimize inernal or swap regre, hen empirical disribuion of play approaches correlaed equilibrium. Foser & Vohra, Har & Mas-Colell, Though doesn imply play is sabilizing. Wha are inernal regre and correlaed equilibria? More general forms of regre. bes eper or eernal regre: Given n sraegies. Compee wih bes of hem in hindsigh. 2. sleeping eper or regre wih ime-inervals : Given n sraegies, k properies. Le S i be se of days saisfying propery i (migh overlap). Wan o simulaneously achieve low regre over each S i. 3. inernal or swap regre: like (2), ecep ha S i = se of days in which we chose sraegy i. Inernal/swap-regre E.g., each day we pick one sock o buy shares in. Don wan o have regre of he form every ime I bough IBM, I should have bough Microsof insead. Formally, regre is wr opimal funcion f:{,,n}!{,,n} such ha every ime you played acion j, i plays f(j). Moivaion: connecion o correlaed equilibria. 5

6 Inernal/swap-regre Correlaed equilibrium Disribuion over enries in mari, such ha if a rused pary chooses one a random and ells you your par, you have no incenive o deviae. E.g., Shapley game. R P S R P S -,- -,,-,- -,- -, -,,- -,- Inernal/swap-regre If all paries run a low inernal/swap regre algorihm, hen empirical disribuion of play is an ap correlaed equilibrium. Correlaor chooses random ime 2 {,2,,T}. Tells each player o play he acion j hey played in ime (bu does no reveal value of ). Epeced incenive o deviae: j Pr(j)(Regre j) = swap-regre of algorihm So, his says ha correlaed equilibria are a naural hing o see in muli-agen sysems where individuals are opimizing for hemselves Inernal/swap-regre, cond Algorihms for achieving low regre of his form: Foser & Vohra, Har & Mas-Colell, Fudenberg & Levine. Can also conver any bes eper algorihm ino one achieving low swap regre. Unforunaely, ime o achieve low regre is linear in n raher han log(n). Inernal/swap-regre, cond Can conver any bes eper algorihm A ino one achieving low swap regre. Idea: Insaniae one copy A i responsible for epeced regre over imes we play i. Each ime sep, if we play p=(p,,p n ) and ge cos vecor c=(c,,c n ), hen A i ges cos-vecor p i c. If each A i proposed o play q i, so all ogeher we have mari Q, hen define p = pq. Allows us o view p i as prob we chose acion i or prob we chose algorihm A i. Congesion games Many muli-agen ineracions have srucure. One nice class: Congesion Games Always have a pure-sraegy equilibrium. Have a poenial funcion s.. whenever a player swiches, poenial drops by eacly ha player s improvemen. So, bes-response dynamics always gives an equilibrium. Le s sar wih an eample. Fair cos-sharing Fair cos-sharing: n players in weighed direced graph G. Player i wans o ge from s i o i, and hey share cos of edges hey use wih ohers. G 6

7 Good equilibria, Bad equilibria Fair cos-sharing: n players in weighed direced graph G. Player i wans o ge from s i o i, and hey share cos of edges hey use wih ohers. Good equilibria, Bad equilibria Fair cos-sharing: n players in weighed direced graph G. Player i wans o ge from s i o i, and hey share cos of edges hey use wih ohers. n s Good equilibrium: all use edge of cos. (cos /n per player) Bad equilibrium: all use edge of cos n. (cos per player) Cos(bad equilib) = n Cos(good equilib) Noe ha here, bad equilb is wha you d epec from naural dynamics (players enering one a ime, ec) cars s s n Shared ransi k n Price of Anarchy and Price of Sabiliy Price of Anarchy: raio of wors equilibrium o social opimum. (wors-case over games in class) We saw for cos-sharing PoA = (n). Also O(n). Price of Sabiliy: raio of bes equilibrium o social opimum. (wors-case over games in class) For cos-sharing, PoS = (log n). Eac Poenial funcion: Funcion s.. if player moves, poenial changes by eacly as much as cos of player who moved. Guaranees ha bes-response dynamics will reach Nash equilibrium Poenial funcions and PoS For cos-sharing, PoS = O(log n): Given sae S, le n e = # players on edge e. Cos(S) = Define poenial (S) = So, cos(s) (S) log(n) cos(s). Now consider bes-response dynamics saring from OPT. can only decrease. So, if could ell people o play OPT, and everyone wen along, hen BR dynamics would lead o good sae. cars s s n Shared ransi k n Congesion games more generally Game defined by n players and m resources. Each player i choses a se of resources (e.g., a pah) from collecion S i of allowable ses of resources (e.g., pahs from s i o i ). Cos of a resource j is a funcion f j (n j ) of he number n j of players using i. Cos incurred by player i is he sum, over all resources being used, of he cos of he resource. Generic poenial funcion: Bes-response dynamics may ake a long ime o reach equil, bu if gap beween and cos is small, can ge o ap-equilib fas. Curren/recen research direcions (esp in relaion o machine learning) How much effor needed o nudge simple bes-response dynamics from bad equilibrium o a good one? Are here naural dynamics ha can manage o reach good equilibria on heir own? Can one say anyhing ineresing abou combining eper advice ypes of problems where he qualiy of an eper depends on wha he oher players are doing? (In paricular, in comparison o bes equilibrium) 7