Subjective Equilibria in Interactive POMDPs: Theory and Computational Limitations

Transcription

1 Subectve Equlbra n Interactve POMDPs: Theory and Computatonal Lmtatons Prashant Dosh and Potr Gmytrasewcz Dept. of Computer Scence, Unversty of Illnos at Chcago, IL {pdosh,potr}@cs.uc.edu Abstract We analyze the asymptotc behavor of agents engaged n a nfnte horzon partally observable stochastc game formalzed by the nteractve POMDP framework. We show that when agents ntal belefs satsfy a truth compatblty condton, ther behavor converges to a subectve ɛ-equlbrum n a fnte tme, and subectve equlbrum n the lmt. Imposng an addtonal assumpton of mutual sngularty on agents ntal belefs makes ther behavor converge to Nash equlbrum. Whle theoretcally sound, the equlbratng process s dffcult to demonstrate computatonally because of the dffculty n comng up wth ntal belefs that satsfy the truth compatblty condton. 1 Introducton We analyze the nteractons takng place between agents partcpatng n an nfnte horzon partally observable stochastc game formalzed wthn the framework of nteractve POMDPs (I-POMDPs) [Gmytrasewcz and Dosh, 2005, Gmytrasewcz and Dosh, 2004]. I-POMDPs represent and solve a partally observable stochastc game (POSG) from the perspectve of an agent playng the game. Ths approach, also called the decson-theoretc approach to game theory [Kadane and Larkey, 1982], dffers from the obectve representaton of POSGs as outlned n [Hansen et al., 2004]. We consder the settng n whch an agent may be unaware of the other agents behavoral strateges, t s uncertan about ther observatons, and t may be unable to perfectly observe the other agents actons. In accordance to Bayesan decson theory, the agent mantans and updates ts belef about the physcal state as well as the strateges of the other agents, and t ts decsons are best responses to ts belefs. Under the assumpton of compatblty of agents pror belefs about future observatons wth the true dstrbuton nduced by the actual strateges of all agents, we show that for agents modeled wthn the I-POMDP framework, the followng propertes hold: () the agents belefs about the future observaton paths of the game concde n the lmt wth the true dstrbuton over the future, and () the agents belefs about the opponents strateges, and hence ther own strateges (whch are best responses to ther belefs), do not change n the lmt. Strateges wth these propertes are sad to be n subectve equlbrum, whch s stable wth respect to learnng and optmzaton. In pror work, [Kala and Lehrer, 1993a, Kala and Lehrer, 1993b] have shown that the strateges of agents engaged n nfntely repeated games wth dscounted payoffs, who are unaware of others strateges, and under the assumptons of perfect observablty of others actons (perfect montorng) and truth compatblty of pror belefs wll converge to a subectve equlbrum.

2 Hahn [Hahn, 1973] ntroduced the concept of a conectural equlbrum n economes where the sgnals generated by the economy do not cause changes n the agents theores, nor do they nduce changes n the agents polces. Fudenberg and Levne [Fudenberg and Levne, 1993] consder a general model of fntely repeated extensve form games wheren strateges of opponents may be correlated (unlke [Kala and Lehrer, 1993a] where strateges are assumed ndependent), and show that behavor of agents that mantan belefs and optmze accordng to ther belefs, converges to a self-confrmng equlbrum. There s a strong lnk between the subectve equlbrum and ts obectve counterpart the Nash equlbrum. Specfcally, under the assumpton of perfect montorng, both [Kala and Lehrer, 1993a] and [Fudenberg and Levne, 1993] show that the strategy profle n subectve and self-confrmng equlbrum nduce a dstrbuton over the future acton paths that concdes wth the dstrbuton nduced by a set of strateges n Nash equlbrum. Of course, ths does not mply that strateges n subectve equlbrum are also n Nash equlbrum; however, the converse s always true. Work of a smlar ven s reported n [Jordan, 1995]. It assumes agents have a common pror over the possble types of agents engaged n a repeated game, and shows that the sequence of Bayesan-Nash equlbrum belefs of agents converges to a Nash equlbrum. The results contaned n ths paper complement pror results. Specfcally, we show the asymptotc exstence of subectve equlbrum n a more general and realstc multagent settng, one n whch the assumptons of perfect observablty of state and others actons have been relaxed. Further, we address the research problem posed n [Kala and Lehrer, 1993a] regardng the exstence of subectve equlbrum n POSGs. We also draw a parallel wth works n multagent learnng [Hu and Wellman, 1998, Bowlng and Veloso, 2002] that show convergence of play to equlbrum. However, our results dffer n that we assume that the state and others actons are partally observable, and the plan s computed offlne usng a gven model of the envronment. Fnally, we comment on the dffcultes n achevng subectve equlbra when a computatonal perspectve s adopted. The rest of ths paper s structured n the followng manner. In the next secton, we brefly revew the Interactve POMDP framework. We focus on the key steps of belef update, and polcy computaton. In Secton 3, we ntroduce the concept of a subectve equlbrum and theoretcally prove that the strategy profle of agents playng a POSG modeled usng an I-POMDP, n the lmt, s n subectve equlbrum. In Secton 4, we remark on the computatonal nfeasblty of arrvng at ths equlbrum. Fnally, we conclude ths paper wth a dscusson of our results and open research ssues n Secton 5. 2 Overvew of Interactve POMDPs Interactve POMDPs [Gmytrasewcz and Dosh, 2005, Gmytrasewcz and Dosh, 2004] generalze POMDPs to account for presence of other agents n the envronment. They do ths by ncludng models of other agents n the state space. Models of other agents, analogous to types n game theory, encompass all prvate nformaton nfluencng ther behavor. For smplcty of presentaton let us consder an agent,, that s nteractng wth one other agent,. The formalsm easly generalzes to a larger number of agents. I-POMDP An nteractve POMDP of agent, I-POMDP, s: I-POMDP = IS, A, T, Ω, O, R where: IS s a set of nteractve states defned as IS = S O M, where S s the set of states of the 2

3 physcal envronment, and O M s the set of pars consstng of a possble observaton functon and a model of agent. Each model, m M, s a par m = h, π, where π : H (A ) s s polcy tree (strategy), assumed computable 1, whch maps possble hstores of s observatons to dstrbutons over ts actons. 2 h s an element of H. 3 O O, also computable, specfes the way n whch the envronment s supplyng the agent wth ts nput. A = A A s the set of ont moves of all agents T s a transton functon, T : S A S [0, 1] whch descrbes results of agents actons on the physcal state Ω s the set of agent s observatons O s an observaton functon O : S A Ω [0, 1] R s defned as R : S A R. We allow the agent to have preferences over physcal states and actons of all agents. The task of computng a soluton for an I-POMDP, smlar to that of a POMDP, can be decomposed nto two steps: (1)Belef update durng whch the agent updates ts belef to reflect newly avalable nformaton. (2)Polcy computaton durng whch the agent computes the optmal acton(s) to perform from each belef state. 2.1 Bayesan Belef Update There are two dfferences that complcate state estmaton n multagent settngs, when compared to sngle agent ones. Frst, snce the state of the physcal envronment depends on the actons performed by both agents, the predcton of how the physcal state changes has to be made based on the predcted actons of the other agent. The probabltes of other s actons are obtaned based on ther models. Thus, as opposed to the lterature on learnng n repeated games, we do not assume that actons are fully observable by other agents. Rather, agents can attempt to nfer what actons other agents have performed by sensng ther results on the envronment. Second, changes n the models of other agents have to be ncluded n the update. Specfcally, update of the other agent s models due to ts new observaton has to be ncluded. In other words, the agent has to update ts belefs based on what t antcpates that the other agent observes and how t updates. Consequently, an agent s belefs record what t thnks about how the other agent wll behave as t learns. For smplcty we decompose the I-POMDP belef update nto two steps: Predcton When an agent, say, wth a prevous belef, b t 1 f the other agent performs ts acton a t 1 P r(s t a t 1, a t 1, b t 1 ) =, the predcted belef state s, b t 1 IS t 1 :(f,o ) t 1 =(f,o ) t ω t O (s t, a t 1, performs a control acton a t 1 (s) P r(a t 1, a t 1, ω t)δ(append(ht 1, ω t) ht ) and m )T (s t 1, a t 1, a t 1, s t ) where δ s the Kronecker delta functon, and APPEND(, ) returns a strng n whch the second argument s appended to the frst. Correcton When agent perceves an observaton, ω t, the ntermedate belef state, 1 We assume computablty n the Turng machne sense,.e. strateges are partal recursve functons. 2 Note that f A 2, then the space of polcy trees s uncountable; however, by assumng π to be computable, we restrct the space to be countable. 3 In [Gmytrasewcz and Dosh, 2005, Gmytrasewcz and Dosh, 2004], we replace O M wth a specal class of models called ntentonal models. These models ascrbe belefs, preferences, and ratonalty to other agents. We do not ntroduce those models here for the purpose of generalty. 3

4 P r( a t 1, a t 1, b t 1 ), s corrected accordng to, P r(s t ω t, a t 1, b t 1 ) = α a t 1 O (s t, a t 1, a t 1, ω t )P r(s t a t 1, a t 1, b t 1 ) where α s the normalzng constant. The update extends to more than two agents n a straghtforward way. We represent possble correlatons between actons of other agents as dependences between ther models, whch are expressed n s belefs. 2.2 Polcy Computaton Each belef state n I-POMDP has an assocated value reflectng the maxmum payoff the agent can expect n ths belef state: { V (b ) = max ER (s, a )b (s) + γ } P r(ω a, b )V (SE(b, a, ω )) (1) a A s ω Ω where, ER (s, a ) = a R (s, a, a )P r(a m ) (snce s = (s, m )). Eq. 1 s a bass for value teraton n I-POMDPs. As shown n [Gmytrasewcz and Dosh, 2005], the value teraton converges n the lmt. Agent s optmal acton, a, for the case of nfnte horzon crteron wth dscountng, s an element of the set of optmal actons for the belef state, OP T (b ), defned as: { OP T (b ) = argmax ER (s, a )b (s) + γ } P r(ω a, b )V (SE(b, a, ω )) (2) a A s ω Ω Equaton 2 enables the computaton of a polcy tree, π, for each belef b. The polcy, π, gves s best response long term strategy for the belef. 3 Subectve Equlbrum n I-POMDPs In Secton 2, we revewed a framework for two-agent POSG n whch each agent computes the dscounted nfnte horzon strategy whch s the subectve best response of the agent to ts belef. Durng each step of game play, the agent startng wth a pror belef revses t n lght of the new nformaton usng the Bayesan belef update process outlned n Secton 2.1, and computes the optmal strategy gven ts belefs. The latter step s equvalent to usng ts observaton hstory to ndex nto ts polcy tree (computed offlne usng the process gven n Secton 2.2) 4 to compute the best response future strategy. 3.1 Background: Stochastc Processes, Martngales, and Bayesan Learnng A stochastc process s a sequence of random varables, {X t }, t = 0, 1,..., whose values are realzed one at a tme. Well-known examples of stochastc processes are Markov chans, as well as sequences of belefs updated usng the Bayesan update. Bayesan learnng turns out to exhbt an addtonal property that classfes t as a specal type of stochastc process, called a Martngale. 4 In the nfnte horzon case, convergence of value teraton allows us to convenently represent the polcy tree as a fnte state machne 4

5 A Martngale s a stochastc process that, for any observaton hstory up to tme t, h t, exhbts the property that for all l t: E[X l h t ] = X t. Consequently, for all future tme ponts l t the expected change, E[X l X t h t ] = 0. A sequence of an agent s belefs updated usng Bayesan learnng s known to be a Martngale. Intutvely, ths means that the agent s current estmate of the state s equal to what the agent expects ts future estmates of the state wll be, based on ts current observaton hstory. Because the Martngale property of Bayesan learnng s central to our results, we sketch a formal proof below. Let an agent s ntal belef over some state space Ξ be X 0 = P r(ξ). The agent receves some observaton, ω, n the future accordng to a dstrbuton φ that depends on θ. Let the future revsed belef be X 1 = P r(ξ ω). By Bayes theorem, P r(ξ ω) = φ(ω ξ)p r(ξ)/p r(ω). We wll show that E[P r(ξ ω)] = P r(ξ): E[P r(ξ ω)] = ω P r(ξ ω)p r(ω) = φ(ω ξ)p r(ξ) ω P r(ω) P r(ω) = ω φ(ω ξ)p r(ξ) = P r(ξ) ω φ(ω ξ) = P r(ξ) = X 0 The above result extends mmedately to observaton hstores of any length t. Formally, E[X t+1 h t ] = X t, and from the law of condtonal expectatons, E[X l h t ] = X t, l t. Therefore the belefs satsfy the Martngale property. All Martngales share the followng convergence property: Theorem 1 (Martngale Convergence Theorem ( 4 of Chapter 7 n [Doob, 1953]). If {X t }, t = 0, 1,... s a Martngale wth E[ X t 2 ] < U < for some U and all t, then the sequence of random varables, {X t } converges wth probablty 1 to some X n mean-square. 3.2 Subectve Equlbrum We nvestgate the asymptotc behavor of agents playng an nfnte horzon POSG, n whch each agent learns and optmzes. Specfcally, each agent starts wth a pror belef whch s revsed on performng an acton and recept of sensory nformaton, followed by computng the strategy whch optmzes ts belefs. In the context of I-POMDPs, each agent uses ts pror belefs to ndex nto ts polcy (computed offlne usng Equatons 1 and 2) resultng n the polcy tree that wll form ts behavor strategy. Sequental behavor of agents n a POSG may be represented usng ther observaton hstores. For an agent, say, let ω t be ts observaton at tme step t. Let ωt = [ω t, ωt ]. An observaton hstory of the game s a sequence, h = {ω t }, t = 1, 2,.... The set of all hstores s, H = t=1ω t where Ω t = Π t 1(Ω Ω ). The set of observaton hstores upto tme t s, H t = Π t 1(Ω Ω ), and the set of future observaton paths from tme t onwards s, H t = Π t (Ω Ω ). Example: We use the multagent tger problem descrbed n [Gmytrasewcz and Dosh, 2005] as a runnng example throughout ths paper. Brefly, the game conssts of two doors, behnd one s a tger and behnd the other s some gold, and two agents and. The agents are unaware of where the tger s (TL or TR), and each can ether open any one of two doors, or lsten(ol,or or L). A tger emts a growl perodcally, whch reveals ts poston behnd a door (GL or GR) but only wth some certanty. Addtonally, each agent can also hear a creak wth some certanty, f the other agent opens a door (CL,CR, or S). We wll assume that nether agent can perceve other s observatons 5

6 nor actons. The game s non-cooperatve snce ether or may open a door, thereby resetng the locaton of the tger, and renderng any nformaton collected by the other agent about the tger s locaton useless to t. Example hstores n the multagent tger problem are shown n Fg. 1. <GL,S> <GL,S> [TL,L,L] <GR,CR><GR,CR> <GL,S><GL,CL> <GR,CR><GR,CL> [TL,L,L] [TL,L,L] [TL,L,L] [TL,L,L] <GL,S><GL,S> <GR,CR><GR,CR> <GL,S><GL,S> <GR,CR><GR,CR> [TL,OR,OR] [TL,L,L] [TL,L,L] [TL,L,L] Fgure 1: Jont observaton hstores n the nfnte horzon multagent tger problem. The nodes represent the state of the game and play of agents, whle the edges are labelled wth the possble observatons. Ths example starts wth the tger on the left and each agent lstenng. Each agent may receve one of sx observatons (labels on the arrows), and performs an acton that optmzes ts resultng belef. In the I-POMDP framework, each agent s belef over the physcal state and others canddate models, together wth the agent s perfect nformaton regardng ts own model, nduces a predctve probablty dstrbuton over the future observaton paths. These dstrbutons play a crtcal role n our analyss; we represent them mathematcally usng a collecton of probablty measures, {µ k }, k = 0,, defned over the space M H, where M = M M and H s as defned prevously, such that, 1. µ 0 s the obectve true dstrbuton over models of each agent and the hstores, 2. pro Mk µ k = pro Mk µ 0 = δ mk k =, Here, condton 2 states that each agent knows ts own model (δ mk s the Kronecker delta functon). Addtonally, pro H µ 0 gves the true dstrbuton over the hstores as nduced by the ntal strategy profle, and pro H µ k ( b 0 k ) for k =, gves the predctve probablty dstrbuton for each agent over the hstores at the start of the game. 5 If the actual sequence of observatons n the game does not proceed along a hstory that s assgned some postve predctve probablty by an agent, then the agent s observatons would contradct ts belefs and the Bayesan update would not be possble. Clearly, t s desrable for each agent s ntal belef to assgn nonzero probablty to each possble observaton hstory; ths s called the truth compatblty condton. To formalze ths condton we need a noton of absolute contnuty of two probablty measures. Defnton 1 (Absolute Contnuty). A probablty measure p 1 s absolutely contnuous wth p 2, denoted as p 1 p 2, f p 2 (E) = 0 mples p 1 (E) = 0, for any measurable set E. Condton 1 (Absolute Contnuty Condton (ACC)). ACC holds for any agent k =, f pro H µ 0 pro H µ k ( b 0 k ). 5 Followng [Nyarko, 1997, Jordan, 1995] the uncondtonal measure µ k may be seen as a pror before an agent knows ts own model, and µ k along wth the condtons as an nterm pror once an agent knows ts own model. 6

7 Condton 1 states that the probablty dstrbuton nduced by an agent s ntal belef on observaton hstores should not rule out postve probablty events accordng to the real probablty dstrbuton on the hstores. A sure way to satsfy ACC s for each agent s ntal belef to have a gran of truth assgn a non-zero probablty to the true model of the other agent. Snce an agent has no way of knowng the true model of ts opponent from beforehand, t must assgn a non-zero probablty to each canddate model of the other agent. Truth compatble belefs of an agent that performs Bayesan learnng tend to converge n the lmt to the opponent model(s) that most lkely generates the observatons of the agent. In the context of the I-POMDP framework, an agent s belef updated usng the process outlned n Secton 2.1, wll converge n the lmt. Formally, Theorem 2 (Bayesan Learnng n I-POMDPs). For an agent n the I-POMDP framework, f ts ntal belef satsfes the ACC, ts posteror belefs wll converge wth probablty 1. Proof. As we mentoned before, Bayesan learnng s a Martngale. In Secton 3.1, set Ξ = IS, and φ = O. Notng that the I-POMDP belef update s Bayesan, ts Martngale property follows from applyng the proof approprately. In order to apply Theorem 1 to the I-POMDP belef update, set X t = P r(s t h t ) where ht s agent s observaton hstory upto tme t. We must frst show that E[ X t 2 ] s bounded. E[ b t 2 ] = ( A Ω ) t k=1 b t = b k 2 P r( b k ) = ( A Ω ) t b k=1 IS k t (s)2 P r(b k ) (L 2 norm) ( A Ω ) t k=1 1 P r( b k ) ( p 2 1) = 1 Theorem 2 now follows from a straghtforward applcaton of Theorem 1. The above result does not mply that an agent s belef converges to the true model of the other agent. Ths s due to the possble presence of observatonally equvalent models of the other agent. For example, for agent, all models of that nduce dentcal dstrbutons over all possble future observaton paths are sad to be observatonally equvalent. When a partcular observaton hstory obtans, agent s unable to dstngush between the observatonally equvalent models of. In other words, observatonally equvalent models generate dstnct behavors for hstores whch are never observed. Example: For an example of observatonally equvalent models, consder a verson of the multagent tger game n whch the tger perssts behnd ts orgnal door once any door has been opened. Addtonally, has superor observaton capabltes compared to, and each agent s able to perfectly observe other s actons but observes the growls mperfectly. Let s utlty dctate that t wll not open any doors untl t s 100% certan that the tger s behnd the opposte door. The correspondng strategy for s to lsten for an nfnte number of tme steps, and then open the door. Suppose that as a best response to ts belef, were to adopt a strategy n whch t would lsten for an nfnte number of steps, but f at any tme opened a door, t would also do so at the next tme step and then contnue openng the same door. The true dstrbuton assgns a probablty 1 to the hstores {[ GL GR, S, GL GR, S ]} 1. Instead of the above mentoned strategy f were to adopt a follow-the-leader strategy,.e. performs the acton whch dd n the prevous tme step, then the true dstrbuton would agan assgn probablty 1 to the prevously mentoned hstores. The two dfferent strateges of turn out to be observatonally equvalent for. An mmedate consequence of the convergence of Bayesan learnng s that the predctve dstrbuton over the future observaton paths nduced by each agent s belef after a fnte sequence 7

8 of observatons h t k, pro H t µ k ( b 0 k, ht k ), k =, becomes arbtrary close to the true dstrbuton, pro Ht µ 0 ( h t ), for a fnte t, and converges unformly n the lmt. Ths s an mportant result, because t establshes that no matter what the ntal belefs of the agents compatble, the agents opnons (about the future) wll merge and correctly predct the true future n the lmt. Ths result was frst noted n [Blackwell and Dubns, 1962]; we present the theorem below and refer the reader to the paper for ts proof. Theorem 3 ( [Blackwell and Dubns, 1962]). Suppose that P s a predctve probablty on X, and Q s absolutely contnuous w.r.t. P. Then for each condtonal dstrbuton P t (x 1,..., x t ) of the future gven the past w.r.t. P, there exsts a condtonal dstrbuton Q t (x 1,..., x t ) of the future gven the past w.r.t. Q such that, P t (x 1,..., x t ) Q t (x 1,..., x t ) t 0 wth Q-probablty 1. We use Theorem 3 to establsh predctve convergence n the context of the I-POMDP framework. Theorem 4 (ɛ-predctve Convergence n I-POMDPs). For all agents n the I-POMDP framework, f ther ntal belefs satsfy the ACC, then for every ɛ > 0, there exsts a fnte T whch s a functon of ɛ, such that for all t T and wth µ 0 -probablty 1, for k =,. pro Ht µ 0 ( h t ) pro Ht µ k ( b 0 k, h t k) ɛ Proof. Referrng to Theorem 3, let X = H. We observe that pro H µ 0 and pro H µ k ( b 0 k ) for k =, are predctve as defned n [Blackwell and Dubns, 1962]. Set Q = pro H µ 0, and P = pro H µ k ( b 0 k ). Subsequently, Qt = pro Ht µ 0 ( h t ), and P t = pro Ht µ k ( b 0 k, ht k ). Theorem 4 then follows mmedately from a straghtforward applcaton of Theorem 3. We have shown that for a POSG modeled usng the I-POMDP formalsm, the players belefs over opponent s models converge n the lmt f they satsfy the ACC property. However, the lmt belefs may be ncorrect, due to the nablty of agents to dstngush between observatonally equvalent models of the opponent on the bass of the observaton hstory. Nevertheless, ther belefs over the future paths come arbtrary close, and reman close, to the true dstrbuton over the future, after a fnte amount of tme. Further observatons wll only confrm ther belefs about the truth, and wll not alter ther belefs. We capture ths noton usng the concept of a subectve equlbrum [Kala and Lehrer, 1993a], defned as follows: Defnton 2 (Subectve ɛ-equlbrum). Let b t k, k =, be the agents belefs at some tme t. A par of polcy trees, π = [π, π ] s a subectve ɛ-equlbrum f, 1. π OP T (bt ), π OP T (bt ) 2. pro Ht µ 0 ( h t ) pro Ht µ k ( b 0 k, ht k ) ɛ, k =, wth a µ 0-probablty 1. For ɛ = 0, subectve equlbrum obtans. Condton 1 of subectve ɛ-equlbrum states that agents are subectvely ratonal,.e. ther strateges are best responses to ther belefs. As we mentoned before, these strateges are the polcy trees computed usng Equatons 1 and 2. The second condton states that the agents belefs have attaned ɛ-predctve convergence. In other words, a strategy profle s n subectve ɛ-equlbrum when the strateges are best responses to agents belefs that have attaned ɛ-predctve convergence. We now establsh the man result of ths paper, whch s that behavor strateges of agents playng a POSG modeled usng the I-POMDP framework, attan subectve ɛ-equlbrum n fnte tme, and subectve equlbrum n the lmt. The followng corollary gves our result. 8

9 Corollary 1 (Convergence to Subectve Equlbrum n I-POMDPs). Let π = [π, π ] be the strateges of agents, and respectvely, playng a POSG modeled usng the I-POMDP formalsm. Let b 0, and b0 be ther ntal belefs. If the followng condtons are met, 1. π OP T (b 0 ), π OP T (b 0 ) 2. pro H µ 0 pro H µ k ( b 0 k ), k =, (ACC) then for any ɛ > 0, and for all µ 0 -postve probablty hstores, there exsts some fnte tme step T whch s a functon of ɛ, such that for all t T, the strategy profle, π = [π, π ] s a subectve ɛ-equlbrum where, b t and bt are the agents belefs at tme t π OP T (bt ), π OP T (bt ) Proof. Corollary 1 follows n part from Theorem 4, and n part from notng that agents n the I- POMDP framework compute strateges that are best responses to ther posteror belefs at each tme step, and that the belefs are updated usng ther observaton hstory. Strategy profles n subectve ɛ-equlbrum for arbtrarly small ɛ 0 are stable. Specfcally, further play wll brng agents belefs over the future closer to the truth statstcally, and the correspondng strategy profles wll reman n the subectve ɛ-equlbrum. Note that ACC s a suffcent condton, but not a necessary one. An example settng n whch even though ACC s volated, yet subectve ɛ-equlbrum stll results s gven n [Kala and Lehrer, 1993a]. 4 Computatonal Lmtatons of Our Results Recall that n Secton 2, we made the assumpton that agent strateges are computable. Ths restrcts the space of possble strateges to be countable. However, as observed n [Nachbar and Zame, 1996], there may exst strateges whch are exact best responses to computable strateges but are themselves not computable, and even when computable best responses do exst, the decson procedure of computng these best responses may not be computable. Consequences of these negatve results lead to a subtle tenson between learnng and optmzaton. Specfcally, f agents best response strateges are not computable, then ther belefs fal to account for such strateges of others, thereby possbly volatng ACC and preventng predctve convergence. On the other hand, f we post that best responses be computable, then the correspondng belefs may be unnatural for example, they may not assgn non-zero probablty to all possble strateges of others. Nachbar [Nachbar, 1997] makes an argument along smlar lnes n the context of repeated games usng the noton of a conventonal set of strateges (analogous to computable set n our settng) attrbuted to each agent. We beleve that these mplausblty ssues are a drect mplcaton of Bnmore s clam n [Bnmore, 1982] that perfect ratonalty s an unattanable deal. Bnmore proves that a Turng machne cannot always predct truthfully the behavor of an opponent Turng machne (gven ts complete descrpton) and optmze smultaneously. Hs clam rests on a partcular constructon of a two-agent game n whch a supposedly ratonal Turng machne when requred to compute the best response s unable to predct truthfully, and when requred to predct truthfully s unable to termnate ts computatons and optmze. Though the above mentoned negatve results are exstental, they serve to show that t may be problematc to fulfll the assumptons lad out n our analyss n practce. Nevertheless, there may be 9

10 ways to overcome these lmtatons. One nterestng drecton s to replace exact optmzaton wth approxmate optmzaton. Specfcally, rather than computng the exact best response to ts subectve belef, an agent may compute an ɛ-best response 6 that s guaranteed to be always computable. However, the effect of ɛ-optmalty on predctve convergence remans an open queston. 5 Dscusson In ths paper we analyzed play of agents engaged n a partally observable stochastc game formalzed usng the nteractve POMDP framework. In partcular, we have consdered subectvely ratonal agents whch may not know others strateges. Therefore, they mantan belefs over the physcal state and models of other agents and optmze wth respect to ther belefs. We have also shown how such agents update ther belefs on performng actons and recevng observatons, and compute best responses to ther belefs. Wthn ths framework, we proved that f agents belefs satsfy a truth compatblty condton, then strateges of agents that learn and optmze converge to the subectve equlbrum n the lmt, and subectve ɛ-equlbrum for arbtrarly small ɛ > 0 n fnte tme. For completeness, we also gave condtons on agents pror belefs that wll allow play to converge to Nash equlbrum. Though the concept of subectve equlbrum s not novel, we beleve that our results complement the exstng lterature. Specfcally, usng the I-POMDP framework for learnng and optmzng, we have shown the exstence of equlbra n a POSG. The POSGs provde a more realstc settng than repeated games, n whch exstence of equlbra was known. We ponted out that attempts to apply these theoretcal results could run nto obstacles. One problem s the nherent dffculty n perfect optmzaton and smultaneous predcton. One may be forced to resort to ɛ-optmalty. Whether any form of equlbrum obtans when the players are boundedly ratonal s a topc of future work. References [Bnmore, 1982] Bnmore, K. (1982). Essays on Foundatons of Game Theory. Pttman. [Blackwell and Dubns, 1962] Blackwell, D. and Dubns, L. (1962). Mergng of opnons wth ncreasng nformaton. Annals of Mathematcal Statstcs, 33(3): [Bowlng and Veloso, 2002] Bowlng, M. and Veloso, M. (2002). Multagent learnng usng a varable learnng rate. Artfcal Intellgence Journal, 136: [Doob, 1953] Doob, J. L. (1953). Stochastc Processes. John Wley and Sons. [Fudenberg and Levne, 1993] Fudenberg, D. and Levne, D. (1993). Self confrmng equlbrum. Econometrca, 61: [Gmytrasewcz and Dosh, 2004] Gmytrasewcz, P. and Dosh, P. (2004). Interactve pomdps: Propertes and prelmnary results. In AAMAS, pages , NYC, NY. [Gmytrasewcz and Dosh, 2005] Gmytrasewcz, P. and Dosh, P. (2005). A framework for sequental plannng n multagent settngs. Journal of AI Research, One way to compute an ɛ-best response s to consder fnte horzons for maxmzaton, rather than nfnty. 10

11 [Hahn, 1973] Hahn, F. (1973). On the Noton of Equlbrum n Economcs: An Inaugural Lecture. Cambrdge Unversty Press. [Hansen et al., 2004] Hansen, E., Bernsten, D., and Zlbersten, S. (2004). Dynamc programmng for partally observable stochastc games. In AAAI. [Hu and Wellman, 1998] Hu, J. and Wellman, M. P. (1998). Multagent renforcement learnng: Theoretcal framework and an algorthm. In 15th Intl Conference on Machne Learnng, pages [Jordan, 1995] Jordan, J. S. (1995). Bayesan learnng n repeated games. Games and Economc Behavor, 9:8 20. [Kadane and Larkey, 1982] Kadane, J. and Larkey, P. (1982). Subectve probablty and the theory of games. Management Scence, 28(2): [Kala and Lehrer, 1993a] Kala, E. and Lehrer, E. (1993a). Ratonal learnng leads to nash equlbrum. Econometrca, 61(5): [Kala and Lehrer, 1993b] Kala, E. and Lehrer, E. (1993b). games. Econometrca, 61(5): Subectve equlbrum n repeated [Nachbar, 1997] Nachbar, J. H. (1997). Predcton, optmzaton, and ratonal learnng n repeated games. Econometrca, 65: [Nachbar and Zame, 1996] Nachbar, J. H. and Zame, W. (1996). Non-computable strateges and dscounted repeated games. Economc Theory, 8: [Nyarko, 1997] Nyarko, Y. (1997). Convergence n economc models wth bayesan herarches of belefs. Journal of Economc Theory, 74: