19 Optimal behavior: Game theory

Transcription

1 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 1 19 Optiml behvior: Gme theory Adversril stte dynmics hve to ccount for worst cse Compute policy π : S A tht mximizes minimum rewrd Let S (, s) = { set of possible sttes s rechble by doing in s } (Assume you cn identify current stte) 19.1 Single step cse: Mxi-min rewrd E.g. s 0 mx 1 2 min Sets of possible next sttes Solve Obtin rewrd of s 1 s 2 s s 4 s 5 s 6 s 7 S ( 1, s 0 ) = {s 1, s 2, s } S ( 2, s 0 ) = {s, s 4, s 5 } S (, s 0 ) = {s 5, s 6, s 7 } = rg mx = min s S (,s 0 ) R(s )

2 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu Sequentil cyclic cse Assume leveled cyclic model (s in cyclic decision theory cse) s 0 s 1 s 2 s t 1 s t s 0 s 1 0 s s t 1 0 s 1 1 s s t s 1 k 1 s 2... k 2 s t 1 k t 1 s t Assume Strt t stte s 0 nd finish t stte s t fter t ctions Model is cyclic: fter executing ction in s 0 we go to one of the sttes s 1 0, s 1 1,..., s 1 k 1 nd fter executing the second ction, we go to one of the sttes Given s 2 0, s 2 1,..., s 2 k 2 nd so on, until fter the tth ction, we rrive in stte s t Stte dynmics S (, s) Rewrd function R(s) Compute Optiml policy π : S A mximizes minimum totl future rewrd for ech stte

3 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu Utility function U(s, π) = minimum future rewrd obtined by running π from s = R(s) + min U(s, π) s S (π(s),s) Compute π tht mximizes U(s, π ) for ll s Efficient lgorithm: Dynmic progrmming Solve for U(s, π) in lst sttes first, nd then recursively bck up π (s i ) = rg mx R(s i ) + min U(s i+1, π ) s i+1 S (,s i ) = rg mx min s i+1 S (,s i ) U(s i+1, π ) U(s i, π ) = R(s i ) + min U(s i+1, π ) s i+1 S (,s i ) where U(s i+1, π ) is lredy computed

4 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu Specil cse: Two-person Zero-sum gme Assume Acyclic stte dynmics 2 plyers MAX plyer tries to mximize rewrd MIN plyer tries to minimize rewrd R(s) = 0 except t lef sttes Then cn drmticlly speed up dynmic progrmming by α-β pruning Note: slight ugmenttion in dynmics Now explicitly model opponent s moves generl cse s 2 plyer s s opp S (, s) opp s

5 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 5 α-β pruning Not every pth hs to be explored E.g Assume the nodes re explored left to right in depth first fshion. Once the children of the left MIN node re explored, the left MIN node will choose rewrd. The MAX node t the top will then know tht it cn obtin rewrd t lest. The second MIN node (in the middle) will then strt to explore its children. Once it sees tht its first child hs vlue 2, it knows tht whtever the other children return, it cn only return rewrd tht is 2 or smller. But this mens the rest of the children of the middle MIN node re irrelevnt, becuse they cnnot cuse this node to obtin lrger vlue thn 2. So the top MAX node will ignore the rest of the middle subtree becuse the MAX node cn lredy chieve rewrd elsewhere. Therefore, the leves with rewrds 4 nd 6 re irrelevnt nd we do not need to check them..

6 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 6 α-cutoff 10 8 irrelevnt β-cutoff α = cutoff vlue for MIN node (if current best vlue α, stop) = current highest vlue of MAX ncestor = lower bound on vlue MIN cn hope to bck up to root = 10 in this exmple irrelevnt β = cutoff vlue for MAX node (if current best vlue β, stop) = current lowest vlue of MIN ncestor = upper bound on vlue MAX cn hope to bck up to root = 10 in this exmple

7 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 7 This method is clled α-β pruning, nd it is implemented by clculting the bounds α nd β of interesting vlues: not interesting interesting vlues. not interesting Algorithms 1 nd 2 describe opertions in mx nd min nodes for α-β pruning. Algorithm 1 α-β pruning lgorithm: lph bet mx(s, α, β) Require: s is mx-node sitution, α nd β re boundries Ensure: the mx vlue nd n optiml ction is returned, ssuming given boundries 1: if s is lef node then 2: return (R(s), no ction ) : end if 4: opt ction not importnt 5: for ll possible ctions do 6: (v, m) lph bet min((s), α, β) 7: if v > α then 8: α v 9: opt ction 10: if α β then 11: return (α, opt ction) 12: end if 1: end if 14: end for 15: return (α, opt ction) If we know the miniml nd mximl vlue of the rewrd, i.e., min s R(s) nd mx s R(s), then those re the initil vlues for α nd β; otherwise, α = nd β = + initilly. The figure 1 illustrtes wht vlues of α nd β re pssed during the serch, nd which nodes re visited (circled ones), in α-β pruning (the initil vlues re α = nd β = +. β α

8 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 8 Algorithm 2 α-β pruning lgorithm: lph bet min(s, α, β) Require: s is min-node sitution, α nd β re boundries Ensure: the min vlue nd n optiml ction is returned, ssuming given boundries 1: if s is lef node then 2: return (R(s), no ction ) : end if 4: opt ction not importnt 5: for ll possible ctions do 6: (v, m) lph bet mx((s), α, β) 7: if v < β then 8: β v 9: opt ction 10: if α β then 11: return (β, opt ction) 12: end if 1: end if 14: end for 15: return (β, opt ction) α = β = + α = β = + α = β = + α = α = β = β = + α = α = β = + β = + α = β = 5 5 α = β = + 1 α = β = Figure 1: α-β pruning exmple

9 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 9 Appliction to rel gmes A problem with lmost ll prcticl gmes is tht the serch tree is too lrge. If the gme is cyclic, it is infinite. In ny cse, it is usully impossible in prctice to serch the whole tree. On improvement is to use memoiztion, i.e., keep cche of clculted situtions (positions). Additionlly, memoiztion cn prevent us from exploring in n infinite loop in cyclic stte spces. The leves of the serch tree re usully to fr wy to be reched by serch lgorithm, so we cnnot usully bck up exct vlues from the leves. (Remember tht the leves re the terminl sttes t the very end of the gme!) In prctice, one lmost lwys uses heuristic pproch: serch to bounded depth, evlute heuristic function t the sttes reched (which estimtes the future min-mx rewrd), tret this heuristic vlue s the terminl rewrd, nd bck up the results. Combined rndom-dversril gmes In some gmes, the sequence of sttes does not depend on the plyers ctions lone, but lso on rndom element, such s dice roll or crd shuffle (e.g. Bckgmmon or Poker). In these cses, the gme tree contins chnce nodes in ddition to MIN nd MAX nodes. The vlue in chnce nodes in clculted s the expected vlue of the child-nodes vlues, using the probbility distribution of the rndom event. In this cse, we use the terms expecti-mini-mx lgorithm, expecti-mini-mx policy, etc Generl cyclic cse Mximize discounted sum of minimum future rewrds Vlue function V π (s) = minimum discounted rewrd obtined by π strting in s = R(s) + γ min V π (s ) s S (π(s),s)

10 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 10 Policy evlution Initilize V π rbitrrily Iterte for ech s Vπ new (s) = R(s) + γ min V s S π old (s ) (π(s),s) Hlt when V new π Policy itertion nd V old π re sufficiently close Initilize π rbitrrily nd evlute V π Iterte π new (s) = rg mx = rg mx R(s) + γ min V π old(s ) s S (,s) min V π old(s ) s S (,s) for ech s Use policy evlution to clculte V π new nd repet policy updte for π new Hlt when π new = π old (or more generlly when V π new = V π old) Vlue itertion Initilize V rbitrrily Iterte for ech s V new (s) = R(s) + γ mx min V old (s ) s S (,s) Hlt when V new nd V old re sufficiently close

11 Intro. to Artificil Intelligence: Dle Schuurmns, Relu Ptrscu 11 Recovering greedy policy Given vlue function V, recover π by π(s) = rg mx = rg mx R(s) + γ min V (s ) s S (,s) min V (s ) s S (,s) for ech s Redings Russell nd Norvig: Sections 6.1, 6.2, 6.5