Cf. Linn Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley Series in Probability & Statistics, 1999.

Transcription

1 Cf. Linn Sennott, Stochstic Dynmic Progrmming nd the Control of Queueing Systems, Wiley Series in Probbility & Sttistics, D.L.Bricker, 2001 Dept of Industril Engineering The University of Iow MDP -- Approximting Sequences pge 1 D.Bricker

2 Assume tht the stte spce of Mrkov Decision Problem (MDP) is countble but infinite. Four different optimiztion criteri re considered: Cses Expected discounted costs Averge cost/stge Finite horizon 1 2 Infinite horizon Expected discounted cost over finite horizon 2. Expected cost/stge over finite horizon 3. Expected discounted cost over infinite horizon 4. Expected cost/stge over infinite horizon MDP -- Approximting Sequences pge 2 D.Bricker

3 Denote the originl MDP by, with infinite (but countble) stte spce S. It is common, for computtionl purposes, to pproximte by MDP with finite stte spce of size. As is incresed, the pproximting MDP is "improved". We re interested in the limit s. MDP -- Approximting Sequences pge 3 D.Bricker

4 Definition Consider the sequence { } 0 of MDPs, where the stte spce of is the nonempty finite set S the ction set for stte i S is A i, nd the cost for ction A is C. i i S, Let { S } be n incresing sequence of subsets of S such 0 tht S = S, nd for ech i S nd Ai distribution on Then { } 0, Pi ( ) S such tht ( ) is probbility lim P = P is n pproximting sequence (AS) for the MDP, nd is the pproximtion level. ij ij MDP -- Approximting Sequences pge 4 D.Bricker

5 The usul wy to define n pproximting distribution is by mens of n ugmenttion procedure: Suppose tht in stte i S, ction Ai is chosen. For j S the probbility Pij is unchnged. Suppose, however, tht P ir > 0 for some r S, i.e., there is positive probbility tht the system mkes trnsition to stte outside of S. This is sid to be excess probbility ssocited with (i,,r,). MDP -- Approximting Sequences pge 5 D.Bricker

6 In order to define vlid MDP, this excess probbility must be distributed mong the sttes of S ccording to some specified ugmenttion distribution qj(i,,r,), where j ( ) q i,, r, = 1 j for ech (i,,r,). The quntity qj(i,,r,) specifies wht portion of the excess probbility Pir is redistributed to stte j S. MDP -- Approximting Sequences pge 6 D.Bricker

7 Definition: The pproximting sequence { } is n ugmenttion-type pproximting sequence (ATAS) if the pproximting distributions re defined s follows: ( ) (,,, ) P P P q i r = + ij ij ij r S otes: The originl probbilities on S re never decresed, but my be ugmented by ddition of portions of excess probbility. Often it is the cse tht there is some distinguished stte z q i,, r, = 1 such tht for ech (i,,r,), ( ) z (Tht is, ll excess probbility is sent to the distinguished stte.) MDP -- Approximting Sequences pge 7 D.Bricker

8 For the discounted-cost MDP with infinite horizon, nd infinite stte spce S, let Vβ () i = min Ci + β PijVβ ( j), j S Ai j Suppose we hve n pproximting sequence { }, with corresponding optiml vlues V β Mjor questions of interest: When does lim V () i V () i β = <+? β If π is the optiml policy for, when does optiml policy for? π converge to n MDP -- Approximting Sequences pge 8 D.Bricker

9 Infinite Horizon Discounted Cost Assumption DC(β): For i S we hve nd () () limsupv i W i β β () () W i V i β β <+ Theorem (Sennott, pge 76): The following re equivlent: lim V β = V <+ β Assumption DC(β) holds. If one (& therefore both) of these conditions re vlid, nd { π β } n optiml sttionry policy for sequence is optiml for.. Then ny limit point of the is MDP -- Approximting Sequences pge 9 D.Bricker

10 The following theorem of Sennot (p. 77) gives sufficient condition for DC(β) to hold (nd hence for the convergence of the pproximting sequence method): Theorem: Assume tht there exists finite constnt B such tht Ci B for every i S nd Ai β ( 0,1). Then DC(β) is vlid for MDP -- Approximting Sequences pge 10 D.Bricker

11 Consider gin our erlier ppliction to inventory replenishment: The dily demnd is rndom, with Poisson distribution hving men of 3 units. The inventory on the shelf (the stte) is counted t the end of ech business dy, nd decision is then mde to rise the inventory level to S t the beginning of the next business dy. There is fixed cost A=10 of plcing n order, holding cost h=1 for ech item in inventory t the end of the dy, nd penlty p=5 for ech unit bckordered. We imposed limits of 7 units of stock-on-hnd nd 3 bckorders, nd found tht the policy which minimizes the expected cost/dy is of type (s,s) = (2, 6), i.e., if the inventory position is 2 or less, order enough to bring the inventory level up to 6. MDP -- Approximting Sequences pge 11 D.Bricker

12 Consider the problem with infinitely-mny sttes, i.e., {, 2, 1, 0, 1, 2, 3, 4, } S = + nd the objective of minimizing the discounted cost, with discount fctor 1 β = = Wht is the optiml replenishment policy? MDP -- Approximting Sequences pge 12 D.Bricker

13 Approximting Sequence Method =1 To define the first MDP in the sequence, 1, use stte spce S 1 = { 2, 1, 0, 1, 2,...6}, i.e., ssume limit of 2 bckorders nd 6 units in stock. The optiml policy is (s, S) = (2, 6): Stte Action V BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= six SOH= MDP -- Approximting Sequences pge 13 D.Bricker

14 =2 We now increse the stte spce to S 2 = { 3, 2, 1, 0, 1, 2,... 6, 7}, i.e., ssume limit of 3 bckorders nd 7 units in stock, nd find tht the optiml policy is (s, S) = (2, 7): Stte Action V BO= three SOH= BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= six SOH= SOH= seven SOH= MDP -- Approximting Sequences pge 14 D.Bricker

15 =3 We now increse the stte spce to S 3 = { 4, 3, 2, 1, 0, 1, 2,... 7, 8}, i.e., ssume limit of 4 bckorders nd 8 units in stock, nd find tht the optiml policy is (s, S) = (2, 8): Stte Action V BO= four SOH= BO= three SOH= BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= six SOH= SOH= seven SOH= SOH= eight SOH= MDP -- Approximting Sequences pge 15 D.Bricker

16 =4 We now increse the stte spce to S 4 = { 5,..., 1, 0, 1, 2,..., 9, 10}, nd find tht the optiml policy is (s, S) = (2, 10): Stte Action V BO= five SOH= BO= four SOH= BO= three SOH= BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= six SOH= SOH= seven SOH= SOH= eight SOH= SOH= nine SOH= SOH= ten SOH= MDP -- Approximting Sequences pge 16 D.Bricker

17 =5 Increse the stte spce to S 5 = { 6,..., 1, 0, 1, 2,... 11, 12}. The optiml policy is gin (s, S) = (2, 10): Stte Action V BO= six SOH= BO= five SOH= BO= four SOH= BO= three SOH= BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= six SOH= SOH= seven SOH= SOH= eight SOH= SOH= nine SOH= SOH= ten SOH= SOH= eleven SOH= SOH= twelve SOH= MDP -- Approximting Sequences pge 17 D.Bricker

18 =6 Increse the stte spce to S 5 = { 7,..., 1, 0, 1, 2,... 11, 15}. The optiml policy is gin (s, S) = (2, 10): Stte Action V BO= seven SOH= BO= six SOH= BO= five SOH= BO= four SOH= BO= three SOH= BO= two SOH= BO= one SOH= SOH= zero SOH= SOH= one SOH= SOH= two SOH= SOH= three SOH= SOH= four SOH= SOH= five SOH= SOH= fourteen SOH= SOH= fifteen SOH= The optiml policies hve converged to (s, S) = (2, 10) MDP -- Approximting Sequences pge 18 D.Bricker

19 For the MDP with finite horizon n nd infinite stte spce S, let vβ, n() i = min Ci + β Pijvβ, n 1( j), j S, n 1 Ai j Suppose we hve n pproximting sequence { }, with v β n corresponding optiml vlues, Mjor questions of interest: When does v () i v () i lim β, n β, n =? If π is the optiml policy for, when does π converge to n optiml policy for? Finite Horizon Assumption FH(β,n): MDP -- Approximting Sequences pge 19 D.Bricker

20 For i S we hve limsup v n w β, β, n <+ nd () () w i v i β, n β, n Theorem (Sennott, pge 43): Let n 1 be fixed. The following re equivlent: lim v β, n vβ, n = < + Assumption FH(β,n) holds. MDP -- Approximting Sequences pge 20 D.Bricker

21 The following theorem of Sennot (p. 45) gives sufficient condition for FH(β,n) to hold (nd hence for the convergence of the pproximting sequence method): Theorem: Suppose tht there exists finite constnt B such tht Ci Fi B B where Fi is the terminl cost of stte i S. Then FH(β,n) holds for ll β nd n 1. MDP -- Approximting Sequences pge 21 D.Bricker