Long Run Average Cost Problem. E x β j c(x j ; u j ). j=0. 1 x c(x j ; u j ). n n E

Transcription

1 Chapter 6. Long Run Average Cost Problem Let {X n } be an MCP with state space S and control set U(x) forx S. To fix notation, let J β (x; {u n }) be the cost associated with the infinite horizon β-discount problem; i.e., J β (x; {u n }) =. E x β j c(x j ; u j ). Its value function, which is the infimum of J β over all control {u n }, is denoted by V β (x). Our main interest in this chapter is the long run average cost. More precisely, we will define I(x; {u n }). =limsup n n 1 1 n E x c(x j ; u j ). The goal is to minimize this quantity, and the value function is denoted by Λ(x); that is, Λ(x). = inf {u n } I(x; {u n}). Remark 1 Sometimes, we will also define another long run average cost I (x; {u n }) =. n 1 1 lim inf n n E x c(x j ; u j ). But mostly, we will focus on cost function I and value function Λ. - We will assume throughout that Condition 1 The running cost c is non-negative. Interested students may want to construct a control problem and a control policy such that I = I. 1 Finite state MCP with finite controls We have the following useful lemma connecting the infinite horizon problem with the long-run average cost problem. 1

2 Lemma 1 Under condition 1, for any control process {u n } and initial state x S we have I (x; {u n }) lim inf (1 β)j β(x; {u n }) lim sup(1 β)j β (x; {u n }) I(x; {u n }). Furthermore, if lim(1 β)j β (x; {u n }) exists and is finite,thenalltheinequalitiesareequalities. Proof. TheproofisthedirectconsequenceoftheTauberiantheorem.. From this lemma, we have the following lemma concerning the stationary policies of a finite state MCP with finite controls. Lemma 2 Suppose {X n } is an MCP with finite state space S and for every x, thecontrolsetu(x) is finite. Let φ : x U(x) be a stationary control policy; that is u n = φ(x n ).Then I(x; φ) =I (x; φ) = lim (1 β)j β (x; φ). Proof. The quantity J β (x; φ), viewed as a function of β, isrational;see the proof of Theorem 1 in Chapter 5. Furthermore, thanks to finiteness, the cost c is bounded from above, say by constant C. It follows that (1 β)j β (x; φ) (1 β) Cβ j = C. It follows that (1 β)j β (x; φ) converges as β 1 and the limit is finite. We complete the proof by Lemma 1. The following result connects the Blackwell optimality and the long-run average cost optimality. Theorem 1 Suppose {X n } is an MCP with finite state space S and for every x, the control set U(x) is finite. Then the Blackwell optimal stationary policy, say φ, is optimal for the long run average cost; that is Λ(x) = I(x; φ ). Moreover, if we further assume that for any x, y S, there exists a stationary policy such that x y under this stationary policy. Then Λ(x) λ for some constant λ. 2

3 Proof. Let {u n } be an arbitrary control. We have I(x; {u n }) lim sup (1 β)j β (x; {u n }) lim sup(1 β)j β (x; φ )=I(x; φ ). This yields that I(x; φ )=Λ(x). Let x, y S be such that x and y achieve the maximum and the minimum respectively over Λ(z) overz S. Assume that x y under the stationary policy φ, and without incurring confusion, let P denote the transition probability matrix under policy φ. It follows that there exists k N such that P xy (k) > 0. Let M denote the upper bound for the running cost c. TheDPEforthe infinite horizon discount problem implies that V β (x) c(x; φ(x)) + β P xz V β (z) M + P xz V β (z). z S z S Repeating this, we arrive at V β (x) km + P xz (k) V β (z). z S Multiplying both sides by (1 β) andlettingβ 1, it follows from the definition of x and y that P (k) xz (k) (k) Λ(z) P xy Λ(y)+[1 P xy ]Λ(x), Λ(x) z S or equivalently Λ(x) Λ(y). This completes the proof. ItcouldhappenthatthevaluefunctionΛ(x) is not a constant across x S, as the following pathological example shows. Example 1 Consider an MCP with state space S = {0, 1, 2, 3}. Thestates {1, 2, 3} is uncontrolled, and transition probability matrix satisfies P 11 = P 23 = P 32 =1. Atstate0,U(0) = {A, B, C} with policy A gives P A 01 = P A 03 =1/2, PB 01 = P B 03 =1/2, PC 01 =1/8 =1 P C 03. The associated cost is c(1) = 0, c(2) = 2, c(3) = 1, and c(0;a) =0, c(0;b) =1, c(0; C) =1/8. The corresponding minimization problem is easy to solve. Let V β denote the value function for the β-discount problem. Clearly, V β (1) = 0, V β (2) = 3

4 (2 + β)/(1 β 2 ), and V β (3) = (2β +1)/(1 β 2 ). It can be shown that A is β-optimal and V β (0) = β(2β +1)/[2(1 β 2 )]. It follows that 3 Λ : {0, 1, 2, 3} 4, 0, 3 2, 3, 2 which is not a constant. In this example clearly there is no stationary policy such that 1 2or The DPE associated with the long-run average problem The DPE for the long-run average problem is not as straightforward as the total expected cost problem. New quantities other than the value function is to be introduced into the equation. Theorem 2 Assume that the state space S is finite and that U(x) is finite for every x S. Suppose Λ(x) λ for some constant λ. Then there exists avectorh : S R such that λ + h(x) = min c(x; u)+ P xy (u)h(y), y S u U(x) where P (u) denotes the transition probability matrix under control u, and the minimum is attained at u = φ (x). Conversely, suppose there exists a constant λ and a vector h satisfies the above equation, then Λ(x) λ and the minimizer u : x U(x) defines an optimal (stationary) policy. Proof. Let φ be the Blackwell optimal policy and P. = P (φ ) the probability transition probability matrix corresponding to φ. It follows that for β close to one, V β (x) =c(x; φ (x)) + β y S P xy (φ )V β (y) Or equivalently, define vectors V β. =[Vβ (x)] and C =[c(x; φ (x))], then V β =(I βp ) 1 C. A result from linear algebra (see Proposition 3) asserts that there exists matrices Q and P such that (1 β)(i βp ) 1 = Q +(1 β)h + o(1 β). 4

5 It follows that (1 β)v β = QC +(1 β)hc + o(1 β). Letting β 1wehaveQC = Λ λ. Define h = HC,wehave which yields V β =(1 β) 1 λ + h + o(1), (1 β) 1 λ + h(x) =c(x; φ (x)) + β y S P xy (1 β) 1 λ + h(y) + o(1), or equivalently, λ + h(x) =c(x; φ (x)) + β y S P xy [h(y)+o(1)]. Letting β 1, we have λ + h(x) =c(x; φ (x)) + y S P xy (φ )h(y). As for the inequality λ + h(x) c(x; u)+ y S P xy (u)h(y), given an arbitrary u U(x). The proof is exactly the same and thus omitted. Thesecondhalfofthetheoremisjustaverification argument. For each control {u n }, construct the process Z n n 1. = c(x j ; u j )+h(x n ) nλ, which is in general a submartingale. We omit the details. Corollary 1 Suppose the MCP {X n } has finite state space S and the control set U(x) is finite for every x S. AssumethatΛ(x) x. Then v n (x) lim λ. n n Here v n is the value function for the corresponding finite horizon problem with horizon n and without discounting. The proof again is the same verification argument and thus omitted. 5

6 1.2 Probabilistic interprtation of h The function h : S R has some nice probabilistic interpretations. For simplicity, we should assume the following condition throughout this subsection. Condition 2 The MCP {X n } is irreducible under arbitrary stationary control policy. Under this condition, clearly Λ(x) λ for some constant λ; see Theorem 1. This further implies that there exist a vector h such that λ + h(x) = min c(x; u)+ P xy (u)h(y). (1) y S u U(x) We have the following result regarding the uniqueness of the vector h. Proposition 1 Suppose {X n } has a finite state space and for each x S the control set U(x) is finite. If Condition 2 holds, then the vector h, asa solution to the DPE (1), is unique up to an additive constant. Proof. Suppose (λ,h)and(λ, h) are both solutions to the DPE (1). Assume that u : x U(x) is the optimal policy (i.e., minimizer) corresponding to (λ,h). We have λ + h(x) = c(x; u (x)) + y S P xy (u )h(y) λ + h(x) c(x; u (x)) + y S P xy (u ) h(y). Letting s. = h h, itfollowsthat s(x) y S P xy (u )s(y). Repeating this to obtain that for any n, s(x) 1 n 1 P n xy(u j ) s(y) π(y)s(y), y S y S where π is the stationary distribution corresponding to the probability transition matrix P (u ). Due to irreducibility, each component of the vector π 6

7 is non-zero. This clearly implies that s is a constant. This completes the proof. Assume that the conditions of Proposition 1 hold. In this case, h admits the following representation. Let φ : x U(x) betheblackwelloptimal policy. Consider the MCP {X n } under the transition probability matrix P (φ ). The proof of Theorem 2 implies that φ is a minimizer of DPE corresponding to h. Fix a generic state z S. For an arbitrary state x S, define τ x. =inf{n 0: Xn = z X 0 = x}. Then τ x 1 h(x) =h(z)+e x [c(x j ; φ (X j )) λ]. (2) The remainder of this subsection is to prove this representation (2). We should define h as in (2) with h(z) an arbitrary constant. We need to show that (λ,h) solves the DPE. Assume from now on x = z, andx 0 = x. We have, for β close to one enough, τ x 1 V β (x) =E x β j c(x j ; φ (X j )) + E x [β τx V β (z)]. It follows that τ x 1 V β (x) V β (z) =E x β j c(x j ; φ (X j )) 1 β τ x (1 β)v β (z)e x 1 β Letting β 1, we have τ x 1 lim[v β (x) V β (z)] = E x [c(x j ; φ (X j )) λ] =h(x) h(z). The DPE for V β implies that (1 β)v β (x) = inf c(x; u)+β P xy (φ )[V β (y) V β (x)]. y S u U(x) Letting β 1, we have λ = inf c(x; u)+ P xy (φ )[h(y) h(x)]. y S u U(x). 7

8 This says that h satisfies the DPE. Sometimes the difference x V β (x) V β (z) is called the relative value function. Thus h also characterizes the limit of this relative value function as β Computational issues Corollary 1 gives an algorithm for computing the value function Λ(x). However, such an algorithm suffers from slow convergence, lack of information on h : S R and the optimal policy. The convergence of infinite horizon value function to the long-run cost value function and the Blackwell optimal policy being long-run cost optimal give rise to another possible algorithm. But it is usually very inconvenient working with the infinite horizon problems. We consider an alternative computation method based on the finite horizon problem v n. Fix a state z S, and define h n (x). = v n (x) v n (z). The DPE for {v n } gives that, for every x S, [v n+1 (z) v n (z)] + h n+1 (x) = inf c(x; u)+ P xy (u)h n (y). u U(x) y S Suppose we can show that {v n+1 (z) v n (z)} and {h n } converge, then the limit pair is a solution to the DPE associated with the long-run average cost problem, whence the limit of the former is the value for the long-run average problem and the limit of {h n } is just h with h(z) =0. Condition 3 Every stationary control policy leads to a Markov process {X n } that is irredcucible and aperiodic. Proposition 2 Suppose Condition 3 holds. Thus Λ(x) λ and let (λ,h) is a solution pair to the DPE associated with the long-run average cost problem such that h(z) =0. Then for every z S, and for every x S, lim n [v n+1 (z) v n (z)] = λ, lim n h n (x) = lim n [v n (x) v n (z)] = h(x). Proof. Define F n (x). = nλ + h(x) v n (x). We first show that {F n } is uniformly bounded. The lower bound is just a verification theorem (very 8

9 similar to Corollary 1. As for the upper bound, let {u n} be the optimal policy for the finite-horizon problem with horizon n. It is clear that Thus V β (x) v n (x)+β n E[V β (X n )]. V β (x) V β (z) v n (x)+β n E[V β (X n ) V β (z)] 1 βn 1 β (1 β)v β(z). Letting β 1, we have h(x) v n (x)+eh(x n ) nλ. The upper bound of {F n } follows readily. Let (λ,h)solvethedpewithu the minimizer of the DPE (whence optimal). It follows that λ + h(x) =c(x; u (x)) + y S P xy (u )h(y). However, we also have v n+1 (x) c(x; u (x)) + y S P xy (u )v n (y). It follows then F n+1 (x) y S P xy (u )F n (y), which further implies that F n+m (x) y S P m xy (u )F n (y). Consider now an arbitrary subsequence of {F n },say{f nk },thatconverges, say to a vector F : S R. We first show that F is constant valued. Fix j N and let k j. It follows that F nk (x) y S P n k n j xy (u )F nj (y). Letting k we have F (x) y S π(y)f nj (y), 9

10 where π is the stationary distribution corresponds to transition probability matrix P (u ). Now letting j,wehave F (x) y S π(y)f (y). Thus it follows easily that F is constant valued. Suppose now that F nk F and F mk F,where{n k } and {m k } are two subsequences. A similar argument yields that F y S π(y)f (y) =F. Exchange the role of F and F we have F F,orF = F. This implies that {F n } converges to vector F. The claims of proposition follow trivially. The algorithm goes as follows: Fix a state z S and set h 0 (x) 0, and then compute inf c(x; u)+ P xy (u)h n (y) =. W n+1 (x). y S u U(x) It follows that h n+1 (x) =W n+1 (x) W n+1 (z)andv n+1 (z) v n (z) =W n+1 (z). Remark 2 We have not discussed the optimal policy yet. Indeed, one can easily show that any limit point of the minimizing policy {u n } in the above infimum is long-run average optimal. We say φ is a limit point of {u n } if there exists a subsequence {n k } such that u nk (x) φ(x) forsufficiently large k. Due to the finiteness, such a limit point always exists. 10

11 A Appendix. The proof of a Tauberian theorem Consider a sequence of non-negative numbers c n [0, ]. For every β [0, 1), define V (β) =. c j β j and S n The Tauberian theorem states that n 1. = c j. Theorem 3 The following inequality holds. lim inf n S n n lim inf (1 β)v (β) lim sup(1 β)v (β) lim sup n S n n. Furthermore, if lim(1 β)v (β) exists and is finite,thenalltheinequalitiesareindeedequalities. Proof. Without loss of generality, we can assume that every c n is finite, otherwise all the quantities are infinity. It follows that V (β) = c j β j = (S j+1 S j )β j = (β j 1 β j )S j. j=1 For every N N, wehave V (β) = N 1 j=1 N 1 j=1 N 1 j=1 (β j 1 β j )S j + (β j 1 β j )S j + (β j 1 β j )S j + sup j N sup j N sup j N S j j j=n S j j S j j (β j 1 β j )j (β j 1 β j )j 1 1 β. This yields S j lim sup(1 β)v (β) sup j N j 11

12 for every N. LettingN, we complete the rightmost inequality. Similarly, we have for every N N, V (β) inf j N S j (β j 1 β j )j inf j j N j=n S j j β N 1 β. Letting β 1andthenN we complete the leftmost inequality. Now assume that lim(1 β)v (β) =. L is finite. We claim that for every continuous function F on interval [0, 1], 1 lim(1 β) c j F (β j )β j = L F (x) dx. (3) 0 This is true for every polynomial. Indeed, for F (x) =x n with n N, we have lim(1 β) c j F (β j )β j 1 β = lim 1 β n+1 (1 βn+1 ) c j (β n+1 ) j 1 β = L lim 1 β n+1 = L 1 n +1 = L x n dx. 0 Since every continuous function can be uniformly approximated by polynomials, the claim follows easily from the standard ε δ language. Now consider a function F (x) =. 0 ; if 0 x<e 1 1/x ; if e 1 x 1. F is not continuous, but we still have the equality (3). The reason is that one can always find sequences of continuous functions {g n } and {G n } such that g n F G n and 1 1 lim g n (x)dx =lim G n (x)dx = n n For this function F,wehave F (x)dx =1. (log β) (1 β) c j F (β j )β j 1 =(1 β) c j =(1 β)s (log β) 1. 12

13 Let β = e 1/n,wehave (1 β) c j F (β j )β j =(1 e 1/n )S n. Letting n,wehaveβ 1, and 1 L = L F (x)dx = lim(1 e 1/n )S n =lim 0 n n This completes the proof. S n n. B Appendix. A result from linear algebra Proposition 3 Let P be an arbitrary probability transition matrix, and β (0, 1). Then lim (I βp ) 1 Q 1 β H =0, with n 1 1 Q = lim P k, H =(I P + Q) 1 Q. n n k=0 Furthermore, Q + H = I + PH = I + HP. Proof. Consider the matrix M(β). =(1 β)(i βp ) 1. Clearly each component of M is a rational function of β. Since (I βp ) 1 = I + βp + β 2 P 2 +, each component of M(β) isboundedby (1 β)(1 + β + β 2 + )=1. Thus, as β 1, M(β) has a limit. Define Q. =lim M(β). The Taylor expansion then implies, for some matrix H, M(β) =Q +(1 β)h + o(1 β). 13

14 Here o(1) denotes a matrix converges to 0 as β 1. It follows that H = lim (I βp ) 1 Q. 1 β Multiplying both sides by (I βp )totheright,wehave Q βqp H βhp = I lim 1 β. We have Q = QP,and H + Q = I + HP. Analogously, multiplying both sides by (I βp ) totheleft, wehaveq = PQ and H + Q = I + PH. This gives P n H + P n Q = P n + P n+1 H,or Q = P n +(P n+1 P n )H. Thus nq = n 1 P j +(P n I)H. Dividing both sides by n and letting n,wehave n 1 1 Q = lim P j, n n since (P n I)H/n 0. Finally, since (I βp )Q = Q βpq =(1 β)q, we have Q = M(β)Q. Lettingβ 1wehaveQ = Q 2.Itiseasytoseenow that (P Q) n = P n Q. This in turn implies that H = lim (I βp ) 1 Q 1 β = lim I Q + β j (P Q) j j=1 = (I P + Q) 1 Q. This completes the proof. =lim β j (P j Q) =lim[i β(p Q)] 1 Q 14