1 Stochastic Dynamic Programming

Transcription

1 1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future are uncertain, the state does not evolve deterministically; instead, states and actions today lead to a distribution over possible states in the future. We ll assume that P (B,x,a) is the induced probability that tomorrow s state will lie in the set B given that today s state is x and today s action is a. Let s take a look at the Bellman equation for a stochastic dynamic program: { } v(x) = max r(x,a)+ v(x )P (dx,x,a). a Γ(x) We have to relate this expression to V (x 0 ) = max {x t,a t} t=0 { [ ]} 0 β t r(x t,a t ), the value function defined over the sequential problem. It turns out to be a bit more work than it was before because we need to be clear how to construct expectations in the recursive case we are taking expectations over tomorrow s states while in the sequential case we are taking expectations over sequences of future states, and it is not obvious that the two ideas can be reconciled. There are three important concepts buried in here that we need conditional expectations, expectations over sequences of events that unfold over time, and the law of iterated expectations. To understand these concepts formally we need to take a detour into measure theory. t=0 1.1 Measure Theory Detour Definition 1 Let S be a space. A family F of subsets of S is called a σ-algebra if (i) A F implies S\A F; (ii) if A n F n then n=1a n F; and (iii) S F. The pair (S,F) is called a measurable space. Note that because the union of sets is equal to the intersection of the complements of those sets, σ-algebras are also closed under countable intersections. Also, F because = S\S and S F. The largest σ-algebra on S consists of all subsets of S and the smallest consists of S and the empty set; in general, we cannot assign size to every member of the largest σ-algebra (for example, in R n the Banach-Tarski paradox constructs a set which cannot be assigned size) so this σ-algebra is only useful if S is countable. The σ- algebra generated by open sets (meaning that it is obtained by taking open sets and then adding whatever is needed to make the collection closed under unions and complements; it is therefore the smallest σ-algebra containing all the open sets) is called the Borel field and is typically denoted B(S) (or B n if S = R n ); this field contains all open and closed sets as well 1

2 as arbitrary unions of open sets and arbitrary intersections of closed sets (but not arbitrary unions of closed sets or arbitrary intersections of open sets, only countable ones). The restriction of B n to a subset of R n is also a σ-algebra, so that ([0,1],B 1 0) is a measurable space, where B 1 0 is the σ-algebra generated by intersecting elements of B with [0,1]. We will call a σ-algebra finer than F (written F ) if every set in F is also in. Definition 2 A sequence of σ-algebras {F t } t with F t F t+1 is called a filtration. Filtrations are a sequence of increasingly-fine σ-algebras, and are the natural method for expressing the flow of information. To see this, let s imagine a world that lasts two periods. Before period 1 nature selects a sequence of two outcomes, good or bad, in each period. Thus, the space of outcomes is S = {(,) (,B) (B,) (B,B)}. at time 0 the agent knows only that nature chose; therefore, the appropriate σ-algebra is F 0 = {,S}. At time 1 the first part is revealed, meaning that the agent now has the σ-algebra F 1 = {{(,) (,B)},{(B,) (B,B)},,S}. Finally, at time 2 everything is revealed, yielding F 2 = {(,),(,B),(B,),(B,B),,S} unioned with all the appropriate unions and complements (which take up too much space to write out). Note that F 0 F 1 F 2, so that this sequence is a filtration. Definition 3 A probability measure on (S,F) is a mapping µ : F [0,1] such that (i) µ( ) = 0; (ii) µ( n=1a n ) = n=1 µ(a n) if A n A m = n m and A n F; and (iii) µ(s) = 1. Thus, µ(a) is interpreted as the probability of event A; sets not in F cannot be assigned probabilities; for the remainder of this part we will just refer to µ as a measure (any finite measure can be normalized into a probability measure). Measures are monotone: µ(a) µ(b) if A B. Also, if {A n } F and A 1 A 2 and A = n A n, then µ(a n ) ր µ(a). Also, if {A n } F and A 1 A 2 and A = n A n, then µ(a n ) ց µ(a). And finally, for any collection of sets {A n } n=1 F we have where the A n may not be disjoint. µ( n A n ) n µ(a n) 2

3 One measure that will be of particular interest is the Lebesgue measure λ(b). Suppose that B can be written as the countable union of disjoint open intervals {(a 1,b 1 ),(a 2,b 2 ),...} (that is, it is a Borel set) and denote the collection of all such sets as I. Then where λ(b) = inf A I,B A {λ(a)} λ(a) = n (b n a n ); that is, the Lebesgue measure of B is the size of the smallest Borel set that contains B. It is not difficult to show that λ is a measure on B 1, the Borel sets of one dimension. Higher dimensions are handled analogously as the cross-product of intervals. In one dimension, countable collections of discrete points (like the rational numbers) have zero Lebesgue measure; in two dimensions, countable collections of discrete points and one-dimensional curves have zero measure; and so on. Definition 4 A triple (S,F,µ) is called a probability space. The key thing we need are concepts of integration and random variables. In particular, we need to know how to decide whether a function can be integrated with respect to a particular σ-algebra, which will matter when we start thinking about expectations. Definition 5 Let (S,F) and (T,) be measurable spaces. A function f : S T is measurable if and only if A we have f 1 (A) F, where f 1 (A) = {s S : f (s) A}. In particular, if we have F = B n then f is measurable if and only if the sets {s S : f (s) r} are contained in F for each r. Any continuous function is measurable, since those sets are all closed and thus must be in B n ; however, certain step functions are also measurable, so the class is clearly larger than the class of continuous functions. The notion of measurable function leads directly to the idea of a random variable. Definition 6 If (S,F,µ) is a probability space and f : S R n is measurable, then we call f a random variable. It is straightforward to apply the definition of measurability to prove the following facts: Proposition 1 Suppose f n : S R is measurable n and that f n (x) f (x) x. Then f is measurable. 3

4 Proposition 2 If f : S R is measurable and g : R R then g f is measurable where (g f)(s) = g(f (s)). Proposition 3 If f : S R and g : S R are measurable then f + g and f g are measurable. Proposition 4 If f : S R is measurable and α is a constant, then αf is measurable. Proposition 5 If f : S R and g : S R are measurable, then fg is measurable where (fg)(s) = f (s)g(s). f/g is measurable if g(s) 0 s, where (f/g)(s) = f (s)/g(s). To define integration, we need to define some types of functions. Definition 7 A function 1 A : S R is called the indicator function for A (sometimes called the characteristic function of A) if it takes the form 1 A (s) = { 1 if s A 0 if s / A. A function f : S R is called a simple function if it takes the form f (s) = n α i 1 Ai (s) i=1 for n <, real numbers {α i } n i=1, and disjoint sets {A i} n i=1 such that n i=1a i = S. Indicator functions are nonzero only over the set A, and simple functions are step functions created through finite linear combinations of indicator functions over disjoint sets(using disjoint sets puts the simple function in the standard representation). It is straightforwardtoseethatindicatorfunctionsaremeasurableifandonlyifa F; thatis,theσ-algebra must allow you to identify whether the event A did happen or did not happen. To see how this process works, consider r 1; then {s S : 1 A (s) r} = S F because indicator functions always take values less than or equal to 1. For r < 0 we have {s S : 1 A (s) r} = F. For any r (0,1) we have {s S : 1 A (s) r} = S\A, so that 1 A is measurable if and only if A F (because it needs to have S\A F). Simple functions are measurable if and only if every A i F; that is, we need to be able to identify whether each A i happened. Thus, measurability of a simple function with respect to a 4

5 particular σ-algebra really comes down to one simple condition the function must be constant on disjoint sets that are not elements of the σ-algebra, because if it changes value we can learn more by observing its value than we are supposed to know. We then define the Lebesgue integral of a simple function f (which is nonnegative) as f (s)λ(ds) = n α i λ(a i ); the notation λ(ds) is meant to make clear that we are summing the height of the simple function times the size of the interval ds. If f is both positive and negative we simply define the positive and negative parts separately and then put them together: { f (s) if f (s) 0 f + (s) = 0 otherwise i=1 and so that f (s) = f (s)µ(ds) = { f (s) if f (s) 0 0 otherwise f + (s)µ(ds) f (s)µ(ds). This function is not measurable if both terms are infinite. For nonsimple functions we define their integral as the limit of the integrals of simple functions that fit under f. That is, { } f (s)µ(ds) = sup g(s)µ(ds) : g is a simple function and 0 g(s) f (s) s. g Again, to integrate a function that has both negative and positive parts we simply do each piece separately and reassemble them. Definition 8 A function f is integrable if f (s) µ(ds) <. It is straightforward to see that f (s)µ(ds) = A f (s)1 A (s)µ(ds), which is the notion of integrating over a limited domain; obviously, if f is integrable over S it is integrable over any subset of S. One helpful property that we will use later is the fact that f (s)µ(ds) f (s) µ(ds); this is just an application of the triangle inequality. Other facts are equally easy to establish, once we note the property of almost everywhere. 5

6 Definition 9 A property P of f is said to hold almost everywhere (a.e) if the set P c = {s S : f (s) does not have property P} has measure µ(p c ) = 0. Properties that hold only on P with µ(p) = 0 are said to hold almost nowhere. We then can easily establish the following. Proposition 6 Let f,g be integrable. Then f +g is integrable, αf is integrable α, and (f +g)(s)µ(ds) = f (s)µ(ds)+ g(s)µ(ds) (αf)(s)µ(ds) = α f (s)µ(ds). Proposition 7 If f = 0 almost everywhere, then f (s)µ(ds) = 0. Proposition 8 If f g almost everywhere, then f (s)µ(ds) g(s)µ(ds). Proposition 9 If f = g almost everywhere, then f (s)µ(ds) = g(s)µ(ds). Proposition 10 If f 0 and µ({s S : f (s) > 0}) > 0 then f (s)µ(ds) > 0. Now we will define a space that will be relevant for our application to dynamic programming. Definition 10 Let (S,F,µ) be a probability space. The space L 1 (S,F,µ) is the space of all functions that are integrable with respect to µ. We will now prove that L 1 is a Banach space; the one caveat is that we must interpret L 1 as a space of equivalence classes, not functions, where an equivalence class is the collection of all functions that equal a reference function f almost everywhere. That is, for our purposes if two functions differ only on a set of Lebesgue measure zero then they are equal (because their expectations will be equal). The proof requires two theorems. Theorem 1 (Monotone Convergence Theorem) Suppose is a measurable set. Let {f n } be a sequence of measurable functions such that 0 f 1 (x) f 2 (x) for x. Let f be defined by f n (x) f (x) as n. Then f n (x)µ(dx) f (x)µ(dx). 6

7 Proof. As n, for some α; since f n f, we have α f n (x)µ(dx) α f (x)µ(dx). Choose c such that 0 < c < 1 and let s be a simple measurable function such that 0 s f. Put n = {x : f n (x) cs(x)}. By monotonicity, 1 2 and by the convergence of f n to f we have For every n, = n=1 n. f n (x)µ(dx) f n (x)µ(dx) c s(x)µ(dx). n n Let n ; since the integral is countably additive set function we can obtain α c s(x)µ(dx). Letting c 1 we have Therefore, α α s(x)µ(dx). f (x)µ(dx). Theorem 2 (Fatou s Lemma) Suppose is measurable. If {f n } is a sequence of nonnegative measurable functions and f (x) = lim inf n {f n(x)} then f (x)µ(dx) lim inf f n (x)µ(dx). n Proof. Set g n (x) = inf i n {f i (x)}. g n is measurable on and 0 g 1 (x) g 2 (x) g n (x) f n (x) g n (x) f (x). 7

8 Therefore, by the Monotone Convergence Theorem g n (x)µ(dx) f (x)µ(dx). The conclusion is now immediate. Theorem 3 (Riesz-Fischer Theorem) L 1 is a Banach space. Proof. We need to show that if {f n } is a Cauchy sequence in L 1, then there exists f L 1 such that {f n } f in L 1. Since f n is Cauchy, there exists a sequence {n k }, k = 1,2,..., such that f nk f nk+1 < 1 2 k. Choose g L 1. By the Schwarz inequality g ( ) f nk f nk+1 µ(dx) g 2. k Hence, k=1 g ( f nk f nk+1 ) µ(dx) g. Interchanging the order of integration and summation, we have g(x) f nk (x) f nk+1 (x) < k=1 almost everywhere on. Therefore, f nk (x) f nk+1 (x) <, k=1 for otherwise we could take g(x) 0 on a subset of of positive measure and contradict the previous inequality. Since the kth partial sum of is ( fnk+1 (x) f n k (x) ) k=1 f nk+1 (x) f nk (x) and the series converges almost everywhere on, we have f (x) = lim k f nk (x) defines f (x) for almost all x ; it does not matter how we define f (x) on those other points. 8

9 To show that f has the desired properties, let ǫ > 0 be given and choose N such that n N and m N implies f n f m ǫ (which is possible because f n is Cauchy). If n k > N, Fatou s lemma shows that f n f nk liminf i f n i f nk ǫ. Thus, f f nk L 1 and sincef = (f f m )+f nk, we have f L 1. Since ǫ is arbitrary, Finally, the inequality lim k f f n k = 0. f f n f f nk + f nk f n shows that f n converges to f in L 1, for if we take n and n k large enough each of the terms on the RHS can be made small. This result will be useful, because it will enable us to apply the Banach fixed point theorem (contraction mapping theorem) to a stochastic dynamic programming problem. Another small point involves the order of integration if we have multiple integrals, it should not matter the order in which we do them. but of course infinities can make things tricky. Define the product σ-algebra F as the smallest σ-algebra that contains all crossproducts of elements of F with elements of. A function f : S T R is measurable if it is measurable with respect to F. We get this result from Fubini s theorem, which we will not prove. Theorem 4 (Fubini s Theorem) Let f : S T R be measurable and let µ be a measure on (S,F) and ν be a measure of (T,). Then ( ) f (s,t)ν(dt) µ(ds) < if and only if S T ( whenever these are finite they are equal. T S ) f (s,t)µ(ds) ν(dt) < ; The final step is to define an expectation, which is just an integral. Definition 11 Let (S,F,µ) be a probability space. If f : S R is a random variable then the expectation of f with respect to µ is defined to be µ [f] = f (s)µ(ds). 9

10 When the measure is not ambiguous, it will be dropped from the expression leaving just [f]. What we are interested in is the conditional expectation of a random variable, which is a bit more challenging to define since we must worry about the possibility that our conditioning event has probability 0 (say, if a random variable has a continuous support the realization of any single real number is 0). What we will do is define a conditional expectation as a random variable over a σ-algebra that is smaller than F so that F; that is, any set that is measurable with respect to is also measurable with respect to F. Definition 12 A measurable partition of S is a collection of subsets of S such that (i) = ; (ii) = S; and (iii) F if. Suppose that is the σ-algebra generated by the partition. If g is a measurable function with respect to, then g must take on constant values over nonempty sets in. As noted before, this idea makes sense as a theory of information if is to represent the things you know then any function which you can take expectations of must not provide you any information about stuff you don t; that is, it cannot take on different values inside sets because then you would know more than you do. We can now define a conditional expectation. Definition 13 The conditional expectation of f relative to isafunction µ [f;](s) that is measurable with respect to and that satisfies the requirement µ [f;](s)µ(ds) = f (s)µ(ds) for any. It can be shown that µ [f;] defined in this way is unique up to a set of measure zero. Since µ [f;] must take on a constant value over, we have µ [f;](s)µ(ds) = µ [f;] µ(ds) If µ() > 0, we have the familiar expression µ [f;] = = µ [f;]µ(). f (s)µ(ds) ; µ() if µ() = 0, we still can employ this concept by thinking in limits (a sequence of sets n with µ( n ) 0). Conditional probabilities are then obtained by letting f be an indicator function: µ [1 A ;] = 1 A(s)µ(ds) µ() 10

11 has the interpretation as the probability of event A conditional on, provided again that µ() > 0. The law of iterated expectations can now be defined that is, the conditional expectation of a conditional expectation. Suppose we have two σ-algebras contained in F such that 1 2 (that is, the sequence { i } defines a filtration). Define the random variables µ [f; 1 ](s) and µ [f; 2 ](s), which are the conditional expectations with respect to each σ-algebra. By the definition of a conditional expectation we have µ [f; i ](s)µ(ds) = f (s)µ(ds) over any set i. Let s now consider the conditional expectation of µ [f; 2 ](s) with respect to 1, which must by definition be µ [ µ [f; 2 ]; 1 ](s)µ(ds) = [f; 2 ](s)µ(ds). Since 1 implies 2 the RHS is [f; 2 ](s)µ(ds) = For any 1 we must also have [f; 1 ](s)µ(ds) = Therefore, for any 1 µ [ µ [f; 2 ]; 1 ](s)µ(ds) = f (s)µ(ds). f (s)µ(ds). [f; 1 ](s)µ(ds). In the sequential problem this law enables us to get rid of expectations of future events with respect to information we gain only in the future; that is, taking the expectation with respect to information today is not inconsistent with the fact that the agent knows that some things will be learned and used to form expectations in the future. We can now be very precise about the term stochastic transition. Definition 14 A mapping P : F A [0,1] is a stochastic transition function on (,F) if, x and a A, P (,x,a) is a probability measure on F and for each B F P (B,, ) is a measurable function on A. An important class of stochastic processes are called Markov processes with stationary transitions. Markov means that the transition probabilities only depend on a finite history of past realizations and stationary means that calender time is irrelevant. These properties are important for dynamic programming, as they imply that t is again irrelevant and that computing expectations does not require the entire history of past events. Some familiar 11

12 stochastic processes are Markovian with stationary transitions. For example, let Φ(x) denote the cdf of a standard normal. Then the transition function describes an AR(1) process P ({x : x r},x) = 1 Φ(r ρx) x t+1 = ρx t +ǫ t+1 with ǫ t+1 iidn (0,1). Another familiar and useful stochastic process is a Markov chain, which arises when is a finite set. The transition function is usually written as P = [p ij ] i,j=1,...,n, with the interpretation that p ij = Pr{x t+1 = x i : x t = x j }; to be consistent with rules of probability requires p ij 0 and i p ij = 1. Stochastic transition functions can be used to define probability measures over sequences, which we need to define the sequential problem. That is, we can define µ T (B;x 0 ) = P (dx T,x T 1 )P (dx T 1,x T 2 ) P (dx 1,x 0 ) A 2 A T A 1 as the probability of B = {x t } T t=1 given the initial state x 0, so µ T ( ;x 0 ) = Pr{x 1 A 1,x 2 A 2,,x T A T x 0 }. It is not hard to show that µ T is in fact a probability measure on the product σ-algebra F T = F F F. As T we can define µ which is technically not a measure on F but something a bit weaker; we use the σ-algebra H which is the smallest σ-algebra that contains all finite unions of sets of the form B = A 1 A 2 A 3 A T where A t F for each t and all but finitely many of the sets equal so that µ (B;x 0 ) = P (dx T,x T 1 )P (dx T 1,x T 2 ) P (dx 1,x 0 ) A 2 A T A 1 where T is the largest date beyond which A t = t > T. We can uniquely extend this measure to all events (that is, those that do not require A t = t T) in H using one of the Hahn-Caratheodory extension theorems; by slight abuse of notion we ll refer to this measure as µ as well. Consider the functions F T (x 1,x 2,...,x T ) = T β t 1 f (x t ) t=1 12

13 and F (x 1,x 2,...) = β t 1 f (x t ); you can interpret F as the lifetime return once a policy function has been selected: F (x 1,x 2,...) = β t 1 r(x t,π(x t )). t=1 ach F T is measurable with respect to H (since it involves events that only differ from at finitely-many dates, namely those up to and including date T) and {F T } converges pointwise to F, so F is measurable with respect to H. We can therefore write [F;x 0 ] = f (x 1 )P (dx 1,x 0 )+ [ ] β f (x 2 )P (dx 2,x 1 ) P (dx 1,x 0 )+ [ [ ] ] β 2 f (x 3 )P (dx 3,x 2 ) P (dx 2,x 1 ) P (dx 1,x 0 )+ and is immediate that [F;x 0 ] =. t=1 [f (x 1 )+β[f;x 1 ]]P (dx 1,x 0 ). That is exactly what we want to have the lifetime expected value is the sum of the firstperiod return and the lifetime expected value from tomorrow onward, discounted back to today. 1.2 Back to Dynamic Programming Again Consider the functional equation { } v(x) = max r(x,a)+β v(x )P (dx,x,a) a Γ(x) where x is a set of events tomorrow. We want to relate this functional equation to the sequential problem { [ ]} V (x 0 ) = max 0 β t r(x t,a t ) {x t,a t} t=0 subject to the restrictions embedded in the state transition and feasible control sets. The sequences which solve the sequential problem should be interpreted as conditional on outcomes that is, they unfold over event trees. This expression is often written as { V (x 0 ) = max β t π ( } ) s t s 0 r(xt,a t ) {x t,a t} t=0 t=0 13 t=0

14 where s t is the history of events up through time t (s t = {s 0,s 1,...,s t }) and π(s t s 0 ) is the probability of that history conditional on s 0. We want to prove that v(x) exists and is unique. Define (Tw)(x) = max a Γ(x) { r(x,a)+β } w(x )P (dx,x,a). We make the same continuity assumptions as before. To apply the Blackwell-Boyd conditionsweneedaφ(x) > 0function; thisfunctionmustadditionallynowsatisfyanintegrability condition. Theorem 5 Assume (1) the one-period return function r is continuous on A and the feasible actions correspondence Γ is continuous and compact-valued; (2) there is a function φ Ξ with φ(x) > 0 such that (i) there exists an M < with x, (ii) and (iii) max { r(x,a) } Mφ(x) (1) a Γ(x) { { }} φ(x ) βsup max x a Γ(x) φ(x) P (dx,x,a) < 1, (2) φ(x )P (dx,x,a) < x and a Γ(x); and (3) for any continuous w Ξ φ w(x )P (dx,x,a) is continuous on A. Under these conditions the Bellman operator is a strict contraction map. Condition (2)(iii) guarantees the integrability of any member of Ξ φ, since { } f (x) f φ = sup x φ(x) and therefore f is bounded in absolute ) value by an) integrable function φ. We now show that T : (Ξ φ, φ (Ξ φ, φ. If w Ξ φ then Tw is continuous by ) ) the maximum theorem, so T : (Ξ φ, φ (Ξ, φ. To see that Tw has finite φ-norm, note that { (Tw)(x) = max r(x,a)+β w(x )P (dx,x,a)} a Γ(x) { } max { r(x,a) }+β max w(x ) P (dx,x,a) a Γ(x) a Γ(x) { } w(x ) = max { r(x,a) }+β max a Γ(x) a Γ(x) φ(x ) φ(x )P (dx,x,a) { } max { r(x,a) }+β w φ max φ(x )P (dx,x,a) a Γ(x) a Γ(x) 14

15 which is finite in the φ-norm by conditions (i) and (iii). Now we check monotonicity. Let v w. Then { } (Tv)(x) = max r(x,a)+β v(x )P (dx,x,a) a Γ(x) { } max r(x,a)+β w(x )P (dx,x,a) a Γ(x) = (Tw)(x). And then discounting constant functions: { } (T (w +Aφ))(x) = max r(x,a)+β [w(x )+Aφ(x )]P (dx,x,a) a Γ(x) { } (Tw)(x)+βA max φ(x )P (dx,x,a) a Γ(x) { } φ(x ) (Tw)(x)+Aφ(x)β max a Γ(x) φ(x) P (dx,x,a) (Tw)(x)+θAφ(x) with θ < 1 by condition (ii). Therefore, we know that v is the unique function that satisfies the Bellman equation. It is not as straightforward to prove that v = V; it turns out that it could be the case that V is not measurable, meaning that it could not satisfy the Bellman equation (the example is due to Blackwell and can be found in SLP). We will now show that our assumptions rule out this pathological case. To begin, let Π denote the collection of all policies π : A such that π(x) Γ(x), the function Q(B,x) = P (B,x,π(x)) is a stochastic transition on (,F,µ), and r(x,π(x)) is a measurable function with respect to F. Π is nonempty under the conditions that r is continuous, P is continuous (condition 3 guarantees this), and that Γ admits a continuous selection (continuity of Γ guarantees this). A given π Π defines a probability measure over realizations {x t } T t=1 for any finite T defined by Q(B,x) = P (B,x,π(x)) = Pr{x t+1 B x t = x}; call this measure µ T ( ;x 0,π) for each T. That is, ( µ T T t=1 A t ;x 0,π ) = Q(dx T,x T 1 ) Q(dx 1,x 0 ). A 2 A T A 1 15

16 For finite T we therefore have ] 0 [ T t=0 β t r(x t,a t ) = r(x 0,π(x 0 ))+ β β 2.β T r(x 1,π(x 1 ))Q(dx 1,x 0 )+ r(x 2,π(x 2 ))Q(dx 2,x 1 )Q(dx 1,x 0 )+ r(x T,π(x T ))Q(dx T,x T 1 ) Q(dx 1,x 0 ). We need to show that this sum converges to a finite number as T. Since we have assumed that r is bounded by φ, we have r( x,π( x)) φ( x) M for every x and π Π. Take last term in the summation and consider the innermost integral: r(x T,π(x T ))Q(dx T,x T 1 ) r(x T,π(x T )) Q(dx T,x T 1 ) r(x T,π(x T )) φ(x T )Q(dx T,x T 1 ) φ(x T ) M φ(x T )Q(dx T,x T 1 ) { } φ(x ) Mφ(x T 1 )sup x φ(x) Q(dx,x). { } φ(x By assumption βsup ) x φ(x) Q(dx,x) = βa < 1. Thus, the innermost integral is bounded by MAφ(x T 1 ) in absolute value. If we repeat this procedure for every integral we get that the entire last term of the summation is bounded by M (βa) T φ(x) in absolute value. If we then move to the next-to-last term we get it is bounded by M (βa) T 1 φ(x), and so on. Thus, the entire summation is bounded by Mφ(x) ( 1+βA+β 2 A 2 + β T A T) = Mφ(x) 1 (βa)t+1 1 βa. 1 AsT wethereforehavethatthesummationisboundedbymφ(x) andtheinfinite 1 βa expected value is well-defined, since βa < 1. Define [ ] w π (x) = π β t r(x t,π(x t )) x 0 = x, t=0 16

17 the lifetime expected utility at x obtained by using π. The value function is V (x) = sup π Π {w π (x)}; since we know that w π Ξ φ then so is V. For any π Π it is true that there exists a unique V π Ξ φ that satisfies V π (x) = r(x,π(x))+β V π (x )P (dx,x,π(x)) by immediate application of the CMT; that is, V π is the unique fixed point of the strict contraction mapping T π defined as (T π u)(x) = r(x,π(x))+β u(x )P (dx,x,π(x)). Now we need to show that V π = w π ; that is, w π is the unique function in Ξ φ that satisfies w π (x) = (T π w π )(x) = r(x,π(x))+β w π (x )P (dx,x,π(x)). That this is true is again an immediate implication of the CMT along with the definition of w π. Tosumup, wehavedefinedthreefunctions. First, wehavethefixedpointofthebellman equation { } v(x) = max r(x,a)+β v(x )P (dx,x,a). a Γ(x) Second, we have the lifetime return to policy π, which satisfies w π (x) = r(x,π(x))+β v(x )P (dx,x,π(x)). And third, we have the value function V (x) = sup{w π (x)}. π Π We want to show that (i) v = V and (ii) V = w π for some optimal policy π. Take any function u Ξ φ and any policy π Π. It is true that w π u φ T πu u φ 1 θ where θ < 1; this property is immediately obtained using the definition of a strict contraction mapping and the triangle inequality. Also ǫ > 0 there exists a policy π Π such that w π v φ ǫ; that is, there exists a policy that nearly delivers lifetime returns equal to v, which should be easy to accept given the form of the fixed point operators that generate v and w π. 17

18 Now suppose v > V. Since there exists a policy π such that w π is arbitrarily close to v we can find one such that w π > V. This contradicts the fact that V is the value function and therefore the supremum over all such policies. So we know that v V. By the definitions of T and T π we must have T π v Tv = v (because T performs a maximization and T π does not). T π is monotone so that Tπv n v as well. Taking the limit yields w π v, which holds for arbitrary π. If v(x) w π (x) for every x and π, we must have v(x) V (x) = sup{w π (x)}. π Π Therefore, v V. Combined with the previous result we get v = V. To check the optimal policy condition v = w π, note that v(x) = r(x,π (x))+β v(x )P (dx,x,π (x)); that is, π attains the maximum on the RHS of the Bellman equation. Since v = V, we have T π V = T π v = Tv = v = V, so that T π V = V. Thus, the value function is the fixed point of T π, which is equal to w π. Therefore, v = w π. 18