The Principle of Optimality

Transcription

1 The Principle of Optimality Sequence Problem and Recursive Problem Sequence problem: Notation: V (x 0 ) sup {x t} β t F (x t, x t+ ) s.t. x t+ Γ (x t ) x 0 given t () Plans: x = {x t } Continuation plans x = {x t } t= Set of feasible plans: Value of a plan Recursive problem: Some terminology: Π (x 0 ) = { x : x t+ Γ (x t ) t} u ( x) = lim T T β t F (x t, x t+ ) V (x) = sup F (x, y) + βv (y) x,y s.t. y Γ (x) (2) The Functional Equation (2) is called a Bellman equation. x is called a state variable. G (x) = {y Γ (x) : V (x) = F (x, y) + βv (y)} is called a policy correspondence. It spells out all the values of y that attain the maximum in the RHS of (2).

2 If G (x) is single-valued (i.e. there is a unique optimum), G is called a policy function. Questions about the recursive problem:. Does (2) have a solution? 2. Is it unique? 3. How do we find it? Questions about the relationship between () and (2):. Is it the case that V (defined from the sequence problem) satisfies (2)? (SLP 4.2) 2. (When) is it the case that a solution to (2) is V? (SLP 4.3) 3. If x solves (), is it the case that it maximizes the RHS of (2), i.e. V (x t ) = F ( x t, xt+) ( ) + βv x t+? (SLP 4.4) 4. If x maximizes the RHS of (2), i.e. V (x t ) = F ( x t, xt+) ( ) + βv x t+ then is it the case that it attains the supremum of problem ()? (SLP 4.5) The assumptions required for each of these statements to be true are slightly different There is more that one set of assumptions that suffice for some of the results. V solves Functional Equation Assumption. Γ(x) is nonempty for all x X. Assumption 2. lim T t βt F (x t, x t+ ) exists for all x Π(x 0 ). Proposition. (SLP 4.2) Suppose Assumptions () and () hold and let V be defined by (). V satisfies (2). Assumption 3. V (x) < for all x X. Assumption 4. For any x 0, there exists a plan x Π(x 0 ) such that u( x) = V (x 0 ). Proof. Proof of Proposition () for the case where Assumptions 3 and 4 hold. See SLP for the general case. 2

3 Let x Γ (x 0 ) and consider sequences x Π (x ) V solving () V (x 0 ) F (x 0, x ) + βu ( x ) x Π (x ) Taking sup x Π(x ) on both sides, we get V (x 0 ) F (x 0, x ) + βv (x ) x Γ (x 0 ) (3) Now the opposite inequality: Let x Π (x 0 ) be such that u ( x) = V (x 0 ). This plan exists by Assumption 4 Then V (x 0 ) = u ( x) = F (x 0, x ) + βu ( x ) where x = {x, x 2,... } Γ (x ) By definition u ( x ) V (x ) Hence there exists some x Γ (x 0 ) such that V (x 0 ) F (x 0, x ) + βv (x ) (4) (3) and (4) imply V solves (2)..2 A Solution to the Functional Equation equals V Proposition 2. (SLP 4.3) Suppose Assumptions () and () hold and let V be a solution to (2) that has the following property: then V = V. lim T βt V (x T ) = 0 xπ (x 0 ), x 0 X (5) Remark. Notice the extra assumption 5. Propositions and 2 are NOT exact converses. Assumption 5. V (x) < for all x X. Assumption 6. For any x, there exists y Γ(x) such that V (x) = F (x, y) + βv (y). Proof. Proof for the case where Assumptions 5 and 6 hold. See SLP for the general case. 3

4 V solves (2) V (x 0 ) F (x 0, x ) + βv (x ) x Γ (x 0 ) F (x 0, x ) + βf (x, x 2 ) + β 2 V (x 2 ) x Γ (x 0 ), x 2 Γ (x )... T β t F (x t, x t+ ) + β T V (x T ) x Π (x 0 ) Take limits and use condition (5): V (x 0 ) u ( x) x Π (x 0 ) (6) Now take a sequence x Π (x 0 ) such that V (x t ) = F (x t, x t+ ) + βv (x t+ ) (by Assumption 6, such a sequence exists) Substituting forward: T V (x t ) = β t F (x t, x t+ ) + β T V (x T ) Taking limits and using condition (5): V (x t ) = u ( x) (7) Conditions (6) and (7) mean that V = V.3 The optimal plan satisfies the Functional equation Proposition 3. (SLP 4.4) Suppose Assumptions () and () hold and suppose V (x 0 ) = u ( x ) for some x Π (x 0 ). Then V (x t ) = F ( x t, xt+) ( ) + βv x t+ Proof. (Following SLP) 4

5 We assumed V (x 0 ) = u ( x ) = F (x 0, x ) + βu ( x ) F (x 0, x ) + βu ( x ) x Π (x 0 ) Holds for any x, in particular those with x = x. Therefore u ( x ) u ( x ) x Π (x ) which implies u ( x ) = V (x ) Continuing by induction gives the result for all t..4 Plans that satisfy the Functional Equation are optimal plans Proposition 4. (SLP 4.5) Suppose Assumptions () and () hold and suppose x Π (x 0 ) satisfies V (x t ) = F ( x t, xt+) ( ) + βv x t+ and Then then V (x 0 ) = u ( x ). lim sup T βt V (x T ) 0 (8) Proof. (Following SLP) Substituting forward: V (x 0 ) = F (x 0, x ) + βv (x )... T = β t F ( x t, xt+) + β T V (x T ) Taking limits: Using condition (8): V (x 0 ) = u ( x ) + lim T βt V (x T ) V (x 0 ) u ( x ) (9) 5

6 Now the opposite inequality: since x t Π (x 0 ) V (x 0 ) u ( x ) (0) Conditions (9) (0) imply the result. 2 Some examples 2. Savings with linear utility, βr = and no borrowing constraint Sequence problem: [ ] V (x 0 ) = sup β t β x t x t+ x t+ β x t x 0 given Since there is no borrowing constraint, V (x 0 ) = Functional Equation V (x) = sup y x β x y + βv (y) β There is more than one solution! V (x) is a solution V (x) = x is also a solution: β V (x) = sup y x β β x y + β β y This example violates condition (5): there are feasible plans such that lim T βt V (x T ) > 0 6

7 2.2 Savings with linear utility, βr = and a borrowing constraint Sequence problem: V (x 0 ) = sup x t+ [ ] β t β x t x t+ [ 0, ] β x t x 0 given V (x 0 ) = β x 0 There are many optimal plans. Examples? Functional equation: V (x) = sup x y + βv (y) y [0, x β ] β Consider the following plan: x t+ = β x t This plan maximizes the RHS of the functional equation BUT u ( x) = 0 (because you never consume!) In this example, plan x violates condition (8). 7