Economics 2010c: Lecture 2 Iterative Methods in Dynamic Programming David Laibson 9/04/2014
Outline: 1. Functional operators 2. Iterative solutions for the Bellman Equation 3. Contraction Mapping Theorem 4. Blackwell s Theorem (Blackwell: 1919-2010, see obituary) 5. Application: Search and stopping problem
1 Functional operators: Sequence Problem: Find ( ) such that ( 0 )= sup { +1 } =0 subject to +1 Γ( ) with 0 given. X =0 ( +1 ) Bellman Equation: Find ( ) such that ( ) = sup { ( +1 )+ ( +1 )} +1 Γ( )
Bellman operator operating on function is defined ( )( ) sup { ( +1 )+ ( +1 )} +1 Γ( ) Definition is expressed pointwise for one value of butappliestoall values of Operator maps function to a new function. Operator maps functions; itisreferredtoasafunctional operator. The argument of the function may or may not be a solution to the Bellman Equation.
Let be a solution to the Bellman Equation. If = then = ( )( ) = sup +1 Γ( ) { ( +1)+ ( +1 )} = ( ) Function is a fixed point of ( maps to ).
2 Iterative Solutions for the Bellman Equation 1. (Guess a solution from last lecture. This is not iterative.) 2. Pick some 0 and analytically iterate 0 until convergence. (This is really just for illustration.) 3. Pick some 0 and numerically iterate 0 until convergence. (This is the way that iterative methods are used in most cases.)
What does it mean to iterate? ( )( ) = sup +1 Γ( ) { ( +1)+ ( +1 )} ( ( ))( ) = sup +1 Γ( ) { ( +1)+ ( )( +1 )} ( ( 2 n ))( ) = sup +1 Γ( ) ( +1 )+ ( 2 )( +1 ) o. =. ( ( ))( ) = sup +1 Γ( ) { ( +1)+ ( )( +1 )}
What does it mean for functions to converge? Notation: lim 0 = Informally, as gets large, the set of remaining elements of the sequence of functions n 0 +1 0 +2 0 +3 0 o are getting closer and closer together. What does it mean for one function to be close to another? maximum distance between the functions is bounded by The We will not worry about these technical issues in this mini-course.
3 Contraction Mapping Theorem Why does converge as? Answer: is a contraction mapping. Definition 3.1 Let ( ) be a metric space and : be a function mapping into itself. is a contraction mapping if for some (0 1) ( ) ( ) for any two functions and. Intuition: is a contraction mapping if operating on any two functions moves them strictly closer together: and are strictly closer together than and
Remark 3.1 A metric (distance function) is just a way of representing the distance between functions (e.g., the supremum pointwise gap between the two functions).
Theorem 3.1 If ( ) is a complete metric space and : is a contraction mapping, then: 1) has exactly one fixed point 2) for any 0 lim 0 = 3) 0 has an exponential convergence rate at least as great as ln( )
Consider the contraction mapping ( )( ) ( )+ ( ) with (0 1) ( )( ) = ( )+ ( ) ( ( ))( ) = 2 ( )+ ( )+ ( ) ( ( 2 ))( ) = 3 ( )+ 2 ( )+ ( )+ ( ) ( )( ) = ( )+ 1 ( )+ + ( ) lim( )( ) = ( ) (1 ) So fixed point, ( ) is ( ) = ( ) (1 ) Note that ( )( ) = ( )
ProofofContractionMappingTheorem Broad strategy: 1. Show that { 0 } =0 is a Cauchy sequence. 2. Show that the limit point is a fixed point of. 3. Show that only one fixed point exists. 4. Bound the rate of convergence.
Choose some 0 Let = 0 Since is a contraction Continuing by induction, ( 2 1 )= ( 1 0 ) ( 1 0 ) ( +1 ) ( 1 0 ) We can now bound the distance between and when ( ) ( 1 )+ + ( +2 +1 )+ ( +1 ) h 1 + + +1 + i ( 1 0 ) = h 1 + + 1 +1 i ( 1 0 ) So { } =0 is Cauchy. X 1 ( 1 0 )
Since is complete We now have a candidate fixed point To show that = note ( ) ( 0 )+ ( 0 ) ( 1 0 )+ ( 0 ) These inequalities must hold for all And both terms on the RHS converge to zero as So ( ) =0 implying that = X
Now we show that our fixed point is unique. Suppose there were two fixed points: 6= Then = and = (since fixed points). Also have ( ) ( ) (since is contraction) So, ( )= ( ) ( ) Contradiction. So the fixed point is unique. X
Finally, note that ( 0 ) = ( ( 1 0 ) ) ( 1 0 ) So ( 0 ) ( 0 ) follows by induction. X This completes the proof of the Contraction Mapping Theorem.
4 Blackwell s Theorem These are sufficient conditions for an operator to be contraction mapping. Theorem 4.1 (Blackwell s sufficient conditions) Let < and let ( ) be a space of bounded functions : <, with the sup-metric. Let : ( ) ( ) be an operator satisfying two conditions: 1) (monotonicity) if ( ) and ( ) ( ), then ( )( ) ( )( ) 2) (discounting) there exists some (0 1) such that [ ( + )]( ) ( )( )+ ( ) 0 Then is a contraction with modulus
Remark 4.1 Note that is a constant and ( + ) is the function generated by adding to the function Remark 4.2 Blackwell sconditionsaresufficient but not necessary for to be a contraction.
ProofofBlackwell stheorem: For any ( ) ( ) ( )+ ( ) We ll write this as + ( ) So property 1) and 2) imply ( + ( )) + ( ) ( + ( )) + ( )
Combining the last two lines, we have ( ) ( ) ( )( ) ( )( ) ( ) sup ( )( ) ( )( ) ( ) ( ) ( ) X
Example 4.1 Check Blackwell conditions for Bellman operator in a consumption problem (with stochastic asset returns, stochastic labor income, and a liquidity constraint). ( )( ) = sup [0 ] n ( )+ ( +1 ( )+ +1 ) o Monotonicity. Assume ( ) ( ). Suppose is the optimal policy when the continuation value function is ( )( ) = sup [0 ] n ( )+ ( +1 ( )+ +1 ) o = ( )+ ( +1 ( )+ +1) ( )+ ( +1 ( )+ +1) n ( )+ ( +1 ( )+ +1 ) o sup [0 ] = ( )( )
Discounting. Adding a constant ( ) to an optimization problem does not affect optimal choice, so n h [ ( + )]( ) = sup ( )+ ( +1 ( )+ +1 )+ io [0 ] n = sup ( )+ ( +1 ( )+ +1 ) o + [0 ] = ( )( )+
5 Application: Search/ stopping. An agent draws an offer, from a uniform distribution with support in the unit interval. The agent can either accept the offer and realize net present value (ending the game) or the agent can reject the offer and draw again a period later. All draws are independent. Rejections are costly because the agent discounts the future with discount factor. This game continues until the agent receives an offer that she is willing to accept. The Bellman equation for this problem is: ( ) =max{ h ( 0 ) i }
In the first lecture, we showed that the solution would be a stationary threshold: REJECT if ACCEPT if This implies that ( ) = ( if if ) Weshowedthatif = 1 µ 1 q1 2 then, ( ) solves the Bellman Equation.
Iteration of Bellman Operator starting with 0 ( ) =0 This is the Bellman Operator for our problem: ( )( ) max{ h ( 0 ) i } Iterating the Bellman Operator once on 0 ( ) yields 1 ( ) = ( 0 )( ) = max{ h 0 ( 0 ) i } = max{ 0} = So associated threshold cutoff is 0 In other words, accept all offers greater than or equal to 0 since the continuation payoff function is 0 =0
General notation. Iteration of the Bellman Operator,. ( ) = ( 0 )( ) = = ( 1 0 )( ) max{ h ( 1 0 )( 0 ) i } Let h ( 1 0 )( 0 ) i, which is the continuation payoff for ( ) is also the cutoff threshold associated with ( ) =( 0 )( ) ( ) ( ) =( if 0 )( ) = if is a sufficient statistic for ( 0 )( )
Another illustrative example. Iterate the functional operator on 0 2 ( ) = ( 2 0 )( ) = ( 0 )( ) = max{ ( 0 )( 0 )} = max{ 1 ( 0 )} = max{ 0 } 2 = 0 = 2 2 ( ) =( 2 0 )( ) =( 2 2 2 )
The proof itself: ( 1 0 )( ) =( 1 if 1 if 1 ) = ( 1 0 )( 0 ) = = 2 " Z 1 1 ( ) + =0 ³ 2 1 +1 Z =1 = 1 ( ) # Set = 1 to confirm that this sequence converges to µ lim = 1 1 q1 2
Iterating functional for search/stopping problem with discount rate =.1 = -ln(delta) 0.7 0.6 0.5 xn value 0.4 0.3 x 2 0.2 0.1 x 1 0 1 2 3 4 5 6 7 8 9 10 iteration number
Review of Today s Lecture: 1. Functional operators 2. Iterative solutions for the Bellman Equation 3. Contraction Mapping Theorem 4. Blackwell s Theorem 5. Application: Search and stopping problem (analytic iteration)