Economics 2010c: Lectures 9-10 Bellman Equation in Continuous Time

Economics 2010c: Lectures 9-10 Bellman Equation in Continuous Time David Laibson 9/30/2014

Outline Lectures 9-10: 9.1 Continuous-time Bellman Equation 9.2 Application: Merton s Problem 9.3 Application: Stopping problem 9.4 Boundary condition: Value Matching 9.5 Boundary condition: Smooth Pasting 10.1 Stopping problem revisited 10.2 Solving second order ODE s 10.3Stoppingproblemresolved

1 Continuous-time Bellman Equation Let s write out the most general version of our problem. of motion of the state variable: Begin with equation = ( ) + ( ) Note that depends on choice of control. Using Ito s Lemma, derive continuous time Bellman Equation: ( ) = ( )+ + ( )+ 1 2 2 2 ( ) 2 = ( ) = optimal value of control variable

Note that value function is a second order partial differential equation (PDE). is the dependent variable and and are the independent variables. To solve this PDE need boundary conditions, since many solutions exist.

1.1 Terminal condition. Suppose problem ends at date Then we know that ( )=Ω( ) Can solve the problem using techniques analogous to backwards induction

1.2 Stationary horizon problem. Now value function doesn t depend on ( ) = ( )+ ( ) 0 + 1 2 ( ) 2 00 We have a second-order ordinary differential equation (ODE). In general a large class of functions are consistent with this ODE. To pin down a solution we need to know something about the economics of the value function This will provide constraints that pick out a single solution.

2 Application: Merton s consumption problem Consumer has CRRA utility: ( ) = 1 1 Consumer has two assets. Risk free: return Equity: return + and proportional variance 2 Consumer invests asset share in equities and consumes at rate =[( + ) ] + so, ( ) =[( + ) ] and ( ) =

Bellman Equation and Ito s Lemma: ( ) = max max { ( ) + ( )} ( ( ) + " + ( )+1 2 2 ( )2 2 # ) For this problem, doesn t depend directly on (why?): ( " ( ) =max ( ) + [( + ) ]+1 2 2 2 ( )2 # ) Boundary condition: ( ) = 1 1 So, 1 ½ =max ( ) + [( + ) ] 2 1 2 1 ( ) ¾

First-order conditions: 2 µ 2 1 ( ) 2 =0 0 ( ) = = Simplification implies: = 2 = 1 Plugging this back into last equation on previous page implies: 1 = Ã + 1 1!Ã! + 2 2 2

Interesting case: =1. = So ' 0 05 = 2 = 0 06 1 (0 16) 2 =2 34! How many households place 2.34 times their wealth (including human capital) in the stock market?

3 Application: Stopping problem Every instant the firm decides whether to continue and get instantaneous flow payoff, ( ) or to stop and get termination payoff, Ω( ) We ll assume that ( ) is increasing in Continuous time value function is given by (limit as 0): ( ) =max n ( ) +(1+ ) 1 ( 0 0 ) Ω( ) o Assume that = ( ) + ( )

The solution to this problem is a stopping rule if ( ) continue if ( ) stop Motivation: Irreversibly closing a production facility.

Distinguish continuation region and stopping region of state space. The stopping region is the set of points h i such that ( ) ( ) In continuation region, use Ito s Lemma to characterize the value function: ( ) = ( ) + ( ) " = ( ) + + ( )+1 2 ( )2 2 2 # We need to solve this partial differential equation.

4 Value Matching Recall that Ω( ) is the termination payoff. One set of boundary conditions is for all boundary points, h ( ) i. lim ( ) ( ) =Ω( ( ) ) This is referred to as a value matching condition (continuity of the value function at the boundary).

Heuristic proof. There exists a neighborhood ( + ) in which: ( ) = ( ) +(1+ ) 1 ( 0 0 ) = ( ) +(1+ ) 1 " ( + 0 )+Ω( 0 ) 2 = 1 2 lim ( )+1 ( ) 2 Ω( ( ) )+ = lim ( )+1 ( ) 2 This implies that lim ( ) = lim ( ) which implies " Ω( ( ) ) ( ) ( )+1 2 " lim ( ) ( ) Ω( ( ) ) lim ( ) ( ) =Ω( ( ) ) # + lim ( ) ( ) # #

5 Smooth Pasting We could now apply backwards induction techniques if we knew the free boundary ( ). To pin down the free boundary, we need another boundary condition, which is derived from optimization and called smooth pasting: ( ( ) )=Ω ( ( ) )

Heuristic proof. Suppose slopes to left and right of stopping point are given by and + where 0 This is a convex kink. If you stop now you get payoff Ω Ifyouwaitanotherinstantandthenstop,yougetpayoff: +(1+ ) 1 " So the net payoff of waiting is +(1+ ) 1 " Ω 2 Ω 2 + ( + ) 2 + ( + ) # 2 " = +(1+ ) 1 Ω # Ω + # 2 Ω Multiply through by (1 + ) simplify and remove terms of at least order to find: (1 + ) + " Ω + 2 # (1 + )Ω = 2

Intuition: At the boundary, the agent should be indifferent between continuation and stopping. If value functions don t smooth paste at ( ), thenstoppingat ( ) can t be optimal. Better to stop an instant later. If there is a (convex) kink at the boundary, then the gain from waiting is in and the cost from waiting is in So there can t be a kink at the boundary. Hence: ( ( ) )=Ω ( ( ) ) How would you rule out concave kinks?

6 Stopping Problem Revisited Assume that a continuous time stochastic process ( ) is an Ito process, = + You might imagine that is the price of a commodity produced by this firm. While in operation, the firm has flow profit ( ) = Assume that the firm can always costlessly (permanently) exit the industry and realize termination payoff Ω =0 Intuitively, this firm will have a stationary stopping rule, if continue if stop (exit)

Let 0 = ( + ) and let represent the interest rate. So ( ) =max n +(1+ ) 1 ( 0 ) 0 o In the stopping region, In the continuation region, ( ) =0 ( ) = +(1+ ) 1 ( 0 ) (1 + ) ( ) =(1+ ) + ( 0 ) ( ) =(1+ ) + ( 0 ) ( ) Multiply out and let 0 Terms of order 2 =0 ( ) = + ( ) (*)

Now substitute in for ( ) using Ito s Lemma: " ( ) = + ( )+1 2 ( )2 2 2 " 0 + 22 # 00 = Substituting this expression into equation (*), we find ( ) = + " 0 + 22 # 00 which is a second-order ordinary differential equation, = + 0 + 2 2 00 #

What are our boundary conditions? Value matching: Smooth pasting: ( )=0 0 ( )=0 As the option value of exiting goes to zero, so converges to the value associated with the policy of never exiting the industry. Hence, lim 1 ( ) ³ =1 + We ll derive this equation later. We could also have written lim 0 ( ) = 1

7 Solving second order ODE s To solve the differential equations that come up in economics, it is helpful to recall a few general results from the theory of differential equations. Consider a generic second order ordinary differential equation: 00 ( )+ ( ) 0 ( )+ ( ) ( ) = ( ) This equation is referred to as the complete equation. Note that ( ), ( ), and ( ), are given functions. We are trying to solve for with independendent variable. Now consider a reduced equation in which ( ) is replaced by 0. 00 ( )+ ( ) 0 ( )+ ( ) ( ) =0 Solving this reduced differential equation will enable us to solve the complete equation. We begin by characterizing the solution of the reduced equation.

Theorem 7.1 Any solution, ˆ ( ), of the reduced equation can be expressed as a linear combination of any two solutions of the reduced equation, 1 and 2 that are linearly independent. ˆ ( ) = 1 1 ( )+ 2 2 ( ) Note that two solutions are linearly independent if there do not exist constants 1 and 2 such that 1 1 ( )+ 2 2 ( ) =0 for all Theorem 7.2 The general solution of the complete equation is the sum of any particular solution of the complete equation and the general solution of the reduced equation.

8 Stopping problem resolved Recall the differential equation that characterizes the continuation region = + 0 + 2 2 00 (complete equation) Consider the reduced equation, 0= + 0 + 2 2 00 (reduced equation) Our first challenge is to find solutions of this equation. Consider the class, To confirm that this is in fact a solution, differentiate and plug in to find 0= + + 2 2 2

This implies that Apply quadratic formula, 0= + + 2 2 2 = ± q 2 +2 2 2 Let + represent the positive root and let represent the negative root. Any solution to the reduced equation can be expressed + + + (general solution to the reduced equation)

The general solution to the complete equation can be expressed as the sum of a particular solution to the complete differential equation and the general solution to the reduced equation. To find a particular solution, consider the payoff function of the policy never leave the industry. The value of this policy is Note that [ ( )] = Z 0 ( ) (0) + Z 0 ( ) Z = (0) + [ + ( )] 0 Z = (0) + + ( ) 0 = (0) +

The value of the policy never leave is derived with integration by parts: Z 0 ( ) = Z 0 = (0) + Ã [ (0) + ] " 1! # 0 Z 0 1 = 1 (0) + Hence, our (candidate) particular solution takes the form ( ) = 1 + Confirm that this is a solution to the complete differential equation. " 1 Ã +!# Ã! = + 1 + 2 2 0 X

We now can draw all of these pieces together. solution to the reduced equation First, we have our general + + + where the roots are given by q + = ± 2 +2 2 2 Second, we have our particular solution: 1 + So the general solution of the complete equation is ( ) = 1 Ã Ã +!! + + + +

Finally, we have boundary conditions. Value matching: ( )=0 Smooth pasting: 0 ( )=0

We know lim 0 ( ) = 1 which implies + =0 Value matching and smooth pasting imply ( )= 1 Ã +! + =0 (1) 0 ( )= 1 + =0 (2) Equation (1) implies = 1 Ã! +

Plugging this expression into equation (2) implies Hence, We have, Hence, 1 Ã +! = 1 =0 q = 2 +2 2 = + 2 2 q 2 +2 2 0

Some interesting special cases (see problem set): ( =0) = 2 0 lim = lim = 0 lim 0 = lim ( if 0 0 if 0 = ( if 0 lim 0 = lim = 0 2 2 if 0 ) )

Illustrative calibration: ( =0) = 2 = 2 02 = 5 Wait until the gets 5 standard deviations below the break-even threshold ( =0) before shutting down.

Finally, now that we have our solution, it is easy to calculate the option value of stopping: Ã! ( ) = ( ) 1 + Substituting in for ( ) yields ( ) = ³ if + if 1 So as the option value of stopping goes to zero.