ECON 582: An Introduction to the Theory of Optimal Control (Chapter 7, Acemoglu) Instructor: Dmytro Hryshko

Transcription

1 ECON 582: An Introduction to the Theory of Optimal Control (Chapter 7, Acemoglu) Instructor: Dmytro Hryshko

2 Continuous-time optimization involves maximization wrt to an innite dimensional object, an entire function: y : [t ; t 1 ]! R. max W (x(t); y(t)) x(t);y(t) Z t 1 f(t; x(t); y(t))dt s.t. _x(t) = G(t; x(t); y(t)) x(t) 2 X(t); y(t) 2 Y (t) for all t; x() = x : y(t) is the control function; x(t) state function. Assume one-dimensional state and control variables.

3 Variational arguments Z t max x(t);y(t);x1 W (x(t); y(t)) 1 f(t; x(t); y(t))dt (7.2) s.t. _x(t) = g(t; x(t); y(t)) (7.3) x(t) 2 X; y(t) 2 Y; x() = x ; x(t 1 ) = x 1 : (7.4) A pair of functions x(t); y(t) is called an admissible pair. Suppose t 1 < 1, f and g are continuously dierentiable functions of t, x and y. Variational approach: assume there exists a continuous solution (function) ^y that lies in the interior of Y (corresponding state ^x lies everywhere in X).

4 W (^x(t); ^y(t)) W (x(t); y(t)) for any admissible pair (x(t); y(t)). ^y(t); ^x(t) never hits the boundary and does not involve discontinuities. Take an arbitrary function (t) and let " 2 R. A variation of the function ^y(t): y(t; ") = ^y(t) + "(t) 2 IntY for all t 2 [; t 1 ] and " 2 [ " ; " ] so that y(t; ") constitutes a feasible variation. " should be suciently small. x(t; ") is the path of the state variable corresponding to the path of the control y(t; ").

5 for all t 2 [; t 1 ], with x(; ") = x. Dene Z t 1 W (") W (t; x(t; "); y(t; ")) = _x(t; ") = g(t; x(t; "); y(t; ")) (7.5) f(t; x(t; "); y(t; "))dt: (7.6) Since ^y(t) is optimal W () W (") for all " 2 [ " ; " ]. Rewrite (7.5) as g(t; x(t; "); y(t; ")) _x(t; ") = : Thus, for any function : [; t 1 ]! R, Z t 1 (t) [g(t; x(t; "); y(t; ")) _x(t; ")] dt = (7.7) {z }

6 Suppose ( ) is continuously dierentiable. This function is the costate variable. Add (7.7) to (7.6): Z t 1 W (") = ff(t; x(t; "); y(t; ")) + (t) [g(t; x(t; "); y(t; ")) _x(t; ")]gdt (7.8) Integration by parts: For dierentiable functions f(x); g(x); F (x); G(x) so that F (x) = f(x) and G (x) = g(x) Z b a F (x)g(x)dx = F (b)g(b) F (a)g(a) Z b a G(x)f(x)dx: Z b a {z } =j b F (x)g(x) a Note that d (F (x)g(x)) R = f(x)g(x) + F (x)g(x). Integrating, dx d +R b (F (x)g(x))dx = a dx G(x)f(x)dx b F (x)g(x)dx. a

7 Z t 1 Thus, (t) _x(t; ")dt = (t 1 )x(t 1 ; ") () x(; ") {z } =x Z t 1 _(t)x(t; ")(t)dt Z t 1 W (") = ff(t; x(t; "); y(t; ")) + (t)g(t; x(t; "); y(t; ")) + x(t; _ ")gdt + ()x (t 1 )x(t 1 ; "): (7.9) Denote partial derivatives of x and y wrt " as x " and y ". Dierentiate W (") wrt " and evaluate the derivative at " =. It should be the case that W () = for all (t):

8 Z t W 1 (") = (f x + g x + )x _ " dt Z t 1 + (f y + g y )y " dt (t 1 )x " (t 1 ; "): Z t W 1 () = f x (t; ^x(t); ^y(t)) + (t)g x (t; ^x(t); ^y(t)) + (t) _ = Z t 1 + (f y (t; ^x(t); ^y(t)) + (t)g y (t; ^x(t); ^y(t))) (t)dt {z } {z } = {z } = (t 1 )x " (t 1 ; ) = : x " (t; )dt

9 Necessary conditions _(t) = [f x (t; ^x(t); ^y(t)) + (t)g x (t; ^x(t); ^y(t))] (7.11) f y (t; ^x(t); ^y(t)) + (t)g y (t; ^x(t); ^y(t)) = (7.12) _x(t) = g(t; x(t); y(t)) (7.3) (t 1 ) = If, instead, the terminal value of x 1 is xed, the necessary conditions are (7.3), (7.11), and (7.12) (Theorem 7.2).

10 Example 7.1 Consider a consumer living between dates and 1; starts with assets a() >, earns interest r, and receives a constant ow of labor earnings w. Assume a(t). max fc(t);a(t)g 1 t= Z 1 exp( t)u(c(t))dt s.t. _a(t) = ra(t) + w c(t) a(t) ; a() given: Consumption is the control variable, asset holdings are the state variable. The (reasonable) terminal value of assets a 1 = a(1) =.

11 (7.11): _(t) = r(t) (7.12): exp( t)u (^c(t)) r(t) = : From (7.11), (t) = exp( rt)(). Plugging this result into (7.12), u (^c(t)) = r() exp (( r)t), and ^c(t) = (u ) 1 [r() exp (( r)t)] :

12 ^c(t) = (u ) 1 [r() exp (( r)t)] : Since u( ) is concave, (u ) 1 is a decreasing function of the argument. Thus, When = r consumption prole is at; when > r consumption is decreasing (front-loaded prole); when < r consumption is increasing (back-loaded prole).

13 The Hamiltonian and the Maximum Principle By analogy with the Lagrangian, we can construct the Hamiltonian: H(t; x(t); y(t); (t)) f(t; x(t); y(t)) + (t)g(t; x(t); y(t)): (7.16) Since f and g are continuously dierentiable so is H. Denote the partial derivatives of H wrt x(t), y(t), and (t) by H x, H y, H.

14 Theorem 7.4 Consider the problem of maximizing (7.2) subject to (7.3) and (7.4), with f and g continuously dierentiable. Suppose this problem has an interior continuous solution (^x(t); ^y(t)). Then there exists a continuously dierentiable function (t) such that the optimal control ^y(t) and the corresponding path of the state ^x(t) satisfy the necessary conditions: H y (t; ^x(t); ^y(t); (t)) = for all t 2 [; t 1 ] (7.17) _(t) = H x (t; ^x(t); (t)) for all t 2 [; t 1 ] (7.18) _x(t) = H (t; ^x(t); (t)) for all t 2 [; t 1 ]; (7.19) with x() = x and (t 1 ) =. Moreover, the Hamiltonian satises the Maximum Principle H(t; ^x(t); ^y(t); (t)) H(t; ^x(t); y; (t)) for all y 2 Y; for all t 2 [; t 1 ].

15 Innite-Horizon Optimal Control max W (x(t); y(t)) x(t);y(t) Z 1 f(t; x(t); y(t))dt (7.32) s.t. _x(t) = g(t; x(t); y(t)) (7.33) x(t) 2 X; y(t) 2 Y; x() = x ; lim t!1 b(t)x(t) x 1 (7.34)

16 Dene the value function, analogous to the value function in discrete-time dynamic programming: V (t ; x(t )) = max x(t);y(t) Z 1 f(t; x(t); y(t))dt (7.35) t s.t. (7.33) and (7.34). V (t ; x(t )) gives the optimal value of the dynamic maximization problem starting at time t with state variable x(t ). Thus, V (t ; x(t )) = Z 1 f(t; ^x(t); ^y(t))dt: t

17 Principle of Optimality Suppose that the pair (^x(t); ^y(t)) is a solution to (7.32) s.t. (7.33) and (7.34). Then Z t 1 V (t ; x(t )) = f(t; ^x(t); ^y(t)) + V (t 1 ; ^x(t 1 )) (7.38) t

18 Hamilton-Jacobi-Bellman Equation Let V (t; x) be as dened in (7.35). Then, under some conditions, when V (t; x) is dierentiable in (t; x), the optimal pair (^x(t); ^y(t)) satises the HJB (t; (t; ^x(t)) f(t; ^x(t); ^y(t)) + + g(t; ^x(t); (7.42) for all t 2 R. R t Proof: Note that V (t ; x ) = f(s; ^x(s); ^y(s))ds + V (t; ^x(t)) t for all t. Dierentiate wrt (t; (t; ^x(t)) f(t; ^x(t); ^y(t)) _x(t) = {z} =g(t;^x(t);^y(t))

19 Transversality condition for innite-horizon problems Theorem 7.12 Suppose that the problem of maximizing (7.32) s.t. (7.33) and (7.34), with f and g continuously dierentiable, has a piecewise continuous optimal control ^y(t) with the path of state variable ^x(t). Let V (t; x(t)) be the value function (t;^x(t)) in (7.35). Suppose that lim t!1 =. Let H(t; x; y; be given by (7.16). Then the pair (^x(t); ^y(t)) satises the necessary conditions (7.39){(7.41) and the transversality condition lim H(t; ^x(t); ^y(t); (t)) = : (7.56) t!1

20 Discounted Innite-Horizon Optimal Control In growth models, utility is discounted exponentially. max W (x(t); y(t)) x(t);y(t) s.t. Z 1 exp( t)f(x(t); y(t))dt; > (7.6) _x(t) = g(t; x(t); y(t)) (7.61) x(t) 2 IntX(t); y(t) 2 IntY (t); x() = x ; lim t!1 b(t)x(t) x 1 b : R +! R + ; lim b(t) < 1: t!1 (7.62)

21 The current-value Hamiltonian The Hamiltonian in this case: H(t; x(t); y(t); (t)) = exp( t)f(x(t); y(t)) + (t)g(t; x(t); y(t)) where {z } current-value Hamiltonian = exp( t) [f(x(t); y(t)) + (t)g(t; x(t); y(t))] ; (t) exp(t)(t): (7.63) We can work with the current-value Hamiltonian: ^H(t; x(t); y(t); (t)) f(x(t); y(t)) + (t)g(t; x(t); y(t)): (7.64)

22 Maximum Principle for Discounted Innite-Horizon Problems Theorem 7.13 Suppose the problem of maximizing (7.6) subject to (7.61) and (7.62). Let ^H(t; x; y; ) be the current-value Hamiltonian given by (7.64). Then except at points of discontinuity of ^y(t), the optimal control pair (^y(t); ^x(t)) satises the following necessary conditions: ^H y (t; ^x(t); ^y(t); (t)) = for all t 2 R + (7.65) (t) _(t) = ^Hx (t; ^x(t); ^y(t); (t)) for all t 2 R + (7.66) _x(t) = ^H (t; ^x(t); ^y(t); (t)) for all t 2 R + ; x() = x ; lim b(t)x(t) t!1 x 1 (7.67) hexp( t) ^H(t; ^x(t); ^y(t); (t)) i lim t!1 where (7.68) is the transversality condition. = : (7.68)

23 Under certain conditions, stronger transversality condition takes the form: lim [exp( t)(t)^x(t)] = : (7.69) t!1