Essentials of optimal control theory in ECON 4140

Size: px

Start display at page:

Download "Essentials of optimal control theory in ECON 4140"

Merryl Taylor
5 years ago
Views:

1 Essentials of optimal control theory in ECON 4140 Things yo need to know (and a detail yo need not care abot). A few words abot dynamic optimization in general. Dynamic optimization can be thoght of as finding the best fnction rather than the best point. We have two tools: Dynamic programming. In ECON4140, that is sed for discrete-time dynamic optimization. The method involves the optimal vale. If vale depends on state x R and time t and the optimal vale is V t (x t ), then one trades off immediate payoff f (direct tility) against ftre optimal vale (indirect tility) V t+1 (x t+1 ). If or control at time t is t, f = f(t, x, ) depends on time, state and control, and so does x t+1 = g(t, x, ), then the best we can do with state x t = x is to maximize f(t, x, ) + V t+1 (g(t, x, )) wrt. or control. If V t+1 is a known fnction, that gives s the optimal t in «feedback form», as a fnction 1 of time and state. In finite horizon T, we can recrse backwards with known V T, then V T 1,... Infinite horizon models has some appealing properties, one of which is that if there is no explicit time in the dynamics and only exponential disconting then the time-dimension vanishes. Using a crrent-vale formlation β t f cv = f and assming f cv a fnction of state and control only (no «t») as well as x t+1 = g(x t, t ) (also withot explicit t), we get the Bellman eqation V (x) = max { f cv (x, ) + βv (g(x, )) } with the same V on the LHS and the RHS (there are infinitely many steps left both today and tomorrow). The optimal is given implicitly in terms of V. Calcls of variations or the Pontryagin maximm principle. These methods work by varying the strategy, and do not reqire the vale fnction. There is no «V» in the Hamiltonian nor in the Eler eqation, there is only state and control (and in the calcls of variations method, the control is ẋ). The discrete-time Eler eqation (yo have seen it in dynamic macro?) does in a way the same thing: Consider a time-homogeneos problem with crrentvale formlation max t=0 βt f cv (x t, x t+1 ). The first-order condition for optimal state x τ at a certain time τ N is fond by taking the two terms that involve it (namely β τ 1 f cv (x τ 1, x τ )+β τ f cv (x τ, x τ+1 )), differentiating wrt. x τ and ptting eqal to zero. Notice: no «V» in that condition. ECON4140 ses dynamic programming in discrete time and the maximm principle in continos time. There exist a continos-time Bellman eqation (often sed in stochastic systems) and a discrete-time maximm principle, bt those are not at all crriclm. 1 if the maximizer is not niqe, is it then a fnction? Then, it does not matter which one we choose, so we can pick a fnction. 1

2 The maximm principle. Necessary conditions. Let the timeframe [t 0, t 1 ] be given 2. Consider the problem to maximize wrt. (t) U the fnctional t 1 t 0 f(t, x(t), (t)) dt where x starts at x(t 0 ) = x 0 (given) and evolves as ẋ(t) = g(t, x(t), (t)); we shall consider the following three possible terminal conditions (i) x(t 1 ) = x 1 (given), (ii) x(t 1 ) x 1, or (iii) x(t 1 ) free. Imagine a trade-off between immediate payoff (or, direct tility) today f(t, x, ) and growth ẋ of the state. With ẋ(t) = g(t, x, ), we weigh immediate payoff at one 3 and weigh growth at p = p(t). Or control is then set to maximize the Hamiltonian H(t, x,, p) = f(t, x, ) + pg(t, x, ) (over in the control region U which we are allowed to choose from it need not be interior). The rest of the maximm principle is abot determining a weight p sch that this gives s a soltion to the dynamic problem. p is often referred to as the «adjoint variable» or «costate» or sometimes «shadow price». The following gives necessary conditions: 0: Form the Hamiltonian H(t, x,, p) = f(t, x, ) + pg(t, x, ). 1: The optimal maximizes H. 2: The adjoint p satisfies ṗ = H (evalated at optimm), with the so-called transversality conditions on p(t 1 ): (i ) no condition on p(t 1 ) if the problem has x(t 1 ) = x 1 ; (ii ) if the problem imposes x(t 1 ) x 1, then p(t 1 ) 0 with eqality if x (t 1 ) > x 1 in optimm; (iii ) if there is no restriction on x(t 1 ), then p(t 1 ) mst be = 0. 3: Also, the differential eqation for x mst hold: an optimal x mst satisfy ẋ (t) = g(t, x (t), (t)) with initial condition x (t 0 ) = x 0 and if applicable, the terminal condition. These conditions may be regarded as a soltion steps recipe althogh in practice it may not be so straightforward as to call it a «cookbook». Next page: 2 there are problems where time can be optimized too. 3 Here there is a theoretical catch which is not exam relevant, except see the second bllet below: Sppose that there is no «optimization», and that there is only one control (t) sch that the terminal condition holds. If yor control has to be reserved to flfill that condition, then yo cannot optimize for tility. Then the weight on f has to be zero. That is the «p 0» constant in the book, which looks a bit akin to the Fritz John type conditions covering the constraint qalification in nonlinear programming. Yo can disregard the p 0 (i.e., pt it eqal to one) for exam prposes. Bt: Do not pt p(t 0 ) eqal to one, becase the constant p 0 is not the same thing as p(t 0 )! (Nor the same as p(0). In case yo wonder what the notation is abot: it is from the case with several states x R n. Then we have an n-dimensional p(t), and the p 0 is then the «zeroeth» dimension.) 2

3 step 0: Form the Hamiltonian H(t, x,, p) = f(t, x, ) + pg(t, x, ). step 1: The optimal maximizes H. Whatever state x and costate p might be, then that gives s a relation between and (t, x, p). With the possible reservation that the maximizer may not be niqe 4, this gives s as a fnction û of (t, x, p) where x = x is the optimal state, and p is the adjoint satisfying the next step. (Note that in practice yo may have to split between cases.) step 2: We have a differential eqation for p: ṗ = H (evalated at optimm), with the so-called transversality conditions on p(t 1 ): (i) In case the terminal vale x(t 1 ) is fixed, there is no condition on p(t 1 ). (ii) In case the problem imposes x(t 1 ) x 1, then we get a complementary slackness condition on p(t 1 ): it is 0, with eqality if x(t 1 ) > x 1 (the latter corresponds to the next item). (iii) If there is no restriction on x(t 1 ), then p(t 1 ) mst be = 0. If we have a fnction û(t, x, p) for the optimal control, then plgging this into H will give ṗ as a fnction of (t, x, p). step 3: Then we have the differential eqation for the state. Inserting û there as well, gives a differential eqation system ẋ = φ(t, x, p), ṗ = ψ(t, x, p) and the conditions on x(t 0 ), x(t 1 ) and p(t 1 ) determine the integration constants. Sfficient conditions. We have two sets of sfficient conditions. Sppose we have fond a pair (x, ) which satisfies the necessary conditions. This pair is a candidate for optimality. We can conclde that it is indeed optimal if it satisfies one of the following: The Mangasarian sfficiency condition: With the p = p(t) that the maximm principle prodces, then H is concave wrt. (x, ) for all t (t 0, t 1 ). The Arrow sfficiency condition: Insert the fnction û(t, x, p) for in the Hamiltonian to get the fnction Ĥ(t, x, p) = H(t, x, û(t, x, p), p). With the p = p(t) that the maximm principle prodces, then Ĥ is concave wrt. x for all t (t 0, t 1 ). The Arrow condition is more powerfl: if Mangasarian applies, then Arrow will always apply. However, the Mangasarian condition cold be easier to verify. 4 in which case several fnctions are possible. For necessary conditions, yo shold consider them all. For sfficient conditions, yo may «gess one and verify that it solves». See the «modified example 2». 3

4 «Example» 1 (not yet [ worked ot completely). A concave problem. In matrix notation, let f(t, x, ) =, k 2 ) ( ) (k1 x 1 (x, )Q( ) ] x 2 e rt where Q = ( q 11 q 12 ) q 12 is symmetric and positive semidefinite with > 0, and let g(t, x, ) = (m 1, m 2 ) ( ) x. Sppose can take any real vale. No matter what the terminal condition on x is, we will have H(t, x,, p) = f +pg is concave in (x, ) regardless of the sign of p, so Mangasarian will apply to the soltion we find as follows 5 : step 0: We have H(t, x,, p) = [ (k1, k 2 ) ( x ) 1 2 (x, )Q( x ) ] e rt + p(m 1, m 2 ) ( x ) = [ k 1 x 1 2 q 11x 2 + (k 2 q 12 x) 1 2 2] e rt + m 1 px + m 2 p step 1: The optimal maximizes H, and can be written as û(t, x, p) where (from the first-order condition for ) û(t, x, p) = (k 2 q 12 x)e rt + m 2 p step 2: The differential eqation for p is ṗ = (q 11 x k 1 + q 12 )e rt m 1 p to be evalated at optimm; that is, we insert x for x and û(t, x, p) for : ṗ = ( q 11 x (k 2 q 12 x )e rt + m 2 p) k 1 + q 12 e rt m 1 p Notice this is linear in (x, p) (we shold reorder coefficients, bt nfortnately it is not homogeneos). Then we have the transversality conditions corresponding to whatever the terminal conditions were for x. step 3: Inserting û in the differential eqation for x, we find ẋ = m 1 x + m 2 û(t, x, p) = m 1 x + m 2 (k 2 q 12 x )e rt + m 2 p Again, this is linear in (x, p). The problem then redces to solving a linear eqation system, which nfortnately is not homogeneos. The crrent-vale formlation below resolves that. 5 Indeed, we can assme U to be an interval as well, and concavity will still hold bt then the problem becomes harder when hits and endpoint. 4

5 Example 2 (H not concave wrt. (x, ), bt Arrow s condition applies) This is nearly one given in class. Sppose can take vales in [0, K] where K > 0 is a constant, and consider the problem max T 0 e δt 2 dt sbject to ẋ =, x(0) = x 0 (a constant > 0) and x(t ) reqired to be 0. We therefore assme x 0 < KT (otherwise, we wold have x(t) 0 atomatically). Let in this problem δ > 0. Notice that the Hamiltonian H(t, x,, p) = e δt 2 p is convex in. That means (a) that the maximizing is either 0 or K, and (b) we cannot se Mangasarian. Bt we can se Arrow: since x does not enter H, then inserting for û (which does not depend on x, as the maximization does not) we get no x in Ĥ. Considering Ĥ a fnction of x only, it is constant, and that is concave. (Not strictly, bt we do not need that.) So whatever we get ot of the following, will indeed be optimal. Let s work ot the steps. step 0: Define H(t, x,, p) = e δt 2 p. step 1: The optimal maximizes H. By convexity, we mst have either the endpoint 0 or the endpoint K, and we jst compare the two, e δt K 2 pk vs. zero. We have K if K > pe δt û = 0 if K < pe δt 0 or K if K = pe δt (the maximization cannot tell which one) This condition can not determine û in the case pe δt = K. If that happens at only one point in time (or never), then it is not a problem, as changing an integrand at a single point does not change any state. If we were to have p(t) = Ke δt on an entire positive interval, we wold be in troble (althogh, by sfficient conditions, we cold try to gess and verify!). step 2: Becase H does not depend on x, then ṗ = 0, so p is a constant P. By the transversality condition, P 0 with eqality if x (T ) > 0. Good news! The «?» case for û will not be an isse: there can only be at most t for which K = P e δt. If there is one sch, then we switch control (and we switch from K to 0... exercise: why?). step 3: We have ẋ = K (if K > P e δt ) or = 0 (if the reverse ineqality holds). We need to determine when we have what. So now we have the conditions, and we can start to «nest» ot what cold happen. Cold we have x(t ) > 0? Then we mst have p(t ) = 0 hence P = 0. Then we wold always have K > P e δt and always = K. Bt then x (T ) = x 0 KT which by assmption is 0. Contradiction! So x(t ) = 0. Indeed, we cannot have K. Therefore, we mst have K = P e δt for some (necessarily niqe) t (0, T ). Then = K on (0, t ) and 0 afterwards. Adjst then t so that we hit zero there: t = x 0 /K. So we mst choose maximally ntil we hit zero, and keep x constant at zero from then on. By Arrow s condition, this is indeed the optimal soltion. 5

6 Modified example 2. Now drop the assmption that δ > 0. The case δ < 0 will not add mch insight we will psh the = K period to the end instead. Bt sppose now δ = 0, and let s see what happens. step 0: Define H(t, x,, p) = 2 p. step 1: The optimal maximizes H. Again, we have endpoint soltion: K if K > p û = 0 if K < p (0 or K if K = p; the maximization cannot tell which one) step 2: Again, p is a constant P 0 (eqal to zero if x (T ) > 0.) step 3: The differential eqation for the state mst hold. Again we get a contradiction if we assme x(t ) > 0. So x(t ) = 0. In particlar, that means we cannot have 0. Therefore, we mst have K P. And we cannot have K, as that yields x (T ) < 0. So K P. With K = P, the necessary conditions cannot tell s whether to se = K. = 0 or Bt we know that we mst = K for precisely long enogh to end p at zero. The necessary conditions cannot tell when to rn at fll throttle. Actally, it does not matter. No matter when, we wold get the same performance Kx 0. Bt that is maybe not so easy to see, was it? Well let s arge as follows: in the case δ > 0, we had = K p to t = x 0 /K. Let s jst make the gess that this is optimal. It does satisfy all the conditions from the maximm principle! By the Arrow condition, it is optimal. (It jst isn t niqely optimal. In fact, all the other «= K for a period totalling x 0 /K in length» will be optimal and Arrow s condition will verify that!) New theory for Monday: Sensitivity! The optimal vale V depends on (t 0, x 0, t 1, x 1 ) althogh no «x 1» if x(t 1 ) is free. With exception for the latter, and to the level of precision of this corse, we have the following sensitivity properties: V 0 = p(t 0 ). V 1 = p(t 1 ) except in the free-end case. Note that the «x 1 > variable is a reqirement yo have to flfll, so the interpretation is that p(t 1 ) is the marginal loss of tightening it by reqiring yo to leave one more nit at the table in the end. 6

7 V t 1 = H(t 1, x 1, (t 1 ), p(t 1 )) is the marginal vale of having one more time nit in the end. That means, that if yo were actally allowed to choose when to stop, then the first-order condition wold be H = 0 at the final time. (Bt for optimal stopping, the sfficient conditions presented herein are no longer valid!) V t 0 = H(t 0, x 0, (t 0 ), p(t 0 )). The mins sign becase increasing t 0 gives yo one nit less of time. Crrent-vale formlation. This is not «crriclm» per se: yo are not reqired to know it (it has been given, bt then with a hint on what to do to get to it). Bt it is helpfl if the rnning tility has exponential disconting and there is no other «explicit time-dependence» and in particlar so in infinite horizon, which is not crriclm. Here is what happens: Sppose that g(t, x, ) does not depend on t directly, and that f(t, x, ) = e rt f cv (x, ), the «cv» for «crrent-vale». Define λ = e rt p. Then H(t, x,, p) = e rt[ f cv + pe rt g ] which eqals e rt H cv (x,, λ) where H cv (x,, λ) = f cv (x, ) + λg(x, ) is called the crrent-vale Hamiltonian. (There is literatre where that is jst called «Hamiltonian» as well.) We have ṗ = d dt λe rt = [ λ rλ]e rt rt Hcv which eqals e, so we get λ rλ = Hcv with the same transversality conditions for λ as for p. The optimal maximizes H cv and will be given as a fnction û cv (x, λ). Example 1 revisited. step 0: Crrent-vale Hamiltonian With crrent-vale formlation, we get H cv (x,, λ) = k 1 x 1 2 q 11x 2 + (k 2 q 12 x) m 1 λx + m 2 λ step 1: The optimal can be written as k 2 q 12 x + m 2 λ step 2: We get the differential eqation λ rλ = q 11 x k 1 + q 12 m 1 λ to be evalated at optimm: λ rλ = q 11 x k 1 + q 12 k 2 q 12 x + m 2 λ Linear in (x, λ), and now the coefficients are constant. step 3: Inserting û in the differential eqation for x, we find ẋ = m 1 x + m 2 k 2 q 12 x + m 2 λ Linear in (x, λ), and now the coefficients are constant. m 1 λ We can solve this system completely. Align the integration constants to x(t 0 ) = x 0 and the terminal/transversality conditions, and we have solved the maximization problem. 7

Optimization via the Hamilton-Jacobi-Bellman Method: Theory and Applications

Optimization via the Hamilton-Jacobi-Bellman Method: Theory and Applications Navin Khaneja lectre notes taken by Christiane Koch Jne 24, 29 1 Variation yields a classical Hamiltonian system Sppose that