CHAPTER 3 THE MAXIMUM PRINCIPLE: MIXED INEQUALITY CONSTRAINTS. p. 1/73

CHAPTER 3 THE MAXIMUM PRINCIPLE: MIXED INEQUALITY CONSTRAINTS p. 1/73

THE MAXIMUM PRINCIPLE: MIXED INEQUALITY CONSTRAINTS Mixed Inequality Constraints: Inequality constraints involving control and possibly state variables. Examples: g(u,t) 0; g(x,u,t) 0. Pure Inequality Constraints of the type h(x,t) 0, i.e., involving only state variables, will be treated in Chapter 4. p. 2/73

PROBLEMS WITH MIXED INEQUALITY CONSTRAINTS 3.1 A Maximum Principle for Problems with Mixed Inequality Constraints State equation: ẋ = f(x,u,t), x(0) = x 0 (1) where x(t) E n and u(t) E m, and f : E n E m E 1 E n is assumed to be continuously differentiable. p. 3/73

PROBLEMS WITH MIXED INEQUALITY CONSTRAINTS CONT. Objective function: max { J = T 0 F(x,u,t)dt + S[x(T),T] }, (2) where F : E n E m E 1 E 1 and S : E n E 1 E 1 are continuously differentiable functions and T denotes the terminal time. For each t [0,T], u(t) is admissible if it is piecewise continuous and satisfies the mixed constraints g(x,u,t) 0, t [0,T], (3) where g: E n E m E 1 E q is continuously differentiable. p. 4/73

TERMINAL STATE The terminal state is constrained by the following inequality and equality constraints: a(x(t),t) 0, (4) b(x(t),t) = 0, (5) where a : E n E 1 E l a and b : En E 1 E l b are continuously differentiable. p. 5/73

SPECIAL CASES An interesting case of the terminal inequality constraints is x(t) Y X, (6) where Y is a convex set and X is the reachable set from the initial state x 0. X={x(T) x(t) obtained by an admissible control u and (1)}. Remarks: 1. The constraint (6) does not depend explicitly on T. 2. The feasible set defined by (4) and (5) need not be convex. 3. (6) may not be expressible by a simple set of inequalities. p. 6/73

FULL RANK CONDITIONS OR CONSTRAINT QUALIFICATIONS rank [ g/ u, diag(g)] = q holds for all arguments x(t), u(t), t, and rank [ a/ x diag(a) b/ x 0 hold for all possible values of x(t) and T. ] = l a + l b The first means that the gradients of active constraints in (3) with respect to u are linearly independent. The second means that the gradients of the equality constraints (5) and of the active inequality constraints in (4) are linearly independent. p. 7/73

HAMILTONIAN FUNCTION The Hamiltonian function H :E n E m E n E 1 E 1 is: H[x,u,λ,t] := F(x,u,t) + λf(x,u,t), (7) where λ E n (a row vector) is called the adjoint vector or the costate vector. Recall that λ provides the marginal valuation of increases in x. p. 8/73

LAGRANGIAN FUNCTION The Lagrangian function L:E n E m E n E q E 1 E 1 as L[x,u,λ,µ,t] := H(x,u,λ,t) + µg(x,u,t), (8) where µ E q is a row vector, whose components are called Lagrange multipliers. Lagrange multipliers satisfy the complimentary slackness conditions µ 0, µg(x,u,t) = 0. (9) p. 9/73

ADJOINT VECTOR The adjoint vector satisfies the differential equation with the boundary conditions λ = L x [x,u,λ,µ,t] (10) λ(t) = S x (x(t),t) + αa x (x(t),t) + βb x (x(t),t), α 0, αa(x(t),t) = 0, where α E l a and β El b are constant vectors. p. 10/73

NECESSARY CONDITIONS The necessary conditions for u (corresponding state x ) to be an optimal solution are that there exist λ, µ, α, and β which satisfy the following: ẋ = f(x, u, t), x (0) = x 0, satisfying the terminal constraints a(x (T), T) 0 and b(x (T), T) = 0, λ = L x [x, u, λ, µ, t] with the transversality conditions λ(t) = S x (x (T), T) + αa x (x (T), T) + βb x (x (T), T), α 0, αa(x (T), T) = 0, the Hamiltonian maximizing condition H[x (t), u (t), λ(t), t] H[x (t), u, λ(t), t] at each t [0, T] for all u satisfying g[x (t), u, t] 0, and the Lagrange multipliers µ(t) are such that L H u u=u (t) := u + µ g u=u u (t) = 0 and the complementary slackness conditions µ(t) 0, µ(t)g(x, u, t) = 0 hold. (11) p. 11/73

SPECIAL CASE In the case of the terminal constraint (6), the terminal conditions on the state and the adjoint variables in (11) will be replaced, respectively, by and x (T) Y X (12) [λ(t) S x (x (T),T)][y x (T)] 0, y Y. (13) p. 12/73

SPECIAL CASE Furthermore, if the terminal time T in the problem (1)-(5) is unspecified, there is an additional necessary transversality condition for T to be optimal (see Exercise 3.5), namely, H[x (T ),u (T ),λ(t ),T ] + S T [x (T ),T ] = 0, (14) if T (0, ). p. 13/73

REMARK 3.1 AND 3.2 Remark 3.1: We should have H = λ 0 F + λf in (7) with λ 0 0. However, we can set λ 0 = 1 in most applications. Remark 3.2: If the set Y in (6) consists of a single point Y = {k}, making the problem a fixed-end-point problem, then the transversality condition reduces to simply λ(t) equals a constant β to be determined, since x (T) = k. In this case the salvage function S becomes a constant, and can therefore be disregarded. p. 14/73

EXAMPLE 3.1 Consider the problem: max { J = 1 0 } udt subject to ẋ = u, x(0) = 1, (15) u 0, x u 0. (16) Note that constraints (16) are of the mixed type (3). They can also be rewritten as 0 u x. p. 15/73

SOLUTION OF EXAMPLE 3.1 The Hamiltonian is H = u + λu = (1 + λ)u, so that the optimal control has the form u = bang[0,x; 1 + λ]. (17) To get the adjoint equation and the multipliers associated with constraints (16), we form the Lagrangian: L = H + µ 1 u + µ 2 (x u) = µ 2 x + (1 + λ + µ 1 µ 2 )u. p. 16/73

From this we get the adjoint equation SOLUTION OF EXAMPLE 3.1 CONT. λ = L x = µ 2, λ(1) = 0. (18) Also note that the optimal control must satisfy L u = 1 + λ + µ 1 µ 2 = 0, (19) and µ 1 and µ 2 must satisfy the complementary slackness conditions µ 1 0, µ 1 u = 0, (20) µ 2 0, µ 2 (x u) = 0. (21) p. 17/73

SOLUTION OF EXAMPLE 3.1 CONT. It is obvious for this simple problem that u (t) = x(t) should be the optimal control for all t [0, 1]. We now show that this control satisfies all the conditions of the Lagrangian form of the maximum principle. Since x(0) = 1, the control u = x gives x = e t as the solution of (15). Because x = e t > 0, it follows that u = x > 0; thus µ 1 = 0 from (20). p. 18/73

SOLUTION OF EXAMPLE 3.1 CONT. From (19) we then have µ 2 = 1 + λ. Substituting this into (18) and solving gives 1 + λ(t) = e 1 t. (22) Since the right-hand side of (22) is always positive, u = x satisfies (17). Notice that µ 2 = e 1 t 0 and x u = 0, so (21) holds. p. 19/73

SUFFICIENCY CONDITIONS: CONCAVE AND QUASICONCAVE FUNCTIONS Let D E n be a convex set. A function ψ : D E 1 is concave, if for all y,z D and for all p [0, 1], ψ(py + (1 p)z) pψ(y) + (1 p)ψ(z). (23) The function ψ is quasiconcave if (23) is relaxed to ψ(py + (1 p)z) min{ψ(y),ψ(z)}. (24) ψ is strictly concave if y z and p (0, 1) and (23) holds with a strict inequality. ψ is convex, quasiconvex, or strictly convex if ψ is concave, quasiconcave, or strictly concave, respectively. p. 20/73

SUFFICIENCY CONDITIONS: THEOREM 3.1 Let (x,u,λ,µ,α,β) satisfy the necessary conditions in (11). If H(x,u,λ(t),t) is concave in (x,u) at each t [0,T], S in (2) is concave in x, g in (3) is quasiconcave in (x,u), a in (4) is quasiconcave in x, and b in (5) is linear in x, then (x,u ) is optimal. p. 21/73

REMARK ON THE CONCAVITY CONDITION IN THEOREM 3.1 The concavity of the Hamiltonian with respect to (x,u) is a crucial condition in Theorem 3.1. Unfortunately, a number of management science and economics models lead to problems that do not satisfy this concavity condition. We replace the concavity requirement on the Hamiltonian in Theorem 3.1 by a concavity requirement on H 0, where H 0 (x,λ,t) = max H(x,u,λ,t). (25) {u g(x,u,t) 0} p. 22/73

THEOREM 3.2 Theorem 3.1 remains valid if H 0 (x (t),λ(t),t) = H(x (t),u (t),λ(t),t), t [0,T], and, if in addition, we drop the quasiconcavity requirement on g and replace the concavity requirement on H in Theorem 3.1 by the following assumption: For each t [0,T], if we define A 1 (t) = {x g(x,u,t) 0 for some u}, then H 0 (x,λ(t),t) is concave on A 1 (t), if A 1 (t) is convex. If A 1 (t) is not convex, we assume that H 0 has a concave extension to co(a 1 (t)), the convex hull of A 1 (t). p. 23/73

3.3 CURRENT-VALUE FORMULATION Assume a constant continuous discount rate ρ 0. The time dependence of the relevant functions comes only through the discount factor. Thus, F(x,u,t) = φ(x,u)e ρt and S(x,T) = σ(x)e ρt. Now, the objective is to maximize { J = T 0 φ(x,u)e ρt dt + σ[x(t)]e ρt } (26) subject to (1) and (3)-(5). p. 24/73

3.3 CURRENT-VALUE FORMULATION CONT. The standard Hamiltonian is H s := e ρt φ(x,u) + λ s f(x,u,t) (27) and the standard Lagrangian is L s := H s + µ s g(x,u,t). (28) p. 25/73

3.3 CURRENT-VALUE FORMULATION CONT. The standard adjoint variables λ s and standard multipliers µ s, α s and β s satisfy λ s = L s x, (29) λ s (T) = S x [x(t),t] + α s a x (x(t),t) + β s b x (x(t),t) = e ρt σ x [x(t)] + α s a x (x(t),t) + β s b x (x(t),t), (30) α s 0, α s a(x(t),t) = 0, (31) µ s 0, µ s g = 0. (32) p. 26/73

The current-value Hamiltonian 3.3 CURRENT-VALUE FORMULATION CONT. H[x,u,λ,t] := φ(x,u) + λf(x,u,t) (33) and the current-value Lagrangian L[x,u,λ,µ,t] := H + µg(x,u,t). (34) We define λ := e ρt λ s and µ := e ρt µ s, (35) so that we can rewrite (27) and (28) as H = e ρt H s and L = e ρt L s. (36) p. 27/73

From (35), we have 3.3 CURRENT-VALUE FORMULATION CONT. λ = ρe ρt λ s + e ρt λs. (37) Then from (29), λ = ρλ L x, λ(t) = σ x [x(t)] + αa x (x(t),t) + βb x (x(t),t), (38) where (38) follows from the terminal condition for λ s (T) in (30), the definition (36), α = e ρt α s and β = e ρt β s. (39) p. 28/73

3.3 CURRENT-VALUE FORMULATION CONT. The complimentary slackness conditions satisfied by the current-value Lagrange multipliers µ and α are µ 0, µg = 0, α 0, and αa = 0 on account of (31), (32), (35), and (39). From (14), the necessary transversality condition for T to be optimal is H[x (T ),u (T ),λ(t ),T ] ρσ[x (T )] = 0. (40) p. 29/73

THE CURRENT-VALUE MAXIMUM PRINCIPLE ẋ = f(x, u, t), a(x (T), T) 0, b(x (T), T) = 0, λ = ρλ L x [x, u, λ, µ, t], with the terminal conditions λ(t) = σ x (x (T)) + αa x (x (T), T) + βb x (x (T), T), α 0, αa(x (T), T) = 0, and the Hamiltonian maximizing condition H[x (t), u (t), λ(t), t] H[x (t), u, λ(t), t] at each t [0, T] for all u satisfying g[x (t), u, t] 0, and the Lagrange multipliers µ(t) are such that L u u=u (t) = 0, and the complementary slackness conditions µ(t) 0 and µ(t)g(x, u, t) = 0 hold. (41) p. 30/73

SPECIAL CASE When the terminal constraint is given by (6) instead of (4) and (5), we need to replace the terminal condition on the state and the adjoint variables, respectively, by (12) and [λ(t) σ x (x (T))][y x (T)] 0, y Y. (42) p. 31/73

Use the current-value maximum principle to solve the following consumption problem for ρ = r: max { J = T subject to the wealth dynamics 0 e ρt ln C(t)dt Ẇ = rw C, W(0) = W 0, W(T) = 0, } EXAMPLE 3.2 where W 0 > 0. Note that the condition W(T) = 0 is sufficient to make W(t) 0 for all t. We can interpret ln C(t) as the utility of consuming at the rate C(t) per unit time at time t. p. 32/73

SOLUTION OF EXAMPLE 3.2 The current-value Hamiltonian is H = ln C + λ(rw C), (43) where the adjoint equation, under the assumption ρ = r, is λ = ρλ H W = ρλ rλ = 0, λ(t) = β, (44) where β is some constant to be determined. The solution of (44) is simply λ(t) = β for 0 t T. p. 33/73

SOLUTION OF EXAMPLE 3.2 CONT. To find the optimal control, we maximize H by differentiating (43) with respect to C and setting the result to zero: H C = 1 C λ = 0, which implies C = 1/λ = 1/β. Using this consumption level in the wealth dynamics gives which can be solved as Ẇ = rw 1, W(T) = 0, β W(t) = W 0 e rt 1 βr (ert 1). p. 34/73

SOLUTION OF EXAMPLE 3.2 CONT. Setting W(T) = 0 gives β = 1 e rt rw 0. Therefore, the optimal consumption C (t) = 1 β = rw 0 1 e rt = ρw 0, since ρ = r. 1 e ρt p. 35/73

3.4 TERMINAL CONDITIONS/TRANSVERSALITY CONDITIONS Case 1: Free-end point. From the terminal conditions in (11), it is obvious that for the free-end-point problem, i.e., when Y = X, If σ(x) 0, then λ(t) = 0. λ(t) = σ x [x (T)]. (45) Case 2: Fixed-end point. The terminal condition is b(x(t),t) = x(t) k = 0, and the transversality condition in (11) does not provide any information for λ(t). λ(t) will be some constant β. p. 36/73

3.4 TERMINAL CONDITIONS/TRANSVERSALITY CONDITIONS CONT. Case 3: One-sided constraints. The ending value of the state variable is in a one-sided interval, namely, a(x(t),t) = x(t) k 0, where k X. In this case it is possible to show that and λ(t) σ x [x (T)] (46) {λ(t) σ x [x (T)]}{x (T) k} = 0. (47) For σ(x) 0, these terminal conditions can be written as λ(t) 0 and λ(t)[x (T) k] = 0. (48) p. 37/73

3.4 TERMINAL CONDITIONS/TRANSVERSALITY CONDITIONS CONT. Case 4: A general case. A general ending condition is x(t) Y X. p. 38/73

TABLE 3.1 SUMMARY OF THE TRANSVERSALITY CONDITIONS Constraint Description λ(t) λ(t) on x(t) when σ 0 x(t) Y = X Free-end λ(t) = σ x [x (T)] λ(t) = 0 point x(t) = k X, Fixed-end λ(t) = a constant λ(t) = a constant i.e., Y = {k} point to be determined to be determined x(t) X [k, ), One-sided λ(t) σ x [x (T)] λ(t) 0 i.e., Y = {x x k} constraints and and x(t) k {λ(t) σ x [x (T)]}{x (T) k} = 0 λ(t)[x (T) k] = 0 x(t) X (, k], One-sided λ(t) σ x [x (T)] λ(t) 0 i.e., Y = {x x k} constraints and and x(t) k {λ(t) σ x [x (T)]}{k x (T)} = 0 λ(t)[k x (T)] = 0 x(t) Y X General {λ(t) σ x [x (T)]}{y x (T)} 0 λ(t)[y x (T)] 0 constraints y Y y Y p. 39/73

EXAMPLE 3.3 Consider the problem: max { J = 2 0 } xdt subject to ẋ = u, x(0) = 1, x(2) 0, (49) 1 u 1. (50) p. 40/73

SOLUTION OF EXAMPLE 3.3 The Hamiltonian is H = x + λu. Clearly the optimal control has the form The adjoint equation is u = bang[ 1, 1;λ]. (51) λ = 1 (52) with the transversality conditions λ(2) 0 and λ(2)x(2) = 0. (53) p. 41/73

SOLUTION OF EXAMPLE 3.3 CONT. Since λ(t) is monotonically increasing, the control (51) can switch at most once, and it can only switch from u = 1 to u = 1. Let the switching time be t 2. Then the optimal control is u (t) = { 1 for 0 t t, +1 for t < t 2. Since the control switches at t, λ(t ) must be 0. Solving (52) we get λ(t) = t t. (54) p. 42/73

SOLUTION OF EXAMPLE 3.3 CONT. There are two cases t < 2 and t = 2. We analyze the first case first. Here λ(2) = 2 t > 0; therefore from (53), x(2) = 0. Solving for x with u given in (54), we obtain x(t) = { 1 t for 0 t t, (t t ) + x(t ) = t + 1 2t for t < t 2. Therefore, setting x(2) = 0 gives x(2) = 3 2t = 0, which makes t = 3/2. Since this satisfies t < 2, we do not have to deal with the case t = 2. p. 43/73

FIGURE 3.1 STATE AND ADJOINT TRAJECTORIES IN EXAMPLE 3.3 p. 44/73

ISOPERIMETRIC OR BUDGET CONSTRAINT It is of the form: T 0 l(x,u,t)dt K, (55) where l : E n E m E 1 E 1 is assumed nonnegative, bounded, and continuously differentiable, and K is a positive constant representing the amount of the budget. To see how this constraint can be converted into a one-sided constraint, we define an additional state variable x n+1 by the state equation ẋ n+1 = l(x,u,t), x n+1 (0) = K, x n+1 (T) 0. (56) p. 45/73

EXAMPLES ILLUSTRATING TERMINAL CONDITIONS Example 3.4 The problem is: max { J = T 0 } e ρt ln C(t)dt + e ρt BW(T) (57) subject to the wealth equation Ẇ = rw C, W(0) = W 0, W(T) 0. (58) Assume B to be a given positive constant. p. 46/73

SOLUTION OF EXAMPLE 3.4 The Hamiltonian for the problem is given in (43), and the adjoint equation is given in (44) except that the transversality conditions are from Row 3 of Table 3.1: λ(t) B, [λ(t) B]W(T) = 0. (59) In Example 3.2 the value of β, which was the terminal value of the adjoint variable, was β = 1 e rt rw 0. We now have two cases: (i) β B and (ii) β < B. p. 47/73

SOLUTION OF EXAMPLE 3.4 CONT. In case (i), the solution of the problem is the same as that of Example 3.2, because by setting λ(t) = β and recalling that W(T) = 0 in that example, it follows that (59) holds. In case (ii), we set λ(t) = B and use (44) which is λ = 0. Hence, λ(t) = B for all t. The Hamiltonian maximizing condition remains unchanged. Therefore, the optimal consumption is: C = 1 λ = 1 B. p. 48/73

SOLUTION OF EXAMPLE 3.4 CONT. Solving (58) with this C gives It is easy to show that W(t) = W 0 e rt 1 Br (ert 1). W(T) = W 0 e rt 1 Br (ert 1) is nonnegative since β < B. Note that (59) holds for case (ii). p. 49/73

EXAMPLE 3.5: A TIME-OPTIMAL CONTROL PROBLEM Consider a subway train of mass m (assume m = 1), which moves along a smooth horizontal track with negligible friction. The position x of the train along the track at time t is determined by Newton s Second Law of Motion, i.e., ẍ = mu = u. (60) Note: (60) is a second-order differential equation. p. 50/73

EXAMPLE 3.5: A TIME-OPTIMAL CONTROL PROBLEM Let the initial conditions on x(0) and ẋ(0) be x(0) = x 0 and ẋ(0) = y 0, Transform (60) to two first-order differential equations: { ẋ = y, x(0) = x 0, ẏ = u, y(0) = y 0. (61) Let the control constraint be u Ω = [ 1, 1]. (62) p. 51/73

EXAMPLE 3.5: A TIME-OPTIMAL CONTROL PROBLEM CONT. The problem is: max subject to { J = T 0 1dt } ẋ = y, x(0) = x 0, x(t) = 0, ẏ = u, y(0) = y 0, y(t) = 0, and the control constraint u Ω = [ 1, 1]. (63) p. 52/73

SOLUTION OF EXAMPLE 3.5 The standard Hamiltonian function in this case is H = 1 + λ 1 y + λ 2 u, where the adjoint variables λ 1 and λ 2 satisfy and λ 1 = 0, λ 1 (T) = β 1 and λ 2 = λ 1, λ 2 (T) = β 2, λ 1 = β 1 and λ 2 = β 2 + β 1 (T t). The Hamiltonian maximizing condition yields the form of the optimal control to be u (t) = bang{ 1, 1; β 2 + β 1 (T t)}. (64) p. 53/73

SOLUTION OF EXAMPLE 3.5 CONT. The transversality condition (14) with y(t) = 0 and S 0 yields H + S T = λ 2 (T)u (T) 1 = β 2 u (T) 1 = 0, which together with the bang-bang control policy (64) implies either λ 2 (T) = β 2 = 1 and u (T) = 1, or λ 2 (T) = β 2 = +1 and u (T) = +1. p. 54/73

TABLE 3.2 STATE TRAJECTORIES AND SWITCHING CURVE (a) u (τ) = 1 for (t τ T) (b) u (τ) = +1 for (t τ T) y = T t y = t T x = (T t) 2 /2 x = (t T) 2 /2 Γ : x = y 2 /2 for y 0 Γ + : x = y 2 /2 for y 0 p. 55/73

SOLUTION OF EXAMPLE 3.5 CONT. We can put Γ and Γ + into a single switching curve Γ as y = Γ(x) = { 2x, x 0, + 2x, x < 0. (65) If the initial state (x 0,y 0 ) lies on the switching curve, then we use u = +1 (resp., u = 1) if (x 0,y 0 ) lies on Γ + (resp., Γ ). In common parlance, we apply the brakes. If the initial state (x 0,y 0 ) is not on the switching curve, then we choose, between u = 1 and u = 1, that which moves the system toward the switching curve. By inspection, it is obvious that above the switching curve we must choose u = 1 and below we must choose u = +1. p. 56/73

FIGURE 3.2 MINIMUM TIME OPTIMAL RESPONSE FOR PROBLEM (63) p. 57/73

SOLUTION OF EXAMPLE 3.5 CONT. The other curves in Figure 3.2 are solutions of the differential equations starting from initial points (x 0,y 0 ). If (x 0,y 0 ) lies above the switching curve Γ as shown in Figure 3.2, we use u = 1 to compute the curve as follows: ẋ = y, x(0) = x 0, ẏ = 1, y(0) = y 0. Integrating these equations gives y = t + y 0, x = t2 2 + y 0t + x 0. Elimination of t between these two gives x = y2 0 y2 2 + x 0. (66) p. 58/73

SOLUTION OF EXAMPLE 3.5 CONT. (66) is the equation of the parabola in Figure 3.2 through (x 0,y 0 ). The point of intersection of the curve (66) with the switching curve Γ + is obtained by solving (66) and the equation for Γ +, namely 2x = y 2, simultaneously. This gives x = y2 0 + 2x 0 4, y = (y0 2 + 2x 0)/2, (67) where the minus sign in the expression for y in (67) was chosen since the intersection occurs when y is negative. The time t to reach the switching curve, called the switching time, given that we start above it, is t = y 0 y = y 0 + (y 2 0 + 2x 0)/2. (68) p. 59/73

SOLUTION OF EXAMPLE 3.5 CONT. To find the minimum total time to go from the starting point (x 0,y 0 ) to the origin (0,0), we substitute t into the equation for Γ + in Column (b) of Table 3.2. This gives T = t y = y 0 + 2(y 2 0 + 2x 0). (69) As a numerical example, start at the point (x 0,y 0 ) =(1,1). Then, the equation of the parabola (66) is 2x = 3 y 2. The switching point (67) is (3/4, 3/2). Finally, the switching time is t = 1 + 3/2 from (68). Substituting into (69), we find the minimum time to stop is T = 1 + 6. p. 60/73

SOLUTION OF EXAMPLE 3.5 CONT. To complete the solution of this numerical example let us evaluate β 1 and β 2, which are needed to obtain λ 1 and λ 2. Since (1,1) is above the switching curve, u (T) = 1, and therefore β 2 = 1. To compute β 1, we observe that λ 2 (t ) = β 2 + β 1 (T t ) = 0 so that β 1 = β 2 /(T t ) = 1/ 3/2 = 2/3. In Exercises 3.14-3.17, you are asked to work other examples with different starting points above, below, and on the switching curve. Note that t = 0 by definition, if the starting point is on the switching curve. p. 61/73

Transversality conditions: Free-end-point: 3.5 INFINITE HORIZON AND STATIONARITY lim T λs (T) = 0 lim T e ρt λ(t) = 0. (70) One-sided constraints: lim x(t) 0. T Then, the transversality conditions are lim T e ρt λ(t) 0 and lim T e ρt λ(t)x (T) = 0. (71) p. 62/73

3.5 INFINITE HORIZON AND STATIONARITY CONT. Stationarity Assumption: f(x,u,t) = f(x,u), g(x,u,t) = g(x,u). (72) Long-run stationary equilibrium is defined by the quadruple { x,ū, λ, µ} satisfying f( x,ū) = 0, ρ λ = L x [ x,ū, λ, µ], µ 0, µg( x,ū) = 0, and H( x,ū, λ) H( x,u, λ) for all u satisfying g( x,u) 0. (73) p. 63/73

3.5 INFINITE HORIZON AND STATIONARITY CONT. Clearly, if the initial condition x 0 = x, the optimal control is u (t) = ū for all t. If x 0 x, the optimal solution will have a transient phase. If the constraint involving g is not imposed, µ may be dropped from the quadruple. In this case, the equilibrium is defined by the triple { x, ū, λ} satisfying f( x, ū) = 0, ρ λ = H x ( x, ū, λ), and H u ( x, ū, λ) = 0. (74) p. 64/73

EXAMPLE 3.6 Consider the problem: max { J = 0 } e ρt ln C(t)dt subject to lim W(T) 0, (75) T Ẇ = rw C, W(0) = W 0 > 0. (76) p. 65/73

SOLUTION OF EXAMPLE 3.6 By (73) we set r W C = 0, λ = β, where β is a constant to be determined. This gives the optimal control C = r W, and by setting λ = 1/ C = 1/r W, we see all the conditions of (73) including the Hamiltonian maximizing condition hold. p. 66/73

SOLUTION OF EXAMPLE 3.6 CONT. Furthermore, λ and W = W 0 satisfy the transversality conditions (71). Therefore, by the sufficiency theorem, the control obtained is optimal. Note that the interpretation of the solution is that the trust spends only the interest from its endowment W 0. Note further that the triple ( W, C, λ) = (W 0,rW 0, 1/rW 0 ) is an optimal long-run stationary equilibrium for the problem. p. 67/73

TABLE3.3: OBJECTIVE, STATE, AND ADJOINT EQUATIONS FOR VARIOUS MODEL TYPES Objective State Current-Value Form of Optimal Function Equation Adjoint Equation Control Policy Integrand φ = ẋ = f = λ = (a) Cx + Du Ax + Bu + d λ(ρ A) C Bang-Bang (b) C(x) + Du Ax + Bu + d λ(ρ A) C x Bang-Bang+Singular (c) x T Cx + u T Du Ax + Bu + d λ(ρ A) 2x T C Linear Decision Rule (d) C(x) + Du A(x) + Bu + d λ(ρ A x ) C x Bang-Bang+Singular (e) c(x) + q(u) (ax + d)b(u) + e(x) λ(ρ ab(u) e x ) c x Interior or Boundary (f) c(x)q(u) (ax + d)b(u) + e(x) λ(ρ ab(u) e x ) c x q(u) Interior or Boundary p. 68/73

3.6 MODEL TYPES In Model Type (a) of Table 3.3 we see that both φ and f are linear functions of their arguments. Hence it is called the linear-linear case. The Hamiltonian is H = Cx + Du + λ(ax + Bu + d) = Cx + λax + λd + (D + λb)u. (77) Model Type (b) of Table 3.3 is the same as Model Type (a) except that the function C(x) is nonlinear. p. 69/73

3.6 MODEL TYPES CONT. Model Type (c) has linear functions in the state equation and quadratic functions in the objective function. Model Type (d) is a more general version of Model Type (b) in which the state equation is nonlinear in x. In Model Types (e) and (f), the functions are scalar functions, and there is only one state equation so that λ is also a scalar function. p. 70/73

REMARKS 3.3 AND 3.4 Remark 3.3: In order to use the absolute value function u of a control variable u in forming the functions φ or f. We define u + and u satisfying the following relations: We write u := u + u, u + 0, u 0, (78) u + u = 0. (79) We need not impose (79) explicitly. u = u + + u. (80) Remark 3.4: Tables 3.1 and 3.3 are constructed for continuous-time models. p. 71/73

REMARK 3.5 Remark 3.5: Consider Model Types (a) and (b) when the control variable constraints are defined by linear inequalities of the form g(u,t) = g(t)u 0. (81) Then, the problem of maximizing the Hamiltonian function becomes: max(d + λb)u subject to g(t)u 0. (82) p. 72/73

REMARKS 3.6 AND 3.7 Remark 3.6:The salvage value part of the objective function, S[x(T),T], makes sense in two cases: (a) When T is free, and part of the problem is to determine the optimal terminal time. (b) When T is fixed and we want to maximize the salvage value of the ending state x(t), which in this case can be written simply as S[x(T)]. Remark 3.7: One important model type that we did not include in Table 3.3 is the impulse control model of Bensoussan and Lions. In this model, an infinite control is instantaneously exerted on a state variable in order to cause a finite jump in its value. p. 73/73