1 Undiscounted Problem (Deterministic)

Transcription

1 Lectre 9: Linear Qadratic Control Problems 1 Undisconted Problem (Deterministic) Choose ( t ) 0 to Minimize (x trx t + tq t ) t=0 sbject to x t+1 = Ax t + B t, x 0 given. x t is an n-vector state, t a k-vector control, R, Q are symmetric, psd and pd respectively. A and B are n n and n k. So, the objective fnction is qadratic and the state transition law is linear, hence this is called a linear qadratic control problem. We gess the vale fnction is qadratic: V (x) = x P x where P is symmetric psd. The vale fnction satisfies Bellman s eqation. V (x) = max [ x Rx Q + V (y)] where y = Ax + B. Sbstitte the gess for V (x) and V (y): x P x = max [ x Rx Q (Ax + B) P (Ax + B)] Get the optimal interior on the RHS throgh an FOC: 2 Q 2(Ax + B) P B = 0. Take the transpose, and rearrange. Q = B P (Ax + B) = B P Ax B P B. So (Q + B P B) = B P Ax. Or = F x, where F = (Q + B P B) 1 B P A. 1

2 Now plg back = F x in Bellman s Eqation. We have Or x P x = x Rx x F QF x [{(A BF )x} P {(A BF )x}] x P x = x [R + A P A + F (Q + B P B)F F B P A A P BF ] x ( ) Now, notice sing symmetry that F = (B P A) (Q + B P B) 1 = A P B(Q + B P B) 1. So, F (Q + B P B)F = A P B(Q + B P B) 1 (Q + B P B)(Q + B P B) 1 B P A = A P B(Q + B P B) 1 B P A = A P BF = F B P A. Ths the last 3 terms of ( ) are the same, and after cancellation, ( ) yields P = R + A P A A P B(Q + B P B) 1 B P A ( ) ( ) is an algebraic matrix Riccati eqation; a fnctional eqation for the matrix P. Under some sfficient conditions on the matrices in the objective fnction and the transition law, it has a niqe soltion, and one that can be obtained sing the recrsion P j+1 = R + A P A A P j B(Q + B P j B) 1 B P j A ( ) with P 0 being a zero matrix, and proceeding to the limit. In fact, this recrsion works to give s the vale fnction for finite-horizon problems as well; only in these, we have to stop after T iterations, where T is the horizon length. To solve a linear qadratic problem, therefore, one can derive its algebraic matrix Riccati eqation, solve it sing a recrsion like the one above, and then sbstitte for it in the matrix F to get the optimal policy fnction t = F x t. 2

3 The optimized system then evolves according to x t+1 = (A BF )x t. It is stable if lim t = 0 starting from any x 0 R n. In fact, the system is stable if all eigenvales of (A BF ) are less than 1 in absolte vale. (This is easy to see in the case that all eigenvales are distinct and less than 1 in absolte vale: then, (A BF ) = DΛD where Λ is the diagonal matrix of eigenvales and D a matrix of eigenvectors. Then x t+1 = DΛD x t = Dλ t D x 0, which converges to 0 as t ). Sfficient conditions for this are discssed in the literatre, and ensre a niqe soltion for the algebraic matrix Riccati eqation as well. 2 Disconted Problem (deterministic) Choose ( t ) 0 to Maximize t=0 β t (x trx t + tq t ) sbject to x t+1 = Ax t + B t, x 0 given, 0 < β < 1. We gess V (x) = x P x for a symmetric psd matrix P. SO V (y) = y P y = (Ax + b) P (Ax + B). Sbstitte these in Bellman s Eqation: x P x = max [ x Rx Q β{(ax + B) P (Ax + B)}] The interior FOC wrt is: 2 Q β{2(ax + B) P B} = 0, or Q = βb P (Ax + B). So, (Q + βb P B) = βb P Ax, or = F x, where F = β(q + βb P B) 1 B P A (#). Sbstitte this in the Bellman Eqation: x P x = [ x Rx x F QF x β {(Ax BF x) P (Ax BF x)}] Or 3

4 x P x = x [R + βa P A + F (Q + βb P B)F βa P BF βf B P A] x and sing a cancelation akin to that in the ndisconted case, we get P = R + βa P A β 2 A P B(Q + βb P B) 1 B P A (##) So, vale iteration in the disconted case iterates recrsively to solve the algebraic Riccati eqation (##), and then ses the reslting P in (#) to get the optimal policy fnction. 3 Stochastic Optimal Linear Reglator The problem is to Maximize E 0 [ t=0 β t (x trx t + tq t )] sbject to x t+1 = Ax t + B t + Cw t+1, x 0 given, 0 < β < 1, where w t+1 is an n-random vector that is i.i.d. N(0, I). The choice is now not one of a seqence of t vectors, t = 0, 1, 2,... t is now permitted to depend on the entire history p ntil time t. Withot nonstandard decision behavior, it can depend on the seqence w 1,..., w t of realized shocks p to that point. Since Bellman s eqation will apply to the vale fnction, t will depend only on the state. In this particlar problem, the shocks w 1,..., w t will all be incorporated in the state x t. Theorem 1 The vale fnction for this problem is V (x) = x P x d, where P is the niqe symmetric psd soltion of the disconted algebraic Riccati eqation: P = R + βa P A β 2 A P B(Q + βb P B) 1 B P A. 4

5 d = β 1 β trace(p CC ). The optimal policy is t = F x t, where F = β(q + βb P B) 1 B P A. Note. P in the vale fnction, and the optimal policy fnction (characterized by F ) are the same as in the disconted deterministic problem. Proof of Theorem 1. Sbstitte the gess for V (y) in the RHS of Bellman s Eqation: V (x) = max [ x Rx Q βe {(Ax + B + Cw) P (Ax + B + Cw)} βd] Inside the braces on the RHS we have x A P Ax + x A P B + x A P Cw + B P Ax + B P B + B P Cw +w C P Ax + w C P B + w C P Cw Taking expectations of the above, the terms 3,6,7 and 8 evalate to zero since E(w x) = 0. Moreover, x A P B is a real nmber and hence eqals its transpose B P Ax. So we have: V (x) = max [x Rx + Q + βx A P Ax + 2βx A P B + β B P B + βe(w C P Cw)] βd The interior FOC wrt is therefore: [2 Q + 2βx A P B + 2β B P B] = 0 or (Q + βb P B) = betab P Ax, so the optimal policy is = β(q + βb P B) 1 B P Ax = F x. Note that if P trns ot to be the same as in the deterministic case, so will this optimal policy fnction. Plg this optimal policy back in Bellman s Eqation. Note also that E(w C P Cw) = trace(p CC ) (proof provided later). We have: 5

6 -x Px - d = - {x Rx + x F QFx + βx A P Ax 2βx A P BF x + βx F B P BF x + βtrace(p CC )} βd Eqating the constants on both sides, d = βtrace(p CC ) + βd or d = β 1 β trace(p CC ) as was to be shown. Now eqate the terms containing x on both sides. Note that F = β(q + βb P B) 1 B P A, so, sing symmetry of Q and B P B, we have F = βa P B(Q + βb P B) 1. Terms 2, 4 and 5 on the RHS together are x F QF x 2βx A P BF x + βx F B P BF x, which eqals: β 2 x A P B(Q + βb P B) 1 Q(Q + βb P B) 1 B P Ax 2β 2 x A P B(Q + βb P B) 1 B P Ax +β 2 x A P B(Q + βb P B) 1 B P B(Q + βb P B) 1 B P Ax Add the first and the third of these terms: β 2 x AP B(Q + βb P B) 1 (Q + βb P B)(Q + βb P B) 1 B P Ax So this mins the second term yeilds β 2 x A P B(Q + βb P B) 1 B P Ax. Now the LHS-RHS comparison of the x terms of Bellman s Eqation yields P = R + βa P A β 2 A P B(Q + βb P B) 1 B P A Note. The proof sed the fact that E(w C P Cw) = trace(p CC ). This is de to the following reslt abot expectations of qadratic forms of random vectors. 6

7 Proposition 1 Let x be a random n-vector with mean µ and Covariance matrix Σ. Then E(x Ax) = trace(aσ) + µ Aµ Proof. x Ax = n i,j=1 a ij x i x j. So, E(x Ax) = n i,j=1 a ij E(x i x j ), de to the linearity of the expectations operator. Since σ ij Cov(x i, x j ) = E(x i x j ) µ i µ j ), E(x Ax) = i,j a ij (σ ij + µ i µ j ) = i,j a ij σji + i,j a ij µ i µ j, since σ ij = σji. This eqals [ n nj=1 ] i=1 a ij σ ji + µ Aµ. Note that in the first of these terms, the inner sm is the dot prodct of the ith row of A and the ith colmn of Σ, which eqals the ith diagonal element of AΣ. So the first term evalates to trace(aσ). The Class lq.py in qantecon implements soltions to finite and infinite horizon linear qadratic problems (both deterministic and stochastic(with iid Normal shocks)). The object attribtes of this class inclde the matrices R, Q, A, B, C and the discont factor β. The class has a method to solve for P, F, d, and a method to generate times series for x t etc. nder the assmption that the decision maker ses the optimal policy fnction t = F x t. We move to this next and se it to solve the problem of an infinitely lived monopoly with otpt adjstment costs, facing stochastic demand each period. 4 Monopoly with Adjstment Costs An infinitely lived monopoly at each time t faces the inverse demand p t = a 0 a 1 q t + d t, where (d t ) follows 7

8 d t+1 = ρd t + σw t+1, with (w t ) iid standard normal. The d t is an intercept shifter. The monopolist chooses (q t ) (fnctions of realized shocks p to that time) to Maximize E { t=0 β t π t }, where π t = p t q t cq t γ(q t+1 q t ) 2. The last term captres convex adjstment costs of changing otpt. If γ = 0, the monopolist wold simply choose the static profit maximizing otpt q t = (a 0 c + d t )/2a 1 in period t. And this wold flctate according to movements in d t. Bt if γ >> 0, it s qite costly to adjst otpt, so q t wold respond less frenetically to shocks in d t, smoothing otpt and price flctations. Now cast this as an LQ control problem. π t sggests that we can let t = q t+1 q t, so that with Q = γ, tq t = γ(q t+1) qt) 2. Now for the state vector. Notice that (p t c)q t = (a 0 a 1 q t + d t c)q t = a 1 q 2 t + (a 0 c + d t )q t. This eqals a 1 q 2 t + (2a 1 q t )q t. We have eliminated c and d t by bringing in q t. This sggests we complete the sqare by adding a 1 q 2 t, to get a qadratic form a 1 q 2 t + (2a 1 q t )q t a 1 q 2 t in the vector ( q t, q t ), and this vector can then be the state vector. We will need to write the evoltion of the state vector. By choice of t, q t+1 = q t + t. And q t+1 = (a 0 c + d t+1 )/2a 1 = m 0 + m 1 d t+1, which eqals m 0 + m 1 (ρd t + σw t+1 ). This eqals m 0 + m 1 ρ ( ) q t m 0 m 1 + m1 σw t+1. So, q t+1 = m 0 (1 ρ) + ρ q t + m 1 σw t+1. The first, constant term on the RHS creates a problem: In order to write the transition las x t+1 = Ax t +B t +Cw t+1 to incorporate the constant term 8

9 for the q t+1 evoltion, we need to expand the state vector. So, let x t = ( q t q t 1). Let ˆπ t = π t a 1 q t 2 = a 1 (q t q t ) 2 γ 2 t. We minimize [ ] E 0 β t (a 1 (q t q t ) 2 + γ 2 t ) t=0. Notice that a 1 (q t qt 2 ) 2 = x trx t, where the symmetric matrix R has r 11 = r 22 = 1, r 12 = r 21 = 1, and all other entries eqal 0. As stated earlier, Q = γ. The objective fnction is then [ ] E 0 β t (x trx t + tq t ) t=0. The transition law is x t+1 = Ax t + B t + Cw t+1 where x t+1 = ( q t+1 q t+1 1), t = q t+1 q t, ρ 0 m 0 (1 ρ) A = B = (0 1 0) and C = (m 1 σ 0 0). We now have the object attribtes R, Q, A, B, C, β to se for this model to be an instance of the Class LQ. 9