1 Linear Quadratic Control Problem

Transcription

1 1 Linear Quadratic Control Problem Suppose we have a problem of the following form: { vx 0 ) = max β t ) } x {a t,x t+1 } t Qx t +a t Ra t +2a t Wx t 1) x t is a vector of states and a t is a vector of controls actions). I will assume that there are n states the first one is a constant and there are k controls the matrices are then dimensioned as Q n n, R k k, and W n k. States evolve according to the transition function x t+1 = Ax t +Ba t. 2) he Bellman equation is x t Px t = max a t { x t Qx t +a t Ra t +2a t Wx t +βx t+1px t+1 } 3) subject to x t+1 = Ax t +Ba t. 4) I have used the fact that the value function will be quadratic; it is trivial to show this fact holds. he value function, x Px, is symmetric the P matrix is symmetric; from now on I will abuse notation a bit and refer to P as the value function). Substituting the constraint into the right-hand-side of the Bellman equation yields x Qx+a Ra+2a Wx+β x A +a B ) P Ax+Ba). 5) Note the matrix differentiation rules that D x y Ax = A y D y y Ax ) = Ax. Maximizing with respect to a yields the solution a = R+βB PB) 1 W +βb PA)x. 6) Substituting this into the Bellman equation and matching coefficients yields the recursive relationship known as the matrix Ricatti equation: x Px = x Q+βA PA W +βa PB)R+βB PB) 1 W +βb PA))x. 7) his equation can be used to solve for the value function by iteration. hink of the RHS as the result of the Bellman operator; it takes the function characterized by P and maps it into one characterized by Q+βA PA W +βa PB)R+βB PB) 1 W +βb PA). 8) 1

2 One way to solve this equation for P is to iterate: start with P 0 = 0, calculate a new P 1 by P 1 = Q+βA P 0 A W +βa P 0 B)R+βB P 0 B) 1 W +βb P 0 A) 9) and repeat until the P are sufficiently close together. his method is extremely fast it will take only a few seconds to solve once you have programmed it. here are faster ways to solve this equation, but none which are so easily seen in light of our discussion of the value function all we are doing here is applying the Bellman operator successively to our linear-quadratic problem. Furthermore, the Contraction Mapping heorem guarantees that only one matrix P can solve the matrix Ricatti equation. he optimal decision rules are then recovered by a = Fx In Matlab, the program would look like this: = R+βB PB) 1 W +βb PA)x. 1. Set initial P 0 a good choice is either the zero matrix or a matrix which is a small negative number times an identity matrix); 2. Compute P 1 according to the Ricatti equation; 3. Compute the norm of P 1 P 0 using the built-in norm function; 4. Set up a while loop: while normp 1 P 0 ) ǫ set P 0 = P 1 and recompute P 1. he while loop will continue until the difference between the two matrices is small. After the loop terminates you can compute F. Some minor points are of some importance here. We require that the value function be concave; this condition was needed to ensure differentiability and many other desirable features such as the transversality condition, and is also implicitly used in the first-order condition above. In the quadratic case, we need P to be negative definite it must possess only negative eigenvalues except for one related to a constant, which typically will be large and positive). A matrix A is called negative definite if and only if x Ax < 0 x 0, and the eigenvalue condition is necessary and sufficient for this condition. We also want our system to be saddle-stable; this will require that the matrix A BF have eigenvalues within the unit circle since x t+1 = Ax t +Ba t = A BF)x t. hus, iterating on the decision rule produces a sequence that converges to the steady state, so transversality is satisfied. 2

3 In general the growth model is not quadratic. herefore, now we will examine how to make it so this is the approach used by Kydland and Prescott in their time-to-build paper and first discussed in the appendix to their rules vs. discretion paper. he growth model has a Bellman equation like this: vk) = max k {ufk)+1 δ)k k )+βvk )}. 10) Let s take the steady state of this model to be k. We know that the model is saddle-stable, at least locally, around this point it will tend to this value over time. herefore, what if we choose to approximate the utility function around this point by a quadratic? We know from before that this steady state is given by the solution to From aylor s theorem we have ufk)+1 δ)k k ) = ufk)+1 δ)k k)+ f k)+1 δ = 1/β. 11) u fk)+1 δ)k k)f k)+1 δ)k k)+ u fk)+1 δ)k k)k k)+ second order terms. We will make the compute calculate this for us; otherwise, the higher-order terms become quite tedious. o do so, we will use the definition of a derivative: f fx+h) fx) x) = lim. 12) h 0 h his is called a one-sided derivative and to calculate it we would choose a small h some around 10 6 x. But in general this is not very accurate so we will employ the two-sided derivative f fx+h) fx h) x) = lim. 13) h 0 2h his will be much more accurate. he second derivative is then f f x+h) f x h) x) = lim h 0 2h = fx+2h) fx) 2h fx) fx 2h) 2h 2h = fx+2h) 2fx)+fx 2h) 4h 2 h should be around 10 3 x. For a function of more than one variable, we calculate partial derivatives by fx,y) x = fx+h 1,y) fx h 1,y) 2h 1 14) 3

4 2 fx,y) x y fx,y) y = fx,y +h 2) fx,y h 2 ) 2h 2 15) = 1 fx+h1,y +h 2 ) fx+h 1,y h 2 ) fx h 1,y +h 2 ) fx h 1,y h 2 ) 2h 1 2h 2 2h 2 = fx+h 1,y +h 2 ) fx+h 1,y h 2 ) fx h 1,y +h 2 )+fx h 1,y h 2 ) 4h 1 h 2 h 1, h 2 should be around 10 3 x or 10 3 y. In general one should avoid computing higher-order derivatives in this manner, but we don t need those terms here. Our goal is to approximate the return function ufk)+1 δ)k k ) Hy) 16) y = x a = we do not augment x with a constant. Let the steady state be denoted y. hen the approximation around the steady state yields k k 17) Hy) = Hy)+f y y)+y y) 1 Vy y). 18) 2 Now define 1 S 2 V = R 19) l k = 1 2 f k 2x 2a R 20) l n = 1 2 f n 2x S 2a 21) G = Hy)+y 1 2 Vy f y 22) f = fn f n refers to the first derivatives with respect to states and f k to controls. Now define and f k 23) W = l k 24) Q = G l n l n S. 25) From here we can use the Ricatti equation above to find P and F. Q now embeds the constant term G as well as the linear terms associated with states times the constant. he 4

5 code provided does all of this work for you, all you need to do is provide it the function u and the steady state y; the code is actually a bit more general, as it separates the states into exogenous here, only the constant 1) and endogenous for later use when we add shocks. Of course one would want to know how well the approximation does are we computing an accurate solution? he standard method for evaluating accuracy is to evaluate the decision rules at points the solution is not set up to work. It is easiest to do this construction with an example. Consider a growth model with a Bellman equation for the planner of vk) = max k {logjk α +1 δ)k k )+βvk )}. he deterministic steady state of the economy satisfies the equation 1 = β αjk α 1 +1 δ ). Set J = 1, α = 0.36, β = 0.99, and δ = 0.025, common parameter values for a growth model with quarterly periods. We now take a quadratic approximation to the period return function and iterate on the Ricatti equation to obtain a quadratic approximation to v. We construct a grid of 500 values for the capital stock that spans the range of 50 percent below to 50 percent above the steady state. he Euler equation for this model is 1 Jk α +1 δ)k πk) = β αjπk) α 1 +1 δ ) Jπk) α +1 δ)πk) ππk)) πk) = F 1 F 2 1 k is the policy function obtained from the LQ regulator problem. Define the Euler equation error as EEk) = 1 Jπk) α +1 δ)πk) ππk))) βjk α +1 δ)k πk)) αjπk) α 1 +1 δ ) for a given value of capital; this equation delivers the utility expressed as a percentage of consumption) that the planner is sacrificing by not using the policy function that sets the Euler equation exactly to zero at k. If we compute and plot the Euler equation errors over the capital grid, expressed in log 10 terms that is, an error of 10 3 is plotted as equal to 3), we get Figure 1; the prominent feature is the dip at the steady state. he average Euler error is and the maximum error is ; that is, over these capital stocks the most an agent would lose is % of consumption by not following the true decision rule, and this occurs when capital is 50 percent below steady state. Note also that the Euler error function is not symmetric; it is more costly to deviate when capital is below the steady state than above, because the household s underlying utility function is concave. Over a smaller range 5 percent above and below, which is roughly the range that one would get from a business cycle model), the maximum error is only , so this method is very accurate for reasonable ranges. 5

6 1.1 Eliminating Cross-erms and Discounting Engineering problems typically do not involve discounting or cross-terms. Suppose we have the problem β t ) x t Q x t +u t Ru t +2u t Wx t = β t x t subject to x t+1 = A x t +Bu t. u t Q W W R xt We can eliminate the cross-terms to match up better with the engineering literature. Define hen and and also a t Ra t = x t x t u t a t = u t +R 1 Wx t. u t Q W W R W R 1 W W W xt u t Q = Q W R 1 W R xt u t = x t Qx t +a t Ra t x t+1 = A x t +Bu t = Ax t +Ba t A = A BR 1 W. Now we have the standard LQ regulator problem subject to β t ) x t Qx t +a t Ra t x t+1 = Ax t +Ba t. We can also eliminate discounting. Define u t x t = βx t ã t = βa t. hen we have β t ) ) x t Qx t +a t Ra t = x t Q x t +ã t Rã t 6

7 subject to x t+1 = à x t + Bã t à = A β B = B β. he solution is ũ t = F x t and F = R+ B P B) 1 B Pà P = Q+à Pà à P B x t+1 = R+ B P B) 1 B Pà ) A B F x t. Now we can apply the results directly; naturally, nothing is affected. 7