Lecture 7: Linear-Quadratic Dynamic Programming Real Business Cycle Models Shinichi Nishiyama Graduate School of Economics Kyoto University January 10, 2019 Abstract In this lecture, we solve and simulate a standard stochastic growth model with linear-quadratic approximation. We start with a linear-quadratic stochastic infinite-horizon optimization problem and derive the Riccati equation. Next, we solve the steady-state conditions of a standard stochastic growth model real business cycle model) to obtain its certainty-equivalent steady-state equilibrium, and we approximate the return function and the state transition function around the steady state. Finally, we obtain the value function and policy function then simulate a stochastic growth economy. 1 An Infinite Horizon Problem Let x t R n be a state vector, and let u t R k be a control vector. We want solve the following infinite horizon problem: 1) max E {u t} subject to β t 1 rx t, u t ) 2) x t+1 = gx t, u t, ε t+1 ) t = 1, 2,...,, where β is a time discount factor, rx t, u t ) is a return function, and gx t, u t, ε t+1 ) is a state transition function. The return function and the state transition function are time-invariant. In this section, we assume that the return function is quadratic and that the state transition function is linear, i.e., 3) rx t, u t ) = x t, u t ) R W W Q ) xt u t ), where R R n n, Q R k k, both of which are symmetric, and W R n k, and 4) x t+1 = Ax t + Bu t + Cε t+1, Graduate School of Economics, Kyoto University; Mailing Address: Yoshida-Honmachi, Sakyo-ku, Kyoto-shi, Japan 606-8501; E-mail: nishiyama@econ.kyoto-u.ac.jp. 1
where A R n n, B R n k, C R n n, and ε t+1 R n such that E t ε t+1 ) = 0 and E t ε t+1 ε t+1 ) = I n. We can express the value function, vx t ), as 5) vx t ) = x t P x t + d, where P R n n is a symmetric negative definite) matrix, and d R. The value function is also time-invariant when the preference, the technology, and the government policy are time-invariant. With 3) and 4), we can rewrite the optimization problem to 6) max E {u t} subject to R W β t 1 x t, u t ) W Q ) xt u t ), 7) x t+1 = Ax t + Bu t + Cε t+1, t = 1, 2...,. Simplifying the notation by x t x and x t+1 x, the Bellman equation of the above problem is 8) x P x + d = max u = max u = max u { ) ) x, u ) R W x + βe [ x P x + d ]} Q u {x Rx + u Qu + 2x W u + βe [ Ax + Bu + Cε ) P Ax + Bu + Cε ) + d ]} W { x Rx + u Qu + 2x W u + β [ Ax + Bu) P Ax + Bu) + trc P C) + d ]}, since E[ 2Ax + Bu) P Cε ] = 0 and 9) E [ ε C P Cε ] = n n [ C P C ] ij Eε iε j) = i=1 j=1 n [ C P C ] ii = trc P C). i=1 When P is negative definite, 10) trc P C) = n n n p ij c ik c jk < 0. i=1 j=1 k=1 The first-order condition is 11) 2Qu + 2W x + 2βB P Ax + Bu) = 0 = Q + βb P B)u = W + βb P A)x, thus, the policy function the decision rule) is 12) u = Q + βb P B) 1 W + βb P A)x F x, 2
where F R k n. Note that the policy function is independent of the size of shocks, C, which is called the certainty equivalent property. Using the policy function, u = F x, the Bellman equation becomes 13) x P x + d = { x Rx + F x) QF x 2x W F x +β [ Ax BF x) P Ax BF x) + trc P C) + d ]} = x { R + F QF 2W F + βa BF ) P A BF ) } x + β [ trc P C) + d ] for all t = 1, 2,.... Since the above equation satisfies for all x, we have 14) 15) P = R + F QF 2W F + βa BF ) P A BF ), d = β [ trc P C) + d ] = d = β1 β) 1 trp CC ) < 0, where F = Q+βB P B) 1 W +βb P A). In the equation of d, the second equality holds because trab) = trba) and, therefore, trabc) = trabc)) = trbc)a) = trbca). Eliminating F from the above equation, we can also obtain the Riccati equation as 16) P = R + βa P A W + βa P B)Q + βb P B) 1 W + βb P A), where we use the following property of a quadratic form: x Ax = x A x R, thus, x A+A )x = 2x Ax = 2x A x. The Riccati equation, or equation 14), is a fixed-point problem, P = fp ), and we have two solution methods: 1. Iterate the value function, lim k P k = P, where P k+1 = fp k ) and P 0 is given, for example P 0 = R; 2. Solve the equation, hp ) fp ) P = 0, for P by the fsolve function or Newton s iteration method. The first method is the value function iteration method, which is usually very slow to converge. The second method is equivalent with the policy function iteration method, which is faster, when we solve hp ) with Newton s iteration method including the fsolve function). When we use Newton s method manually, we need to convert the n n matrix P into an n 2 vector p by using reshapep, n^2, 1) and convert p into P by using reshapep, n, n). 3
2 A Stochastic Growth Model We will analyze the following stochastic growth economy by using the linear-quadratic approximation. The social planner s problem is 17) max E {c t,l t} subject to β t 1 uc t, l t ) 18) 19) 20) 21) n t = 1 l t, y t = z t F k t, n t ), ln z t+1 = ρ ln z t + σε t+1, where ε t+1 N0, 1), k t+1 = 1 δ)k t + y t c t. Eliminating c t, l t, y t, the objective function becomes 22) max E {n t,k t+1 } β t 1 u[ z t F k t, n t ) + 1 δ)k t k t+1, 1 n t ]. The first-order conditions are 23) 24) n t : u l c t, l t ) = u c c t, l t )z t F n k t, n t ), k t+1 : u c c t, l t ) = βe { u c c t+1, l t+1 )[ z t+1 F k k t+1, n t+1 ) + 1 δ ] z t }. The two conditions are the marginal rate of substitution condition and the Euler equation. 2.1 The Certainty Equivalent Steady-State Economy As we saw in Section 1, if the return function, rx, u), is quadratic and the state transition function, gx, u, ε ), is linear, the problem has a certainty-equivalent property, i.e., the optimal decision does not depend on the size of shocks. Let the optimization problem be 25) max E {n t,k t+1 } β t 1 rz t, k t, n t, k t+1 ) = E β t 1 u ) z t F k t, n t ) + 1 δ)k t k t+1, 1 n t. Then, the first-order condition with respect to k t+1 is 26) r 4 z t, k t, n t, k t+1 ) + βe [ r 2 z t+1, k t+1, n t+1, k t+2 ) z t ] = 0, which is identical to the Euler equation 24). When the return function, rz t, k t, n t, k t+1 ), is quadratic, its partial derivative, r j z t, k t, n t, k t+1 ), is linear for all arguments and j = 1,..., 4. Thus, the Euler 4
equation can be shown as 27) r 4 z t, k t, n t, k t+1 ) = βr 2 Ezt+1 z t ), k t+1, En t+1 z t ), Ek t+2 z t ) ), = u c c t, l t ) = βu c Ect+1 z t ), El t+1 z t ) ){ Ez t+1 z t )F k k t+1, En t+1 z t )) + 1 δ }, which has the certainty-equivalent property. Note that a quadratic utility function is not sufficient for the certainty equivalence. Let z = Ez t ), k = Ek t ), and n = En t ). Then, if the return function is quadratic, we can obtain the certainty-equivalent steady state of this economy by solving the unconditional expectations of the first-order conditions, 28) 29) u l z F k, n ) δk, 1 n ) = u c z F k, n ) δk, 1 n )z F n k, n ), 1 = β[ z F k k, n ) + 1 δ ], for k and n, given z. We calculate z = Ez t ) from ρ and σ as 30) ε t N0, σ 2 ) ln z t N0, σ 2 /1 ρ 2 )) z = Ez t ) = e 0.5σ2 /1 ρ 2). Then, we solve the above system of two equations for k and n by the fsolve function, Newton s iteration method, or possibly analytically. When the utility function is Cobb-Douglas/CRRA and the production function is Cobb-Douglas, 31) uc, l) = [ cα l 1 α ] 1 γ, F k, n) = k θ n 1 θ, 1 γ then, we can solve the above steady-state conditions for k and n analytically. From the second condition, 32) k n = β 1 ) 1/θ 1) 1 + δ ; z θ from the law of motion of capital, [ ) k 33) c = z k θ n 1 θ δk = z θ ) ] k δ n ; and from the first equation, we get ) k 34) 1 α)c = α1 n )z θ 1 θ). Eliminating c, we can solve the last equation for n and obtain k. n n n 5
Differences from the Steady State. as We define the differences changes) from the steady-state values 35) ẑ = ẑ t z t z, ˆk = ˆkt k t k, ˆn = ˆn t n t n, ˆk = ˆk t+1 k t+1 k. Then, the return function the utility function) is defined as 36) rẑ, ˆk, ˆn, ˆk ) u [ z + ẑ)f k + ˆk, n + ˆn) + 1 δ)k + ˆk) k + ˆk ), 1 n + ˆn) ]. Log Differences from the Steady State. from the steady-state values. We can alternatively define the log differences % changes) 37) ẑ = ẑ t ln z t ln z, ˆk = ˆkt ln k t ln k, ˆn = ˆn t ln n t ln n, ˆk = ˆk t+1 ln k t+1 ln k. Then, the return function is defined as 38) rẑ, ˆk, ˆn, ˆk ) u [ e ln z +ẑ F e ln k +ˆk, e ln n +ˆn ) + 1 δ)e ln k +ˆk e ln k +ˆk, 1 e ln n +ˆn ]. In a Matlab function file or a Fortran function, we first calculate the original values, z, k, n, and k, from the differences or log differences, ẑ, ˆk, ˆn, and ˆk ; then, we calculate the return utility) level with u[ zf k, n) + 1 δ)k k, 1 n ]. 2.2 The Linear-Quadratic Approximation around the Steady State Let x be ẑ, ˆk, ˆn, ˆk ). The quadratic approximation around the certainty-equivalent steady state, i.e., the second-order Taylor expansion of the return function around x = 0 is 39) rx) r0) + Dr0)x + 1 2 x D 2 r0)x 1 r0). = 1, x................ 2. Dr0)......... ) ) 1 x ). x, u )M, 1 2 Dr0) 1. 2 D2 r0) x u. where x = 1, ẑ, ˆk), u = ˆn, ˆk ), Dr0) is a 1 4 gradient vector Jacobian matrix), and D 2 r0) is a 4 4 Hessian matrix. You can calculate the Jacobian matrix and Hessian matrix by using the numerical ) R W differentiation. Since M =, we can obtain R, Q, and W as W Q 40) R = M [1:3,1:3], Q = M [4:5,4:5], W = M [1:3,4:5]. 6
The law of motion of the state variables, z and k, are 41) ln z = ρ ln z + σε, k = k. When z = z + ẑ and k = k + ˆk, using the first-order Taylor expansion, 42) lnz + ẑ ) = ρ lnz + ẑ) + σε ln z + ẑ z ρ ln z + ẑ ) z + σε ẑ ρ 1)z ln z + ρẑ + σz ε. When ln z = ln z + ẑ and ln k = ln k + ˆk, 43) ẑ = ρ 1) ln z + ρẑ + σε, ˆk = ˆk. In the second case, we have the linear state transition function as 1 1 0 0 1 0 0 ) 0 44) x ẑ = ρ 1) ln z ρ 0 ẑ + ˆn 0 0 + ˆk ˆk 0 0 0 ˆk σ ε Ax + Bu + Cε. 0 1 0 When σ is small, e.g., σ = 0.007 as often assumed in the RBC literature, ρ 1) ln z 0 and z 1. We can replace these with 0 and 1 without making significant approximation errors. 3 Simulating a Stochastic Growth Economy The following is an example of simulating a stochastic growth economy. 1. Choose a large integer T, for example, T = 2000. Generate a sequence of standard normal random numbers, ε t for t = 1,..., T. Note: Use randn function in the Matlab statistics toolbox. Alternatively, you can create a standard normal sequence from a sequence of uniform random numbers by the Box-Muller method Judd, p.290). 2. Set x 1 = 1, 0, 0), which is the steady state of the economy. For t = 1,..., T, calculate u t = F x t, then calculate x t+1 = Ax t + Bu t + Cε t. Note: It does not matter whether we use ε t or ε t+1, and the code will be slightly simpler if we use ε t. 3. Calculate z t and n t for t = 1,..., T and k t for t = 1,..., T + 1 from ẑ t = x t 2); ˆk t = x t 3); ˆn t = u t 1); and ˆk t+1 = u t 2); then calculate y t, c t, and iv t investment) for t = 1,..., T. Also, calculate y, c, and iv by using z, k, and n. 4. Calculate ẑ t, ˆk t, ˆn t, ŷ t, ĉ t, and ˆ iv t for t = 1,..., T as log differences from the steady state. You don t have to recalculate ẑ t, ˆk t, ˆn t if those are already log differences from the steady state. 7
5. Calculate the deviations from trend, z d t, k d t, n d t, y d t, c d t, and iv d t, by using the Hodrick-Prescott filter the H-P filter) with parameter λ = 1600 as z d = ẑ hpfilterẑ, 1600); k d = ˆk hpfilterˆk, 1600);... 6. Calculate the standard deviations and the cross correlations with output of the variables and construct a table similar to Table 1.2 of Cooley and Prescott 1995). References [1] Cooley, Thomas F. Editor), Frontiers of Business Cycle Research, Princeton University Press: Princeton, 1995 [Chapter 1, pp. 1-38; Chapter 2, pp. 39-64] [2] Judd, Kenneth, Numerical Methods in Economics, MIT Press: Cambridge, 1998 [Chapter 12, pp. 431-433] [3] Ljungqvist, Lars and Thomas J. Sargent, Recursive Macroeconomic Theory Third Edition), MIT Press: Cambridge, 2012 [Chapter 5, pp. 127-133, 142-147] [4] Miranda, Mario J. and Paul L. Fackler, Applied Computational Economics and Finance, MIT Press: Cambridge, 2004 [Chapter 9, pp. 221-227] 8