Lecture 5: Some Informal Notes on Dynamic Programming

Transcription

1 Lecture 5: Some Informal Notes on Dynamic Programming The purpose of these class notes is to give an informal introduction to dynamic programming by working out some cases by h. We want to solve subject to {c t,k t+ } β t Uc t c t = fk t + δk t k t+ with k 0 given. Equivalently, the problem that we want to address is the selection of a sequence {k t+ } that solves {k t+ } β t Ufk t + δk t k t+ with k 0 given. Here f is a production function assumed to satisfy the stard Inada conditions. Note that the sum is converging as 0 < β < k is bounded at every t. Finite Horizon Problem To begin, assume that the time horizon is finite equal to T, so that the problem is to find a sequence {k t+ } T to solve {k t+ } T T β t Ufk t + δk t k t+ with k 0 given. A simple way to address this problem is by backward induction. Time T : To keep notation simple, let F k t fk t + δk t, assume to be at time T. Then, k 0,..., k T are given by history, the imization problem to solve the last period problem is simple: β T UF k T k T +. k T + The solution to this problem sets k T + = 0 because at time T + there is no utility from

2 consumption, so there is no point in accumulating capital. Define V 0 k T = UF k T ; this expression gives the imum attainable utility at time T, given the capital level k T inherited historically the optimal choice of k T +. That is, V 0 represents the value of solving the problem at T, when there are zero periods to go the capital stock available, k T, is given. Time T : Consider the situation at T. Now, k 0,..., k T are historically given. The problem can be written as k T β T UF k T k T + β T V 0 k T. The first-order condition is given by U F k T k T = βv 0k T, regularity of U F imply that the first-order condition has a solution given by k T = g T k T. Clearly, the solution will be a function of k T, we want to know something about the map g T. Note that differentiating both sides with respect to k T yields U F dk T dk T = βv 0 dk T dk T, which implies dk T = g T k t = dk T U U + βv 0 F > 0. Therefore, a higher value of k T leads to a higher choice of k T. In addition, g T k T < F k T, because C T = F k T k T, we have dc T dk T = F g T > 0. Note that V 0 = U F > 0 V 0 = U F + U F < 0.

3 Define V k T = UF k T g T k T + βv 0 g T k T, where V = U F g T + βv 0g T = U F > 0 V = U F g T F + U F < 0, k T = g T k T =. Time T : At t = T, k 0,..., k T are given, the problem is to choose k T, k T, k T +. But we already know that k T + = 0 k T = g T k T. Then the problem is β T UF k T k T + βuf k T g T k T + β V 0 g T k T k T = k T UF k T k T + βv k T. The first-order condition for this problem is U F k T k T = βv k T, which gives a solution k T = g T k T. In addition U F g T = βv g T, or with 0 < g T < F a before. g T = U U + βv F, 3

4 Define V k T = UF k T g T k T + βv g T k T. with V = U F g T + βv g T = U F + g T βv U = U F > 0 V = F U + F U F g T < 0. Note that the value function always summarizes what will happen in the future that more or less the same thing is happening in every period. That is, the problem displays a recursive structure. The arguments are not the same every period but the structure of the problem is. We want a sequence V 0 k T, V k T,..., V T k 0 with associated optimal choices g T k T = k T, g T k T = k T,..., g k 0 = k. In general, V n k T n = UF k T n k T n+ + βv n k T n+ V T k 0 = {k t+ } T T β t UF k t k t+ V T k 0 = k {UF k 0 k + βv T k }. Infinite Horizon Problem Suppose now that the time horizon of the problem is infinite. Consider V n. At any time n there are an infinite number of periods to go. The issue is whether there is a time invariant value function V that is the limit of the finite horizon case. The answer to this 4

5 question is positive for the problems that we will consider. Associated with V there is a time-invariant policy function g such that k t+ = gk t. In general, this is the value of knowing the value function it allows to know the policy function, which tells us what the behavioral implication of the model are. More formally, V T k 0 = {k t+ } T T β t UF k t k t+ lim V T = V. T This is the idea behind Bellman s principle of optimality the idea, more or less, is that if you consider a sequence of imal payoffs remove the first n elements, the remaining tail of the sequence must still be optimal. Formally we can express this idea as follows: V k 0 = {k t+ } T = {k t+ } T β t UF k t k t+ { UF k 0 k + β } β t UF k t k t+ t= = k {UF k 0 k + βv k }. Maximization of the infinite stream of payoffs is equivalent to imizing today s payoff followed by the imal continuation value. The equation V k 0 = k {UF k 0 k + βv k } is an example of a functional equation it is referred to as the Bellman equation. The unknown in the Bellman equation is the function V. For a special class of payoff functions transition equations the Bellman equation can be solved explicitly by h. These case include Uc t = ln c t We will mention the problem of proving the existence, uniqueness, differentiability of V. 5

6 F k t = Ak α, with δ =. In this case there are two easy methods that can be applied to obtain the value function: i the method of undetermined coefficients, ii the value function iteration.. Undetermined coefficients For the method of undetermined coefficients we guess that V k t = B + b ln k t. We then verify that this is an admissible solution of the Bellman equation. To do so we solve: Uc t + βv k t+ {c t,k t+ } or, substituting c t = Ak α t k t+ our guess for V k, k t+ lnak α t k t+ + βb + b ln k t+ taking k t as given. The first-order condition is Ak α t k t+ = βb k t+ = k t+ = βbakα t + βb Plugging the first-order condition into V k t = k t+ {Uc t + βv k t+ } 6

7 using our guess for V k t, we know that V k t = ln Akt α βbakα t + βb + βb ln βbakα t + βb + βb. Our guess for V implies that V k t = B + b ln k t = ln Akt α βbakα t + βb + βb ln βbakα t + βb + βb it holds for all k t. We choose B b so that our guess is verified. Exping the right-h side of this equation we rewrite it as: V k t = α ln k t + ln = ln A A βba + βb βba + βb + βb + βb ln βba + βb + βbα ln k t + βb ln + βb βba + βb + α + βbk t. Since this must be equal to our guess V k t = B + b ln k t, collecting the terms yield the following two equations: α + βb = b, which equates coefficient associated with ln k t in both equations, ln A βba βba + βb + βb ln = B, + βb + βb which equate the remaining terms. Solving the first equation for b yields b = α αβ. 7

8 The second equation can be rewritten as A + βb βba βba βb = ln + βb + βb ln + βb + βb βb = ln A ln + βb + βb lnβba βb ln + βb βb = + βb ln A + βb ln + βb + βb lnβb. Substituting the expression for b yields αβ βb = ln A ln + αβ ln αβ αβ αβ βb = ln A αβ ln + αβ lnαβ αβ αβ αβ βb = ln A + ln αβ + αβ αβ lnαβ finally, after adding subtracting at the expression for B: αβ αβ ln A from the right-h side, we arrive B = lna αβ + αβ β αβ lnaαβ. We have shown, therefore, that the value function is given by V k t = B + b ln k t. Substituting the value for b in the policy function we get k t+ = βαak α t. This is the same result that we get when we solve the imization problem with the Euler equation approach. To see that, consider the problem {k t+ } β t lnakt α k t+. 8

9 The first order condition at time t, the Euler equation, is β t Ak α t k t+ = β t+ Ak α t+ k t+ αak α t+. Rearranging the terms yields Ak α t+ k t+ = β Ak α t k t+ αak α t+. Since c t = Ak α t k t+ we get c t+ = βc t αak α t+. Given the resource constraint k t+ + c t = Ak α t, we have the system of two first order difference equations that gives the solution to our problem. From the solution to the Bellman equation remember that the solution to the Bellman equation is the value function as the solution to a functional equation is a function we know that k t+ = βαakt α. The resource constraint then tells us that c t = αβak α t. Shifting it by one period c t+ = αβak α t+ plugging it into the first-order condition yields αβak α t+ = βc t αak α t+, or αβk t+ = αβc t. 9

10 Using the resource constraint c t = Ak α t k t+, we obtain k t+ = βαak α t, which is policy function we obtained with the Bellman equation.. Value Function Iteration An important result in dynamic programming is that the sequence of value functions that solves the finite horizon problem converges uniformly under some conditions to the value function that solves the infinite horizon problem. A consequence of this result is that another method to solve dynamic programming finding the policy function is iteration of the value function. We will apply this method to the problem that we have addressed so far, show that we obtain the same solution. The problem is, once again, subject to {c t,k t+ } β t Uc t c t = Ak α t k t+ with k 0 given. An easy way to start the iteration process is to choose V 0 k T = 0, where the meaning of V 0 is the same as before there are no periods left. Since at the end of time the value of future capital is zero this guess is reasonable. With this initial choice, working backwards as in the finite horizon case, we know: V k T = k T lnak α T k T + βv 0 k T which has an easy solution given that V 0 k T = 0: k T = 0. With this solution we have that c T = Ak α T, therefore, V k T = ln A + α ln k T. 0

11 The next problem is then to determine V. The Bellman equation is V k T = k T lnak α T k T + βv k T that is V k T = k T lnak α T k T + β ln A + α ln k T. The first-order condition for this problem is Ak α T k T = αβ k T which implies k T = c T = Ak α T k T = αβ AkT α AkT α. The value function is then V k T = ln = ln αβ Akα T + β ln A + α ln Akα T A αβa + β ln A + αβ ln + α ln k T. Now let A αβa v0 = ln + β ln A + αβ ln v = α so the V k T can be expressed as V k T = v 0 + v ln k T.

12 Moving another step backward, V 3 k T 3 = k T ln c T 3 + βv k T or equivalently, V 3 k T 3 = k T lnak α T 3 k T + β v 0 + v ln k T. The first-order condition for this problem is given by Ak α T 3 k T = αβ k T or k T = Ak α T 3 k T αβ. This yields k T = + αβ Ak α T 3 αβ + αβ c T 3 = + αβ AkT α 3. Given this expressions, the value function can be written as αβ + αβ V 3 k T 3 = ln + β ln + αβ Akα T 3 A + β ln A + αβ ln αβa + αβ ln + αβ Akα T 3. This can be simplified to αβ + αβ V 3 k T 3 = ln + αβ A A + β ln + β ln A + αβ ln A + αβ ln + αβ αβa + α + αβ ln k T 3.

13 As before, define + αβ A αβa αβ + αβ v0 3 = ln + αβ ln v 3 =α + αβ A + β ln + αβ ln + β ln A A + αβ so that V 3 k T 3 = v v 3 ln k T 3. We move another step backward to compute V 4. The problem we have to address is V 4 k T 4 = k T 3 ln c T 4 + βv 3 k T 3 or, equivalently, V 4 k T 4 = k T 3 lnak α T 4 k T 3 + β v v 3 ln k T 3. The first-order condition for this problem is given by Ak α T 4 k T 3 = αβ + αβ k T 3 or k T = Ak α T 3 k T αβ. This yields k T 3 = + αβ + αβ 3 Ak α T 4 αβ + αβ + αβ 3 c T 3 = + αβ + αβ 3 AkT α 4. We can then write αβ + αβ + αβ 3 V 4 k T 4 = ln + αβ + αβ 3 Akα T 4 + βv 3 k T 3 + β v0 3 + v 3 ln k T 3. 3

14 Writing it out yields αβ + αβ + αβ 3 V 4 k T 4 = ln + αβ + αβ 3 Akα T 4 A + β ln + β 3 ln A + αβ 3 ln + αβ A ln + αβ + αβ + αβ + αβ 3 ln αβ + αβ + β ln αβa + αβ A + αβ + αβ 3 Akα T 4. We can collect terms to write αβ + αβ + αβ 3 αβ + αβ V 4 k T 4 = ln + αβ + αβ 3 A + β ln + αβ A A αβa + β ln + β 3 ln A + αβ 3 ln + αβ A ln + αβ + αβ + αβ + αβ 3 ln + αβ + αβ 3 A + α αβ + αβ + αβ 3 k T 4. As before, define αβ + αβ v0 4 + αβ 3 αβ + αβ = ln + αβ + αβ 3 A + β ln + αβ A A αβa + β ln + β 3 ln A + αβ 3 ln + αβ A ln + αβ + αβ + αβ + αβ 3 ln + αβ + αβ 3 A, v 4 =α αβ + αβ + αβ 3 so that V 4 k T 4 = v v 4 ln k T 4. Note that some patterns are beginning to emerge. In general, we can guess that 4

15 when we are solving the problem t periods from the end of time we get v t = α so for infinitely many periods, t αβ i, i= v = lim t α t αβ i = i= α αβ. For the other series, we can see that v t 0 is composed by two sub series. Let x i A t = ln + αβ αβ i t yt i = αβ + αβ αβ i t + αβ αβ i t ln + αβ αβa αβ i t v i 0 Since can be written as we get the value function is V k t = i lim i i lim i i i v0 i = β t x i t + αβ β t yt. i β t x i t = β t yt i = ln A αβ β αβ ln Aαβ β αβ lim i vi 0 = ln A αβ + αβ ln Aαβ β αβ ln A αβ + β αβ ln Aαβ + αβ α αβ ln k t as we had determined with the method of undetermined coefficients. 5