Lecture 7: Stochastic Dynamic Programing and Markov Processes

Transcription

1 Lecture 7: Stochastic Dynamic Programing and Markov Processes Florian Scheuer References: SLP chapters 9, 10, 11; LS chapters 2 and 6 1 Examples 1.1 Neoclassical Growth Model with Stochastic Technology Production function y = A f k where A is random Let A s t be productivity in history s t We could have A be a function of s t only, i.e. A s t but allocations will still be functions of histories Sequence problem V k 0, A 0 = 1.2 A Model of Job Search A worker is unemployed max cs t,ks t t=0 s t β t Pr s t u c s t s.t. k s t+1 = A s t f k s t + 1 δ k s t c s t k s t 0 c s t 0 k 0, A 0 given Each period a worker begins by receving a wage offer w s t. 1

2 We can interpret w s t = 0 as not getting an offer If he accepts the offer, he gets paid w per period forever If he rejects the offer he gets unemployment benefits b instead The worker cannot borrow or save, so he consumes either w or b respectively Decision: acceptance decision as t {0, 1} as a function of history s t Sequence problem: c s t = { V = max as t t=0 s t β t Pr s t u c s t s.t. b if a s j = 0 for all j t w s τ if a s j = 0 for all j < τ and a s τ = Consumption-Savings Under Uncertainty Household gets income y s t in history s t Again, we could simplify this to y s t Can borrow up to a limit and save at the risk-free interest rate R Does not have access to insurance i.e. there are no Arrow-Debreu securities he can buy Sequence problem 2 Markov Processes V A 0, y 0 = max As t,cs t t=0 s t β t Pr s t u c s t s.t. A s t+1 = RA s t + y s t c s t A s t+1 B A 0 and y 0 given Stochastic process: sequence of random vectors x t X 2

3 Example: x t = {A t, k t } capital and productivity x t could be composed of exogenous variables, endogenous variables or a combination thereof Markov property: Pr x t+1 x t, x t 1,..., x t k = Pr x t+1 x t for all k The state today is a sufficient statistic for forecasting the state tomorrow This is not as restrictive as it seems To avoid going too much into measure theory, we are going to focus mostly on cases where X is a finite set In those cases, a Markov process can also be called a Markov chain We ll briefly mention how some of the results generalize to a continuous state space 2.1 Markov chains Discrete state space x {x 1,..., x N } The evolution of x is governed by a matrix P sometimes called transition matrix, Markov matrix or stochastic matrix Definition 1. A stochastic matrix is an N N matrix P such that: N P ij = 1 j=1 i P ij = Probability that tomorrow you ll be in state j given that today you are in state i Suppose we represent the probability distribution over states in period t by π t where π t is a row vector such that N π ti = 1 i=1 then πp represents the probability distribution over states in period t + 1 3

4 Example 1: π = i.e. there are three possible states and we start at x = x 2 for sure P = p 11 p 12 p p 31 p 32 p 33 so πp = so the second row of P tells us the probability distributions over states tomorrow given the state today Example 2: π = P = p 31 p 32 p 33 πp = so if in period t there s a fifty-fifty chance of being is states x 1 or x 2, then πp gives us the probabilities of being in the three states in period t + 1 Notation note: Sometimes people use the opposite row/column convention for transition matrices: P i,j = Probability that tomorrow you ll be in state i given that today you are in state j For this you should write π as a column vector and π = Pπ Of course, this is all equivalent 2.2 The Markov Assumption Suppose I have the following process: x X = {Rain, Sun} 0.1 if x t = x t 1 = Rain Pr {x t+1 = Rain x t, x t 1 } 0.6 if x t = Rain and x t 1 = Sun 0.3 if x t = Sun 4

5 This seems not to satisfy the Markov assumption because the weather today is not a sufficient statistic to forecast the weather tomorrow But a simple transformation restores that Markov property: X = {NewRain, OldRain, Sun} Transition Matrix P = Many history-dependent processes can be rewritten this way as Markov Processes For now, we ll use the Markov assumption to have a justification for saying Pr x x and knowing that we can ignore the history of x Later we ll ask questions about Markov processes themselves: What do they look like in the long run? Under what conditions do they converge? What do we mean by converge? 3 Recursive Setup Often, there is more than one way to set things up Typical sequence: Stuff happens Make decisions Stuff happens Make decisions... Do I compute the value function at the time I m about to make a decision or when something is about to happen? What do I define as the state variable? 5

6 The results on the equivalence of the sequence problem and the functional equation extend to the stochastic case, s.t. some technical caveats See SLP Theorems 9.2 and Neoclassical Growth Model with Stochastic Technology State variables: capital stock endogenous productivity exogenous Assume A follows a Markov process Bellman equation: or simply where V k, A = for a discrete space for A V k, A = max c,k u c + βe V k, A A s.t. k = A f k + 1 δ k c k 0 c 0 max u A f k + 1 δ k k + βe V k, A A k [0,A f k+1 δk] E V k, A A = A Pr A A V k, A FOC for c: FOC for k : Envelope condition: and therefore u c λ = 0 β A Pr A A V K k, A λ = 0 V K k, A = λ [ A f k + 1 δ ] V K k, A = λ [ A f k + 1 δ ] 6

7 Putting these together u c k, A = β A Pr A A u c k, A [ A f k + 1 δ ] which is the Euler equation that we also encountered when we looked at an endowment economy: u c t = βe [ Ru c t+1 ] where R = A f k + 1 δ is the stochastic return of a unit of capital The solution defines a system of stochastic difference equations in k, c and A: u c t = βe [ A t+1 f k t δ u c t+1 ] k t+1 = A t f k t + 1 δ c t A t+1 exogenous Markov process This is a Markov process for the vector k, c, A Unfortunately, unless we restrict k and c to grids, this Markov process lives in a continuous state space even if A lives in a discrete state space Questions: How does this system behave over time? Does it tend to a steady state? In what sense? Are there values that it will never reach / reach repeatedly? 3.2 Job search model Assume w follows a Markov process Two ways to do it: 1. Define value function after receiving today s offer 2. Define value function before receiving today s offer Option 1: w V w = max au a {0,1} 1 β + 1 a E V w w 7

8 Option 2: V w [ = max E a u w aw 1 β ] + 1 a V w w Option 2 seems more complicated but can be easier. Suppose w is iid Option 1: Option 2: w V w = max au a {0,1} 1 β + 1 a E V w [ V = max E a u w ] + 1 a V aw 1 β Exercise Show that job acceptance policy is simply a reservation wage 2. Show that the reservation wage increases if G becomes more dispersed in a SOSD sense 3.3 Consumption-Savings Assume y follows a Markov process State variables: A : Level of assets y: today s income VA, y = max uc + β Pry yva, y 1 c,a s s.t. A y c + RA A b Alternatively, let x RA + y cash on hand : Vx, y = max uc + β Pry yvx y, y 2 c,x y y s.t. x y R [x c] + y c x + b This relabeling is especially useful if y is iid: reduce problem to a single state variable 8

9 because with x, y as the state variables,y only matters through Pr y y FOC: u c y λ y R µ = 0 β Pr y y Vx y, y x with equality if c < x + b λ y = 0 u c βr Pry y Vx y, y y x Envelope condition Vx, y x = u cx, y Euler equation: with equality if c < x + b u c βr s Prs su cx y, y 4 Convergence of Markov Processes Starting from π 0 which could be degenerate, a Markov chain will evolve according to π t = π 0 P t π belongs to the set N = { π R n + : } N π i = 1 i=1 Define the following norm in N : π = N π i i=1 We say a Markov chain converges to π if lim π t π = 0 T In this case we call π an invariant distribution and the support of π an ergodic set 9

10 Define the operator T : N N as T π = πp Proposition 1. SLP Lemma 11.3 Let ɛ j contraction modulus 1 ɛ. = min i P ij and ɛ = N j=1 ɛ j. If ɛ > 0 then T is a Example: P = Row i: probabilities over states at t + 1 if the state at t is i must add up to 1 Column j: probabilities of being in state j at t + 1 depending on what is the state at period t do not need to add up to 1 ɛ j : minimum probability of reaching state j in period t + 1 over all possible states at t ɛ sum of all ɛ j Here ɛ j = ɛ = 0.3 Interpretation: ɛ > 0 if there is at least one value of x that you reach with positive probability from any other state Proof. The distance between T π and T µ is: ρ T π, T µ = T π T µ = π µ P = π i µ i P ij j i 10

11 Example: j π = µ = π µ = π i µ i P ij = i π i µ i P ij = = i continue proof = j j = j i i i i π i µ i P ij ɛ j + π i µ i ɛ j i π i µ i P ij ɛ j + π i µ i ɛ j j i π i µ i P ij ɛ j π i µ i j = ρ π, µ 1 ɛ Pij ɛ j Proposition 1 immediately implies: There is a unique fixed point, i.e. unique solution to π = T π denote it π Starting from any π 0, lim n Tn π 0 = π You can find π by solving π = πp π I P = 0 11

12 i.e. π is the eigenvector corresponding to the eigenvalue 1 of matrix P, scaled so that i=1 N π i = 1. Generalizations: If the condition ɛ > 0 holds for the matrix P n, then T n is a contraction mapping, so the limiting condition holds too Interpretation: there is a state that you reach with positive probability in n steps starting from anywhere For states spaces that are not finite, SLP provide what they call condition M, which ensures that the Markov process converges. Definition 2. A Markov process satisfies condition M if there exist ɛ > 0 and n 1 such that for any set A X, either: 1. Pr x t+n A x t ɛ for all x t X or 2. Pr x t+n A C x t ɛ for all xt X Example: suppose there is a recurrent element x R X such that Pr x t+n = x R x t ɛ x t Then condition M is satisfied because for any A, either x R A or x R A C Proposition 2. SLP Lemma If a Markov process satisfies condition M, then T n is a contraction of modulus ɛ. 12