NUMERICAL DYNAMIC PROGRAMMING

Transcription

1 NUMERICAL DYNAMIC PROGRAMMING Kenneth L. Judd Hoover Institution and NBER July 29,

2 Dynamic Programming Foundation of dynamic economic modelling Individual decisionmaking Social planners problems, Pareto efficiency Dynamic games Computational considerations Applies a wide range of numerical methods: Optimization, approximation, integration Can exploit any architecture, including high-power and high-throughput computing Outline Review of Dynamic Programming Necessary Numerical Techniques Approximation Integration Numerical Dynamic Programming

3 Discrete-Time Dynamic Programming Objective: E ( TX t=1 X is set of states and D is the set of controls π(x t,u t,t)+w (x T +1 ) ), (12.1.1) π(x, u, t) payoffs inperiodt, forx X at the beginning of period t, andcontrolu D is applied in period t. D(x, t) D: controls which are feasible in state x at time t. F (A; x, u, t) :probability that x t+1 A X conditional on time t control and state Value function definition ( TX V (x, t) sup E U(x,t) s=t π(x s,u s,s)+w (x T +1 ) x t = x ). (12.1.2) Bellman equation V (x, t) = sup u D(x,t) π(x, u, t)+e {V (x t+1,t+1) x t = x, u t = u} (12.1.3) Existence: boundedness of π is sufficient

4 Autonomous, Infinite-Horizon Problem: Objective: max u t ( ) X E β t π(x t,u t ) t=1 (12.1.1) Value function definition: if U(x) is set of all feasible strategies starting at x. ( ) X V (x) sup E β t π(x t,u t ) U(x) x0 = x, (12.1.8) Bellman equation for V (x) t=0 V (x) = sup u D(x) π(x, u)+βe V (x + ) x, u ª (TV)(x), (12.1.9) Optimal policy function, U(x), ifitexists, isdefined by U(x) arg max π(x, u)+βe V (x + ) x, u ª u D(x) Standard existence theorem: If X is compact, β<1, and π is bounded above and below, then TV = sup u D(x) π(x, u)+βe V (x + ) x, u ª ( ) is monotone in V, and a contraction mapping with modulus β in the space of bounded functions, and has a unique fixed point.

5 Deterministic Growth Example Problem: Euler equation: V (k 0 )=max ct P t=0 βt u(c t ), k t+1 = F (k t ) c t k 0 given u 0 (c t )=βu 0 (c t+1 )F 0 (k t+1 ) ( ) Bellman equation V (k) =max u(c)+βv (F (k) c). ( ) c Solution to ( ) is a policy function C(k) and a value function V (k) satisfying 0=u 0 (C(k))F 0 (k) V 0 (k) ( ) V (k)=u(c(k)) + βv (F (k) C(k)) ( ) ( ) defines the value of an arbitrary policy function C(k), not just for the optimal C(k). The pair ( ) and ( ) expresses the value function given a policy, and a first-order condition for optimality.

6 Stochastic Growth Accumulation Problem: ( X ) V (k, θ) =max c t, t E t=0 β t u(c t ) k t+1 = F (k t,θ t ) c t θ t+1 = g(θ t,ε t ) ε t : i.i.d. random variable k 0 = k, θ 0 = θ. State variables: k: productive capital stock, endogenous θ: productivity state, exogenous The dynamic programming formulation is V (k, θ) =max c u(c)+βe{v (F (k, θ) c, θ + ) θ} ( ) θ + = g(θ, ε) The control law c = C(k, θ) satisfies the first-order conditions 0=u c (C(k, θ)) βe{u c (C(k +,θ + ))F k (k +,θ + ) θ}, ( ) where k + F (k, L(k, θ),θ) C(k, θ),

7 Discrete State Space Problems State space X = {x i,i=1,,n} Controls D = {u i i =1,..., m} qij t (u) =Pr(x t+1 = x j x t = x i,u t = u) Q t (u) = qij t (u) : Markov transition matrix at t if u i,j t = u. Value Function Iteration: Discrete-State Problems State space X = {x i,i=1,,n} and controls D = {u i i =1,..., m} Terminal value: V T +1 i = W (x i ),i=1,,n. Bellman equation: time t value function is V t i =max u [π(x i,u,t)+β nx j=1 q t ij(u) V t+1 j ], i=1,,n Bellman equation can be directly implemented - called value function iteration. Only choice for finite T.

8 Infinite-horizon problems Bellman equation is now a simultaneous set of equations for V i values: nx V i =max π(x i,u)+β q ij (u) V j,i=1,,n u Value function iteration is V k+1 i U k+1 i =max u =argmax u π(x i,u)+β j=1 nx j=1 π(x i,u)+β q ij (u) V k j nx j=1 Can use value function iteration with arbitrary V 0 i Error is given by contraction mapping property: V k V 1 1 β q ij (u) V k j,i=1,,n,i=1,,n and iterate k. V k+1 V k Stopping rule: continue until V k V <εwhere ε is desired accuracy.

9 Policy Iteration (a.k.a. Howard improvement) Value function iteration is a slow process Linear convergence at rate β Convergence is particularly slow if β is close to 1. Policy iteration is faster Current guess: V k i,i=1,,n. Iteration: compute optimal policy today if V k is value tomorrow: nx Ui k+1 =argmax π(x i,u)+β q ij (u) Vj k,i=1,,n, u Compute the value function if the policy U k+1 is used forever, which is solution to the linear system Vi k+1 = π nx x i,ui k+1 + β q ij (Ui k+1 ) Vj k+1,i=1,,n, j=1 Policy iteration depends on only monotonicity If initial guess is above or below solution then policy iteration is between truth and value function iterate Works well even for β close to 1. j=1

10 Linear Programming Approach If D is finite, we can reformulate dynamic programming as a linear programming problem. (12.3.4) is equivalent to the linear program P min n Vi i=1 V i s.t. V i π(x i,u)+β P n j=1 q ij(u)v j, i, u D, Computational considerations ( ) ( ) may be a large problem Trick and Zin (1997) pursued an acceleration approach with success. Recent work by Daniela Pucci de Farias and Ben van Roy has revived interest. Continuous states: Discretization Method: Replace continuous X with a finite X = {x i,i=1,,n} X Proceed withafinite-state method. Problems: Sometimes need to alter space of controls to assure landing on an x in X. A fine discretization often necessary to get accurate approximations

11 Continuous Methods for Continuous-State Problems Basic Bellman equation: V (x) = max u D(x) π(u, x)+βe{v (x+ ) x, u)} (TV)(x). (12.7.1) Discretization essentially approximates V with a step function Approximation theory provides better methods to approximate continuous functions. General Task Choose a finite-dimensional parameterization V (x). = ˆV (x; a), a R m (12.7.2) and a finite number of states X = {x 1,x 2,,x n }, (12.7.3) Find coefficients a R m such that ˆV (x; a) approximately satisfies the Bellman equation.

12 General Parametric Approach: Approximating T For each x j, (TV)(x j ) is defined by v j =(TV)(x j )= max u D(x j ) π(u, x j)+β Z ˆV (x + ; a)df (x + x j,u) (12.7.5) In practice, we compute the approximation ˆT v j =(ˆTV)(x j ). =(TV)(x j ) Integration step: for ω j and x j for some numerical quadrature formula Z E{V (x + ; a) x j,u)}= ˆV (x + ; a)df (x + x j,u) Z = ˆV (g(x j,u,ε); a)df (ε). = X ω ˆV (g(x j,u,ε ); a) Maximization step: for x i X, evaluate Fitting step: v i =(T ˆV )(x i ) Data: (v i,x i ),i=1,,n Objective: find an a R m such that ˆV (x; a) best fits the data Methods: determined by ˆV (x; a)

13 Approximation Methods General Objective: Given data about f(x) construct simpler g(x) approximating f(x). Questions: What data should be produced and used? What family of simpler functions should be used? What notion of approximation do we use? Comparisons with statistical regression Both approximate an unknown function and use a finite amount of data Statistical data is noisy but we assume data errors are small Nature produces data for statistical analysis but we produce the data in function approximation

14 Interpolation Methods Interpolation: find g (x) from an n-d family of functions to exactly fit n data items Lagrange polynomial interpolation Data: (x i,y i ),i=1,.., n. Objective: Find a polynomial of degree n 1, p n (x), which agrees with the data, i.e., y i = f(x i ),i=1,.., n Result: If the x i are distinct, there is a unique interpolating polynomial Does p n (x) converge to f (x) as we use more points? Consider f(x) = 1 1+x 2,x i uniform on [ 5, 5] Figure 1:

15 Hermite polynomial interpolation Data: (x i,y i,y 0 i),i=1,.., n. Objective: Find a polynomial of degree 2n 1, p(x), which agrees with the data, i.e., y i =p(x i ),i=1,.., n yi 0 =p 0 (x i ),i=1,.., n Result: If the x i are distinct, there is a unique interpolating polynomial Least squares approximation Data: A function, f(x). Objective: Find a function g(x) from a class G that best approximates f(x), i.e., g =argmax g G kf gk2

16 Orthogonal polynomials General orthogonal polynomials Space: polynomials over domain D weighting function: w(x) > 0 Inner product: hf,gi = R D f(x)g(x)w(x)dx Definition: {φ i } is a family of orthogonal polynomials w.r.t w (x) iff φi,φ j =0,i6= j We like to compute orthogonal polynomials using recurrence formulas φ 0 (x)=1 φ 1 (x)=x φ k+1 (x)=(a k+1 x + b k ) φ k (x)+c k+1 φ k 1 (x)

17 Chebyshev polynomials [a, b] =[ 1, 1] and w(x) = 1 x 2 1/2 T n (x) =cos(n cos 1 x) T 0 (x) =1 T 1 (x) =x T n+1 (x) =2xT n (x) T n 1 (x), General Orthogonal Polynomials Few problems have the specific intervals and weights used in definitions One must adapt interval through linear COV: If compact interval [a, b] is mapped to [ 1, 1] by y = 1+2 x a b a then φ i 1+2 x a b a are orthogonal over x [a, b] with respect to w 1+2 x a b a iff φi (y) are orthogonal over y [ 1, 1] w.r.t. w (y)

18 Regression Data: (x i,y i ),i=1,.., n. Objective: Find a function f(x; β) with β R m., m n, with y i = f(xi ),i=1,.., n. Least Squares regression: Chebyshev Regression Chebyshev Regression Data: X min (yi f (x i ; β)) 2 β R m (x i,y i ),i=1,.., n > m, x i are the n zeroes of T n (x) adapted to [a, b] Chebyshev Interpolation Data: (x i,y i ),i=1,.., n = m, x i are the n zeroes of T n (x)adapted to [a, b]

19 Algorithm 6.4: Chebyshev Approximation Algorithm in R 1 Objective: Given f(x) defined on [a, b], find its Chebyshev polynomial approximation p(x) Step 1: Compute the m n +1Chebyshev interpolation nodes on [ 1, 1]: µ 2k 1 z k = cos 2m π,k=1,,m. Step 2: Adjust nodes to [a, b] interval: µ b a x k =(z k +1) + a, k =1,..., m. 2 Step 3: Evaluate f at approximation nodes: w k = f(x k ),k=1,,m. Step 4: Compute Chebyshev coefficients, a i,i=0,,n: P m k=1 a i = w kt i (z k ) P m k=1 T i(z k ) 2 to arrive at approximation of f(x, y) on [a, b]: nx µ p(x) = a i T i 2 x a b a 1 i=0

20 Minmax Approximation Data: (x i,y i ),i=1,.., n. Objective: L fit Problem: Difficult to compute min max β R m i ky i f (x i ; β)k Chebyshevminmaxproperty Theorem 1 Suppose f :[ 1, 1] R is C k for some k 1, andleti n be the degree n polynomial interpolation of f based at the zeroes of T n (x). Then µ 2 k f I n k log(n +1)+1 π (n k)! ³ µ π k b a k k f (k) k n! 2 2 Chebyshev interpolation: converges in L essentially achieves minmax approximation easy to compute does not approximate f 0

21 Splines Definition 2 Afunctions(x) on [a, b] is a spline of order n iff 1. s is C n 2 on [a, b], and 2. there is a grid of points (called nodes) a = x 0 <x 1 < <x m = b such that s(x) is a polynomial of degree n 1 on each subinterval [x i,x i+1 ], i =0,...,m 1. Note: an order 2 spline is the piecewise linear interpolant. Cubic Splines Lagrange data set: {(x i,y i ) i =0,,n}. Nodes: The x i are the nodes of the spline Functional form: s(x) =a i + b i x + c i x 2 + d i x 3 on [x i 1,x i ] Unknowns: 4n unknown coefficients, a i,b i,c i,d i,i=1, n.

22 Conditions: 2n interpolation and continuity conditions: y i =a i + b i x i + c i x 2 i + d i x 3 i, i =1,.,n y i =a i+1 + b i+1 x i + c i+1 x 2 i + d i+1 x 3 i, i =0,.,n 1 2n 2 conditions from C 2 at the interior: for i =1, n 1, b i +2c i x i +3d i x 2 i =b i+1 +2c i+1 x i +3d i+1 x 2 i 2c i +6d i x i =2c i+1 +6d i+1 x i Equations (1 4) are 4n 2 linear equations in 4n unknown parameters, a, b, c, and d. construct 2 side conditions: natural spline: s 0 (x 0 )=0=s 0 (x n ); it minimizes total curvature, R x n x 0 s 00 (x) 2 dx, among solutions to (1-4). Hermite spline: s 0 (x 0 )=y0 0 and s 0 (x n )=yn 0 (assumes extra data) Secant Hermite spline: s 0 (x 0 )=(s(x 1 ) s(x 0 ))/(x 1 x 0 ) and s 0 (x n )=(s(x n ) s(x n 1 ))/(x n x n 1 ). not-a-knot: choosej = i 1,i 2, such that i 1 +1<i 2, and set d j = d j+1. Solve system by special (sparse) methods; see spline fit packages

23 Shape-preservation Concave (monotone) data may lead to nonconcave (nonmonotone) approximations. Example Schumaker Procedure: 1. Take level (and maybe slope) data at nodes x i 2. Add intermediate nodes z + i [x i,x i+1 ] 3. Run quadratic spline with nodes at the x and z nodes which intepolate data and preserves shape. 4. Schumaker formulas tell one how to choose the z and spline coefficients (see book and correction at book s website) Many other procedures exist for one-dimensional problems, but few procedures exist for twodimensional problems

24 Spline summary: Evaluation is cheap Splines are locally low-order polynomial. Can choose intervals so that finding which [x i,x i+1 ] contains a specific x is easy. Finding enclosing interval for general x i sequence requires at most dlog 2 ne comparisons Good fits even for functions with discontinuous or large higher-order derivatives. E.g., quality of cubic splines depends only on f (4) (x), not f (5) (x). Can use splines to preserve shape conditions

25 Multidimensional approximation methods Lagrange Interpolation Data: D {(x i,z i )} N i=1 Rn+m,wherex i R n and z i R m Objective: find f : R n R m such that z i = f(x i ). Need to choose nodes carefully. Task: Find combinations of interpolation nodes and spanning functions to produce a nonsingular (well-conditioned) interpolation matrix.

26 Tensor products General Approach: If A and B are sets of functions over x R n, y R m, their tensor product is A B = {ϕ(x)ψ(y) ϕ A, ψ B}. Given a basis for functions of x i, Φ i = {ϕ i k (x i)} k=0, the n-fold tensor product basis for functions of (x 1,x 2,...,x n ) is ( ny ) Φ = ϕ i k i (x i ) k i =0, 1,,i=1,...,n Orthogonal polynomials and Least-square approximation Suppose Φ i are orthogonal with respect to w i (x i ) over [a i,b i ] Least squares approximation of f(x 1,,x n ) in Φ is X hϕ, fi hϕ, ϕi ϕ, i=1 where the product weighting function defines h, i over D = Q i [a i,b i ] in ϕ Φ W (x 1,x 2,,x n )= hf(x),g(x)i = Z D ny i=1 w i (x i ) f(x)g(x)w (x)dx.

27 Algorithm 6.4: Chebyshev Approximation Algorithm in R 2 Objective: Given f(x, y) defined on [a, b] [c, d], find its Chebyshev polynomial approximation p(x, y) Step 1: Compute the m n +1Chebyshev interpolation nodes on [ 1, 1]: µ 2k 1 z k = cos 2m π,k=1,,m. Step 2: Adjust nodes to [a, b] and [c, d] intervals: µ b a x k =(z k +1) + a, k =1,..., m. 2 µ d c y k =(z k +1) + c, k =1,..., m. 2 Step 3: Evaluate f at approximation nodes: w k, = f(x k,y ),k=1,,m., =1,,m. Step 4: Compute Chebyshev coefficients, a ij,i,j =0,,n: P m P m k=1 =1 a ij = w k, T i (z k )T j (z ) ( P m k=1 T i(z k ) 2 )( P m =1 T j(z ) 2 ) to arrive at approximation of f(x, y) on [a, b] [c, d]: nx nx µ p(x, y) = a ij T i 2 x a µ b a 1 T j 2 y c d c 1 i=0 j=0

28 Multidimensional Splines B-splines: Multidimensional versions of splines can be constructed through tensor products; here B-splines would be useful. Summary Tensor products directly extend one-dimensional methods to n dimensions Curse of dimensionality often makes tensor products impractical Complete polynomials Taylor s theorem for R n produces the approximation f(x). =f(x 0 )+ P n i= P n i 1 =1 P n i 2 =1 f x i (x 0 )(x i x 0 i ) 2 f x i1 x ik (x 0 )(x i1 x 0 i 1 )(x ik x 0 i k )+... For k =1, Taylor s theorem for n dimensions used the linear functions P1 n {1,x 1,x 2,,x n } For k =2, Taylor s theorem uses P2 n P1 n {x 2 1,,x 2 n,x 1 x 2,x 1 x 3,,x n 1 x n }. In general, the kth degree expansion uses the complete set of polynomials of total degree k in n variables. nx Pk n {x i 1 1 x i n n i k, 0 i 1,,i n } =1

29 Complete orthogonal basis includes only terms with total degree k or less. Sizes of alternative bases degree k Pk n Tensor Prod. 2 1+n + n(n +1)/2 3 n 3 1+n + n(n+1) 2 + n 2 + n(n 1)(n 2) 6 4 n Complete polynomial bases contains fewer elements than tensor products. Asymptotically, complete polynomial bases are as good as tensor products. For smooth n-dimensional functions, complete polynomials are more efficient approximations Construction Compute tensor product approximation, as in Algorithm 6.4 Drop terms not in complete polynomial basis (or, just compute coefficients for polynomials in complete basis). Complete polynomial version is faster to compute since it involves fewer terms

30 Integration Most integrals cannot be evaluated analytically Integrals frequently arise in economics Expected utility and discounted utility and profits over a long horizon Bayesian posterior Solution methods for dynamic economic models Gaussian Formulas All integration formulas choose quadrature nodes x i [a, b] and quadrature weights ω i : Z b f(x) dx =. nx ω i f(x i ) (7.2.1) a Newton-Cotes (trapezoid, Simpson, etc.) use arbitrary x i Gaussian quadrature uses good choices of x i nodes and ω i weights. Exact quadrature formulas: Let F k bethespaceofdegreek polynomials A quadrature formula is exact of degree k if it correctly integrates each function in F k Gaussian quadrature formulas use n points and are exact of degree 2n 1 i=1

31 Theorem 3 Suppose that {ϕ k (x)} k=0 is an orthonormal family of polynomials with respect to w(x) on [a, b]. Then there are x i nodes and weights ω i such that a<x 1 <x 2 < <x n <b,and 1. if f C (2n) [a, b], then for some ξ [a, b], Z b a w(x) f(x) dx = nx i=1 ω i f(x i )+ f (2n) (ξ) q 2 n(2n)! ; 2. and P n i=1 ω if(x i ) is the unique formula on n nodes that exactly integrates R b a f(x) w(x) dx for all polynomials in F 2n 1.

32 Gauss-Chebyshev Quadrature Domain: [ 1, 1] Weight: (1 x 2 ) 1/2 Formula: Z 1 f(x)(1 x 2 ) 1/2 dx = π 1 n for some ξ [ 1, 1], with quadrature nodes µ 2i 1 x i = cos 2n Arbitrary Domains Want to approximate R b a nx f(x i )+ i=1 π f (2n) (ξ) 2 2n 1 (2n)! (7.2.4) π, i =1,..., n. (7.2.5) f(x) dx for different range, and/or no weight function Linear change of variables x = 1+2(y a)(b a) Multiply the integrand by (1 x 2 ) ± 1/2 (1 x 2 ) 1/2. Z b f(y) dy = b a Z 1 µ (x +1)(b a) f 2 2 a 1 1 x 2 1/2 + a dx (1 x 2 1/2 ) Gauss-Chebyshev quadrature uses the x i Gauss-Chebyshev nodes over [ 1, 1] Z b f(y) dy =. π(b a) nx µ (xi +1)(b a) 1 f + a x 2 i 2n 2 a i=1 1/2

33 Gauss-Hermite Quadrature Domain is [, ] and weight is e x2 Formula: for some ξ (, ). Z f(x)e x2 dx = nx i=1 ω i f(x i )+ n! π f (2n) (ξ) 2 n (2n)! N x i ω i (1) (1) N x i ω i (1) ( 3) (1) ( 1) Normal Random Variables Y is distributed N(µ, σ 2 ). Expectation is integration. Use Gauss-Hermite quadrature: Linear COV x =(y µ)/ 2 σ implies E{f(Y )}= Z. =π 1 2 f(y)e (y µ)2 /(2σ 2) dy = nx ω i f( 2 σx i + µ) i=1 Z f( 2 σx+ µ)e x2 2 σdx where the ω i and x i are the Gauss-Hermite quadrature weights and nodes over [, ].

34 Multidimensional Integration Most economic problems have several dimensions Multiple assets Multiple error terms Multidimensional integrals are much more difficult Simple methods suffer from curse of dimensionality There are methods which avoid curse of dimensionality

35 Product Rules Build product rules from one-dimension rules Let x i,ω i, i =1,,m, be one-dimensional quadrature points and weights in dimension from a Newton-Cotes rule or the Gauss-Legendre rule. The product rule Z [ 1,1] d f(x)dx. = mx mx i 1 =1 i d =1 ω 1 i 1 ω 2 i 2 ω d i d f(x 1 i 1,x 2 i 2,,x d i d ) Gaussian structure prevails Suppose w (x) is weighting function in dimension Define the d-dimensional weighting function. dy W (x) W (x 1,,x d )= w (x ) Product Gaussian rules are based on product orthogonal polynomials. Curse of dimensionality: m d functional evaluations is m d for a d-dimensional problem with m points in each direction. Problem worse for Newton-Cotes rules which are less accurate in R 1. =1

36 General Parametric Approach: Approximating T For each x j, (TV)(x j ) is defined by v j =(TV)(x j )= max u D(x j ) π(u, x j)+β Z ˆV (x + ; a)df (x + x j,u) (12.7.5) In practice, we compute the approximation ˆT v j =(ˆTV)(x j ). =(TV)(x j ) Integration step: for ω j and x j for some numerical quadrature formula Z E{V (x + ; a) x j,u)}= ˆV (x + ; a)df (x + x j,u) Z = ˆV (g(x j,u,ε); a)df (ε). = X ω ˆV (g(x j,u,ε ); a) Maximization step: for x i X, evaluate v i =(T ˆV )(x i ) Hot starts Concave stopping rules Fitting step: Data: (v i,x i ),i=1,,n Objective: find an a R m such that ˆV (x; a) best fits the data Methods: determined by ˆV (x; a)

37 Approximating T with Hermite Data Conventional methods just generate data on V (x j ): Z v j = max π(u, x j)+β ˆV (x + ; a)df (x + x j,u) (12.7.5) u D(x j ) Envelope theorem: If solution u is interior, Z vj 0 = π x (u, x j )+β ˆV (x + ; a)df x (x + x j,u) If solution u is on boundary Z vj 0 = µ + π x (u, x j )+β ˆV (x + ; a)df x (x + x j,u) where µ is a Kuhn-Tucker multiplier Since computing vj 0 is cheap, we should include it in data: Data: (v i,vi 0,x i), i=1,,n Objective: find an a R m such that ˆV (x; a) best fits Hermite data Methods: determined by ˆV (x; a)

38 General Parametric Approach: Value Function Iteration Comparison with discretization guess a ˆV (x; a) (v i,x i ),i=1,,n new a This procedure examines only a finite number of points, but does not assume that future points lie in same finite set. Our choices for the x i are guided by systematic numerical considerations. Synergies Smooth interpolation schemes allow us to use Newton s method in the maximization step. They also make it easier to evaluate the integral in (12.7.5). Finite-horizon problems Value function iteration is only possible procedure since V (x, t) depends on time t. Begin with terminal value function, V (x, T ) Compute approximations for each V (x, t), t = T 1,T 2, etc.

39 Algorithm 12.5: Parametric Dynamic Programming with Value Function Iteration Objective: Solve the Bellman equation, (12.7.1). Step 0: Choose functional form for ˆV (x; a), and choose the approximation grid, X = {x 1,..., x n }. Make initial guess ˆV (x; a 0 ), and choose stopping criterion >0. Step 1: Maximization step: Compute v j =(T ˆV ( ; a i ))(x j ) for all x j X. Step 2: Fitting step: Using the appropriate approximation method, compute the a i+1 R m such that ˆV (x; a i+1 ) approximates the (v i,x i ) data. Step 3: If k ˆV (x; a i ) ˆV (x; a i+1 ) k<, STOP; else go to step 1.

40 Convergence T is a contraction mapping ˆT may be neither monotonic nor a contraction Shape problems An instructive example Figure 2: Shape problems may become worse with value function iteration Shape-preserving approximation will avoid these instabilities

41 Summary: Discretization methods Easy to implement Numerically stable Amenable to many accelerations Poor approximation to continuous problems Continuous approximation methods Can exploit smoothness in problems Possible numerical instabilities Acceleration is less possible