Numerical Dynamic Programming

Transcription

1 Numerical Dynamic Programming Kenneth L. Judd Hoover Institution Prepared for ICE05 July 20, lectures 1

2 Discrete-Time Dynamic Programming Objective: X: set of states E ( TX D: the set of controls t=1 π(x t,u t,t)+w(x T +1 ) ), (12.1.1) π(x, u, t) payoffs inperiodt, forx X at the beginning of period t, and control u D is applied in period t. D(x, t) D: controls which are feasible in state x at time t. F (A; x, u, t) : probability that x t+1 A X conditional on time t control and state Value function ( TX V (x, t) sup E U(x,t) s=t π(x s,u s,s)+w(x T +1 ) x t = x ). (12.1.2) Bellman equation V (x, t) = sup u D(x,t) π(x, u, t)+e {V (x t+1,t+1) x t = x, u t = u} (12.1.3) Existence: boundedness of π is sufficient 2

3 Autonomous, InÞnite-Horizon Problem: Objective: X: set of states D: the set of controls max u t ( ) X E β t π(x t,u t ) t=1 D(x) D: controls which are feasible in state x. (12.1.1) π(x, u) payoff in period t if x X at the beginning of period t, and control u D is applied in period t. F (A; x, u) : probability that x + A X conditional on current control u and current state x. Value function deþnition: if U(x) is set of all feasible strategies starting at x. ( ) X V (x) sup E β t π(x t,u t ) U(x) x0 = x, (12.1.8) t=0 3

4 Bellman equation for V (x) V (x) = sup u D(x) π(x, u)+βe V (x + ) x, u ª (TV)(x), (12.1.9) Optimal policy function, U(x), if it exists, is deþned by U(x) arg max π(x, u)+βe V (x + ) x, u ª u D(x) Standard existence theorem: Theorem 1 If X is compact, β<1, andπ isboundedaboveandbelow, then the map TV = sup u D(x) π(x, u)+βe V (x + ) x, u ª ( ) is monotone in V, is a contraction mapping with modulus β in the space of bounded functions, and has a unique Þxed point. 4

5 Deterministic Growth Example Problem: Euler equation: V (k 0 )=max ct P t=0 βt u(c t ), k t+1 = F (k t ) c t k 0 given u 0 (c t )=βu 0 (c t+1 )F 0 (k t+1 ) ( ) Bellman equation V (k) =max c u(c) +βv (F(k) c). ( ) Solution to ( ) is a policy function C(k) and a value function V (k) satisfying 0=u 0 (C(k))F 0 (k) V 0 (k) ( ) V (k)=u(c(k)) + βv (F(k) C(k)) ( ) ( ) deþnes the value of an arbitrary policy function C(k), not just for the optimal C(k). The pair ( ) and ( ) expresses the value function given a policy, and a Þrst-order condition for optimality. 5

6 Stochastic Growth Accumulation Problem: ( X ) V (k, θ) =max c t,7 t E t=0 β t u(c t ) k t+1 = F(k t,θ t ) c t θ t+1 = g(θ t,ε t ) ε t : i.i.d. random variable k 0 = k, θ 0 = θ. State variables: k: productive capital stock, endogenous θ: productivity state, exogenous The dynamic programming formulation is V (k, θ) =max c u(c)+βe{v (F (k, θ) c, θ + ) θ} ( ) θ + = g(θ, ε) The control law c = C(k, θ) satisþes the Þrst-order conditions 0=u c (C(k, θ)) βe{u c (C(k +,θ + ))F k (k +,θ + ) θ}, ( ) where k + F (k, L(k, θ),θ) C(k, θ), 6

7 Objectives of this Lecture Formulate dynamic programming problems in computationally useful ways Describe key algorithms Value function iteration Policy iteration Gauss-Seidel methods Linear programming approach Describe approaches to continuous-state problems Point to key numerical steps Approximation of value functions Numerical integration methods Maximization methods Describe how to combine alternative numerical techniques and algorithms 7

8 Discrete State Space Problems State space X = {x i,i=1,,n} Controls D = {u i i =1,..., m} qij(u) t =Pr(x t+1 = x j x t = x i,u t = u) Q t (u) = qij(u) t : Markov transition matrix at t if u i,j t = u. Value Function iteration Terminal value: V T +1 i = W(x i ),i=1,,n. Bellman equation: time t value function is nx Vi t =max[π(x i,u,t)+β q t u ij(u) Vj t+1 ], i=1,,n j=1 which deþnes value function iteration Value function iteration is only choice for Þnite-horizon problems InÞnite-horizon problems Bellman equation is a set of equations for V i values: nx V i =max π(x i,u)+β q ij (u) V j,i=1,,n u j=1 Value function iteration is now V k+1 i =max u π(x i,u)+β nx j=1 q ij (u) V k j,i=1,,n 8

9 Can use value function iteration with arbitrary Vi 0 Error is given by contraction mapping property: V k V 1 1 β V k+1 V k and iterate k. Algorithm 12.1: Value Function Iteration Algorithm Objective: Solve the Bellman equation, (12.3.4). Step 0: Make initial guess V 0 ; choose stopping criterion A>0. Step 1: For i =1,..., n, compute Vi 7+1 =max u D π(x i,u)+β P n j=1 q ij(u)vj 7. Step 2: If k V 7+1 V 7 k<a,thengotostep3;elsegotostep1. Step 3: Compute the Þnal solution, setting U = UV 7+1, Pi = π(x i,ui ), i =1,,n, V =(I βq U ) 1 P, and STOP. Output: 9

10 Policy Iteration (a.k.a. Howard improvement) Value function iteration is a slow process Linear convergence at rate β Convergence is particularly slow if β is close to 1. Policy iteration is faster Current guess: V k i,i=1,,n. Iteration: compute optimal policy today if V k is value tomorrow: nx Ui k+1 =argmax π(x i,u)+β q ij (u) Vj k,i=1,,n, u Compute the value function if the policy U k+1 is used forever, which is solution to the linear system Vi k+1 = π nx x i,ui k+1 + β q ij (Ui k+1 ) Vj k+1,i=1,,n, j=1 Comments: j=1 Policy iteration depends on only monotonicity Policy iteration is faster than value function iteration If initial guess is above or below solution then policy iteration is between truth and value function iterate Works well even for β close to 1. 10

11 Gaussian acceleration methods for inþnite-horizon models Key observation: Bellman equation is a simultaneous set of equations nx V i =max π(x i,u)+β q ij (u) V j,i=1,,n u Idea: Treat problem as a large system of nonlinear equations Value function iteration is the pre-gauss-jacobi iteration nx Vi k+1 =max π(x i,u)+β q ij (u) Vj k,i=1,,n u True Gauss-Jacobi is V k+1 i =max u pre-gauss-seidel iteration j=1 j=1 " π(xi,u)+β P j6=i q # ij(u) Vj k,i=1,,n 1 βq ii (u) Value function iteration is a pre-gauss-jacobi scheme. Gauss-Seidel alternatives use new information immediately Suppose we have Vi 7 At each x i,givenvj 7+1 for j<i, compute Vi 7+1 in a pre-gauss-seidel fashion V 7+1 i =max u π(x i,u)+β X j<i q ij (u)v 7+1 j +β X j i q ij (u)v 7 j (12.4.7) Iterate (12.4.7) for i =1,.., n 11

12 Gauss-Seidel iteration Suppose we have V 7 i If optimal control at state i is u, then Gauss-Seidel iterate would be V 7+1 i = π(x i,u)+β Pj<i q ij(u)vj P j>i q ij(u)vj 7 1 βq ii (u) Gauss-Seidel: At each x i,givenvj 7+1 for j<i, compute Vi 7+1 V 7+1 i =max u π(x i,u)+β P j<i q ij(u)vj β P j>i q ij(u)vj 7 1 βq ii (u) Iterate this for i =1,.., n Gauss-Seidel iteration: better notation No reason to keep track of 7, number of iterations At each x i, V i max u π(x i,u)+β P j<i q ij(u)v j + β P j>i q ij(u)v j 1 βq ij (u) Iterate this for i =1,.., n, 1,..., etc. 12

13 Upwind Gauss-Seidel Gauss-Seidel methods in (12.4.7) and (12.4.8) Sensitive to ordering of the states. Need to Þnd good ordering schemes to enhance convergence. Example: Two states, x 1 and x 2, and two controls, u 1 and u 2 u i causes state to move to x i, i =1, 2 Payoffs: π(x 1,u 1 )= 1, π(x 1,u 2 )=0, π(x 2,u 1 )= 0,π(x 2,u 2 )=1. β =0.9. (12.4.9) Solution: Optimal policy: always choose u 2,movingtox 2 Value function: V (x 1 )=9,V(x 2 )=10. x 2 is the unique steady state, and is stable Value iteration with V 0 (x 1 )=V 0 (x 2 ) = 0 converges linearly: V 1 (x 1 )=0,V 1 (x 2 )=1,U 1 (x 1 )=2,U 1 (x 2 )=2, V 2 (x 1 )=0.9, V 2 (x 2 )=1.9, U 2 (x 1 )=2,U 2 (x 2 )=2, V 3 (x 1 )=1.71, V 3 (x 2 )=2.71, U 3 (x 1 )=2,U 3 (x 2 )=2, Policy iteration converges after two iterations V 1 (x 1 )=0,V 1 (x 2 )=1,U 1 (x 1 )=2,U 1 (x 2 )=2, V 2 (x 1 )=9,V 2 (x 2 )=10,U 2 (x 1 )=2,U 2 (x 2 )=2, 13

14 Upwind Gauss-Seidel Value function at absorbing states is trivial to compute Suppose s is absorbing state with control u V (s) =π(s, u)/(1 β). With absorbing state V (s) wecomputev (s 0 )ofanys 0 that sends system to s. V (s 0 )=π (s 0,u)+βV (s) With V (s 0 ), we can compute values of states s 00 that send system to s 0 ;etc. 14

15 Alternating Sweep It may be difficult to Þnd proper order. Idea: alternate between two approaches with different directions. W = V k, W i =max u π(x i,u)+β P n j=1 q ij(u)w j,i=1, 2, 3,..., n W i =max u π(x i,u)+β P n j=1 q ij(u)w j,i= n, n 1,..., 1 V k+1 = W Will always work well in one-dimensional problems since state moves eitherrightorleft,andalternating sweep will exploit this half of the time. In two dimensions, there may still be a natural ordering to be exploited. Simulated Upwind Gauss-Seidel It may be difficult to Þnd proper order in higher dimensions Idea: simulate using latest policy function to Þnd downwind direction Simulate to get an example path, x 1,x 2,x 3,x 4,..., x m Execute Gauss-Seidel with states x m,x m 1,x m 2,..., x 1 15

16 Linear Programming Approach If D is Þnite, we can reformulate dynamic programming as a linear programming problem. (12.3.4) is equivalent to the linear program P min n Vi i=1 V i s.t. V i π(x i,u)+β P n j=1 q ij(u)v j, i, u D, ( ) Computational considerations ( ) may be a large problem Trick and Zin (1997) pursued an acceleration approach with success. OR literature did not favor this approach, but recent work by Daniela Pucci de Farias and Ben van Roy has revived interest. 16

17 Continuous states: discretization Method: Replace continuous X with a Þnite X = {x i,i=1,,n} X Proceed with a Þnite-state method. Problems: Sometimes need to alter space of controls to assure landing on an x in X. A Þne discretization often necessary to get accurate approximations 17

18 Continuous States: Linear-Quadratic Dynamic Programming Problem: max u t TX µ 1 β t 2 x> t Q t x t + u > t R t x t u> t S t u t x> T +1W T +1 x T +1 t=0 (12.6.1) x t+1 = A t x t + B t u t, Bellman equation: 1 V (x, t) =max u t 2 x> Q t x + u > t R t x u> t S t u t + βv (A t x + B t u t,t+1). (12.6.2) Finite horizon Key fact: We know solution is quadratic, solve for the unknown coefficients The guess V (x, t) = 1 2 x> W t+1 x implies f.o.c. 0=S t u t + R t x + βb t > W t+1 (A t x + B t u t ), F.o.c. implies the time t control law u t = (S t + βb t > W t+1 B t ) 1 (R t + βb t > W t+1 A t )x (12.6.3) U t x. Substitution into Bellman implies Riccati equation for W t : W t = Q t + βa > t W t+1 A t +(βb t > W t+1 A t + R t > )U t. (12.6.4) Value function method iterates (12.6.4) beginning with known W T +1 matrix of coefficients. 18

19 Autonomous, InÞnite-horizon case. Assume R t = R, Q t = Q, S t = S, A t = A, andb t = B The guess V (x) 1 2 x> Wx implies the algebraic Riccati equation Two convergent procedures: Value function iteration: W =Q + βa > WA (βb > WA+ R > ) (12.6.5) (S + βb > WB) 1 (βb > WB + R > ). W 0 : a negative deþnite initial guess W k+1 =Q + βa > W k A (βb > W k A + R > ) (S + βb > W k B) 1 (βb > W k B + R > ). (12.6.6) Policy function iteration: W 0 :initialguess U i+1 = (S + βb > W i B) 1 (R + βb > W i A) : optimal policy for W i 1 2 W i+1 = Q U i+1su > i+1 + U i+1r > : value of U i 1 β 19

20 Lessons We used a functional form to solve the dynamic programming problem We solve for unknown coefficients We did not restrict either the state or control set Can we do this in general? Continuous Methods for Continuous-State Problems Basic Bellman equation: V (x) = max u D(x) π(u, x)+βe{v (x+ ) x, u)} (TV)(x). (12.7.1) Discretization essentially approximates V with a step function Approximation theory provides better methods to approximate continuous functions. General Task Find good approximation for V Identify parameters 20

21 Parametric Approach: Approximating V (x) Choose a Þnite-dimensional parameterization and a Þnite number of states V (x). = ˆV (x; a), a R m (12.7.2) X = {x 1,x 2,,x n }, (12.7.3) polynomials with coefficients a and collocation points X splines with coefficients a with uniform nodes X rational function with parameters a and nodes X neural network specially designed functional forms Objective: Þnd coefficients a R m such that ˆV (x; a) approximately satisþes the Bellman equation. Data for approximating V (x) Conventional methods just generate data on V (x j ): Z v j = max j)+β u D(x j ) ˆV (x + ; a)df (x + x j,u) (12.7.5) Envelope theorem: If solution u is interior, vj 0 = π x (u, x j )+β Z ˆV (x + ; a)df x (x + x j,u) If solution u is on boundary Z vj 0 = µ + π x (u, x j )+β ˆV (x + ; a)df x (x + x j,u) where µ is a Kuhn-Tucker multiplier Since computing vj 0 is cheap, we should include it in data We review approximation methods. 21

22 Approximation Methods General Objective: Given data about a function f(x) (which is difficult to compute) construct a simpler function g(x) that approximates f(x). Questions: What data should be produced and used? What family of simpler functions should be used? What notion of approximation do we use? How good can the approximation be? How simple can a good approximation be? Comparisons with statistical regression Both approximate an unknown function Both use a Þnite amount of data Statistical data is noisy; we assume here that data errors are small Nature produces data for statistical analysis; we produce the data in function approximation Our approximation methods are like experimental design with very small experimental error 22

23 Types of Approximation Methods Interpolation Approach: Þnd a function from an n-dimensional family of functions which exactly Þts n data items Lagrange polynomial interpolation Data: (x i,y i ),i=1,.., n. Objective: Find a polynomial of degree n 1, p n (x), which agrees with the data, i.e., y i = f(x i ),i=1,.., n Result: If the x i are distinct, there is a unique interpolating polynomial Hermite polynomial interpolation Data: (x i,y i,y 0 i),i=1,.., n. Objective: Find a polynomial of degree 2n 1, p(x), which agrees with the data, i.e., y i =p(x i ),i=1,.., n yi 0 =p 0 (x i ),i=1,.., n Result: If the x i are distinct, there is a unique interpolating polynomial Least squares approximation Data: A function, f(x). Objective: Find a function g(x) fromaclassg that best approximates f(x), i.e., g =argmaxkf gk2 g G 23

24 Chebyshev polynomials - Example of orthogonal polynomials [a, b] =[ 1, 1] w(x) = 1 x 2 1/2 T n (x) =cos(n cos 1 x) Recurrence formula: T 0 (x)=1 T 1 (x)=x T n+1 (x)=2xt n (x) T n 1 (x), Figure 1: 24

25 General intervals Few problems have the speciþc intervals used in deþnition of Chebshev polynomials Map compact interval [a, b] to[ 1, 1] by y = 1+2 x a b a then φ i 1+2 x a b a are the Chebyshev polynomials adapted to [a, b] Regression Data: (x i,y i ),i=1,.., n. Objective: Find β R m., m n, with y i = f(xi ; β),i=1,.., n. Least Squares regression: Chebyshev Regression Chebyshev Regression Data: X min (yi f (x i ; β)) 2 β R m (x i,y i ),i=1,.., n > m, x i are the n zeroes of T n (x) adapted to [a, b] Chebyshev Interpolation Data: (x i,y i ),i=1,.., n = m, x i are the n zeroes of T n (x)adapted to [a, b] 25

26 Minmax Approximation Data: (x i,y i ),i=1,.., n. Objective: L Þt min max β R m i Problem: Difficult to compute ky i f (x i ; β)k Chebyshevminmaxproperty Theorem 2 Suppose f :[ 1, 1] R is C k for some k 1, andleti n be the degree n polynomial interpolation of f based at the zeroes of T n (x). Then k f I n k µ 2 π Chebyshev interpolation: converges in L log(n +1)+1 (n k)! n! essentially achieves minmax approximation easy to compute does not approximate f 0 ³ µ π k b a k k f (k) k

27 Splines DeÞnition 3 Afunctions(x) on [a, b] is a spline of order n iff 1. s is C n 2 on [a, b], and 2. there is a grid of points (called nodes) a = x 0 <x 1 < <x m = b such that s(x) is a polynomial of degree n 1 on each subinterval [x i,x i+1 ], i =0,...,m 1. Note: an order 2 spline is the piecewise linear interpolant. Cubic Splines Lagrange data set: {(x i,y i ) i =0,,n}. Nodes: The x i are the nodes of the spline Functional form: s(x) =a i + b i x + c i x 2 + d i x 3 on [x i 1,x i ] Unknowns: 4n unknown coefficients, a i,b i,c i,d i,i=1, n. 27

28 Conditions: 2n interpolation and continuity conditions: y i =a i + b i x i + c i x 2 i + d i x 3 i, i =1,.,n y i =a i+1 + b i+1 x i + c i+1 x 2 i + d i+1 x 3 i, i =0,.,n 1 2n 2 conditions from C 2 at the interior: for i =1, n 1, b i +2c i x i +3d i x 2 i =b i+1 +2c i+1 x i +3d i+1 x 2 i 2c i +6d i x i =2c i+1 +6d i+1 x i Equations (1 4) are 4n 2 linear equations in 4n unknown parameters, a, b, c, andd. construct 2 side conditions: natural spline: s 00 (x 0 )=0=s 00 (x n ); it minimizes total curvature, R xn x 0 s 00 (x) 2 dx, among solutions to (1-4). Hermite spline: s 0 (x 0 )=y0 0 and s 0 (x n )=yn 0 (assumes extra data) Secant Hermite spline: s 0 (x 0 ) = (s(x 1 ) s(x 0 ))/(x 1 x 0 )and s 0 (x n )=(s(x n ) s(x n 1 ))/(x n x n 1 ). not-a-knot: choose j = i 1,i 2, such that i 1 +1<i 2, and set d j = d j+1. Solve system by special (sparse) methods; see spline Þt packages 28

29 B-Splines: A basis for splines Put knots at {x k,,x 1,x 0,,x n }. Order 1 splines: step function interpolation spanned by 0, x < x i, Bi 0 (x) = 1, x i x<x i+1, 0, x i+1 x, Order 2 splines: piecewise linear interpolation and are spanned by 0, x x i or x x i+2, Bi 1 x x (x) = i x i+1 x i, x i x x i+1, x i+2 x x i+2 x i+1, x i+1 x x i+2. The B 1 i -spline is the tent function with peak at x i+1 and is zero for x x i and x x i+2. Both B 0 and B 1 splines form cardinal bases for interpolation at the x i s. Higher-order B-splines are deþned by the recursive relation µ x Bi k xi (x)= Bi k 1 (x) x i+k x µ i xi+k+1 x + Bi+1 k 1 x i+k+1 x (x) i+1 29

30 Shape-preservation Concave (monotone) data may lead to nonconcave (nonmonotone) approximations. Example Figure 2: 30

31 Schumaker Procedure: 1. Take level (and maybe slope) data at nodes x i 2. Add intermediate nodes z + i [x i,x i+1 ] 3. Create quadratic spline with nodes at the x and z nodes to interpolate data and preserves shape. Many other procedures exist for one-dimensional problems Few procedures exist for two-dimensional problems Higher dimensions are difficult, but many questions are open. Spline summary: Evaluation is cheap Splines are locally low-order polynomial. Can choose intervals so that Þnding which [x i,x i+1 ]containsaspeciþc x is easy. Good Þts even for functions with discontinuous or large higher-order derivatives. Can use splines to preserve shape conditions 31

32 Multidimensional approximation methods Lagrange Interpolation Data: D {(x i,z i )} N i=1 R n+m,wherex i R n and z i R m Objective: Þnd f : R n R m such that z i = f(x i ). Task: Find combinations of interpolation nodes and spanning functions to produce a nonsingular (well-conditioned) interpolation matrix. Tensor products If A and B are sets of functions over x R n, y R m, their tensor product is A B = {ϕ(x)ψ(y) ϕ A, ψ B}. Given a basis for functions of x i, Φ i = {ϕ i k (x i)} k=0, the n-fold tensor product basis for functions of (x 1,x 2,...,x n )is ( ny ) Φ = ϕ i k i (x i ) k i =0, 1,,i=1,...,n i=1 Multidimensional Splines B-splines: Multidimensional versions of splines can be constructed through tensor products; here B-splines would be useful. Summary Tensor products directly extend one-dimensional methods to n dimensions Curse of dimensionality often makes tensor products impractical 32

33 Complete polynomials In general, the complete set of polynomials of total degree k in n variables. nx Pk n {x i 1 1 x i n n i 7 k, 0 i 1,,i n } Sizes of alternative bases 7=1 degree k Pk n Tensor Prod. 2 1+n + n(n +1)/2 3 n 3 1+n + n(n+1) 2 + n 2 + n(n 1)(n 2) 6 4 n Complete polynomial bases contains fewer elements than tensor products. Asymptotically, complete polynomial bases are as good as tensor products. For smooth n-dimensional functions, complete polynomials are more efficient approximations Construction Compute tensor product approximation, as in Algorithm 6.4 Drop terms not in complete polynomial basis (or, just compute coefficients for polynomials in complete basis). Complete polynomial version is faster to compute since it involves fewer terms 33

34 Parametric Approach: Approximating T - Expectations For each x j,(tv)(x j )isdeþned by v j =(TV)(x j )= max u D(x j ) π(u, x j)+β The deþnition includes an expectation Z E{V (x + ; a) x j,u)} = How do we approximate the integral? Integration Most integrals cannot be evaluated analytically Z ˆV (x + ; a)df (x + x j,u) (12.7.5) ˆV (x + ; a)df (x + x j,u) We examine various integration formulas that can be used. Newton-Cotes Formulas Trapezoid Rule: piecewise linear approximation nodes: x j = a +(j 1 2 )h, j =1, 2,...,n, h =(b a)/n for some ξ [a, b] Z b a f(x) dx= h 2 [f 0 +2f f n 1 + f n ] h2 (b a) f 00 (ξ) 12 Simpson s Rule: piecewise quadratic approximation Z b a f(x) dx= h 3 [f 0 +4f 1 +2f 2 +4f f n 1 + f n ] h4 (b a) f (4) (ξ) 180 Obscure rules for degree 3, 4, etc. approximations. 34

35 Gaussian Formulas All integration formulas are of form Z b a f(x) dx. = nx i=1 ω i f(x i ) (7.2.1) for some quadrature nodes x i [a, b] andquadrature weights ω i. Newton-Cotes use arbitrary x i Gaussian quadrature uses good choices of x i nodes and ω i weights. Exact quadrature formulas: Let F k be the space of degree k polynomials A quadrature formula is exact of degree k if it correctly integrates each function in F k Gaussian quadrature formulas use n points and are exact of degree 2n 1 Gauss-Chebyshev Quadrature Domain: [ 1, 1] Weight: (1 x 2 ) 1/2 Formula: Z 1 1 f(x)(1 x 2 ) 1/2 dx = π n nx f(x i )+ i=1 π f (2n) (ξ) 2 2n 1 (2n)! (7.2.4) for some ξ [ 1, 1], with quadrature nodes µ 2i 1 x i = cos 2n π, i =1,..., n. (7.2.5) 35

36 Arbitrary Domains Want to approximate R b a f(x) dx Different range, no weight function Linear change of variables x = 1+2(y a)(b a) Multiply the integrand by (1 x 2 ) 1/2 ± (1 x 2 ) 1/2. C.O.V. formula Z b a a f(y) dy = b a 2 Z 1 1 f i=1 µ (x +1)(b a) 2 1 x 2 1/2 + a dx (1 x 2 1/2 ) Gauss-Chebyshev quadrature produces Z b f(y) dy =. π(b a) nx µ (xi +1)(b a) 1 f + a x 2 i 2n 2 where the x i are Gauss-Chebyshev nodes over [ 1, 1]. 1/2 36

37 Gauss-Hermite Quadrature Domain: [, ]] Weight: e x2 Formula: Z for some ξ (, ). f(x)e x2 dx = nx i=1 ω i f(x i )+ n! π f (2n) (ξ) 2 n (2n)! Example formulas Table 7.4: Gauss Hermite Quadrature N x i ω i (1) (1) (1) ( 1) (1) ( 3) (1) ( 1)

38 Normal Random Variables Y is distributed N(µ, σ 2 ) Expectation is integration: E{f(Y )} =(2πσ 2 ) 1/2 Z Use Gauss-Hermite quadrature linear COV x =(y µ)/ 2 σ COV formula: Z f(y)e (y µ)2 /(2σ 2) dy = COV quadrature formula: E{f(Y )}. = π 1 2 Z f(y)e (y µ)2 2σ 2 dy f( 2 σx+ µ)e x2 2 σdx nx ω i f( 2 σx i + µ) i=1 where the ω i and x i are the Gauss-Hermite quadrature weights and nodes over [, ]. 38

39 Multidimensional Integration Many dynamic programming problems have several dimensions Multidimensional integrals are much more difficult Simple methods suffer from curse of dimensionality There are methods which avoid curse of dimensionality Product Rules Build product rules from one-dimension rules Let x 7 i,ω 7 i, i = 1,,m, be one-dimensional quadrature points and weights in dimension 7 from a Newton-Cotes rule or the Gauss-Legendre rule. The product rule Z [ 1,1] d f(x)dx. = mx mx i 1 =1 i d =1 ω 1 i 1 ω 2 i 2 ω d i d f(x 1 i 1,x 2 i 2,,x d i d ) Curse of dimensionality: m d functional evaluations is m d for a d-dimensional problem with m points in each direction. Problem worse for Newton-Cotes rules which are less accurate in R 1. 39

40 Monomial Formulas: A Nonproduct Approach Method Choose x i D R d,i=1,..., N Choose ω i R, i=1,..., N Quadrature formula Z D f(x) dx. = NX i=1 ω i f(x i ) (7.5.3) A monomial formula is complete for degree 7 if NX Z ω i p(x i )= p(x) dx (7.5.3) i=1 for all polynomials p(x) of total degree 7; recall that P 7 was deþned in chapter 6 to be the set of such polynomials. For the case 7 = 2, this implies the equations D P N i=1 ω i = R P D 1 dx N i=1 ω ix i j = R D x j dx, j =1,,d P N i=1 ω ix i jx i k = R D x jx k dx, j, k =1,,d (7.5.4) 1+d + 1 2d(d +1)equations N weights ω i and the N nodes x i each with d components, yielding a total of (d +1)N unknowns. 40

41 Simple examples Let e j (0,...,1,...,0) where the 1 appears in column j. 2d points and exactly integrates all elements of P 3 over [ 1, 1] d Z [ 1,1] d f. =ω u= dx f(ue i )+f( ue i ) i=1 µ 1/2 d,ω= 2d 1 3 d For P 5 the following scheme works: R [ 1,1] d f =ω. P 1 f(0) + ω d 2 i=1 f(ue i )+f( ue i ) +ω 3 P1 i<d, f(u(e i ± e j )) + f( u(e i ± e j )) i<j d where ω 1 =2 d (25 d d + 162), ω 2 =2 d (70 25d) ω 3 = d, u =( 3 5 )1/2. 41

42 Parametric Approach: Approximating T - Maximization For each x j,(tv)(x j )isdeþned by v j =(TV)(x j )= max u D(x j ) π(u, x j)+β Z ˆV (x + ; a)df (x + x j,u) (12.7.5) In practice, we compute the approximation ˆT using some integration method. v j =(ˆTV)(x j ). =(TV)(x j ) E{V (x + ; a) x j,u)}. = X 7 ω 7 ˆV (g(x j,u,ε 7 ); a) We now come to the maximization step: for x i X, evaluate v i =(T ˆV )(x i ) Use appropriate optimization method, depending on the smoothness of the maximand Hot starts Concave stopping rules When we have computed the v i (and perhaps v 0 i) we execute the Þtting step: Data: (v i,v 0 i,x i), i=1,,n Objective: Þnd an a R m such that ˆV (x; a) bestþts the data Methods: determined by ˆV (x; a) 42

43 Parametric Approach: Value Function Iteration Comparison with discretization guess a ˆV (x; a) (v i,x i ),i=1,,n new a This procedure examines only a Þnite number of points, but does not assume that future points lie in same Þnite set. Our choices for the x i are guided by systematic numerical considerations. Synergies Smooth interpolation schemes allow us to use Newton s method in the maximization step. They also make it easier to evaluate the integral in (12.7.5). Finite-horizon problems Value function iteration is only possible procedure since V (x, t) depends on time t. Begin with terminal value function, V (x, T ) Compute approximations for each V (x, t), t = T 1,T 2, etc. 43

44 Algorithm 12.5: Parametric Dynamic Programming with Value Function Iteration Objective: Solve the Bellman equation, (12.7.1). Step 0: Choose functional form for ˆV (x; a), and choose the approximation grid, X = {x 1,..., x n }. Make initial guess ˆV (x; a 0 ), and choose stopping criterion A>0. Step 1: Maximization step: Compute v j =(T ˆV ( ; a i ))(x j )forallx j X. Step 2: Fitting step: Using the appropriate approximation method, compute the a i+1 R m such that ˆV (x; a i+1 ) approximates the (v i,x i ) data. Step 3: If k ˆV (x; a i ) ˆV (x; a i+1 ) k< A, STOP; else go to step 1. 44

45 Convergence T is a contraction mapping ˆT may be neither monotonic nor a contraction Shape problems An instructive example Figure 3: Shape problems may become worse with value function iteration Shape-preserving approximation will avoid these instabilities 45

46 Comparisons We apply various methods to the deterministic growth model Relative L2 Errors over [0.7,1.3] N (β,γ) : (.95,-10.) (.95,-2.) (.95,-.5) (.99,-10.) (.99,-2.) (.99,-.5) Discrete model e e e e e e e e e e e e-04 Linear Interpolation 4 7.9e e e e e e e e e e e e e e e e e e-05 Cubic Spline 4 6.6e e e e e e e e e e e e e e e e e e e e e e e e-09 Polynomial (without slopes) 4 DNC 5.4e e e e e e e e e e e-09 Shape Preserving Quadratic Hermite Interpolation 4 4.7e e e e e e e e e e e e e e e e e e-08 Shape Preserving Quadratic Interpolation (ignoring slopes) 4 1.1e e e e e e e e e e e e e e e e e e-07 46

47 General Parametric Approach: Policy Iteration Basic Bellman equation: Policy iteration: V (x) = max u D(x) π(u, x)+βe{v (x+ ) x, u)} (TV)(x). Current guess: a Þnite-dimensional linear parameterization V (x). = ˆV (x; a), a R m Iteration: compute optimal policy today if ˆV (x; a) isvaluetomorrow U (x) =π u (x i,u(x),t)+β d ³ n E ˆV x + ; a o x, U (x)) du using some approximation scheme Û(x; b) Compute the value function if the policy Û(x; b) is used forever, which is solution to the linear integral equation ˆV (x; a 0 )=π(û(x; b),x)+βe{ ˆV (x + ; a 0 ) x, Û(x; b))} that can be solved by a projection method 47

48 Summary: Discretization methods Easy to implement Numerically stable Amenable to many accelerations Poor approximation to continuous problems Continuous approximation methods Can exploit smoothness in problems Possible numerical instabilities Acceleration is less possible 48