2 Numericl integrtion This is pge i Printer: Opque this 2. Introduction Numericl integrtion is problem tht is prt of mny problems in the economics nd econometrics literture. The orgniztion of this chpter is s follows. The rst section covers qudrture procedures, which re the dominnt wy to solve models. The second section covers (pseudo) Monte Crlo integrtion techniques. The lst section discusses qusi Monte Crlo integrtion. 2.2 Qudrture techniques Suppose we wnt to clculte I = ; (2.) where f(x) is sclr function. This could be di cult problem, e.g., becuse the functionl form is nsty or becuse we do not even hve functionl form, but only set of function vlues. Qudrture techniques re numericl integrtion techniques for which the formul of the numericl integrl cn be written s I = w i f i ; (2.2) i=
ii 2. Numericl integrtion where f i is the function vlue of f t node x i nd w i is weight. We will discuss two types of qudrture techniques. The rst is Newton-Cotes. Newton-Cotes is not very creful bout choosing the loction of the nodes, but is clever bout choosing the weights. The second is Gussin Qudrture. This procedure is clever bout choosing the weights s well s the nodes. To implement qudrture methods you cn forget the detils of the derivtion. All you hve to remember is how to construct wht kind of weights nd this is esy. 2.2. Newton-Cotes Qudrture Consider the integrtion problem given in Eqution 2. nd suppose tht one hs three function vlues t three nodes. Given this informtion, how would one clculte the integrl? Well, one could clculte n pproximting function nd clculte the integrl for this pproximting function. Since we hve been given three points, we cn clculte second-order polynomil, P 2 (x), nd get n estimte for the integrl using P 2 (x)dx: (2.3) Since integrting polynomils is esy, this procedure is strightforwrd. But one still hs to nd the pproximtion nd do the integrtion. It turns out tht these procedures cn be stndrdized. Tht is, one cn nd the weights in (2.2) independent of the functionl form of f. They do depend to some extend on the loction of the nodes. We ssume tht x 0 =, x = ( + b)=2, nd x 2 = b. Tht is, we hve equidistnt nodes nd the rst (lst) node is the left (right) boundry. We hve two segments of equl length, h nd we cn write x = x 0 + h nd x 2 = x 0 + 2h. Recll from the chpter on function pproximtion, tht using Lgrnge interpoltion the second-order polynomil cn be written s P 2 (x) = f 0 L 0 (x) + f L (x) + f 2 L 2 (x): (2.4) This mens tht our pproximting integrl is given by P 2 (x)dx = (f 0 L 0 (x) + f L (x) + f 2 L 2 (x)) dx = f 0 L 0 (x)dx + f L (x)dx + f 2 L 2 (x)dx The right-hnd side lredy hs the qudrture form s in Eqution (2.2). The weights re the integrls. Key is tht the integrls, i.e., the weights, do not depend on the function vlues. The nodes re pinned down by the vlue of x 0 nd h. The beuty is tht the integrls do not depend on x 0
2.2 Qudrture techniques iii either. Try to do the integrtion for one of them to ensure yourself tht this is true. In prticulr, it is not di cult to show tht Simpson qudrture Combining the results we get L 0 (x)dx = 3 h L (x)dx = 4 3 h L 2 (x)dx = 3 h P 2 (x)dx = 3 f 0 + 4 3 f + 3 f 2 h: This will give you n ccurte nswer if the function f cn be pproximted well with second-order polynomil over the intervl [; b]. It will give you n exct nswer for ny second-order polynomil. But clerly this will not give you n ccurte nswer if the function you re integrting is more complex. Sticking to the originl ide of integrting pproximting polynomils, there re two wys to proceed. The rst is to extend the ide to higher-order polynomils. The other is to use the sme ide but to smller intervls. In prticulr, suppose tht one hs n + equidistnt nodes nd the distnce between the nodes is h. The totl number of nodes must be odd so tht there re n=2 segments of length 2h. On ech of these segments of length 2h one then pplies the bove procedure. This would give 3 f 0 + 4 3 f + 3 f 2 h + 3 f 2 + 4 3 f 3 + 3 f 4 h + + 3 f n 2 + 4 3 f n + 3 f n h = 3 f 0 + 4 3 f + 2 3 f 2 + 4 3 f 3 + 2 3 f 4 + 2 3 f n 2 + 4 3 f n + 3 f n h 2.2.2 Gussin qudrture In constructing the Simpson weights no thought went into choosing the loction of the nodes. We simply strted with equidistnt nodes nd clculted the formuls for the weights. Writing the code to implement Newton- Cotes is so esy, becuse the weights only depend on h nd not on x 0. But
iv 2. Numericl integrtion by being smrt bout choosing the nodes we cn do even better. Tht is we cn get more ccurte nswer with the sme number of points. To be precise, if n is the number of nodes, then the following is true. With Newton-Cotes qudrture we get n exctly correct nswer if the function we re integrting is polynomil of order n, wheres with Gussin qudrture we get n exctly correct nswer if the function we re interested in is polynomil of order 2n. For exmple, if we hve 5 nodes then we get n exct nswer for ll polynomils of order 9 (or less) nd we get n ccurte nswer for functions tht cn be pproximted well with 9 th -order polynomil. Given tht we cn cover quite few functions with 9 th -order polynomils, you better be impressed bout the power nd simplicity of Gussin qudrture. To understnd the procedure, suppose we wnt to integrte sclr function de ned on [ ; ] using the qudrture formul with n nodes. Thus, Z! i f( i ): (2.5) Note tht we hve 2n free prmeters, nmely the! i s (the weights) nd the i s (the nodes). We wnt to get the correct nswer for ny polynomil of order 2n. To ccomplish this, we choose the vlues of! i nd i so tht by construction we get the correct nswer for ll the bsis functions, tht is, for, x, x 2,, nd x 2n. But if we get the correct nswer for ll bsis functions, we get the correct nswer for ny combintion. Tht is, one gets the correct nswer for ny polynomil of order 2n. To see why, suppose tht we hve found the! i s nd the i s such tht pplying the formul in Eqution 2.5 for f(x) = x 4 gives the right nswer, tht is Z x 4 dx = i=! i 4 i : (2.6) But this mens tht we lso get the right nswer for f(x) = x 4 for ny vlue of. To see why our pproximtion now would be! i 4 i =! i 4 i : (2.7) i= Tht is our pproximtion is the nswer for f(x) = x 4 times. Since the integrl of f(x) is indeed equl times the integrl of f(x) we get the right nswer. Similrly, we get the right nswer for ny combintion of polynomil bsis functions. But we still hve to nd the! i s nd the i s tht give us the correct nswer for the bsis functions. Tht will be the cse if the following is true: Z x j dx =! i j i j = 0; ; ; 2n (2.8) i= i= i=
2.2 Qudrture techniques v This is system of 2n equtions in 2n unknowns. The importnt thing to relize is tht the solution to this system of equtions does not depend on f. Tht is, independent of the prticulr function one is considering, one uses the sme vlues for the! i s nd the i s. In fct, there re stndrd subroutines vilble to solve for the qudrture nodes nd weights. Guss-Legendre The procedure discussed bove tht clculted the integrl of function over the intervl [ ; ] is clled Guss-Legendre. So in prctice one would do the following. One would use numericl procedure to generte the! i s nd the i s. The generted vlues will stisfy (2.8), but you don t hve to worry bout how the lgorithm mkes tht hppen. Let the solution be! GL i nd GL i. The only thing tht you hve to do is to obtin function vlues t the indicted nodes nd clculte the pproximtion to the integrl using the qudrture formul, tht is Guss-Hermite Z i=! GL i f( GL i ): (2.9) Guss-Legendre will give n ccurte nswer if f(x) cn be pproximted well with polynomil. Now suppose tht one wnts to integrte function f(x) tht cn be written s g(x) W (x) nd one knows tht g(x) cn be pproximted well with polynomil but g(x) W (x) cnnot. In this cse, it would not be smrt to use Guss-Legendre. Insted one would wnt to djust the procedure to tke this into ccount. There re di erent Gussin qudrture procedure tht do exctly this for di erent weighting functions, W (x), nd di erent domins. An importnt one is Guss-Hermite for which the weighting function is e x2 nd the domin is the rel line. For Guss- Hermite the weights nd the nodes re chosen to stisfy Z x j e x2 dx =! i j i j = 0; ; ; 2n (2.0) Let the solution be! GH i by Z i= nd GH i. So the pproximtion would be given g(x)e x2 dx i=! GH i g( gh;i ): (2.) Mke sure you understnd why there is n = in Eqution (2.0) nd n in Eqution (2.). In the rst eqution we re choosing the! i s nd the i s so tht our pproximting formul, i.e. Eqution (2.) will give the correct nswer for prticulr choices of g(x), nmely polynomil bsis functions. But unless g(x) is polynomil, the qudrture formul is n pproximtion.
vi 2. Numericl integrtion Guss-Chebyshev Another Gussin qudrture procedure is Guss-Chebyshev tht dels with Z g(x) dx ( x 2 ) =2 Qudrture nodes The nodes tht solve the problems discussed here turn out to be the zeros of the bsis functions of the corresponding Orthogonl polynomil. Tht is, the Chebyshev nodes tht solve Z x j ( x 2 ) dx = X n! =2 i j i j = 0; ; ; 2n i= re exctly the sme s the Chebyshev nodes discussed in the chpter on pproximting functions, lthough it goes bit to fr to explin why. 2.2.3 Chnge in vrible Suppose one wnts to clculte the expecttion of h (y), i.e., E[h(y)] ; where y is rndom vrible with N(; 2 ) distribution. Tht is one wnts to clculte Z (y ) 2 p h(y) exp 2 2 2 dy Also, suppose tht h(y) is function tht one expects cn be pproximted well with polynomil. This problem resembles Guss-Hermite qudrture problem but not exctly. One might be tempted to mke it Guss-Hermite problem simply by de ning (y ) 2 h(y) = h(y) exp p 2 2 2 exp ( y 2 ) nd considering the identicl integrl Z h(y) exp y 2 dy: But note tht it ws given tht h(y) could be pproximted well with polynomil, not tht h(y) cn be. In fct, given tht exp( y 2 ) is not like polynomil, h(y) my be pproximted very poorly with polynomil. So the pproprite wy to go is to do chnge of vribles. This is very esy but don t forget the Jcobin. Tht is, if y = (x) then g(y)dy = Z (b) () g((x)) 0 (x)dx
2.3 Monte Crlo Integrtion vii The trnsformtion we use here is x = y p 2 or y = p 2x + This gives E [h(y)] = = = Z Z Z (y ) 2 p h(y) exp 2 2 2 dy p 2 h(p 2x + ) exp x 2 p 2dy p h( p 2x + ) exp x 2 dy Wht to do in prctice? So wht would you do in prctice if one wnts to evlute E[h(y)]. First, one obtins n Guss-Hermite qudrture weights nd nodes using numericl lgorithm. Second, one gets n pproximtion using E [h(y)] p p! GH i h 2 GH i + i= (2.2) nd do not forget to divide by p! Well, how often do you get something in life so complex s n integrl so esily? 2.3 Monte Crlo Integrtion The ide behind Monte Crlo integrtion is very simple. Consider rndom vrible x with CDF F (x). Then one cn pproximte the integrl of the function h(x) with P T t= h(x)df (x) h(x t) ; (2.3) T where fx t g T t= is series drwn from rndom number genertor corresponding to the distribution of x. Although very simple there is one importnt disdvntge: it is not very ccurte. Above we sw tht we cn get n ccurte nswer with just few qudrture nodes for lrge clss of functions. Monte Crlo is subject to smpling vrition nd this only disppers t root n. Suppose we clculte the men of rndom vrible with uniform distribution on the unit intervl. With T = 00 the stndrd error is 0.029 which is 5.8% of the true men. Even with T = ; 000 we hve stndrd error tht is.8% of the true men.
viii 2. Numericl integrtion If one doesn t hve CDF then one cn use uniform distribution. Tht is, h(x)dx = (b ) h(x)f b (x)dx; (2.4) where f b is the density of rndom vrible with uniform distribution over [; b], tht is, f b = (b ). Thus, one could pproximte the integrl with P T t= h(x)dx (b ) h(x t) ; (2.5) T where x t is generted using rndom number genertor for vrible tht is uniform on [; b]. Typiclly one doesn t hve ccess to true rndom numbers nd one only hs ccess to computer progrm tht genertes them. Therefore, these procedures re lso referred to pseudo rndom numbers. The computer progrm genertes dt tht re (if it is good progrm) indistinguishble from true series of rndom numbers. But the function tht genertes the series is deterministic (nd chotic) so tht one should be creful in using theorems for true rndom numbers to think bout things like rtes of convergence. 2.4 Multivrite problems nd qusi Monte Crlo integrtion It is strightforwrd to extend the ide of qudrture techniques to higher dimensionl problems. The number of nodes increses exponentilly, however, which mens tht it becomes computtionlly quickly very expensive. With Monte Crlo integrtion one does not seem to hve this problem. Tht is the men of h(x t ) nd the men of h(x t ; z t ) both converge towrds its men t rte p T. Think of numericl integrtion problem s choosing nodes nd then tking (weighted) verge. By extending the qudrture techniques derived for sclr functions to multivrite problems one doesn t ll in the spce with nodes in the most e cient wy. Building on the ide of pseudo Monte Crlo new techniques hve been developed tht ll in the spce better. The strting point of qusi Monte Crlo integrtion is to generte equidistributed series. A sclr sequence fx t g T t= is equidistributed over [; b] i b lim T! T TX f(x t ) = t= (2.6) for ll Riemn-integrble f(x). Note the similrity with Eqution (2.5). There re severl exmples of equidistributed series. For exmple the se-
2.4 Multivrite problems nd qusi Monte Crlo integrtion ix quence (; 2; 3; 4; ) is equidistributed modulo for ny irrtionl number. Another exmple is the sequence of prime numbers multiplied by n irrtionl number (2; 3; 5; 7; ). For d-dimensionl problem, n equidistributed sequence fx t g T t= D R d stis es (D) TX Z lim f(x t ) = ; (2.7) T! T t= where (D) is the Lebesque mesure of D. 2 Some exmples for equidistributed vectors on the d-dimensionl unit hypercube re the following. Weyl: x t = (t p p ; t p p 2 ; ; t p p d ) modulo, (2.8) where p i is the i th positive prime number. Neiderreiter: x t = (t2 =(d+) ; 2 2=(d+) ; ; t2 d=(d+) ) modulo Equidistributed vectors for other hypercubes cn be done using liner trnsformtions. D F rc(x) (or x Modulo ) mens tht we subtrct the lrgest integer tht is less thn x. For exmple, frc(3:564) = 0:564: 2 To see why you hve to multiply the sum with (D) just consider the cse when f(x) = 8x. Then we know the integrl should be equl to (D) which we get becuse P T t= f(x)=t = :