Orthogonal Polynomials, Quadratures & Sparse-Grid Methods for Probability Integrals

1/33 Orthogonal Polynomials, Quadratures & Sparse-Grid Methods for Probability Integrals Dr. Abebe Geletu May, 2010 Technische Universität Ilmenau, Institut für Automatisierungs- und Systemtechnik Fachgebiet Simulation und Optimale Prozesse

2/33 Topics Part II: Sparse Grid Methods for Multidimensional Integrals Introduction - Multidimensional Integrals Full-gird integration Tensor-product sparse-grid integration Fully-symmetric integration formulas Software Literature

1. Introduction Problem: Given a (continuous) function f : R n R and nonnegative weight function φ : R n R how to compute the integral: b1 bn I[f] =... f(x)φ(x)dx? a 1 a n In many practical applications the indefinite integral... f(x)φ(x)dx does not have analytic expression = computation of I[f] requires a numerical method. The domain of integration Ω := n i=1 [a i, b i ] = [a 1, b 1 ]...[a n, b n ]. Commonly a 1 = a 2 =... = a n and b 1 = b 2 =... = b n so that Ω = [a, b] n. 3/33

Standard weight functions and integration domains 4/33 Example: If x is a random variable w.r.t. the probability measure μ such dμ(x) = φ(x)dx and {x R n φ(x) = 0} Ω. E[f] =... f(x)φ(x)dx, Ω represents the expected value of f w.r.t. the probability measure μ. Example: Some standard integration domains and corresponding weight functions: Ω = [ 1, 1] n, with φ(x) = 1; Ω = (,+ ) n, with φ(x) = e x x ; Ω = [ 1, 1] n, with φ(x) = n i=1 (1 x i) α i(1+x i ) β i,α i,β i > 1; Ω = [0,+ ) n, with φ(x)

5/33 Basic Assumptions Assumptions: (A1): φ(x) = φ(x 1, x 2,..., x n ) can be written as φ(x) = n φ k (x k ) product weight function; k=1 where φ k : R R +. (A2): the domain of integration where Ω k R, k = 1,...,n. Ω = Ω 1 Ω 2...Ω n ; Assumption (A1) holds true when x 1, x 2,..., x n are independent random variables.

6/33 Example:(Integration with Gaussian-weight function) n φ(x) = e x x = k=1 e x2 k with φk (x k ) = e x2 k and Ω 1 = Ω 2 =... = Ω n = (,+ ). Remark: In assumptions (A1) and (A2), in general, it is not necessary to have: φ 1 = φ 2 =... = φ n ; also Ω 1 = Ω 2 =... = Ω n.

7/33 Major approaches for the numerical computation of I[f] (a) Monte-Carlo & Quasi-Monte-Carlo (QMC) techniques (Caflisch 1998, Niederreiter 1992) (b) Lattice Rules (Sloan & Joe 1994) (c) Full-grid integration (cubature) techniques or product rules (Cools 2002) (d) Sparse-Grid integration (cubature) techniques The state-of-the-art: Fewer integration nodes & higher polynomial exactness = Sparse-grid techniques. Note: Cubature techniques are constructed based one dimensional quadrature rules. One-dimensional interpolatory Gauss quadrature rules (and their extensions) are found to be efficient, due to their higher degree of accuracy.

2. Full-grid integration techniques 8/33 Suppose assumptions (A1) and (A2) hold true. Let, for each k = 1,...,n, X i = {x (i) 1, x(i) 2,...,x(i) N i } Ω i are Gauss quadrature nodes; w (i) 1, w(i) 2,..., w(i) corresponding weights N i for the one-dimensional integral on Ω i with weight function φ i. A (tensor-product) full-grid integration rule: ( ) Q[f] = Q (1) 1 Q (2) 1... Q (n) 1 [f] = N 1 N 2 k 1 =1 k 2 =1... N n k n=1 ( ) ( ) w (1) k 1 w (2) k 2... w (n) k n f x (1) k 1 x (2) k 2... x (n) k n Also called full-grid tensor-product of one-dimensional quadrature rules Q (i) 1, i = 1,...,n; or product rule.

How good are full-grid integration rules? 9/33 How many grid-points are there in Q[ ]? The number of elements of the set X := X 1 X 2... X n {( ) = x (1) k 1, x (2),..., x (n) x (i) k n k 2 k i } X i, k i = 1,...,N i, i = 1,...,n Number of grid-points (integration-nodes) in Q[ ] : If N 1 = N 2 =... = N n, then #X = N 1 N 2... N n. #X = N n. the number of gird-points in Q[ ] grows exponentially w.r.t. with the dimension of the integral known as the curse of dimension.

10/33 Example: for a 5-dimensional integral with 11-quadrature nodes in each dimension #X = N n = 11 5 = 161051. Is it necessary to use all the grid points in X? Is there redundancy in the full-grid scheme? (see Mysovskikh s Theorem, Mysovskikh 1968). What is the polynomial exactness (accuracy) of Q[ ]?

11/33 2.1. Multidimensional polynomials and polynomial-exactness of cubature rules Definition (monomial) For the variables x 1, x 2,..., x n a monomial of degree d of the n variables is an expression x d = x j 1 1 x j 2 2... x jn n, where (j 1, j 2,..., j n ) N n 0 and j 1 + j 2 +...+j n = d. Example: For the two variables x 1, x 2, (a) the following are monomials of degree 3 x 3 1 x 0 2 = x 3 1, x 2 1 x 2, x 1 x 2 2, x 0 1 x 3 2 = x 3 2.

12/33 (b) all monomials of degree less than or equal to 3 are degree monomials 0 1 1 x 1, x 2 2 x 1 x 2, x1 2, x 2 2 3 x1 3, x 1 2x 2, x 1 x2 2, x 2 3 The number of distinct monomials in n variables of degree less than equal to d is equal to ( ) n+d = (n+d)!. d n!d!

13/33 Definition (multidimensional polynomial) A multidimensional polynomial p (d) n (x 1, x 2,...,x n ) of degree d in the variables x 1, x 2,...,x n is a linear combination of monomials of x 1, x 2,...,x n of degree less or equal to d;i.e. { p (d) n P d n := span x j 1 1 x j 2 2... xn jn (j 1, j 2,...,j n ) N n 0, Hence, dim(p d n) = (n+d)! n!d!. Examples: Observe that j 1 + j 2 +...+j n = d} P 3 2 = span {1, x 1, x 2, x 1 x 2, x 2 1, x 2 2, x 2 1 x 2, x 1 x 2, x 3 1, x 3 2 }.

14/33 Examples: Some multidimensional polynomials: (a) degree 2: p 1 (x 1, x 2 ) = 2x 1 x 2 x 2 + 5x 1 P 2 2 ; (b) degree 3: p 2 (x 1, x 2 ) = x 2 1 x 2 + 3x 1 x 2 2 + 10x 2 2 + 12x 1 10 P 3 2 ; (c) degree 3: p(x 1, x 2, x 3 ) = 1 3 x 1x 2 x 3 2x 1 +π P 3 3. Definition The cubature rule Q[ ] is exact for the polynomial pn d if [ ] Q p (d) n =... p (d) n (x 1,..., x n )φ(x)dx Ω 1 Ω n [ = Q p (d) n ] = [ = I N 1 N 2 k 1 =1 k 2 =1 p (d) n ].... N n k n=1 p (d) n ( w (1) k 1 w (2) k 2... w (n) k n ) ( ) x (1) k 1 x (2) k 2... x (n) k n ;

15/33 Definition (polynomial exactness, Cools 2002) A cubature rule Q[ ] has polynomial exactness (or degree of precision or accuracy) d if it is exact for all polynomials degree at most d. Theorem (Cools 2002) If a cubature rule Q[ ] is constructed as a tensor product of one-dimensional Gaussian quadrature rules: Q[ ] = (Q 1 Q 2... Q n )[ ] with degree of exactness of the qudarature Q i [ ] equal to 2N i 1, i = 1,...,n, then the degree of exactness of Q[ ] is equal to max 1 i n {2N i 1} In particular, if N 1 = N 2 =... = N n = N, then the degree of exactness of Q[ ] is 2N 1.

16/33 Good Idea: to approximate I[f] with a degree of accuracy equal to d, compute I[p n (d) ], where p n (d) is an n-dimensional interpolating polynomial of f degree d; where p (d) n (x 1, x 2,..., x n ) = f(x 1, x 2,..., x n ), for each (x 1, x 2,..., x n ) X 1 X 2... X n. = I[f] I[p (d) n ]. Example: To achieve a degree of accuracy d in n dimensions from a full-grid tensor product of Gauss-quadrature rules, one can use equal ( ) d + 1 N i =, i = 1,...,n; 2 nodes in each dimension. For instance, N i = 2 m 1 + 1, m 1, number of Gaussian-quadrature nodes in each dimension yield a degree of accuracy equal to d = 2 m + 1.

17/33 NB: Higher accuracy for computing I[f] can only be achieved by using higher degree cubature rules. Is it necessary to use all the N n grid-points to obtain this degree to accuracy? Mysovskikh observed that, it is not necessary to use the full grid. Theorem (Mysovskikh 1968, Möller 1976) To attain a polynomial exactness equal to d, the (optimal) required number of grid-points in Q[ ] has lower and upper bounds given by N min = ( ) n+ d/2 N d/2 opt ( ) n+d = N d max. The expression x = the largest integer less than or equal to x. E.g. 3.4 = 3 and 2/3 = 0.

18/33 N min is known to be Möller s lower bound while is N max Mysovsikikh upper bound. Example: A full-grid tensor-product of 7-point Gauss-quadrature rules in 3 dimensions. Has a polynomial exactness d = 2 7 1 = 13; number of full-grid nodes 7 3 = 342; maximal required nodes N max = (2+13)! 2!13! = 105; minimal required nodes N min = (2+ 13/2 )! 2! 13/2! = (2+6)! 2!6! = 28.

19/33 Remark: Mysoviskikh s theorem is not constructive. Question: How to construct cubature rules with minimal number of nodes; i.e. number of nodes near or equal to N min? If not, rules with number of nodes satisfying the bounds N min and N max? There is so much research to be done! But one approach is the tensor-product sparse-grid integration technique.

20/33 3. Tensor-product sparse-grid integration rules In addition of assumptions (A1) & (A2) Assumption (A3): Ω 1 = Ω 2 =... = Ω n and φ 1 = φ 2 =... = φ n. = the same quadrature rule on each Ω i using φ i, so drop the index i. Assumption (A4) (Nested one-dimensional integration nodes) For each one-dimensional cubature rule on Ω R, there is a sequence of sets of nodes X (i),x (i+1),... such that X (i) X (i+1),..., i = 1, 2,...

For f 1 : { R R, the quadrature } rule Q (i) 1 with nodes in X (i) = x (i) 1, x(i) 2,..., x(i) N i is Q (i) 1 [f 1] = N i k i =1 w (i) k i f 1 (x (i) k i ). Observe that: Q (i) 1 [f 1] Q (i+1) 1 [f 1 ], i = 1, 2,... Question: How to construct embedded quadrature rules? Curtis-Clenshaw type: N 1 = 1 and N i = 2 i 1 + 1, i = 2, 3,... Degree of exactness of Q (i) CC is N i. Kronord-Patterson type: N 1 = 1, N 2 = 3, and N i = 2 i 1, i = 3, 4,... Degree of exactness of Q (i) KP is 3 2i 1, i > 1. 21/33

22/33 Petras sequences (Knut Petras 2003) X (i) PD = X(i) KP, with 3 2j + 4 8i 6j, j = 1, 2, 3, SMOLPACK - c code to generate Petra s delayed sequences (visit John Burkardt s Hompage, Florida State University) Construction of embedded quadrature rules, in general, is not trivial.

23/33 Define a multi-index i = (i 1, i 2,...,i n ) N n such that i = i 1 + i 2 +...+i n. Definition (Smolyak tensor-product sparse-grid cubature rule, Smolyak 1963, Wasilkowski & Wozniakowski 1995) The sparse-gird integration technique for a given integer d (with d n) is defined as ( ) S n,d [f] = ( 1) d i n 1 d i d n+1 i d ( ) Q (i 1) 1 Q (i 2) 1... Q (in) 1 [f]; where ( ) Q (i 1) 1 Q (i 2) 1... Q (in) 1 [f] N i1 k i1 N i2 N i n ( ) ( ) =... w ki1 w ki2... w ki f x n ki1 x ki2... x ki. n k i2 k i n

3.2. Properties of sparse-grid integration techniques The above is more of a theorem than a definition, for details see Heiss & Winchel 2006 and the references therein. The set of all nodes in the sparse gird rule S n,d [ ] is ( X d,n = X (i1) X (i1)... X (in)). d n+1 i d The number of nodes in the sparse gird rule S n,d [ ] is #X d,n 2d d! nd. #X d,n has a polynomial dependence w.r.t. the dimension n, for a fixed d. increasing the level of precession d does not incur so much function evaluation (polynomial complexity) sparse-grid techniques need few integration nodes as compared to full-grid rules 24/33

Properties... Theorem (polynomial exactness for sparse-grid rules, Novak & Ritter 1999) If each quadrature rule Q (i) 1, in the sparse-grid integration rule S n,d [ ], has a degree of exactness at least 2i 1, i = 1, 2,..., then S n,d [ ] has a degree of polynomial exactness at least 2d 1. Theorem (Novak & Ritter 1996) If f has a smoothness of order k and each quadrature rule Q (i) 1 has an approximation error of order N 1 i, then Sn,d [f] I[f] Cn,k,f (N) k (logn) (n 1)(k+1) where N is the number of nodes in the sparse-grid and C n,k,f is a constant depending on n, k, f. Sparse-grid methods based on the Clenshaw-Curtis and 25/33

3.3. Advantages and disadvantages of sparse-grid integration methods Advantages a few integration nodes are enough to yield a good approximation of integrals; thus, saving CPU time; the construction of the integration nodes & weights is done using the weight function φ( ) and the integration domain Ω, independent of the function to be integrated. Thus, grid-points and weights can be computed only once and used repeatedly; functions which are polynomial with respect to the uncertain variable can be integrated exactly. Disadvantages of Sparse-Grid less degree of polynomial exactness as compared to full-grid methods pure-sparse grid integration may not be good for non-smooth function (i.e. for functions with singularities) 26/33

27/33 4. Fully-symmetric integration rules Sometimes the number of integration nodes can be significantly reduced by using symmetric properties of the weight function and the domain of integration. Definition (full symmetry, Davis & Rabinowitz 1984) An integration domain Ω R n is fully symmetric if for each (x 1,..., x n ) Ω (±x i1,...,±x in ) Ω where (i 1,..., i n ) is any permutation of the n-tuple index (1, 2,..., n) with all permutation signs being considered. A weight function φ is fully symmetric on Ω R n if Ω is fully symmetric and for each (x 1,...,x n ) Ω φ(x 1,..., x n ) = φ(±x 1,...,±x n )

4.2. Some examples of fully-symmetric integrals 28/33 The weight and integration domain of I n [f] =... f(x)φ(x)dx, are symmetric if Ω Ω = (,+ ) n and normal density (Gaussian weight) function: φ(x) = e x x ; Ω = [ 1, 1] n and product Beta-density function: φ(x) = n i=1 (1 x i) α i(1+x i ) β i,α i,β i > 1; while if Ω = [0,+ ) n and product Weibul weight function: φ(x) = ( ) ki 1 n k i xi i=1 α i α i e symmetry does not hold. ( ) xi ki α i ;

29/33 4.3. Fully-Symmetric integration formulas Classical fully-symmetric integration formulas: See Stroud 1973 for an extensive compilation of such formulas. Lu & Darmofal 2004 give fully-symmetric formulas for the Gaussian weight function. Modern and efficient fully-symmetric integration formulas: Genz & Keister 1996 (also Genz 1986) give efficient fully-symmetric formulas for the Gaussian weight function. Hinrichs & Novak 2007 verify that the Genz-Keister rules are sparse-grid integration methods. Fully-symmetric formulas for general symmetric φ and Ω are not yet known.

30/33 5. Software Sparse Grid integration techniques: Spinterp - Matlab Toolbox from Andreas Klimke, University of Stutgart, Germany. visit http://sparse-grids.de/ for Matlab codes that generate readily evaluated nodes and weights SPGLib = Sparse Grid Library - FORTRAN 90 codes from Haijun Yu, University of Purdu, USA. C++, FORTRAN, Matlab codes from John Burkardt, Florida State Universities, USA. Fully Symmetric Integration Formulas: HintLib=High-dimensional Integration Library - includes C++ implementations of the rules in Stroud 1973, Rudolf Schürer, University of Salzburg, Austria. FORTRAN codes from Alan Genz, Washington State University, USA.

6. Literatur 31/33 Bungartz, H.-J.; Griebel, M. Sparse grids. Acta Numerica (2004), pp. 1-123. Caflisch, R. E. Monte Carlo and quasi-monte Carlo Methods. Acta Numerica 7(1998) 1-49. Cools, R. Advances in multidimensional integration. J. Comput. Appl. Math. 149(2002) 1-12. Davis, P. J.; Rabinowitz, P. Methods of numerical integration (2nd. Ed.). Academic Press, 1984. Genez, A.; Keister, B. D. Fully symmetric interpolatory rules for multiple integrals over infinite regions with Gaussian weights. SIAM J. Numer. Anal., 71(1996) 299 309. Genez, A. Fully symmetric interpolatory rules for multiple integrals. SIAM J. Numer. Anal., 23(1986), 1273 1283. Gerstner, T.; Griebel, M. Numerical integration using sparse grids. Numer. Algorithms 23(1998) 209 232.

32/33 Heiss, F., Winschel, V. Esitimation with numerical integration on sparse grids. Münchner Wirtschaftswissenschaftliche Beiträge(VWL), 2006-15. Hinrichs, A.; Novak, E. Cubature formula for symmetric measures in high dimensions with few points. Math. Comput. 76(2007) 1357 1372. Lu, J.; Darmofal, D. L. Higher-dimensional integration with Gaussian weight for applications in probabilistic design. SIAM J. Sci. Comput., 26(2004) 613 624. Möller, H. M. Kubaturformeln mit minimaler Knotenzahl. Numer. Math. 25(1976) 185 200. Mysovskikh, I. P. On the construction of cubature formulas with the smallest number of nodes. Soviet Math. Dokl. 9(1968) 277 280. Niederreiter, H. Random Number Generation and Quasi-Monte Carlo Methods. SIAM 1992.

Novak, E.; Ritter, K. Simple cubature formulas with high polynomial exactness. Constructive Approximation 15(1999) 499 522. Novak, E.; Ritter, K. High dimensional integration of smooth functions over cubes. Numer. Math. 75(1996) 79 97. Petras, K. Smolyak cubature of given polynomial degree with few nodes for increasing dimension. Numer. Math. 93(2003) 729 753. Sloan, I. H.; Joe, S. Lattice methods for multiple integration. Calerndon Press, Oxford, 1994. Smolyak, S. A. Quadrature and interpolation for tensor products of certain classes of functions. Soviet Math. Dokl. 4(1963) 240 243. Stroud, A. H. Approximate calculation of multiple integrals. Printc-Hall Inc., Englewood Cliffs, N. J., 1971. Wasilkowski, G. W.; Wozniakowski, H. Explicit cost bounds of algorithms for multivariate tensor product problems. J. Complexity 11(1995) 1-56. 33/33