Fast evaluation of mixed derivatives and calculation of optimal weights for integration. Hernan Leovey

Transcription

1 Fast evaluation of mixed derivatives and calculation of optimal weights for integration Humboldt Universität zu Berlin MCQMC2012 Tenth International Conference on Monte Carlo and Quasi Monte Carlo Methods in Scientific Computing

2 Contents Algorithmic Differentiation (AD) 1 Algorithmic Differentiation (AD) Basics Complexity 2 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions 3 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension 4

3 Basics Algorithmic Differentiation (AD) Basics Complexity Frequently we have a program that calculates numerical values for a function, and we would like to obtain accurate values for derivatives of the function as well. The usual divided difference approach is given by : D +h f (x) f (x + h) f (x) h or D ±h f (x) f (x + h) f (x h) 2h For h small, truncation and round-off errors reduce the number of significant digits If h is not small, normally no good approximation to a derivative is expected

4 Basics Algorithmic Differentiation (AD) Basics Complexity Typically h = ɛ is taken, for ɛ = working accuracy. Expected accuracy: 1 2 of the significant digits of f for D +h 2 3 of the significant digits of f for D ±h In contrast, AD methods incur no truncation errors at all and usually yield derivatives with working accuracy. AD 0: Algorithmic Differentiation does not incur truncation errors AD 1: Difference quotients may sometimes be useful too AD 2: What is good for function values is good for their derivatives

5 Basics Algorithmic Differentiation (AD) Basics Complexity Standard setting for AD: Vector function F is the composition of a sequence of once continuously differentiable elemental functions ϕ i. Basic set of functions (polynomial core): {+,, (unary sign op.), c (const. init.)} A typical example of a library containing elemental functions: {c, +,,, /, exp, log, sin, cos, tan, tan 1,..., Φ, Φ 1,...}

6 Basics Complexity

7 Basics Complexity Basic complexity results Consider temporal complexity measure TIME, TIME{task(F )} = w WORK{task(F )} (1) with w = (w 1, w 2, w 3, w 4 ) a vector of platform dependent weights, and MOVES of fetches and stores WORK{task} ADDS MULTS of additions and subtractions of multiplications (2) NLOPS of nonlinear operations Forward mode AD: TIME{F (x), F (x)ẋ} ω tang TIME{F (x)} with a constant ω tang [2, 5/2]

8 Basics Complexity Reverse mode AD: Cheap Gradient Principle TIME{F (x), ȳ F (x)} ω grad TIME{F (x)} (3) for a constant ω grad [3, 4]. As consequence, the cost to evaluate a gradient f is bounded above by a small constant ω grad [3, 4] times the cost to evaluate the function itself. Random Access Memory requirements, in forward and reverse, are bounded multiples of those for the functions. Sequential Access Memory requirement of basic reverse mode is proportional to temporal complexity of the function.

9 Algorithmic Differentiation (AD) Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions (Automatic Evaluations of Cross-Derivatives. Griewank, L, L, Z) With the term cross derivatives we refer to those mixed partial derivatives where differentiation w.r.t. each variable is done at most once. f i (x) = j i x j f (x) = k f x i1... x ik (x), i = {i 1, i 2,..., i k }. There are 2 d cross derivatives if we take f (x) = f (x). We create a data structure with all 2 d cross derivatives of a function u in a flat array with 2 d entries. We call such data structure an d dimensional cube. d = 3 u u {1} u {2} u {1,2} u {3} u {1,3} u {2,3} u {1,2,3}

10 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions Basic Operations For a function u we denote by U its cube. For a constant function u(x) = c we set U[0] = c and zero everywhere else. For a coordinate function reps. input variable u(x) = x j we initialize its cube by U[0] = x j and U[2 j ] = 1. The rest of the entries are set to zero. Addition and Subtraction: V[i]=U[i] ± W[i] for all 0 i < 2 d. Scalar Multiplication: For v(x) = cu(x) the propagation rule is V[i]=c*U[i]. Scalar Addition/Subtraction is applied only to U[0]. The complexity of the above basic operations is (O(2 d )).

11 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions Nonlinear Operations Multiplication: The Leibniz formula for the multiplication of two functions v = u w states that: v i (x) = j i u j (x)w i j (x). Assume now that n / i. Then the above convolution sum can be split into v i {n} (x) = j i u i j (x)w j {n} (x) + j i u j {n} (x)w i j (x) Fixing the same subset i, the sums have the same structure. They all operate inside separate halves of cubes.

12 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions This leads to a possible implementation: void crossmult (int h, double U, double W, double V) { if (h == 1) { V[0] + = (U[0] W[0]); return; } h/ = 2; crossmult(h,u,w+h,v+h); crossmult(h,u+h,w,v+h); crossmult(h,u,w,v); } Due to the recursive nature of this procedure, there will be 3 d overall function calls at h = 1 resulting in 3 d multiplications and the same number of additions.

13 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions Exponential function: v = exp(u) has a very simple identity for the first partial derivatives, v k = vu k. This generalizes for k / i to: v i {k} = j i v i j (x)u j {k} (x) The second half cube of v is thus obtained by multiplying the previously computed first half cube of v and the second half cube of u. void exponent(int h, double U, double V) { int i; for(i= 0;i<h;i++) { V[i] = 0.0; } V[0]=exp(U[0]); for(i=1;i<h;i =2) { crossmult (i,v,u+i,v+i); } } There are d calls to the multiplication function and the final relative cost is 1 2 of the cost of a full multiplication.

14 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions Complexity: Nonlinear differentiable functions ϕ(u) included in math.h exhibit cost proportional to cross multiplication Given a library exhibiting cost proportional to cross multiplication, extend it by considering any nonlinear ϕ(u) satisfying differential equation ϕ (u) a(u)ϕ(u) = b(u) with functions a(.), b(.) in original library (ODE extension). Proposition The direct computation of all cross derivatives f of a function f given as an evaluation procedure (with elementals in ODE extended library) is itself an evaluation procedure with complexity OPS(f ) = O(3 d ) OPS(f ) for the runtime and with a factor of 2 d in the memory size. The unit is one multiplication, which is also the cost of addition or subtraction.

15 Arithmetic operations and nonlinear functions Complexity Comparison with other methods: univariate Taylor polynomial expansions Comparison with other methods: Proposition Method of interpolation of all cross derivatives from Taylor coefficients via univariate expansions exhibits complexity OPS(f ) = O(d 2 2 d ) (OPS(f ) + c), c 4, for the runtime and with a factor of (d + 1) 2 d in the memory size. The cross over between the methods occur at d 14. For large dimensions d, the Taylor method will have better runtimes. Advantages of direct new method: more accurate than Taylor univariate method faster for d 14

16 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Quasi Monte Carlo Methods (QMC): Q N,n (f ) := 1 N N f (x i ) I (f ) := f (x)dx, [0,1] n i=1 with x 1,, x N deterministically and cleverly chosen from [0, 1] n. Lattice Rules Q N,n,z (f ) := 1 N 1 f N i=0 ({ }) i N z Where N (usually prime) is the number of selected points and z is a carefully selected integer vector in Z n. Shifted Lattice Rules Q N,n,z, (f ) := 1 N 1 ({ }) i f N N z + for [0, 1] n. i=0

17 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension (Weighted) Reproducing Kernel Hilbert spaces (Sloan&Woźniakowski 98) Integration over particular RKHS F n of functions over [0, 1] n. Reproducing kernel K n (x, t) is function defined over [0, 1] n [0, 1] n, such that K n (., t) F n for all t [0, 1] n and f (t) = f (.), K n (., t) n, f F n ; t [0, 1] n. Worst Case Error of QMC algorithm over F n e(q N,n ) := sup I (f ) Q N,n (f ) f F n: f Fn 1 Assume integration functional I (.) is continuous over F n, then e(q N,n ) is bounded and I (f ) Q N,n (f ) e(q N,n ). f Fn

18 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Weighted Unanchored Sobolev Space F n,γ Consider weights 0 γ n,i, for i {1,, n}. ( 1 K n,γ (x, y) = 1+ γ n,i 2 B 2({x j y j }) + (x j 1 2 )(y j 1 ) 2 ) f Fn,γ = =i {1,,n} i {1,,n} j i ( ) 2 γ 1 i n,i f (x i, x D i )dx D i dx i [0,1] i [0,1] x n i i 1 2 Product weights γ n,i = j i γ {n,j} Tensor Product RKHS n F n,γ = H n,γ := H 1,γ1 H 1,γn n times K n,γ (x, y) = K 1,γj (x i, y i ) j=1

19 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Theorem (Novak&Woźniakowski 10, Kuo, Sloan, Joe,..) Let 0 γ n,i, i D, D := {1,, n}, f F n,γ. Given a prime number N, there exits a shifted rank-1 lattice rule Q N,n,z, with generator vector z constructed by the Component by Component algorithm (CBC), such that ( ( ) ) i τ =i D (γ n,i) 1/(2τ) 2ζ(1/τ) ( 2π) 1/τ I (f ) Q N,d,z, (f ) (N 1) τ f F n,γ for any τ [ 1 2, 1). For fixed f, we need the weights to construct a generator vector z for a lattice rule, using CBC algorithm. How should we choose the weights in practice? What is an optimal embedding for a given function f in a practical problem?

20 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Common Approach: Choose the weights such that the integration error bound is minimized. In general case, 2 n 1 terms inside f F = n,γ Approach: =i D ( ) 2 γ 1 i n,i f (x i, x D i )dx D i dx i [0,1] i [0,1] x n i i 1 2 Very often, problems in applications exhibit low effective dimension d << n. Effective dimension refers to essential ANOVA part of the function that accumulates most of the variance ( 99%).

21 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Assume f is a square integrable function. Then we can write f as the sum of 2 n ANOVA terms: f (x) = f i (x), f i (x) = f (x i, x D i )dx D i f j (x) i D [0,1] n i j i For a given family T of subsets of D, let us define now f T (x) = i T f i (x). Then, the integration error of a QMC algorithm Q N,n is given by (I Q N,n )(f ) (I Q N,n )(f T ) + (I Q N,n) i {1,...,n},i T f i (x)

22 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Theorem Let T be a given family of subsets of D. Let f i F n,γ for i T. Then for the function f T defined above it holds f T F = n,γ =i T ( γ 1 i ) 2 n,i f (x i, x D i )dx D i dx i [0,1] x i i [0,1] n i Moreover, if f F n,γ, it holds for i D b f,i := = [0,1] i ( i ) 2 f (x i, x D i )dx D i dx i x i [0,1] n i ( ) 2 i f (x i, x D i )dx D i dx i [0,1] x n i i [0,1] i [0,1] n 1 2 ( ) i 2 f (x) dx x i

23 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Integrands with low effective dimension Remark Note that any (good) upper bound b f,i R, b f,i b f,i, conduces also to an integration error upper bound of the form ( ( ) ) i τ i D (γ n,i) 1/(2τ) 2ζ(1/τ) ( 1 2π) 1/τ (I Q N,n )(f T ) 2 γ 1 (N 1) τ n,i b f,i i T Product Weights (γ n,i = j i γ n,{j}): Let d denote effective dimension of f in truncation sense (say d 14). App.-1 f EFFTd := f i (x). i {1,...,d}

24 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Set 0 0 = 0, c 0 = + for c > 0. Assume that at least one term b i,f > 0 for some i {1,..., d}. For f EFFTd consider bound objective function ψ : R n 0 [0, + ] (where the variables are the weights). For simplicity set ψ(0) = +, and for (x 1,..., x n ) R n 0 \ {0} define ( n ( ψ(x 1,..., x n) 2τ = 1 + x 1 2ζ(1/τ) j ( 2π) 1/τ j=1 ) ) τ 1 i {1,...,d}( j i x 1 j )b i,f 1 2 Clearly we have: minimize ψ(x 1,..., x n) minimize ψ(x 1,..., x n) (x 1,...,x n) R n 0 (x 1,...,x n) R n 0 subject to x j = 0, d + 1 j n.

25 Reproducing Kernel Hilbert Spaces (RKHS) Integrands with low effective dimension Choice for non important weights (in ANOVA sense): Lemma For fixed τ [1/2, 1), let γn = (γn,1,..., γn,d, 0,..., 0) be an optimal feasible solution of problem above. Let ɛ 0 > 0, and let a 1,..., a n d be any sequence of nonnegative real numbers with n d i=1 a i M. Define R 0 = ( 2π) 1/τ M2ζ(1/τ) log τ ɛ 1 0 d j=1 (1 + (γ n,j) 1 2τ d j=1 (1 + (γ n,j ) 1 2τ ) 2ζ(1/τ) ( 2π) 1/τ ) 2ζ(1/τ) ( 2π) 1/τ Then it follows for γ n = (γ n,1,..., γ n,d, R 0a 1,..., R 0a n d ) that ψ(γ n) (1 + ɛ 0)ψ(γ n )

26 Option valuation problem for arithmetic average Asian options Asset S t follows the geometric Brownian motion model. ) ) S t = S 0 exp ((r σ2 t + σw t 2 Simulating asset prices reduces to simulating paths W t1,..., W td. e rt V = (2π) d/2 max 1 d S j (w) K, 0 e 1 2 wt C 1w dw det(c) R d d with w = (W t1,..., W td ). After a factorization C = AA T of the covariance matrix, transform integral using Φ 1 (.). V = e [0,1] rt max 1 d S j (AΦ 1 (x)) K, 0 dx, d d j=1 For the tests, we simplify the problem assuming K = 0. Consider principal components (PCA) and Brownian Bridge (BB) factorization of C. j=1

27 Sensitivity tests for effective dimension (Algo. Wang&Fang 03) K = 0, S 0 = 100, T = 1 real dimension n = 16, 64, 128 Domain Truncation (Kuo&Sloan&Griebel 10) ɛ = 0.1, 0.01, 0.001, ( I (f ) I (f ɛ ) ɛs 0 ) For (σ, r) [0.05, 0.35] [0.05, 0.35] (tests on 7 7 uniform grid) Using 2 16 Sobol points, all tests resulted with effective dimension in truncation sense d 3 for PCA, and d 8 for BB construction. ( ) b ˆ i 2 f,i b f,i := f (x) dx for i {1,, d} [0,1] x n i CrossAD cost for simplified examples without strike (K = 0): Example \n = Runtime(crossPCA) (d = 4) Runtime(PCA) Runtime(crossBB) (d = 8) Runtime(BB)

28 Fixed K = 0, S 0 = 100, T = 1,σ = 0.1,r = 0.1, domain truncation ɛ = 0.1, (b i,f estimates using cross AD for d first variables) Table: Weights for τ = 0.9 (runtime for opt. solver approx seconds) BB n8 S11 n8 S14 n8 acc n16 S11 n16 S14 n16 acc n128 S11 n128 S14 γn, γn, γn, γn, γn, γn, γn, γn, ψ(γ ) 2.6e e e e e e e e+04 ψ( 1, 0) 9.5e e e e e e e+05 j 3.5e+05 ψ( 1, 0) 1.0e e e e e e e+04 j 2 5.2e+04

29 Table: Weights for τ = 0.9 (runtime for opt. solver approx seconds) PCA n8 S11 n8 S14 n8 acc n16 S11 n16 S14 n16 acc n128 S11 n128 S14 γn, γn, γn, γn, ψ(γ ) 6.0e e e e e e e e+03 ψ( 1, 0) 4.0e e e e e e e+04 j 1.1e+04 ψ( 1, 0) 1.6e e e e e e e+03 j 2 4.5e+03

30 Further investigations: f EFF + T d := i {1,...,d} f i (x) + d+1 j n f {j} (x). Using cross AD + reverse mode AD for cheap gradients (Optimization problem remains n dimensional) Product and order dependent Weights for functions with low effective superposition dimension using forward and reverse AD (No need for numerical Optimization) Good bounds for functions with kinks (K 0, eff. sup. dim. d = 2 and P.O.D. weights) Improved sampling strategy for squared mixed derivatives strongly diverging at small sub-cube borders Domain truncation alternative Thank you for your attention!