A Guaranteed, Adaptive, Automatic Algorithm for Univariate Function Minimization

Transcription

1 A Guaranteed, Adaptive, Automatic Algorithm for Univariate Function Minimization Xin Tong Department of Applied Mathematics, IIT May 20, 2014 Xin Tong 0/37

2 Outline Introduction Introduction Conclusion and Future work Xin Tong 1/37

3 Problem Description Introduction Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery Univariate function minimization on the unit interval S(f) = min f(x) R, 0 x 1 where the input functions space is the Sobolev space W 2, = {f C[0,1] : f < }. Xin Tong 2/37

4 Background Introduction Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery fminbnd in Matlab: find the minimum of a function of one variable within a fixed interval. It may give the local solution. It is not guaranteed. Clancy et al [1]: Numerical analysis on cone of input functions L approximation of univariate functions Xin Tong 3/37

5 Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery Basic concepts and Fixed-cost Algorithms The basic linear spline algorithm: A n (f)(x) = (n 1)[f(x i )(x i+1 x)+f(x i+1 )(x x i )], x i x x i+1, x i = i 1 n 1, i = 1,2,...,n. Definition (Cone of input functions) C τ := {f W 2, : f τ f f(1)+f(0) }. Xin Tong 4/37

6 Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery Basic concepts and Fixed-cost Algorithms cont d One can approximate f f(1)+f(0) by following algorithm: F n (f) : = A n (f) A 2 (f) = sup (n 1)[f(x i+1 ) f(x i )] f(1)+f(0). i=1,,n 1 And next algorithm approximates f which is also served as a lower bound: F n (f) := (n 1) 2 sup i=1,,n 2 f(x i ) 2f(x i+1 )+f(x i+2 ) f. Xin Tong 5/37

7 Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery Approximation theory from Clancy et al[1] I 1 The difference between f(x) and its linear spline algorithm A n (f)(x) f(x) A n (f)(x) = (n 1) xi+1 x i v i (t,x)f (t)dt, ( ) where the continuous, piecewise differentiable kernel v is defined as { (x i+1 x)(x i t), x i t x, v i (t,x) := (x x i )(t x i+1 ), x < t x i+1. Xin Tong 6/37

8 Problem Description Background Basic concepts and Fixed-cost Algorithms Approximation Thoery Approximation theory from Clancy et al[1] II 2 Bounds on f f(1)+f(0) F n (f): 0 f f(1)+f(0) F n (f) 3 An upper bound on f : 1 2(n 1) f. f τc n Fn (f), ( ) where C n = 2(n 1) 2(n 1) τ. 4 A necessary condition that f lies in the cone C τ : f C τ = F n (f) f τc n Fn (f) = τ min,n := 2(n 1)F n (f) 2(n 1) F n (f)+f n (f) τ. Xin Tong 7/37

9 How do we get the algorithm? Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Bounds on f(x) Bounds on min f(x) 0 x 1 estimated minimum value and error possible optimal solutions } Stopping condition The algorithm Xin Tong 8/37

10 Lower bound on f(x) Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm For x [x i,x i+1 ], the difference between f(x) and its linear spline algorithm A n (f)(x) becomes xi+1 f(x) A n (f)(x) = (n 1) v i (t,x)f (t)dt ( ) x i xi+1 (n 1) f v i (t,x) dt x i = f (x x i )(x i+1 x) 2 τc n Fn (f) (x x i)(x i+1 x). ( ) 2 The second inequality follows using Hölder s inequality [2, Theorem p.356]. Xin Tong 9/37

11 Lower bound on f(x) cont d Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Lower bound on f(x) fnl(x) i := A n (f)(x) τc n Fn (f) (x x i)(x i+1 x) f(x), 2 x [x i,x i+1 ]. Note that this lower bound fnl i [x i,x i+1 ]. is a quadratic function on each Xin Tong 10/37

12 Lower bound on min 0 x 1 f(x) Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Rewrite the quadratic function fnl i by completing square and rescaling ranges: fnl(x) i = f(x i+1) f(x i ) 4Cn i (t i +Cn) i 2 f(x i+1) f(x i ) [f(x i+1)+f(x i )], where ( ) t i = 2(n 1) x x i+x i+1 2 Cn i = 2(n 1)2 [f(x i+1 ) f(x i )]. τc n Fn(f) [ 1,1] and ( Cn i + 1 ) Cn i Xin Tong 11/37

13 Lower bound on min 0 x 1 f(x) cont d Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm The minimum value of f i nl on [x i,x i+1 ] follows l i n :=1 2 [f(x i+1)+f(x i )] 1 4 f(x i+1) f(x i ) The lower bound on min f(x) is 0 x 1 ( ) min( Cn,1)+ i 1 min( Cn i,1). L n := min 1 i n 1 li n. Xin Tong 12/37

14 Bounds on min 0 x 1 f(x) Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm The upper bound on min f(x) is 0 x 1 U n := min 0 x 1 A n(f)(x) = min 1 i n f(x i) min 0 x 1 f(x). Bounds on min 0 x 1 f(x) L n min 0 x 1 f(x) U n Xin Tong 13/37

15 Possible Optimal solution set Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Definition (Optimal solution set for an input function f) X := {t [0,1] : f(t) f(x), x [0,1]} Step 1, find all intervals may contain the optimal solutions. Now we have L n ln i fi nl (x) f(x). And if l i n U n, there exist some x in [x i,x i+1 ] such that L n l i n fi nl (x ) U n. These x may be the possible optimal solutions. Define an index set J n := {j : l j n U n}. Xin Tong 14/37

16 Possible optimal solution set Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Step 2, find all possible optimal solutions. Collect all the possible points on [x j,x j+1 ], j J n such that {x : f j nl (x) U n}. Then the possible solution set on [0,1] is given by X n := j Jn {x : f j nl (x) U n} X. Xin Tong 15/37

17 Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Estimate error and compute volume of possible solution set The estimated minimum value is chosen as min 0 x 1 A n(f)(x) (or the upper bound U n ). The error for min 0 x 1 A n(f)(x) is err n (f) := U n min 0 x 1 f(x) U n L n. The possible optimal solution set is a collection of intervals. To compute its volume is to compute the length of each interval and sum them up, i.e. Vol(X n ) = j J n Vol{x : f j nl (x) U n}. Xin Tong 16/37

18 Stopping condition Introduction Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm This algorithm is guaranteed to satisfy the error tolerance ε or the X tolerance δ. Stopping condition U n L n ε or Vol(X n ) δ Xin Tong 17/37

19 Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Xin Tong 18/37

20 Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Algorithm: Adaptive Univariate Function Minimization I Let the sequences of algorithms {A n } n I, { F n } n I, {F n } n I be as described above. Let τ 2 be the cone constant. Set k = 1. Let n 1 = (τ +1)/2 +1. For any error tolerance ε, X tolerance δ and input function f, do the following: Stage 1. Estimate f f(1)+f(0) and bound f. Compute F nk (f) and F nk (f). Stage 2. Check the necessary condition for f C τ. Compute τ min,nk = F nk (f) F nk (f)+f nk (f)/(2n k 2). Xin Tong 19/37

21 Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Algorithm: Adaptive Univariate Function Minimization II If τ τ min,nk, then go to stage 3. Otherwise, set τ = 2τ min,nk. If n k (τ +1)/2, go to stage 3. Otherwise, choose τ +1 n k+1 = 1+(n k 1). 2n k 2 Go to stage 3. Stage 3. Check for convergence. Check whether n k is large enough to satisfy either the error tolerance or X tolerance, i.e. a) U nk L nk ε or b) Vol(X nk ) δ Xin Tong 20/37

22 Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Algorithm: Adaptive Univariate Function Minimization III If it is true, return min 1 i n k f(x i ) and terminate the algorithm. If it is not true, choose: Go to stage 1. n k+1 = 1+2(n k 1). Xin Tong 21/37

23 Theorem Introduction Bounds on f(x) and minf(x) Intervals containing optimal solutions Stopping condition Algorithm Theorem For any n, L n min 0 x 1 f(x) U n, X n X; Moreover, given the error tolerance ε and X tolerance δ, there exists a N such that for any n k > N, either err nk (f) = U nk min f(x) ε or Vol(X) δ is satisfied. 0 x 1 Proof. See P in the thesis. Xin Tong 22/37

24 A family of bump test functions Bump test functions Two local minimum points 1 [ 4a 2 (x z) 2 (x z a) x z a 2a 2 f(x) = +(x z +a) x z +a ], z 2a x z +2a, 0, otherwise with log 10 (a) U[ 4, 1] and z U[2a,1 2a]. The minimum value of this family functions f is 1 at x = z. It follows that f f(1)+f(0) = 1/a and f = 1/a 2. Xin Tong 23/37

25 An example of bump test functions Bump test functions Two local minimum points For instance, let a = 0.03 and z = 0.4, then the function f is depicted below Figure : One example of the bump test functions. Xin Tong 24/37

26 Bump test functions Two local minimum points Experiment with ε = 10 8 and δ = random test functions initial τ values of 10,100,1000 a cost budget of N max = 10 7 success: the exact and approximate minimum value agree to within ε. warning: the proposed n k+1 exceeds N max. The algorithm returns a warning and falls back to the previous n k. Xin Tong 25/37

27 Bump test functions Two local minimum points Experiment with ε = 10 8 and δ = 0, result Table : The empirical success rates with ε = 10 8 and δ = 0. Success Success Failure Failure τ Prob(f C τ) No Warning Warning No Warning Warning % 20.58% 20.58% 0.00% 79.42% 0.00% % 52.79% 52.77% 0.02% 47.21% 0.00% % 85.62% 80.96% 2.46% 14.38% 2.20% It was successful for almost all f that finally lies inside C τ. It diverged for a small percentage of functions lying inside the cone. Xin Tong 26/37

28 Bump test functions Two local minimum points Experiment with ε = 0 and δ = 10 6 success: the exact optimal solution is contained in the possible solution set whose volume is less than δ. Table : The empirical success rates with ε = 0 and δ = Success Success Failure Failure τ Prob(f C τ) No Warning Warning No Warning Warning % 20.75% 20.75% 0.00% 79.25% 0.00% % 52.27% 52.28% 0.00% 47.72% 0.00% % 85.47% 85.49% 0.00% 14.51% 0.00% It was successful without warning for all f that finally lies inside C τ. Xin Tong 27/37

29 Bump test functions Two local minimum points Experiment with ε = 10 8 and δ = 10 6 success: the exact and approximate minimum value agree to within ε or the exact optimal solution is contained in the solution set whose volume is less than δ. Table : The empirical success rates with ε = 10 8 and δ = Success Success Failure Failure τ Prob(f C τ) No Warning Warning No Warning Warning % 21.21% 21.21% 0.00% 78.79% 0.00% % 53.21% 53.22% 0.00% 46.78% 0.00% % 85.66% 85.69% 0.00% 14.31% 0.00% It was successful without warning for all f that finally lies inside C τ. Xin Tong 28/37

30 Bump test functions Two local minimum points Functions with two local minimum points Another family of test functions is defined as f(x) = 5exp( [10(x a 1 )] 2 ) exp( [10(x a 2 )] 2 ), with a 1 U[0,1] and a 2 U[0,1]. f has two local minimum points at a 1 and a 2. It attains its minimum at x = a 1. 0 x 1, Xin Tong 29/37

31 Bump test functions Two local minimum points One example of functions with two local minimum points For instance, let a 1 = 0.3 and a 2 = 0.75, then the function f is f(x) = 5exp( [10(x 0.3)] 2 ) exp( [10(x 0.75)] 2 ), 0 x 1 and depicted by Figure below: Figure : One example of functions with two local minimum points. Xin Tong 30/37

32 Bump test functions Two local minimum points Experiment using our algorithm compared to fminbnd random test functions ε = 0 and δ = 10 2,10 4,10 7. a cost budget of N max = 10 7 success: the exact optimal solution is contained in the possible solution set whose volume is less than δ. success for fminbnd: the difference between its solution and the exact solution is less than δ. Xin Tong 31/37

33 Bump test functions Two local minimum points Experiment using our algorithm compared to fminbnd,result For this expetiment, our algorithm performed much better than fminbnd to solve global minimization problem. Table : Success rates of our algorithm compared to fminbnd. Our Algorithm fminbnd Success Success Success δ No Warning Warning Success % 76.91% 0.00% 0.03% % 52.95% 0.00% 0.00% % 0.00% 40.58% 0.00% Xin Tong 32/37

34 Conclusions Introduction Conclusions Acknowledgement Questions Breakthroughs Soving the globle minimization problem. Guaranteed either error tolerance ε or X tolerance δ. Restrictions Input function f should have finite second order derivative on [0,1]. It needs a huge number of data solving problems with flat functions. Xin Tong 33/37

35 Future work Introduction Conclusions Acknowledgement Questions Interval extension. Computational cost. Xin Tong 34/37

36 Acknowledgement Introduction Conclusions Acknowledgement Questions Professor Fred J. Hickernell Professor Sou-Cheng Choi GAIL team members: Yuhan Ding, Yizhi Zhang, and Lan Jiang Xin Tong 35/37

37 Conclusions Acknowledgement Questions Thank you! Any Questions? Xin Tong 36/37

38 References I Introduction Conclusions Acknowledgement Questions N. Clancy, Y. Ding, C. Hamilton, F. J. Hickernell, and Y. Zhang. The cost of deterministic, adaptive, automatic algorithms: Cones, not balls. Journal of Complexity, 30:21 45, J. K. Hunter and B. Nachtergaele. Applied Analysis. World Scientific, University of California, Davis, CA, Xin Tong 37/37