Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 1 / 30 S. Smooth Optimization

Size: px

Start display at page:

Download "Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 1 / 30 S. Smooth Optimization"

Oscar Burns
5 years ago
Views:

1 Numerical Experience with a Class of Trust-Region Algorithms for Unconstrained Smooth Optimization XI Brazilian Workshop on Continuous Optimization Universidade Federal do Paraná Geovani Nunes Grapiglia Universidade Federal do Paraná May 23, 2016 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 1 / 30 S

2 1 NSC Method Introduction NSC Method Complexity 2 Numerical experiments Implementation Time and iterations Individually Robustness and efficiency Performance Profiles Best on full unconstrained set Reproducibility 3 Finalizing Conclusions Future work Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 2 / 30 S

3 NSC Method Introduction Unconstrained Optimization min f(x), f : R n R is twice continuously differentiable. Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 3 / 30 S

4 NSC Method Introduction Classical Trust-Region Method (Powell [1]) 1 q k (d) = f(x k ) + f(x k ) T d dt B k d 2 d k such that d k δ k and q k (0) q k (d k ) κ f(x k ) min 3 ρ k = f(xk ) f(x k + d k ) q k (0) q k (d k ) { f(x k } ) 1 + B k, δ k. 4 If ρ k η 1, do x k+1 = x k + d k. Otherwise x k+1 = x k. 5 Choose δ k+1 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 4 / 30 S

5 NSC Method Introduction Modified Trust-Region Method (Fan and Yuan [2]) 1 q k (d) = f(x k ) + f(x k ) T d dt B k d 2 d k such that d k δ k f(x k ) { and f(x q k (0) q k (d k ) κ f(x k k } ) ) min 1 + B k, δ k f(x k ). 3 ρ k = f(xk ) f(x k + d k ) q k (0) q k (d k ) 4 If ρ k η 1, do x k+1 = x k + d k. Otherwise x k+1 = x k. 5 Choose δ k+1 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 4 / 30 S

6 NSC Method Introduction ARC Method (Cartis, Gould, and Toint [3], [4]) 1 q k (d) = f(x k ) + f(x k ) T d dt B k d + 1 3δ k d 3 2 d k such that d k δ 1 2 k f(x k ) 1 2 and { f(x q k (0) q k (d k ) κ f(x k k } ) ) min 1 + B k, δ 1 2 k f(x k ) ρ k = f(xk ) f(x k + d k ) q k (0) q k (d k ) 4 If ρ k η 1, do x k+1 = x k + d k. Otherwise x k+1 = x k. 5 Choose δ k+1 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 4 / 30 S

7 NSC Method NSC Method NSC Method Nonlinear stepsize control, trust regions and regularizations for unconstrained optimization, Toint (2013) [5] Generalizes trust-region and regularization methods; Provides unified convergence theory; Suggests new methods. Let φ, ψ, χ : R n R be nonnegative functions such that min{φ(x), ψ(x), χ(x)} = 0 x is a critical point Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 5 / 30 S

8 NSC Method NSC Method NSC Method 1 Find a model q k (d) such that q k (0) = f(x k ) and f(x k + d) q k (d) κ m d 2 2 d k such that d k (δ k, { χ k ) = δk αχβ k and } q k (0) q k (d k φ k ) κψ k min 1 + B k, (δ k, χ k ). 3 ρ k = f(xk ) f(x k + d k ) q k (0) q k (d k ) 4 If ρ k η 1, do x k+1 = x k + d k. Otherwise x k+1 = x k. [γ 1 δ k, γ 2 δ k ] ρ k < η 1 5 δ k+1 [γ 2 δ k, δ k ] η 1 ρ k < η 2 [δ k, + ) ρ k η 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 6 / 30 S

9 NSC Method NSC Method NSC Method (Particular cases) Classical Trust-Region Method { α = 1 and β = 0 φ k = ψ k = χ k = f(x k ) = (δ k, χ k ) = δ k Modified Trust-Region Method { α = β = 1 φ k = ψ k = χ k = f(x k ) = (δ k, χ k ) = δ k f(x k ) ARC Method { α = β = 1/2 φ k = ψ k = χ k = f(x k ) = (δ k, χ k ) = δ 1 2 k f(xk ) 1 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 7 / 30 S

10 NSC Method Complexity How α and β affect the method? Theorem Suppose that φ k χ k and ψ k χ k ; {f(x k )} is bounded below; and B k κ. Then the NSC method takes at most O(ɛ 2 ) iterations to achieve χ k ɛ. Nonlinear stepsize control algorithms: complexity bounds for first and second order optimality (TR) - Grapiglia, Yuan, and Yuan ([6]) Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 8 / 30 S

11 Implementation How does α and β affect the method Similar to what Gould, Orban, Sartenaer, et al. [7] did; Discretize (0, 1] [0, 1] to a grid; Define algorithm for each (α, β); Run algorithm for 58 CUTEst problems (small); Analyze how sensitive it is; Anything better than traditional. Numerical Experience with a Class of Trust-Region Algorithms May 23, for 2016 Unconstrained 9 / 30 S

12 Implementation Implementation q(d) = f(x k ) + f(x k ) T d dt 2 f(x k )d Find d k by Steihaug-Toint ɛ = 10 8, maximum number of iteration 1000 η 1 = 1 4, η 2 = 3 4 σ 1 = 1 6, σ 2 = 4 σ 1 δ k ρ k < η 1 δ k+1 = δ k η 1 ρ k < η 2 σ 2 δ k ρ k η 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 10 / 30 S

13 Implementation Measurement of time Some problems have very small elapsed time (smallest: ); Run problem many times (how many?); 0.1 First time: t 0, define N = ; Run N times, get average; Get 3 averages, keep the best. t 0 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 11 / 30 S

14 Time and iterations Elapsed time and number of iterations 30 problems converged for every choice of (α, β); Number of iterations and elapsed time for these problems; Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 12 / 30 S

15 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

16 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

17 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

18 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

19 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

20 Time and iterations Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 13 / 30 S

21 Individually Time and iterations individually Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 14 / 30 S

22 Individually time Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 15 / 30 S

23 Individually time Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 15 / 30 S

24 Individually iter Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 15 / 30 S

25 Individually iter Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 15 / 30 S

26 Robustness and efficiency Robustness and efficiency Robustness varies between 45 and 56 problems; Performance profile c s,p r s,p = s S, p P; min{c s,p s S} ρ s (t) = #{r s,p t p P} ; #P ρ s (1) is an efficiency measure; ρ s (+ ) is the robustness (independent of S); Run (1, 0) vs s = (α, β), get ρ s (1) and ρ s (+ ). Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 16 / 30 S

27 Robustness and efficiency Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 17 / 30 S

28 Robustness and efficiency Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 17 / 30 S

29 Robustness and efficiency Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 17 / 30 S

30 Robustness and efficiency Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 17 / 30 S

31 Robustness and efficiency Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 17 / 30 S

32 Robustness and efficiency Rob Eff (1), (2) 94.8% 77.6% (3), (4) 96.6% 70.7% Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 18 / 30 S

33 Performance Profiles Performance Profiles Number of iterations; (1, 0) and (1, 1); Best of iterations and elapsed time; Best of robustness and efficiency (vs (1, 0)); Full set of 58 problems; Used Perprof-py [8]. Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 19 / 30 S

34 Performance Profiles (1,0) vs (1,1) 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

35 Performance Profiles Iter best vs Time best 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Time Iter (1): Time Iter (8): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

36 Performance Profiles (1,0) vs Best of iter 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Time Iter (1): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

37 Performance Profiles (1,0) vs Best of time 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Time Iter (8): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

38 Performance Profiles (1,0) vs Profile top 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Iter prof. (1): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

39 Performance Profiles (1,0) vs Profile top 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Iter prof. (2): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

40 Performance Profiles (1,0) vs Profile top 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Iter prof. (3): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

41 Performance Profiles (1,0) vs Profile top 1 Performance Profile 0.8 Percentage of problems solved Performance ratio Iter prof. (4): Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 20 / 30 S

42 Best on full unconstrained set Best on full unconstrained set All 173 unconstrained problems without bounds. (1,0) vs (1,1); (1,0) vs (0.78,0.18), (1) (1,0) vs (0.96,0.04), (3) 1 minute, 1000 iterations. Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 21 / 30 S

43 Best on full unconstrained set 1 Performance Profile Percentage of problems solved Performance ratio (1,0) (1,1) Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 22 / 30 S

44 Best on full unconstrained set 1 Performance Profile Percentage of problems solved Performance ratio (0.78,0.18) (1,0) Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 22 / 30 S

45 Best on full unconstrained set 1 Performance Profile Percentage of problems solved Performance ratio (0.96,0.04) (1,0) Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 22 / 30 S

46 Reproducibility How reproducible are these results? Ran the complete grid again; Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 23 / 30 S

47 Reproducibility Run 1 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

48 Reproducibility Run 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

49 Reproducibility Run 1 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

50 Reproducibility Run 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

51 Reproducibility Run 1 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

52 Reproducibility Run 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

53 Reproducibility Run 1 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

54 Reproducibility Run 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

55 Reproducibility Run 1 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

56 Reproducibility Run 2 Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 24 / 30 S

57 Finalizing Conclusions Conclusions The algorithm is indeed very dependent on (α, β); There are appears to be superior choices to be made; There are many better choices than (1, 0) in the small set; The results, naturally, also depend on P. Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 25 / 30 S

58 Finalizing Future work Future work Optimize other parameters for each (α, β); Optimize (α, β)? (Too many local minima); Sensitivity of this analysis regards the set of problems; Best choices for specific class of problems; Some algorithm modification that reduces sensitivity? (non-monotone); Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 26 / 30 S

59 Finalizing Future work M. J. D. Powell, Convergence properties of a class of minimization algorithms, in Nonlinear Programming 2, O. L. Mangasarian, R. R. Meyer, and S. M. Robinson, Eds., Academic Press, New York, J. Fan and Y. Yuan, A new trust region algorithm with trust region radius converging to zero, in Proceedings of the 5th International Conference on Optimization: Techniques and Applications (ICOTA 2001, Hong Kong), D. Li, Ed., 2001, pp C. Cartis, N. I. M. Gould, and P. L. Toint, Adaptive cubic overestimation methods for unconstrained optimization. part I: Motivation, convergence and numerical results.,, Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 27 / 30 S

60 Finalizing Future work, Adaptive cubic overestimation methods for unconstrained optimization. part II: Worst-case function - and derivative - evaluation complexity, Mathematical Programming, vol. 130, no. 2, pp , DOI: /s y. P. L. Toint, Nonlinear stepsize control, trust regions and regularizations for unconstrained optimization, Optimization Methods and Software, vol. 28, no. 1, pp , DOI: / G. N. Grapiglia, J. Yuan, and Y. Yuan, Nonlinear stepsize control algorithms: Complexity bounds for first and second order optimality, UFPR, Tech. Rep., Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 28 / 30 S

61 Finalizing Future work N. I. M. Gould, D. Orban, A. Sartenaer, and P. L. Toint, Sensitivity of trust-region algorithms to their parameters, 4OR, vol. 3, no. 3, pp , DOI: /s y. A. S. Siqueira, R. G. C. da Silva, and L.-R. Santos, Perprof-py: A python package for performance profile of mathematical optimization software, Journal of Open Research Software, vol. 4, no. 1, e12, DOI: /jors.81. Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 29 / 30 S

62 Finalizing Future work Thanks This presentation is licensed under the Creative Commons Attributions-ShareAlike 4.0 International License. Numerical Experience with a Class of Trust-Region Algorithms May 23, 2016 for Unconstrained 30 / 30 S

Parameter Optimization in the Nonlinear Stepsize Control Framework for Trust-Region July 12, 2017 Methods 1 / 39

Parameter Optimization in the Nonlinear Stepsize Control Framework for Trust-Region Methods EUROPT 2017 Federal University of Paraná - Curitiba/PR - Brazil Geovani Nunes Grapiglia Federal University of