arxiv: v1 [math.oc] 18 May 2018

Transcription

1 Derivative-Free Optimization Algoritms based on Non-Commutative Maps * Jan Feiling,, Amelie Zeller, and Cristian Ebenbauer arxiv: v [mat.oc] 8 May 08 Institute for Systems Teory and Automatic Control, University of Stuttgart, Germany {jan.feiling@ist,st967@stud,ce@ist}.uni-stuttgart.de Dr. Ing..c. F. Porsce AG, Stuttgart, Germany jan.feiling@porsce.de Abstract. A novel class of derivative-free optimization algoritms is developed. Te main idea is to utilize certain non-commutative maps in order to approximate te gradient of te objective function. Convergence properties of te novel algoritms are establised and simulation examples are presented.. INTRODUCTION In te course of continuously increasing computational power, optimization algoritms and optimization-based control play more and more a central role in solving control and real-time decision making problems. In many tasks, te resulting optimization problems are very callenging, i.e., tey are ig-dimensional, non-convex, nonsmoot, or of stocastic nature. Hence, improving existing optimization algoritms and developing novel algoritms is of central importance. An interesting class of algoritms are derivative-free algoritms, wic typically need only evaluations of te objective function for optimization. Hence, derivative-free optimization algoritms are simple and appealing in many callenging applications, for example in real-time optimization, were only noisy measurements and no matematical description of te objective function are available and terefore no derivative, i.e., no gradient can be computed directly. Consequently, derivative-free optimization and adaptive control tecniques are often used in tis applications [, 3, 4, 5, 6, 7]. Moreover, in te field of macine learning, derivative-free optimization problems [8, 9, 0, ] gain renewed interest, wic are often tackled wit so-called stocastic gradient approximation metods []. In tis paper, we propose a novel class of derivative-free discrete-time optimization algoritms. Te key caracteristic of te proposed * Tis article is a sligtly extended version of [] wit an extra Figure 6. algoritms is te use of deterministic non-commutative maps to evaluate te objective function at certain points suc tat a gradient descent step is approximated. In comparison to [8, 9, 0, ] our proposed metod is not a randomized or stocastic algoritm. Interestingly and to te best of our knowledge, te use of suc a deterministic non-commutative sceme in discrete-time optimization algoritms as not been studied so far in te optimization literature. In te geometric control teory literature, owever, te effect of non-commutative vector fields is of fundamental importance [] and as been used in control design since decades. Recently, non-commutative vector fields ave also been used in te design of continuous-time optimization algoritms. For example, Lie brackets, i.e., te infinitesimal caracterization of non-commutativity between two vector fields, ave been used to approximate gradients and to design extremum seeking algoritms for unconstrained and constrained optimization and adaptive control problems, see, e.g., [3, 4, 5, 6, 7]. Furtermore, in [8], Lie brackets ave been exploited to tackle te problem of distributed optimization over directed graps. Te proposed class of discrete-time derivative-free algoritms are inspired by tese Lie bracket based continuoustime, explorative optimization metods, but our approac is not simply a discretization of continuous-time metods. For example, utilizing non-commutative maps based on Euler-integration steps lead to approximation results wic are different from continuous-time Lie bracket approximation results (see, e.g., [9]), yet we sow ow tey can be used for gradient approximation. Indeed, as sown in te paper, suitable integration scemes need to be employed in order to approximate te results known from continuous-time metods. One advantage wic arises from utilizing non-commutative maps in gradient approximation is te robustness wit respect to noisy ob-

2 jective functions as we will see in simulations. Tis is not observed if finite differences are used for gradient approximations, like it was done in [0], were extremum seeking and a version of [] is combined. Overall, te contributions of tis work are as follows: Firstly, utilizing non-commutative maps, we develop novel discretetime gradient approximation scemes and corresponding derivative-free optimization algoritms. Secondly, we perform a convergence analysis of te proposed algoritms. Finally, we study te proposed algoritm in different simulation examples. Te sequel of te paper is structured as follows: In te next section we state te problem setup and give te main idea ow te discrete-time derivative-free optimization algoritms are derived. Section III contains te main lemmas and teorems providing te gradient approximation and te asymptotic convergence beavior of te algoritms. In Section IV we analyze te algoritm in various simulation examples. Finally, we give a summary and outlook of furter work. Notation. N 0 denote te natural numbers including zero, i.e., {0,,,...}. C n wit n N 0 stands for te set of n-times differentiable functions. Te norm denotes te Euclidean norm. A compact set denoted by S δ x wit δ (0, ) and x R n is defined as {x R n : x x δ}. Te gradient of a function J C : R n R is represented by x J(x) := ( J/ x(x)) and its Hessian by x J(x) := J/ x (x). A function f (ɛ) : R R n is said to be of order O(ɛ), if tere exist k, ɛ R suc tat f (ɛ) kɛ, for all ɛ [0, ɛ]. Te operator k mod n denotes te modulo operation.. MAIN IDEA AND ALGORITHM Te aim of tis section is to motivate and to develop te basic steps of te proposed derivative-free optimization algoritms for unconstrained multidimensional optimization problems of te form min J(x), () x Rn were J : R n R is te so-called objective or cost function... Non-Commutative Vector Fields and Continuous-time Gradient Approximation Consider te scalar continuous-time system ẋ(t) = f (x(t))u (t) + f (x(t))u (t) () wit state x(t) R, vector fields f, f C : R R and periodic inputs u i (t) = u i (t + T) R, i =,, wit period T R. It is very well-known tat, if for example periodic rectangular-saped input signals suc as, t [0, ) 0, t [, ) u (t) =, (3), t [, 3) 0, t [3, 4) 0, t [0, ), t [, ) u (t) = 0, t [, 3), t [3, 4) are applied to system (), as sown in Figure (a), ten, for > 0 small, a second order Taylor expansion reveals (4) x(4) x(0) = [ f, f ](x(0)) + O( 3 ), (5) were [ f, f ] = f x f f x f is te Lie bracket between te vector fields f, f. Scaling te vector fields f, f by gives x(4) x(0) = [ f, f ](x(0)) + O( 3/ ). Ten, by successively repeating (5) wit new initial values x(l), l =,,... and taking te limit 0 leads to te approximate (Lie bracket) system x(t) = [ f, f ]( x(t)). (6) Regarding te (scalar) optimization problem (), one desirable metodology to find an extremum, would be a simple continuous-time gradient descent equation ẋ = x J(x). Interestingly, by defining i =,, and coosing f, f : R R as f i (x) := f i (J(x)), (7) f (x) = J(x), f (x) =, or (8) f (x) = sin(j(x)), f (x) = cos(j(x)), (9) we ave [f, f ](x) = x J(x). Hence, by substitution f i by / f i, i =,, in (5) and (), respectively, we obtain by () a derivative-free approximation of a gradient flow. Tis observation as been utilized and generalized for te design and analysis of continuous-time extremum seeking algoritms, see [3, 5, 7] and references terein. Especially, [5] presents a broad class of generating vector fields f, f for gradient approximation. Non-commutativity comes into play since Lie brackets naturally arise wen studying te commutativity of flows. In particular, if we denote wit ϕ t f (x i 0 ) : R n R n te flow map of te vector field f i : R n R n at time t + t 0 and initial condition x(t 0 ) = x 0 R n, ten we can express equation (5) as ) x(4) = (ϕ ϕ f ϕ ϕ (x f f f 0 ), (0)

3 u (t) u (t) (a) 4 t x(3) ϕ f x(4) x() ϕ f x(0) (b) ϕ f ϕ f x() Figure. Switcing vector fields: (a) periodic inputs u (t), u (t) depicted for one period 4; (b) non-commutative flow maps of vector fields ± f, ± f. as sown in Figure (b). It is easy to see tat x(4) = x(0) if and only if te flow maps ϕ f,ϕ f commute, since te flow is a bijection wit (ϕ f ) = ϕ f and ence x(4) = x(0) if and only if (ϕ f ϕ f )(x(0)) = (ϕ f ϕ f )(x(0)). Furter, it can be sown tat te flow maps commute if and only if te Lie bracket between f, f vanises [], ence te Lie bracket is an infinitesimal measure for te commutativity of vector fields... Non-Commutative Maps In principle, one could just numerically integrate () wit appropriate vector fields and inputs in order to get a derivative-free discrete-time algoritm. However, rectangular-saped inputs as sown in Figure (a) often lead to bad numerical beavior in combination wit standard numerical integration scemes. Hence, in continuous-time algoritms [3, 5], te rectangularsaped inputs are often replaced by sinusoidal inputs in order to avoid tese problems. A key idea of te proposed discrete-time optimization algoritm is to incorporate non-commutative maps more directly into a tailored integration sceme, wic approximates a gradient descent step wit te elp of composite maps of te form (0). Hereby, two callenges arise: i) How to efficiently obtain te maps? ii) Wat are suitable discrete-time input functions in an n-dimensional optimization problem? i) Maps. We construct te composition of flow maps by suitable numerical integration metods, were we consider two approaces. Firstly, we utilize te Eulerintegration metod. It does not lead to a Lie bracket approximation as in (5), since te Euler-integration metod is a first order metod wile Lie brackets are second order effects. However, we sow tat te Euler-integration metod still can be used for gradient approximation. Secondly, we utilize te so-called Heun-integration metod, also known as te trapezoidal-integration metod, wic preserve properties like (5) (see Lemma in Section 3). Accordingly, for an ODE ẋ = g(x) wit x R n and vector field g : R n R n we use te Euler- and Heun-integration steps given by E g(x k ) := x k + g(x k ), and () H g(x k ) := x k + (g(x k) + g(x k + g(x k ))), () as an approximation of te flow map ϕ g. ii) Inputs. As depicted in Figure (a), we consider periodic rectangular-saped inputs. Tese inputs can simply be interpreted as switcing f, f : R n R at every time step (interval) k (). Hence, a switced vector field is defined as f (x)e l if k mod 4 = 0 f g k (x) = (x)e l if k mod 4 =, (3) f (x)e l if k mod 4 = f (x)e l if k mod 4 = 3 were l = k/4 mod n + wit e l te l-t unit vector in R n. Functions f, f : R n R, as for example in (8) and (9) are components of vector fields g k (x) = f i (x)e l, but for te sake of convenience we sometimes call f, f vector fields and use te notation [f, f ] = x f f x f f. Regarding (3), one can tink of executing four steps in every coordinate direction successively, ence te switcing rule represents a coordinate wise approximation sceme. To keep te main idea clear, we consider only tis single switcing function and refer to furter work were we will investigate more complex switcing policies..3. Derivative-free Optimization Algoritms Combining te integration steps () and (), respectively, wit te sequential switcing rule (3) and scaling te vector fields f, f wit /, as discussed in te previous section, results in our discrete-time derivative-free optimization algoritms: x k+ = M g k (x k ) = E g k (x k ) (4a) H g k (x k ). (4b) As sown in te next section, every four steps (scalar case) x k+4 = (Mg k+3 Mg k+ Mg k+ Mg k )(x k ) (5) a gradient descent step under suitable conditions on te vector fields f, f is approximated. Summarizing, two algoritms presented in (4) and also in Algoritm, were derived by utilizing te effect of non-commutative maps, introduced by a sequential switcing rule of vector fields and proper integration metods. In te next section we analyze bot scemes by sowing te gradient approximation property and prove asymptotic convergence to a neigborood of an extremum. 3

4 Algoritm Derivative-free optimization algoritm wit non-commutative maps (4b) : Required: x 0,,f, f, stop criterion : Init: k = 0 3: wile stop criterion is not fulfilled do 4: l n mod k/4 + 5: e l [0 l,, 0 n l ] f (x)e l if k mod 4 = 0 f 6: g k (x) (x)e l if k mod 4 = f (x)e l if k mod 4 = f (x)e l if k mod 4 = 3 7: c g k (x k ) 8: c g k (x k + c ) 9: x k+ x k + (c + c ) 0: k k + : end wile : return [x 0, x,...] 3. MAIN RESULTS In te sequel, basic convergence and approximation properties of bot discrete-time derivative-free optimization algoritms presented in (4) are analyzed. Note, convergence rate results of derivative-free algoritms are almost absent in literature [8]. Te algoritms are applicable for broad classes of objective functions J, including nondifferentiable objective functions, given a coice of proper algoritmic parameters x 0,, f, f, as we observe in simulations, since only function evaluations of J are necessary. However, for te approximation and convergence analysis we assume: Assumption. Te objective function J : R n R in () and te vector fields f, f : R n R in (7) are class C functions. First, we state two lemmas, wic sow te approximated optimization direction after 4n steps, induced by te non-commutative maps of our proposed algoritms (4): Lemma. Let Assumption old and consider algoritm (4a). Ten te evolution of x k R n at step k + 4n is given by x k+4n = x k + ([f, f ](x k ) x f (x k )f (x k ) for all k N 0 x f (x k )f (x k )) + O( 3/ ). (6) Lemma. Let Assumption old and consider algoritm (4b). Ten te evolution of x k R n at step k + 4n is given by x k+4n = x k + [f, f ](x k ) + O( 3/ ). (7) for all k N 0 Te proofs of Lemma and Lemma are given in Appendix A. and Appendix A., respectively. We observe tat te Heun-integration steps Hg k (x k ) preserve te Lie bracket approximation property, as sown in (7). However, as mention in Section, successively applying te Euler-integration steps Eg k (x k ) reveals te Lie bracket but additional terms of te same order O() are present, as sown in (6). Interestingly, coosing proper vector fields f, f, bot algoritms approximate a gradient descent metod: Teorem. Let Assumption old and consider te algoritms in (4) wit te pair of vector fields (9). Ten te evolution of x k R n at step k + 4n is given by for all k N 0 x k+4n = x k x J(x k ) + O( 3/ ) (8) Proof. Tis result follows directly from Lemma and Lemma and simple calculations. Remark. Te pair of vector fields (8) wit algoritm (4a), as commonly employed in continuous-time metods [3], yield an approximation sceme of te form x k+4n = x k x J(x k )( + J(x k )) + O( 3/ ). (9) Hence, tis sceme is not a gradient descent step but it is still suitable for optimization if, for example, te objective function is positive semi-definite. For te convergence analysis of (4), we impose te following additional assumptions on te cost function J : R n R in () : Assumption. J(x) is radially unbounded and tere exists a x R n suc tat x J(x) (x x ) > 0 for all x R n \{x }. Remark. Assumption A implies tat x is te unique global minimizer and tere exists no oter local minimum. Te following teorem sows asymptotic convergence to a neigborood of te minimum x of J(x): Teorem. Let Assumption and Assumption old. Consider te discrete-time derivative-free optimization algoritm (4) and let te vector fields f, f satisfy for (4a) and (4b) respectively: x J(x) = ([f, f ] x f f x f f ) (x), (0) x J(x) = [f, f ](x). () Ten for all 0 < δ < δ tere exists an > 0 suc tat for all (0, ) and all initial conditions x 0 S δ x, x k converges to S δ x. 4

5 .5 4 x k, y k, z k.5 J(x) k Figure. Convergence of x k in (4) for setup P wit = 0.5 to a neigborood of x =. Te flow maps as in (5) for M = E are igligted wit E f ( ), E f ( ), E f ( ), E f ( ) and te same color sceme (dotted) for (5), M = H. Tis is compared wit te trajectory resulting from a gradient descent algoritm wit te exact gradient ( ). By filtering x k, a good asymptotic convergence beavior of y k to x can be observed ( ). Te proof of Teorem is given in Appendix A.3. Note, tat Teorem states te semi-global uniform practical asymptotic stability property [3] of S δ x of (4) under te given assumptions. 4. SIMULATIONS In tis section we study our discrete-time derivative-free optimization algoritms (4) in various simulation examples and provide some tuning rules. To tis end, we consider te cost function J(x) = (x ) + 6. If not oterwise specified we use te vector fields (9) and initialize te algoritms wit x 0 = x(0) = 0.5. For te sake of convenience we define tis setup as P := (J, x 0, f, f ). Note tat te results obtained wit suc a quadratic cost function also provide insigt into te results tat would be obtained wit an arbitrary function of class C near a minimum wit positive definite Hessian. We start by analyzing (4), applied to te basic setup P, wic is depicted for (a large) step size = 0.5 in Figure, were te switcing of te vector fields f, f is igligted. As it can be observed, te trajectories of x k are converging into a neigborood of x, wic is of order O( ) (see proof of Teorem ). To eliminate te steady-state oscillatory beavior, we propose te filter y k+ = y k + 4n k i=k 4n x i+ x i were y 0 = x 0 and x l = x 0, l < 0, wic sows a good asymptotic convergence beavior of te algoritms (4) to te extremum x as illustrated in Figure. Obviously, y k is a windowed filter wit lengt 4n, wic is not fed back into te algoritm. Tis beavior can also be observed in a multidimensional optimization problem as depicted in Figure 3. 0 x 3 Figure 3. A two-dimensional optimization problem wit J(x) = (x ) + (x ) + 6. Trajectories x k ( ) of (4a) and y k ( ) converge to te extremum x = [, ] ( ) initialized wit x 0 = y 0 = [0.5, 0.5] ( ). x k k Figure 4. Coosing smaller step sizes for algoritm (4a) or (4b) yield to a smaller neigborood of convergence. Trajectories of x k for setup P wit step sizes = 0.5 ( ), = 0. ( ), = 0.0 ( ), = 0.00 ( ). As known from continuous-time metods, see for example [3], increasing te (switcing) frequency, and ence reducing te step size, leads to a convergence into a smaller neigborood around x as depicted for different s in Figure 4. Clearly, for decreasing te step size, more iterations are needed, ence more function evaluations. As mentioned in Section, an advantage of te proposed class of algoritms is robustness wit respect to noisy objective functions. Terefore, consider te setup P, but wit a noisy cost function J : R R suc tat J(x k ) N (J(x k ), 0.04), () were N (J(x k ), 0.04) is a normal distribution wit mean J(x k ) and standard derivation 0.04 = 0.. In Figure 5, algoritm (4b) is compared wit te well known difference quotient metod to approximate te gradient x 0 5

6 x k, x k k Figure 5. Study of noisy objective function: trajectory x k of algoritm (4b) ( ), trajectory x k of gradient descent wit gradient approximation via difference quotient ( ), trajectory of gradient descent wit exact gradient ( ). D(J( x)) := (J( x k ) J( x k + )) used in a gradient descent algoritm x k+ = x k D(J( x)) wit step size = 0.0 and initial condition x 0 = x 0. Apparently, approximating te gradient via te difference quotient metod is muc more sensitive regarding noisy objectives as algoritms (4). Tis is reasoned by te averaging beavior of te proposed algoritms, implied by utilizing non-commutative maps. Concluding, we illustrated bot algoritms (4) on simple examples and we introduced a filtering metod. Secondly, we sowed te impact of decreasing and te performance under noisy measurements. Regarding te convergence beavior, algoritm (4a) and (4b) sow similar convergence beavior for te considered vector fields (9) and switcing function (3). Neverteless, algoritm (4b) needs two times more function evaluations tan (4a). 5. CONCLUSION In tis work we introduced a metodology for te design of novel discrete-time derivative-free optimization algoritms. Based on non-commutative maps, designed by two different numerical integration metods and a sequential switcing rule, it was sown ow to approximate a gradient descent step. Furtermore, practical convergence and stability towards an extremum of te objective function was proven. Te algoritms are illustrated using numerical examples. Te introduced class of algoritms raise several new researc questions, wic will be part of future work. For example, wat are more efficient switcing rules in multidimensional optimization problems compared to te introduced coordinate wise switcing rule. Moreover, it is interesting to caracterize a general class of noncommutative maps wic sow gradient approximation properties or to study convergence rates or accelerated gradient scemes. References [] J. Feiling, A. Zeller, and C. Ebenbauer, Derivativefree optimization algoritms based on noncommutative maps, IEEE Control Systems Letters, submitted, 08. [] K. B. Ariyur and M. Krstic, Real-time optimization by extremum-seeking control. Jon Wiley & Sons, 003. [3] M. Guay and T. Zang, Adaptive extremum seeking control of nonlinear dynamic systems wit parametric uncertainties, Automatica, vol. 39, no. 7, pp , 003. [4] Y. Tan, W. Moase, C. Manzie, D. Nešić, and I. Mareels, Extremum seeking from 9 to 00, in Control Conference (CCC), 00 9t Cinese. IEEE, 00, pp [5] K. T. Atta, A. Joansson, and T. Gustafsson, Extremum seeking control based on pasor estimation, Systems & Control Letters, vol. 85, pp , 05. [6] J. Poveda, M. Benosman, and A. Teel, Distributed extremum seeking in multi-agent systems wit arbitrary switcing graps, IFAC-PapersOnLine, vol. 50, no., pp , 07. [7] J. Poveda and A. Teel, A robust event-triggered approac for fast sampled-data extremization and learning, IEEE Transactions on Automatic Control, vol. 6, no. 0, pp , 07. [8] Y. Nesterov and V. Spokoiny, Random gradient-free minimization of convex functions, Foundations of Computational Matematics, vol. 7, no., pp , 07. [9] J. C. Duci, M. I. Jordan, M. J. Wainwrigt, and A. Wibisono, Optimal rates for zero-order convex optimization: Te power of two function evaluations, IEEE Transactions on Information Teory, vol. 6, no. 5, pp , 05. [0] V. Torczon, On te convergence of pattern searc algoritms, SIAM Journal on optimization, vol. 7, no., pp. 5, 997. [] J. C. Spall, Multivariate stocastic approximation using a simultaneous perturbation gradient approximation, IEEE Transactions on Automatic Control, vol. 37, no. 3, pp , 99. 6

7 [] R. Brockett, Te early days of geometric nonlinear control, Automatica, vol. 50, no. 9, pp. 03 4, 04. [3] H.-B. Dürr, M. S. Stanković, C. Ebenbauer, and K. H. Joansson, Lie bracket approximation of extremum seeking systems, Automatica, vol. 49, no. 6, pp , 03. [4] H.-B. Dürr, M. S. Stanković, K. H. Joansson, and C. Ebenbauer, Extremum seeking on submanifolds in te Euclidian space, Automatica, vol. 50, no. 0, pp , 04. [5] V. Gruskovskaya, A. Zuyev, and C. Ebenbauer, On a class of generating vector fields for te extremum seeking problem: Lie bracket approximation and stability properties, Automatica, vol. 94, pp. 5 60, 08. [6] H.-B. Dürr, C. Zeng, and C. Ebenbauer, Saddle point seeking for convex optimization problems, IFAC Proceedings Volumes, vol. 46, no. 3, pp , 03. [7] A. Sceinker and M. Krstić, Extremum seeking wit bounded update rates, Systems & Control Letters, vol. 63, pp. 5 3, 04. [8] S. Micalowsky, B. Garesifard, and C. Ebenbauer, A Lie bracket approximation approac to distributed optimization over directed graps, Automatica, 08 (provisionally accepted). [9] C. Altafini, Nonintegrable discrete-time driftless control systems: geometric pases beyond te area rule, in Decision and Control (CDC), 06 IEEE 55t Conference on. IEEE, 06, pp [0] D. Popovic, M. Jankovic, S. Magner, and A. R. Teel, Extremum seeking metods for optimization of variable cam timing engine operation, IEEE Transactions on Control Systems Tecnology, vol. 4, no. 3, pp , 006. [] H. Nijmeijer and A. Van der Scaft, Nonlinear dynamical control systems. Springer, 990, vol. 75. A. APPENDIX A.. Proof of Lemma We consider te derivative-free optimization algoritm (4a) and assume witout loss of generality k mod 4 = 0 and k/4 mod n = 0. Due to our assumption on k, g k (x k ) = f (x k )e. Since, te switcing policy (3) is based on a coordinate-wise implementation, it is sufficient to analyze te evolution of x k in only one dimension. Hence, it olds x k+ = x k + f (x k )e. (3) For te sake of readability in te sequel we define fi k := f i (x k )e and Fi k := ( f i x (x k)) e and neglect e. Terefore, te next step is given by x k+ = x k + (f k + f (x k + f k )). (4) Performing a Taylor expansion on f (x k + f k ) in (4) reveals x k+ = x k + (f k + fk ) + Fk fk + O(3/ ), (5) were all iger order terms are pused in O( 3/ ). Repeating tis procedure, as presented above, x k evolutes as follows: x k+3 = x k + f k + (Fk fk Fk fk Fk fk ) + O( 3/ ), (6) x k+4 = x k + (F k fk Fk fk Fk fk Fk fk ) + O( 3/ ). (7) Eventually, repeating te same procedure for dimensions {, 3,..., n} delivers te same expression (7) for eac dimension, ence (6) can be directly concluded. A.. Proof of Lemma As in te proof of Lemma we repeat te same procedure and only state te main equations. For notations and assumptions (w.l.o.g) reread Appendix A.. Terefore it olds x k+ = x k + (f k + f (x k + ) f k). (8) Performing a Taylor expansion on f (x k + f k ) in (8) reveals x k+ = x k + f k + Fk fk + O(3/ ). (9) Repeat te procedure, as presented above, x k evolutes as follows: x k+ = x k + (f k + fk ) + ) (F k fk + Fk (fk + fk ) + O( 3/ ), (30) x k+3 = x k + f k + O(3/ ) + ( Fk fk + Fk fk + Fk fk ), (3) x k+4 = x k + (F k fk Fk fk ) + O(3/ ). (3) Eventually, repeating te same procedure for dimensions {, 3,..., n} delivers te same expression (3) for eac dimension, ence (7) can be directly concluded. 7

8 A.3. Proof of Teorem Let 0 δ 3 δ δ δ 0. Te proof is separated in: ) Define a Lyapunov-like function V(x), suc tat V(x k+4n ) V(x k ) < 0 for S δ 0 x \S δ 3 x, ) Practical invariance of S δ x, 3) Convergence of x k to S δ x wit x 0 S δ x. ) Consider te algoritms (4) and teir evolution as given in (6) and (7) under conditions (0), () on vector fields f, f as stated in Teorem. W.l.o.g., let te Lagrange remainder of te Taylor expansion (6) and (7), respectively, be of te form 3/ R k = O( 3/ ), were R k R n depends on te vector fields f, f and teir Jacobians and Hessians. Hence, (4) is given by 3) Due to (36) wit > 0, it olds tat for every initial value x 0 S δ x, tere exist a maximum number of iterations k > 0, suc tat for all (0, min{, }), it olds x k S δ 3 x. Furtermore, as in ), one can sow tat tere exists a 3 > 0, suc tat for all (0, 3 ) and x 0 S δ 3 x, x k S δ x for all k 0. Tus, for all (0, min{,, 3 }), if x 0 S δ x, ten x k converges to S δ x. Figure 6 illustrates te proof and te introduced compact sets S x. x 0 x k+4n = x k x J(x k ) + 3/ R k. (33) Consider te Lyapunov-like candidate function V(x k ) = J(x k ) J(x ), ten it olds for all k 0, V(x k+4n ) V(x k ) = ) = J (x k x J(x k ) + 3/ R k J(x k ) = x J(x k ) x J(x k ) + 3/ x J(x k ) R k + γ H k γ, (34) were te last equality is gained by a second order Taylor expansion at x k, wit te Lagrange reminder γ H k γ, were γ = x J(x k ) + 3/ R k and H k R n n. R k and H k depend on J, f, f and teir Jacobians and Hessians, Hence, R k, H k, and x J(x) are bounded on every compact set S δ 0 x, due to Assumption. Terefore, for every δ 0 tere exists a 0 R δ0 R suc tat for all x k S δ 0 x old x S δ 3 x S δ x S δ x S δ 0 x Figure 6. Convergence of x k R ( ) wit initial value x 0 ( ) S δ x S δ 0 x to S δ x. For all k > k, x k stays in S δ x. Note, every 4n-t step is marked wit. V(x k+4n ) V(x k ) x J(x k ) + 3/ R δ0. (35) Ten under Assumption, tere exists a > 0, suc tat for all (0, ) old V(x k+4n ) V(x k ) ɛ on L = L 0 \L 3, (36) wit ɛ > 0, S δ 0 x \S δ 3 x L and sub-level sets of V(x), L 0 S δ 0 x and L 3 S δ 3 x. ) By Assumption and te first order Taylor expansions, it olds tat for all k 0, 0 < l < 4n, tere exist an upper bound on te Lagrange reminders 0 R k+l R in S δ 0 x suc tat V(x k+l ) = V(x k + k+l i=k g i (x i )) V(x k ) + R k+l (37) Hence, in combination wit ) tere exist a > 0 suc tat for all (0, ), x 0 S δ x x k S δ 0 x for all k 0. 8