Simulated Annealing with Time dependent. Matthias Lowe. Universitat Bielefeld. Fakultat fur Mathematik. Postfach Bielefeld.

Size: px

Start display at page:

Download "Simulated Annealing with Time dependent. Matthias Lowe. Universitat Bielefeld. Fakultat fur Mathematik. Postfach Bielefeld."

Neal Greer
5 years ago
Views:

1 Simulated Annealing with Time dependent Energy Function via Sobolev inequalities Matthias Lowe Universitat Bielefeld Fakultat fur Mathematik Postfach Bielefeld Abstract We analyze the Simulated Annealing Algorithm with an energy function U t that depends on time. Assuming some regularity conditions on U t (especially that U t does not change too quickly in time), and choosing a logarithmic cooling schedule for the algorithm, we derive bounds on the Radon-Nikodym density of the distribution of the annealing algorithm at time t with respect to the invariant measure at time t. Moreover we estimate the entrance time of the algorithm into typical subsets V of the state space in terms of t (V ). c Keywords: Simulated Annealing, Sobolev inequalities, Spectral gap, Markov processes Introduction Let X be a nite set. The well known Simulated Annealing (SA) algorithm is an inhomogeneous Markov process Y t on X with the aim to minimize a given function U : X! R. The idea behind SA is to think of U as an energy function and to choose the Markov process in such a way that the transition kernel at time t has at its invariant measure t, the Gibbs distribution with

2 energy U at temperature = t. So, if t diverges to innity in the limit for large times the invariant measure becomes concentrated on the set of minima of U. On the other hand one has to "freeze" the process slowly enough to guarantee convergence of its distribution at time t towards equilibrium. Since the introduction of SA by Kirkpatrick et al. in [K83] it has found a wide range of applications (see [GG84],[V93]), and the problems mentioned above, i.e. the optimal choice of t and the asymptotic behavior of SA have been studied by several authors. The problem of an asymptotically optimal "cooling schedule" (i.e. the dependence of t from t) has been settled by Hajek [Ha88], who showed the optimality of a logarithmic freezing schedule t = log(a + Bt), where A and B are positive constants and m is, roughly m speaking, the largest hill to be climbed to reach the global minimum. The rate of convergence has been investigated by Gidas in [Gi85] and Mitra et al. in [M85] on the basis of Dobrushin coecients for Markov chains. Cantoni has obtained bounds of the type P (Y t E) O(t? ) in [C90] by using Freidlin-Vencel techniques (see [FV84]) on perturbed dynamical systems. In [HS88] Holley and Stroock were able to give bounds for P x (Y t = y)= t (y) for reversible SA processes based on Sobolev inequalities. Additionally they obtained sharp estimates for the spectral gap of the operators associated to the SA process. Based on rened spectral gap estimates Gotze in [G9] proved eective bounds for the waiting time of escaping from an unlikely set (i.e. a set of high energy). These techniques were generalized by Deuschel and Mazza in [DM9] to Markov processes on nite sets similar to SA. In this paper we are going to analyze that variant of the SA algorithm, where U depends on t, i.e. U = U t. These processes have rst been studied by Frigerio and Grillo in [FG93]. Not only they are a natural generalization of SA, but also they can be regarded as a discrete version of quantum diusions (see [FG93], Section 4). Moreover these processes become interesting when thinking of SA as a sort of genetic algorithm (the reader is referred to the book of Holland [H75] for an introduction and a survey on these algorithms). Then the SA process can be regarded as an individual undergoing a mutation, the mutation being governed by the "tness-function" U. In such a model the time dependence of U comes up naturally, if one e.g. thinks of the tness of an individual being given by its environment, that usually changes in time.

3 In [FG93] Frigerio and Grillo have proved that for SA with time dependent energy function the distribution of the process and its invariant measure become indistinguishable in the weak topology for large times t, if the following catalogue of conditions is fullled: the SA process is driven by a bistochastic matrix, U t (x) is uniformly bounded in x and t by a nonnegative number M and has a derivative with respect to t bounded by C t for some C; > 0, the generator L t associated to the process at time t admits a bound on its spectral gap of the form e?tm for some ; m > 0, and we choose a logarithmic cooling schedule t = log( + t) with h > m and m=h <. These h conditions, that are especially satised, when U t admits an uniform limit U look rather natural from a point of view of Sobolev inequalities and spectral gap estimates. On one hand one has chosen a cooling schedule that also is close to optimal, when U is constant in t. On the other hand in some sense m and M are the only values which enter in these techniques for time independent energy function. So if the variation of U t is moderate one should expect spectral gap estimates to work. The aim of this paper is to show that under the conditions formulated by Frigerio and Grillo in [FG93] this is indeed the case. By Sobolev inequalities we obtain bounds for the density of the distribution of the process with respect to the invariant distribution at time t and are able to analyze the speed of convergence towards equilibrium. It turns out the conditions formulated above cannot essentially be weakened. Our main tools for this approach are those of Holley and Stroock in [HS88] and Gotze in [G9]. We organize the rest of the paper in the following way. Section contains our basic denitions and a formal description of the model. In Section 3 we describe the behavior of the Radon-Nikodym derivative of the distribution of the process at time t with respect to the Gibbs-distribution at that time with the help of Sobolev inequalities and check, under which conditions these are fullled. Moreover we use spectral gap estimates to bound the probability of large entrance times into typical sets. The Model Let us now give a formal description of the algorithm we have mind. To this end let q(x; ) be a probability distribution on X. Assume that the 3

4 random walk associated with q(x; y) is irreducible (i.e. for all pairs x; y X there is a number n := n(x; y) such that q n (x; y) > 0 where we have put q n (x; y) = P zx q(x; z)q n? (z; y)). Moreover for simplicity we assume that q(x; y) is time-reversible with respect to a probability measure on X, i.e. (x; y) := (x)q(x; y) = (y)q(y; x) = (y; x) () for all x; y X. Note that is q-stationary and therefore irreducibility of q implies that charges every point in X. This assumption of reversibility often makes life easier in convergence theorems for stochastic processes. On the other hand a comparison of the spectral gap estimates for time homogeneous, reversible Markov chains by Diaconis and Stroock [DS9] with those for homogeneous, irreversible chains by Fill [F9] shows that there is some hope to drop the reversibility condition in future work. Let (U t ) t be a sequence (indexed by R + ) of functions U t : X! R. Moreover let us assume that t 7! t is a monotonely increasing function in t, which is strictly positive ( t will play the role of the inverse temperature at time t). Corresponding to q, U t, and t we may dene a transition probability q t (x; y) by q t (x; y) = ( exp(?t (U t (y)? U t (x)) + )q(x; y) if x 6= y? P z6=y q t (x; z) if x = y () where we have put (x) + := max(x; 0). It is easy to check that q t is reversible with respect to the Gibbs measure t (with reference measure ) with energy U t at inverse temperature t dened by t (y) := e?tut(y) Z t (y) (3) where Z t := P y e?tut(y) (y) is the partition function. Note also that q t again is irreducible and reversible and that t is its stationary distribution, or, equivalently, that the operator Q t given by [Q t ](x) := X yx (y)q t (x; y) (4) is a self-adjoint contraction on L ( t ). 4

5 If we now dene the operator L t := Q t? I on L ( t ), i.e. [L t ](x) := X yx((y)? (x))q t (x; y); x X (5) we are able to describe the continuous time SA algorithm. Namely let its transition probabilities P s;t (x; y), i.e the probabilities to arrive in y at time t when starting in x at time s, for s t be given by the [P s;t](x) = [P s;t [L t ]](x) for s < t (6) and [P s;s ](x) := (x) for : X! R. Here [P s;t ](x) := X yx (y)p s;t (x; y); x X: The intuitive meaning of.6 is that starting in x at time s the process waits for exponential time with mean one and then chooses a neighbor y according to q(x; y). If then U t (y) U t (x) (where t is the instantaneous time after having waited) the algorithm moves to y. Otherwise it moves to y with probability exp(? t (U t (y)? U t (x))) and stays in x with probability? exp(? t (U t (y)? U t (x))). Obviously this is the continuous time analogue of the ordinary discrete time SA process where we have chosen U to depend on the time. The advantage of using continuous time instead of discrete time becomes especially apparent when studying the time evolution of f t, the Radon-Nikodym derivative of the distribution of Y t with respect to t, which then can be described by means of dierential equations instead of dierence equations that are harder to solve. Let us remark that together with Q t also L t is a self-adjoint operator on L ( t ) with largest eigenvalue 0. Due to the irreducibility of q t the eigenspace of the eigenvalue 0 is one-dimensional and the constants are the only eigenfunctions. The key observation on which most of the following techniques are based is the relation of second smallest eigenvalue t of?l t and the Dirichlet form associated with?l t. This relation is given by t = inff E t(; ) V ar t () : : X! R non constantg: (7) 5

6 Here E t (; ) :=? < ; [L t ] > t = X x;yx ((x)? (y)) t (x; y) (8) is the discrete Dirichlet form associated to L t, < ; > t denotes the scalar product in L ( t ), V ar t () is the variance of with respect to t and t is given by t (x; y) := t (x)q t (x; y) = t (y)q t (y; x) = t (y; x): (9) Note that E t is related to the Dirichlet form E 0 (; ) := P x;yx ((x)? (y)) (x; y) via the relation E t (; ) = X x;yx ((x)? (y)) (x; y) exp(? t max(u t (x); U t (y))) Z t : (0) (.7) is derived by the well known variational characterization of eigenvalues (see e.g. [HJ85]). For the rest of the paper let us assume that the (U t ) t fulll the following conditions : First we may without loss of generality assume that min x U t (x) = 0. Moreover let us choose such sequences of energy functions that there exist universal constants 0 and ; C; m; M > 0 such that 0 U t (x) M for all x X, for all t 0 ; U t(x) C ( + t) for all x X; () t e?tm for all t 0: (3) We will conclude this section with the observation that assumption (.3) is automatically fullled if we choose m similar to [Ha88] as the maximal hill to be climbed at any time to reach to global minimum. More formally for xed t and any two points x; y X let us call a sequence (x i ) ik a path from x to y, if x = x, x k = y and q t (x i? ; x i ) > 0 for all i = ; : : : ; k?. Dene m t (x) := minfe 0 : there exists y X with U(y) < U(x) and a path (x i ) ik from x to y with U(x i ) U(x) + E for all i kg: 6

7 Finally put m t := max xx m t(x): (4) With these denitions we are able to give the precise order (up to a factor) of the spectral gap t : Theorem. There exist constants ;? > 0 such that e?tmt t?e?tmt (5) Proof: Note that the statement of Theorem. is the analogue to the spectral gap estimate Theorem. in [HS88] for the ordinary SA. As a matter of fact, since we analyze the spectral gap of the instantaneous time operator L t, which does not "feel" the change of U t in t, the proof stays the same for our case. Since we are not going to make use of the upper bound in the rest of the paper, we will skip its proof here. The proof of the lower bound follows the lines of the proof of Lemma.7 in [HS88]. By (.7) it suces to show the existence of a constant, such that for all t, and for all f : X! R E t (f; f) V ar t (f) =? P xx f(x)[l t f](x) t (x) V ar t (f) This is done by bounding the variance as follows : For any two x; y X take a path p x;y t with vertices p x;y t that where the % x;y t length of p x;y t max p x;y t (i) U t (p x;y t (i)) = min % x;y t max % x;y t (j) e?tmt : (6) U t (% x;y t (j)) are paths form x to y with vertices % x;y t and put N t := max x;yx n t(x; y): (i) from x to y such (j). Let n t (x; y) be the We introduce the indicator ( if there exists 0 i t nt (x; y) : p x;y t (i) = z and p x;y t (i + ) = w z;w(x; y) = 0 otherwise. In the following we put t z;w (x;y) X (x;y) V ar t (f) = x;yx = 0 if t z;w (x; y) = 0. Then for any f : X! R (f(x)? f(y)) t (x) t (y) 7

8 = X x;yx X x;yx 0 nt(x;y) i= n t(x;y) n t (x; y) f(p x;y t (i))? f(p x;y t (i? )) X i= (f(p x;y t A t (x) t (y) (i))? f(p x;y t (i? ))) t (x) t (y) by Jensen's inequality. Therefore X V ar t (f) N t t z;w(x; y)(f(z)? f(w)) e?t max(ut(z);ut(w)) (z; w) x;y;z;wx t (x) t (y) e?t max(ut(z);ut(w)) (z; w) X N t [ max t t (x) t (y)z t z;w(x; y) e?t max(ut(z);ut(w)) (z; w) ] z;wx x;yx [ X z;wx (f(z)? f(w)) e?t max(ut(z);ut(w)) (z; w) Z t ]: (7) Since N t is bounded by the size of the vertex set jxj (because of irreducibility) and the second term in brackets in (.7) is twice the Dirichlet form E t (f; f) (due to (.0)) it suces to calculate that t t (x) t (y)z t z;w(x; y) e?t max(ut(z);ut(w)) (z; w) = t z;w(x; y) (z; w) e tmt [ t z;w(x; y) (z; w) (x)(y) e?t(max(ut(z);ut(w))?ut(x)?ut(y)) Z t (x)(y) min vx (v) ]; since by denition m t (max(u t (z); U t (w))?u t (x)?u t (y)) whenever t z;w (x; y) = for all x; y; z; w X. Since [ t z;w (x;y) (x)(y) ] is bounded independently (z;w) minvx (v) of t, this proves the lower bound. Remark. Note that under condition (.) especially m := max t m t exists and therefore we obtain for all t. t e?tm 8

9 3 Convergence of the Algorithm via Sobolev inequalities In this section we are going to analyze the behavior of the density of the distribution of the process with respect to the invariant measure on the basis of a Sobolev inequality, which we rst assume and then show to hold true. The results of this section are similar to those for ordinary SA proved by Holley and Stroock in [HS88]. Like they we start with a result on a dierential inequality, on which the rest of the section is based. Lemma 3. Let a; b; c be non-negative numbers, let a > 0, let " > 0, and 0 <. Let u be a dierentiable function u : R +! R + 0 fullling d dt u(t) =: u0 (t) ( + t ) [?au +" + bu + cu ] for t > 0 (where we adopt the convention that primes denote derivatives with respect to t). Then u(t) where := max(b; a + c). 4 4" a? exp( " (? ( +? t)? )) Remark 3. Lemma 3. generalizes Lemma. in [HS88] where =. Our proof is similar. 3 5 " Proof: Put v(t) := (u(t)). Then v 0 (t) = u0 (t) (u(t)) ( + t ) [?a( (v(t) )+" ) + b v(t) + c] = ( + t ) [?a( + ( v(t) )+" + b v(t) + a + c] ( + t ) [?a v(t) ( + 4 " )+" + ( + v(t) )]: 9

10 Set w(t) := ( + v(t) ) exp(? (+t)? ). Then? w 0 (t) = exp(? ( + t)?? exp(? ( + t)?? =?a + t)? exp("( 4 "? Dene z(t) := (w(t))?" and obtain and therefore Hence )[ v0 (t)? )[? a v(t) ( + 4 " z 0 (t) =?"w0 (t) (w(t)) +" v(t) ( + ( + t) )] )+" ]( + t ) )(w(t)) +" ( + t ) : z 0 (t) a" + t)? exp("( )( 4 "? + t ) : z(t) a" Z t ( + s)? exp(" )( 4 " 0? + s ) ds?? = a" Z (+t) 4 " = a" 4? exp("s)ds + t)? [exp("( "? Recalling the denitions of z; w and v this yields exp(" which implies ( + t)?? u(t) )(u(t))?" a" 4 4 4" a )? exp("? )]: + t)? [exp("( "?? exp( " (? ( +? t)? )) )? exp("? )]; To make use of Lemma 3. let us assume that the following hypothesis on the SA are fullled: There is a 0 and " :

11 (A) there is a constant B < and such that where R t := sup x U 0 (x). 0 tm + t R t B ( + t) (A) there is a constant A and a < p < such that for all t 0 jjf? < f > t jj p;t A( + t) E t (f; f) for all f L p ( t ), where < > t denotes expectation with respect to t. The following Lemma shows, how to use Lemma 3. and assumption (A) as well as the Sobolev inequality (A) to bound the density of the distribution of the process at time t with respect to the invariant measure at time t. To this end set f t (y) := P x(y t = y) t (y) where x is the starting point of the SA process. Lemma 3.3 Assume that hypotheses (A) and (A) are fullled and set " := p? =p as well as K := 4 " (4AB + )? exp(? "(B + =A)) (? e))! :? Then after a waiting time of t e??. Proof: Therefore Note that jjf t jj ;t + K " d dt Z t =?Z t ( 0 t < U t > t + t < U 0 t > t): d dt t(y) =? t (y)( 0 t ^U t (y) + t ^U 0 t (y))

12 where ^Ut (y) := U t (y)? < U t > t. Using the Fokker-Planck equations (.6) we arrive at d dt f t(y) = X (f t (z)? f t (y)) t (z; y) + f t (y)( 0 t t (y) ^U 0 t (y) + t ^U t (y)): zx Putting u(t) := jjf t? jj ;t = jjf tjj ;t?,this implies u 0 (t) =?E t (f t ; f t ) + X y X f t (y)( 0 t ^U t (y) + t ^U 0 t (y)) t (y) =?E t (f t ; f t ) + X y X(f t (y)? ) ( 0 t ^U t (y) + t ^U 0 t (y)) t (y) + X y X(f t (y)? )( 0 t ^U t (y) + t ^U 0 t (y)) t (y)?e t (f t ; f t ) + X y X(f t (y)? ) ( 0 t M + t R t ) t (y) + X y X(f t (y)? )( 0 t M + t R t ) t (y) ( + t ) [? A u+" + Bu + Bu ]: Thus the estimate of the lemma follows from Lemma 3.. It remains to show that under conditions (.)-(.3) we have modeled the SA in such a way that assumptions (A) and (A) are fullled. This is the content of the following lemma. Lemma 3.4 Under the assumptions (.)-(.3) for any p > there exists a cooling schedule t such that the associated SA algorithm satises (A) and (A). Proof: We follow the ideas of the proof of Theorem. in [HS88]. So we choose < and according to assumption (.) and we put t = log( + t); m + " where " = p? M. Now for any xed t 0 on one hand we have p 0 t M + tr t M (m + ")( + t) const: ( + t) + log( + t) m + " C ( + t)

13 with a constant depending on for any 0 < <. On the other hand for every f : X! R X X < f > t ) yx(f(y)? p t (y) = (f(y)? < f > t ) j(f(y)? < f > t )j p? t (y) yx X (f(y)? < f > t ) t (y)jj(f? < f > t )jj p? yx emt E t (f; f)jjf? < f > t jj p? Since Z t we obtain Hence jjf? < f > t jj p e?tm min xx jjf? < f > t jj p min t(x) xx jjf? < f > t jj p p;t: jjf? < f > t jj p p;t M (p?) emt t jjf? < f > t jj p? p;t E t (f; f)e p : (min xx (x)) p? p Thus for the nite constant A = jjf? < f > t jj p;t (minx (x)) p?=p M (p?) Aet(m+ p E t (f; f): So the choice of t and " implies the statement of the lemma. Summarizing we have proved the following theorem Theorem 3.5 Let Y t be the SA process with time dependent energy function U t. Under conditions (.)-(.3) the cooling schedule t = log( + t) m+" with 0 < < implies the following bound on the Radon-Nikodym density f t (y) := Px(Yt=y) t(y) jjf t (y)jj + K " for t e??, and any starting point x X. Here the constants " and K are chosen according to lemmas above. 3

14 Remarks 3.6. Following the same ideas as in [HS88], Section, it is possible to derive uniform bounds for f t as well. We omit a detailed discussion here.. The conditions of Theorem 3.5 are not only sucient to bound f t in L ( t ) (and therefore to ensure convergence of the algorithm), but also are nearly necessary conditions to make SA sensible. Especially if m t and therefore any bound on U t diverges to innity with t! even the cooling schedule t = log( + t), which we expect to be optimal, is in fact a "heating" schedule, i.e. t! 0 and therefore the m equilibrium measures at time t converge to the uniform distribution on X. Obviously, in this situation SA does not make sense any longer. Also note that M being nite does not imply that Mt 0 = 0 (where M t = sup x U t (x)), but only that, roughly speaking, Mt 0 t? for some >, i.e M t changes noticeably slower than U t does. 3. Note that if in condition (.) > U t admits a limit (for t! ) U. As one would expect, SA with time dependent energy function U t then asymptotically behaves like SA with energy function U. Let us nally show, how to use Theorem 3.5 to derive a bound on the entrance time of the algorithm into typical sets. Note that the following theorem especially becomes interesting, when the minima of U t become concentrated on a subset V X. Theorem 3.7 Let V X and denote by V c the rst entrance time of the algorithm into V c when starting in x V. Then under the conditions and with the notations of Theorem 3.5 for t e??. P ( V c > t) ( + K " ) t (V ) + t (V ) Proof: We have P ( V c > t) = P (Y s V 8s t) inf st P (Y s V ); 4

15 because of fy r V 8r tg fy s V g if s t. Moreover (P (Y t V )? t (V )) = X t? ) V (y) t (y) A yx(f jjf t? jj ;t t (V ): Hence the assertion follows form Theorem 3.5. Surprisingly the SA with time dependent energy function (and < ) seems to resist calculations as in [G9] for the ordinary SA, where more applicable constants than our ( + K ") were obtained. On the other hand a rough estimate of K yields the following corollary. Corollary 3.8 With the notations of Theorem 3.7 we have 0 P ( V c > 56C + ( A e q (min x (x)) )=4 t (V ) + t (V ) for t e?=?. Proof: Estimating the constants in K with p = 4, i.e " = and = = yields the bound. Remark 3.9 Throughout the paper we have used a bound for the spectral gap of the form e?tm. As Theorem. shows these bounds are optimal up to a constant. On the other hand, since usually m, or m t resp. are as hard to determine as the argument where U t attains its minimum, these bounds as well as the associated cooling schedules are not very applicable. For practicle use Poincar-type or Cheeger-type bounds as have been obtained in [DS9] and [G9] seem to be preferable. References [C90] O. Cantoni, Asymptotics of simulated annealing algorithms, Ph.D. Thesis, Universit Paris-Sud, Orsey (990) 5

16 [DM9] J.D. Deuschel, C. Mazza, L Convergence of Time Non- Homogeneous Markov Processes I: Spectral Estimates, Preprint (99) [DS9] P. Diaconis, D. Stroock, Geometric Bounds for Eigenvalues of Markov chains, Ann. Appl. Prob., 36-6 (99) [F9] J.A. Fill, Eigenvalue bounds on Convergence to stationarity for nonreversible Markov chains, with an application to exclusion processes, Ann. Appl. Prob., 6-87 (99) [FV84] M.I. Freidlin, A.D. Vencel, Random Pertubations of Dynamical Systems, Springer, Grundlehren d. math. Wissenschaften 60 (984) [FG93] A. Frigerio, G. Grillo, Simulated annealing with time-dependent energy function, Math. Zeit. 3, 97-6 (993) [GG84] S. Geman, D. Geman, Stochastic Relaxation, Gibbs Distribution, and the Bayesian Restoration of images, IEEE Trans. on Pattern Anal. and Machine Intelligence, PAMI-6, 7-74 (984) [Gi85] [G9] B. Gidas Nonstationary chains and the convergence of the annealing algorithm, J. Stat. Phys. 39, 73-3 (985) F. Gotze, Rate of convergence for simulated annealing processes, Preprint 9-008, SFB 343, Bielefeld (99) [Ha88] B. Hajek, Cooling Schedules for optimal Annealing, Math. of Operations Research 3, 3-39 (988) [H75] J. Holland, Adaption in Natural and Articial Systems, Ann Arbor, University of Michigan Press (975) [HS88] R. Holley, D. Stroock, Simulated Annealing via Sobolev inequalities, Comm. Math. Phys. 5, (988) [HJ85] R. Horn, C. Johnson, Matrix analysis, Camb. Univ. Press (985) [K83] S. Kirkpatrick, C. Gelatt, M.P. Vecchi, Optimization by Simulated Annealing, Science 0, (983) 6

17 [M85] [V93] D. Mitra, F. Romeo, A. Sangiovanni-Vincentelli, Convergence and nite time behavior of simulated annealing, Proc. 4th Conf. Dec. a. Control, (985) R.V.V. Vidal (Ed.), Applied Simulated Annealing, Lecture Notes in Economics and Math. Syst. 369, Springer, Berlin (993) 7

OPTIMIZATION BY SIMULATED ANNEALING: A NECESSARY AND SUFFICIENT CONDITION FOR CONVERGENCE. Bruce Hajek* University of Illinois at Champaign-Urbana

OPTIMIZATION BY SIMULATED ANNEALING: A NECESSARY AND SUFFICIENT CONDITION FOR CONVERGENCE Bruce Hajek* University of Illinois at Champaign-Urbana A Monte Carlo optimization technique called "simulated