Asymptotic Properties of a Generalized Cross Entropy Optimization Algorithm

Transcription

1 1 Asymptotic Properties of a Generaized Cross Entropy Optimization Agorithm Zijun Wu, Michae Koonko, Institute for Appied Stochastics and Operations Research, Caustha Technica University Abstract The discrete cross entropy optimization agorithm iterativey sampes soutions according to a probabiity density on the soution space The density is adapted to the good soutions observed in the present sampe before producing the next sampe The adaptation is controed by a so-caed smoothing parameter We generaize this mode by introducing a fexibe concept of feasibiity and desirabiity into the samping process In this way, our mode covers severa other optimization procedures, in particuar the ant based agorithms The focus of this paper is on some theoretica properties of these agorithms We examine the first hitting time τ of an optima soution and give conditions on the smoothing parameter for τ to be finite with probabiity one For a simpe test case, we show that runtime can be poynomiay bounded in the probem size with a probabiity converging to 1 We then investigate the convergence of the underying density and of the samping process We show in particuar, that a constant smoothing parameter, as it is often used, makes the sampe process converge in finite time, freezing the optimization at a singe soution which need not be optima Moreover, we define a smoothing sequence that makes the density converge without freezing the sampe process and that sti guarantees the reachabiity of optima soutions in finite time This settes an open question from iterature Index Terms Evoutionary computation, cross entropy optimization, ant coony optimization, heuristic optimization, discrete optimization, mode based optimization I INTRODUCTION CROSS entropy CE) optimization is a widey used too for heuristic optimization in particuar for discrete probems, see 1 for an overview As was noted before see eg 2), it aso has much in common with ant agorithms ACOs) In this paper we use a generaized version of the CE agorithm that incudes most of the features of ACOs We concentrate on theoretica properties of the underying processes, that hod under very genera conditions and therefore appy to a types of agorithms covered by our mode The paper is inspired by 3 where for a particuar CE mode convergence properties were proven In a CE mode, we are considering a cost function f on a finite set S of soutions A soution is a tupe of fixed ength Manuscript received 28 Nov 2013; revised 16 Apr 2014 and 21 Jun 2014 The work of Z Wu was supported by China Schoarship Counci, No The authors are with Institut für Angewandte Stochastik und Operations Research, TU Caustha, Erzstr1, D Caustha-Zeerfed, Germany emai: zwu10@tu-causthade, koonko@mathtu-causthade) Copyright c) 2012 IEEE Persona use of this materia is permitted However, permission to use this materia for any other purposes must be obtained from the IEEE by sending a request to pubs-permissions@ieeeorg We use a probabiity distribution or density on S as a mode for producing good soutions This density is iterativey adapted with the goa to give an increasing weight to optima soutions Roughy, the method proceeds in two steps: Samping: We take a sampe of fixed size N from S, based on the present distribution Π t, starting with a given density Π 0 for t = 0 eg the uniform distribution) Adaptation: We evauate the soutions in the sampe with the cost function f and determine the reative frequencies of components in the best soutions We define Π t+1 by adapting Π t to these frequencies Here, a so-caed smoothing parameter ϱ t contros the reative weight of the sampe during the adaptation process In this way, Π t is expected to give increasing weight to the better part of the soution space In 3, the impact of the smoothing parameter on the convergence of the process Π t ) t 0 is investigated The authors give conditions on ϱ t under which an optima soution is samped with probabiity one after finitey many iterations reachabiity) In particuar, they found that for a constant smoothing parameter ϱ t ϱ, the distribution Π t wi converge to a one point mass convergence of density), but that one has to make ϱ arbitrariy sma to increase the probabiity of reachabiity Soutions are restricted to 0-1-tupes in 3 We extend these resuts in severa directions We aow soutions to be strings of arbitrary eements and do not require the optima soution to be unique Most importanty, we introduce a notion of feasibiity and desirabiity to restrict the set of soutions and to bias the samping The sampes in the Samping step above are actuay drawn according to a mixture of the present distribution Π t and a given feasibiity measure This aso aows to incude a greedy aspect into the construction of the soutions, as it is common in ACOs under the name of visibiity However, this additiona feature makes the mathematica mode much more compex compared to that in 3 Sti, we are abe to show that the resuts from 3 hod for this generaized mode We investigate τ, the number of iterations unti an optima soution is samped for the first time We show that, if the smoothing parameter ϱ t is a constant, then τ has a stricty positive probabiity to be infinite, and therefore the expected runtime is To get more insight into the finite time behavior of the agorithm, we consider a standard test probem LeadingOne, see 4) We show that for this probem, even if ϱ t is constant, the runtime, ie the number Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

2 2 of soutions samped unti an optima one is found, is bounded by a poynomia in the probem size with a probabiity that is converging fast against one In genetic agorithms, the phenomenon of genetic drift is we-known, see eg 5, it describes the oss of a variation in the soutions A simiar thing may aso happen in our mode We show that, for smoothing parameters ϱ t bounded away from 0, the distribution Π t as we as the samped soutions converge Thus, for fixed ϱ t ϱ > 0, the sampe process is absorbed into a fixed point after finitey many iterations, and this fix point need not be an optima soution This may be the theoretica proof for the stagnation in samping that has been reported in 6, where CE was appied to a Maxima Cut probem In CE, it is of great importance whether amost sure reachabiity of optima soutions and convergence of the underying density are compatibe This important question remained open in 3 We sove this question by giving smoothing sequences that show both, reachabiity of the optima soution and convergence of the density with probabiity one In this sense, these sequences baance what is sometimes caed expoitation and exporation However, at present we are not abe to show that the imiting density is concentrated on optima soutions Our resut aso shows the difference between the two types of convergence: convergence of the density does not necessariy mean that the sampe process is absorbed after finitey many iterations Appied to ACOs, our resuts compement the convergence resuts of 7 and 8 to the case where the update of pheromones ony uses current soutions They aso show that, with this update, ACOs wi end in suboptima soutions with a positive probabiity if a constant evaporation rate is used See Section IV for detais on ACOs The paper starts with an exact definition of the encoding of the soutions, the CE agorithm and the stochastic mode describing its evoution in Section II In Section III we state the main resuts and discuss their impications In Section IV, we discuss some extensions of our mode and how it appies to the ACO agorithms The very compex and technica proofs together with some auxiiary resuts are coected in Section V Section VI contains a concusion and an outook on further research A Probem Encoding II THE MATHEMATICAL MODEL We are considering a probem of discrete optimization with a finite set of feasibe soutions S and a cost function f : S R Let S := {s S fs) = min s S fs )} be the set of optima soutions To excude trivia cases, we assume that S > 1 and that S S We assume further that S has a particuar structure: each feasibe soution s S can be written as a finite string s = s 1,, s L ) of symbos from a finite aphabet A := {a 1,, a K } L is the fixed ength of the strings Often, CE modes eg in 3) use an unconstrained soution space S = A L We generaize this mode introducing a feasibiity function C i y, a) that assigns a weight to each a A for every possibe partia soution y of ength i Thus, if C i y, a) = 0, the partia soution y cannot be continued by adding the symbo a The arger the vaue of C i y, a), the more desirabe it seems, from a greedy point of view, to continue with symbo a We assume that these vaues are normed, an exampe is given after the forma definition beow Our CE method, described in Subsection II-B beow, constructs soution strings stepwise by adding new feasibe symbos to the right end of a partia soution unti it is compete More formay, we define the set R i of feasibe partia soutions of ength i recursivey as foows: Let denote the empty string over A We assume that we are given C 0, ) : A 0, 1, C 0, a) = 1, a A expressing the feasibiity or desirabiity of a symbo at the first position of a soution We then define R 1 := {a A C 0, a) > 0} as the set of feasibe partia soutions of ength 1 Assume now that we have defined R i for some i {0,, L 1} and that we are given C i y, ) : A 0, 1, C i y, a) = 1 for each y R i a A Let y, a) denote the concatenation of symbo a A to the right end of the partia soution string of symbos) y R i Then we define R i+1 := { y, a) y R i, a A, C i y, a) > 0 } and put S := R L For y R i, i {0,, L 1}, et C i y) := {a A C i y, a) > 0} be the support of C i y, ) We use the abbreviation I := {0,, L 1} in the seque As an exampe, we ook at a Traveing Saesman Probem TSP), where the aim is to find a tour of minima ength through K cities {a 1,, a K } Here, C 0, a) = 0 coud be used to prevent tours from starting in city a If y R i is a partia tour of ength i, then C i y) contains a cities that coud be visited next C i y, a) woud be zero if a has been visited before or cannot be reached from y 0 < C i y, a) coud be chosen as the normed distance from the end of) y to a Here, the distance has to be normed by the sum of distances from y to a a C i y) This feasibiity concept aows to incude the case where there is ony a set C i y) A of feasibe continuations of the partia soution y R i and no desirabiity information In this case, C i y, ) is chosen to be the uniform distribution on C i y) In particuar, if there are no constraints at a, we have C i ) A for a i {0,, L 1} and put C i y, a) 1 A, for a y R i, a A II1) We sha refer to II1) as the unrestricted case Athough the construction of S = R L seems to be very particuar, it imposes no restrictions on the optimization probems Formay, we can incude any finite set S of feasibe Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

3 3 soutions into our mode by choosing A := S, L := 1 and C 0, s) > 0 for a s S Thus we can force any probem into our framework, but the efficiency of the method which is not discussed in this paper) may not be high It is more reasonabe to use our mode in cases where A S B The Generaized CE Agorithm The cross entropy agorithm essentiay evoves a distribution on the set S = R L of a feasibe soutions with the aim to give a high probabiity to the optima soutions from S Let PA) denote the set of a probabiity measures on the set A Then p PA) L, p = p1),, pl)) is a product probabiity measure on A L, that describes the seection of a soution s = s 1,, s L ) A L, where the L symbos s 1,, s L are chosen independenty of each other Here, pi) = pa; i) ) PA) is the distribution for the symbo a A on the i-th position of the string Our CE agorithm takes the foowing items as input: the feasibiity distributions C i, ), i I; a sequence of smoothing parameters ϱ t ) t 1 with ϱ t 0, 1) ; a sampe size N N and a subsampe size N; a starting distribution p 0 PA) L Starting: For time-step t = 0, put p := p 0, then run iterativey through the foowing steps for t = 1, 2, unti some stopping criterion is fufied Samping: If the present distribution is p PA) L, a soution s = s 1,, s L ) S is drawn according to the probabiity L Q p s) := Q p s 1 ; 1, ) Q p s i ; i, s 1,, s i 1 )), i=2 II2) where pa, i)c i 1 y, a) Q p a; i, y) := a A pa, i)c i 1 y, a II3) ) is the probabiity that the feasibe symbo a A is added at position i to the feasibe partia soution y R i 1 We use the convention 0 0 = 0 throughout this paper In this way, the CE agorithm draws N soutions s 1),, s N) independenty and identicay distributed iid) Evauation: This sampe x := s 1),, s N) ) is ordered according to its cost vaues fs n1) ) fs n2) ) fs n N ) ) and the better soutions are seected: := {s n1), s n2),, s n N ) b } Then we define the reative frequency of symbo a at position i {1,, L} in the seected part of the sampe: wa; i, x) := 1 1 {a} s i ) II4) s and coect these frequencies for a a A to form wi, x) := wa; i, x) ) a A and wx) := w1, x),, wl, x) ) II5) Then wx) is a product probabiity measure from PA) L, that gives the reative frequencies of symbos in the better part of the sampe x drawn with Q p Update: We update the present distribution p as a convex combination of p and the reative frequencies wx): p := 1 ϱ t+1 )p + ϱ t+1 wx) II6) Next, the counter t is increased by 1 and the step Samping is performed with the new p Some extensions to this mode and a comparison to other approaches can be found in Section IV beow C The Soution Process Appying the above agorithm iterativey resuts in a sequence of probabiity measures and sampes that form a stochastic process Π t ; X t ) t=0,1, where Π t = Π t 1),, Π t L) ) is a random variabe taking on vaues in PA) L, that describes the distribution underying the samping in the t-th iteration Π t is caed the density of the agorithm with Π 0 = p 0 X t takes on vaues in S N and is the sampe of N soutions produced in the t-th Samping step of the agorithm, using Q Πt as defined in II2) In the seque, we put X t = X 1) t,, X N) ) t and X n) t = X n) t 1),, X n) t L) ) Then X n) t i) denotes the symbo at position i, 1 i L, drawn in the n-th soution in iteration t Simiary, X n) t 1,, i) denotes the partia soution up to the i-th position in that soution The process Π t, X t ) t 0 is we defined on a joint probabiity space Ω, P ) It can be shown, using II2) and II6), that ) Π t, X t as we as the margina process Π t 0 t )t 0 are Markov processes Due to the deterministic nature of the update mechanism, we may aso write Π t+1 = 1 ϱ t+1 )Π t + ϱ t+1 wx t ) on a vector eve and on the more detaied eve Π t+1 a ; i) = 1 ϱ t+1 )Π t a ; i) + ϱ t+1 wa ; i, X t ) II7) for each a A, i = 1,, L We sha refer to II7) as the basic recursion, most of our resuts are based on it D Genera assumptions Throughout this paper we assume that the starting distribution Π 0 = p 0 is given such that any item from A has a positive probabiity at any position i : p 0 a; i) > 0 for a a A, i = 1,, L II8) Aso, without oss in generaity we may assume that there are at east two soutions s = s 1,, s L ), s = s 1,, s L) with s 1 s 1 II9) Otherwise, a soutions woud start with the same symbo, which coud then be dropped from the encoding of the soutions Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

4 4 III MAIN RESULTS In this Section we coect the main resuts of this paper and discuss their meaning The quite compicated proofs are given in Section V A Resuts on the Reachabiity of Optima Soutions Let X t := {X 1) t,, X N) t } be the set of soutions samped in the t-th iteration, then τ := min{t 0 X t S } III1) denotes the first iteration, in which an optima soution is samped We now investigate whether we can guarantee to reach an optima soution in finite time, ie whether Pτ < ) = 1 Theorem 1 shows, that the resuts of 3 hod in our more genera mode Theorem 1 a) If t 1 ϱ m) L =, then Pτ < ) = 1 t 1 ϱ m) = b) If Pτ < ) = 1, then c) If ϱ t ϱ > 0 is a constant, then we have Pτ < ) < 1, but Pτ < ) 1, if either N or ϱ 0 The proof of this Theorem aong the ines of 3 is given in Section V-B beow Theorem 1 b) shows in particuar that for a constant smoothing parameter, ie ϱ t ϱ, it cannot be guaranteed that the optima soution is reached in finite time, as we have t 1 ϱ) = 1/ϱ 1 < But then aso the expected runtime, ie the expected number of soutions samped before an optima soution is reached, wi be infinite for arbitrary optimization probems For a particuar probem however, namey the LeadingOne probem, see eg 4, we are abe to show that the runtime is bounded by a poynomia in the probem size with a probabiity converging to 1, even if ϱ t is constant In the LeadingOne probem we consider the unrestricted case with A = {0, 1}, S = {0, 1} L and fs) := L L i=1 s i for s = s 1,, s L ) III2) Minimizing fs) then means to maximize the number of consecutive 1s counted from the eft of the soution Theorem 2 extends the runtime resuts of 4, who consider a specia case of the CE agorithm with ϱ t 1 on a LeadingOne probem The Theorem essentiay states that if we take ϱ t ϱ 0, 1) and et the sampe size grow as N = L 2+ɛ for some ɛ > 0, then we can reach the optima soution in L iterations with a probabiity converging to 1, ie the runtime is OL 3+ɛ ) in a stochastic sense Theorem 2 Let ϱ t ϱ, sampe size N = L 2+ɛ for some ɛ > 0 and = βn for some 0 < β < 1 ) 3e 1 1 ϱ) m Let Π 0 1, i) 1 2, ie we start with the uniform distribution Then for a LeadingOne probem, defined in III2), we have Pτ < L) 1 as L For a proof of Theorem 2, see Subsection V-C beow Theorem 1 b) and the ower bound in Lemma 10 b) beow show that a constant smoothing parameter reduces the goba exporation of the search space Theorem 2 however shows that we can compensate for that by increasing the oca expoitation in terms of a growing sampe size, at east in specific probems B Resuts on Convergence of Density and Sampes An important question, aso addressed in 3, is whether the density Π t wi converge to a density concentrated on one point, and whether this point is an optima soution For the unrestricted case, 3 showed that the agorithm with ϱ t ϱ has a convergent density in the foowing sense: Definition 3 We say that the agorithm has convergent densities, if Π t converges amost surey against a product )t 0 of one-point measures, ie ) P i = 1,, L a A im Π t a ; i) {0, 1} = 1 A convergent density means that asymptoticay, we ony sampe one identica soution But this may aso happen after finitey many iterations: Definition 4 We say that the agorithm has convergent sampes, if X t converges against some fixed sampe )t 0 x = s,, s) S N of identica soutions, amost surey, ie if the foowing event has probabiity one: s S T N m 1 n = 1,, N X n) T +m = s In this case, the samping freezes to a singe soution after the finite random time T and no more progress is possibe after that time Theorem 5 shows that this does happen, if a constant smoothing parameter is used Theorem 5 a) If ϱ t ϱ for some ϱ > 0, then the agorithm has convergent sampes b) For S = 1, convergent sampes impy Pτ < ) < 1, hence convergence of sampes and Pτ < ) = 1 are mutuay excusive in this case c) Convergent sampes impy a convergent density d) If the density converges, then ϱ t = The quite compex proof of Theorem 5 is given in Section V-D beow Theorem 5 a) shows that the phenomenon of genetic drift may aso occur in this generaized CE provided a constant smoothing parameter is empoyed From Theorem 5 b), we see that in this case an optima soution may not be found if it is unique C Reachabiity and Convergent Density are Compatibe In 3, the question was raised whether convergence of densities and reachabiity of optima soutions are mutuay excusive, and this remained an open question In Theorem 5, a partia answer was given with respect to the stronger concept of convergent sampes At east for the unrestricted case, we can give a compete answer: there are smoothing sequences such that the agorithm has Pτ < ) = 1, it has convergent densities but no convergent sampes This aso shows that the opposite direction in Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

5 5 Theorem 5 c) does not hod in genera, densities may converge without freezing the sampes in finite time Theorem 6 In the unrestricted case the foowing hods: Assume that ϱ 0, 1) and c k N for k = 0, 1,, with c 0 = 1, are chosen such that c k 1 ϱ) kl = k=1 Define e k := k i=1 c i 1, k 1, and et x k 0, 1) be any sequence such that k=1 x k < We may now define a smoothing sequence ϱ if t = e k for some k 1 ϱ t := 1 1 x k ) 1 c k 1 if e k < t < e k+1 for some k 1 III3) for t 1 Then the agorithm has convergent densities, it has Pτ < ) = 1, and its sampes do not converge The proof of Theorem 6 is again given in Section V-E As an exampe for the vaues in Theorem 6, take an arbitrary ϱ > 0, then one may choose c k = 1 ϱ) kl to obtain c k 1 ϱ) kl = 1 = k=1 k=1 For ϱ < 1/2, one coud, for exampe, use c k := 2 kl Theorem 6 shows that a goba exporation of the search space that impies reachabiity) and the abiity for thorough oca search, that incudes some kind of convergence, can be baanced in this type of agorithm It aso shows that the two types of convergence are different, convergence in the continuous space of densities does not impy convergence in the finite space of sampes We must admit, however, that we are currenty not abe to give a smoothing sequence under which the imiting density is concentrated on some optima soution IV EXTENSIONS AND COMPARISON TO ACOS The evauation step in II4) simpy cacuates the reative frequencies in the better part of the sampe A thorough inspection of the proofs beow shows that the resuts of the above Theorems except Theorem 2) sti hod, if we repace the reative frequencies wa; i, x) by some other measure 0 wa; i, x) 1 not depending on t with the foowing properties: i) a A wa; i, x) = 1, ie w ; i, x) is a probabiity measure on A; ii) wa; i, x) = 1 if a soutions in the sampe x have symbo a at their i th position; iii) wa; i, x) = 0 if none of the soutions in the sampe x has symbo a at its i th position For exampe, w coud be the reative frequencies of a randomy seected subset N of x, or the weighted reative frequencies where the weights refect the cost fs) of the soution as in s N wa; i, x) := 1 {a}s i )gfs)) s N gfs)), IV1) here g is some decreasing function w may even incude some kind of memory, as ong as properties i)-iii) hod In the proofs of Section V beow, we wi stick to the origina frequency mode as used in 3 With these extensions, our mode covers many heuristic optimization agorithms of the mode-based type Therefore the resuts given above wi appy to a these agorithms In particuar, they appy to ACO agorithms as described in 9 and 10 Here soutions are maxima oop-free paths in a so-caed construction graph The arcs k, ) of the path are seected according to pheromones τ k and a visibiity η k on the arcs Taking arcs as symbos a := k, ), pheromones can be expressed by the density in our mode, and the visibiity together with the oop-freeness can go into our feasibiity measure C i y, a) The evaporation rate for the update of pheromones is identica to the smoothing parameter, hence soutions in our CE mode are samped with the same probabiity as they are constructed by ants, if we choose IV1) as evauation of the soutions observed Note that our mode requires density and feasibiity to be normed probabiity measures, but this does not affect the samping probabiities as given, for exampe, by 9 Convergence resuts of ACOs can be found in 10, 7 and 8 They concentrate on ACOs with an update of pheromones that ony uses the best soution found so far and restricted pheromone vaues in 7, whereas our mode has no restrictions on the density and uses the present sampe for update With these differences, our resuts on reachabiity compement those found in 10 and 8 In particuar, the assumptions made in 7 fufi the sufficient condition of reachabiity in Theorem 1 a) The resuts on absorption for constant evaporation rate and the possibe baancing in Theorems 5 and 6 aso carry over to ACOs Our runtime resut in Theorem 2 compements findings in 11, 12 and 13, who inspect the runtime of ACOs with restricted pheromone vaues max-min ant system) on the so-caed OneMax probem Using best-so-far update, as in the ACO modes cited above, introduces a monotonicity that aows to show convergence to an optima soution once it has been samped, see 7 Currenty, we are not abe to show such a resut for our agorithm A Some Auxiiary Resuts V PROOFS OF THE THEOREMS We start with the basic recursion in II7), which is crucia for our work For a fixed pair a; i), this recursion fufis the condition V1) of the foowing Lemma 7, and therefore the concusions V2) - V4) hod This wi be used throughout this paper We assume as usua k i=m Lemma 7 Let 0 < r t < 1 for t = 1, 2, a) r t = 1 r t ) = 0 1 for m > k 1 cr t ) = 0 for any 0 < c < 1 Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

6 6 b) t r m i=m+1 1 r i ) = 1 1 r m ) for any t 1 c) For a given sequence w t 0, 1, t = 0, 1,, and q 0 0, 1), the recursion q t+1 = 1 r t+1 )q t + r t+1 w t, t 0, V1) has the unique soution q t = q 0 1 r m ) + with t r m w m 1 i=m+1 1 r i ) V2) 0 < q 0 1 r m ) q t V3) 1 1 q 0 ) 1 r m ) < 1, t 0 If, in particuar, w m w 0, 1 for m = 0,, t 1, then q t = w w q 0 ) 1 r m ) V4) Proof: see 3) a) The first part is a standard resut, the second foows as i=0 r i = i=0 cr i = b) foows, if r m = 1 1 r m ) is used on the eft hand side c) V2) can be proven using induction on t V4) foows from V2) using b) Now, V3) appied to II7) and II8) shows that 0 < Π t a; i) < 1 V5) for a t 0, i {1,, L} and a A We then aso have 0 < C i y, a)π t a; i+1) < 1 V6) for a t 0, i I, y R i and a C i y) In the unrestricted case as in 3, we have Q Πt a; i+1, y) = Π t a; i+1), ie the probabiity to continue a feasibe partia soution y R i with a symbo a in the t-th iteration coincides with the density Πa; i+1) We may therefore directy use the basic recursion II7) to derive the asymptotic behavior of the procedure from Lemma 7 In the genera constrained case this is not possibe, as Q Πt does not fufi an equation as II7) But we can find a bounding probabiity Q Π t for Q Πt, for which a simiar recursion hods under certain conditions see Lemma 11 beow) For p PA) L, i I, y R i and a A define { pa;i+1) Q pa; i+1, y) := a C i y) pa ;i+1) if a C i y), 0 otherwise, V7) with 0 0 = 0 This is simiar to Q p except that the feasibiity distribution C i y, a) has been repaced by a constant on its support C i y) Note, that for the unrestricted case of II1), Q p, Q p and p a coincide To bound Q p a; i+1, y) with the hep of Q pa; i+1, y), we first need bounds on C i y, a) Let η := max { C i y, a) y R i, a A, i I } V8) and λ := min { C i y, a) > 0 y R i, a A, i I } V9) Obviousy, 0 < λ η 1 Now define the foowing two bounding functions for x 0, 1 hx) := ηx λ + η λ)x, x) := λx η η λ)x V10) Then x) x hx) and in the unrestricted case hx) = x) = x Lemma 8 coects some properties of h and Lemma 8 Let h, be as defined in V10) a) h and are stricty increasing and continuous taking vaues in 0, 1 h is concave, is convex with x) x hx), x 0, 1 b) For η = λ we have hx) = x) = x for x 0, 1, for η > λ we have hx) = x = x) x { 0, 1 } c) hx) = 1 1 x) and x) = 1 h1 x) d) Let 0 x n 1, n N, be a convergent sequence, then for any 0 < c < 1, the foowing series are either a convergent or a divergent x n, hx n ), x n ), n=1 n=1 hcx n ), n=1 n=1 cx n ) n=1 Proof: Assertions a) - c) are straightforward to show Assertion d) foows immediatey from the we-known imit comparison test Lemma 9 beow coects some properties of Q Πt that are needed throughout the paper Reca that is the empty string that we use as a starting vaue for the recursions Note that some assertions ony hod if C i y) > 1 This excudes the case that there is just one possibe continuation of y, which then must have probabiity one Lemma 9 Let i I, y R i and a C i y) Then the foowing hods a) Q Π t a; i + 1, y) ) Q Πt a; i + 1, y) h Q Π t a; i + 1, y) ) for a t N b) If C i y) = 1 then Q Πt a; i + 1, y) = Q Π t a; i + 1, y) = 1 If C i y) > 1, then 0 < Q Π t a; i + 1, y) < 1 and 0 < Q Πt a; i + 1, y) < 1 for any t 0 Proof: a) In view of Lemma 8 b) we ony have to consider the case C i y) > 1 Π t a; i + 1)C i y, a) Q p a; i + 1, y) = a A Π ta ; i + 1)C i y, a ) ηπ t a; i + 1) ηπ t a; i + 1) + λ a C Π iy),a a ta ; i + 1) = h Q Π t a; i + 1, y) ), where we use V5) Simiary, the eft-hand inequaity of a) is derived b) From V5) we have Q Π t a; i+1, y) > 0 If C i y) > 1, we see that Q Π t a; i+1, y) < 1 for any a C i y) and t N, this impies the concusion using Lemma 8 b) and part a) Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

7 7 B Proof of Theorem 1 We first give ower bounds on the probabiities of a) reaching the set S of optima soutions and b) of staying forever with one possiby non-optima) soution Lemma 10 a) For s = s 1,, s L ) S and p 0 PA) L et L ) cs) := C 0, s 1 ) C i 1 s1,, s i 1 ), s i and i=2 δs) := p 0 s) cs) Then for τ as defined in III1) Pτ < ) 1 min 1 δs ) s S t=0 V11) 1 ϱ m ) L) N b) Let s S, then, with h as defined in V10), PX n) t = s for n = 1,, N and t = 0, 1, ) Q Π0 s) N 1 h t 1 ϱ m ) )) LN Note that the bound in part a) hods for the first hitting time of any subset of soutions, so it is in fact a bound for the probabiity that arbitrary soutions wi be visited in finite time Proof of Lemma 10: a) cf 3) We fix an arbitrary optima soution s = s 1,, sl) S, then we have ) Pτ = ) = P S X t = P t=0 t=0 ) s / X t = Ps / X 0 ) P s / X t s / X m, m = 0,, t 1 V12) We now derive an upper bound for the factors in V12) First we have P s / X t s / X m, m = 0,, t 1 V13) = E P s / X t Π t s / X m, m = 0,, t 1 From the definition II3) of Q p and V3) of Lemma 7 appied with q 0 = Π 0 a; i) we have Q Πt a; i, y) Π t a; i)c i 1 y, a) Π 0 a; i)c i 1 y, a) 1 ϱ m ) for a a A, y R i 1, i {1,, L} and t 0 As the soutions are samped iid, we may now concude P s / X t Π t V14) = P s X 1) ) N t Π t = 1 P s = X 1) ) N t Π t 1 δs ) 1 ϱ m ) L) N For the first factor in V12), we have from V14) for t = 0 N Ps / X 0 ) 1 δs )) Combining these resuts, we obtain Pτ < ) = 1 Pτ = ) 1 1 δs ) 1 ϱ t ) L) N t=0 V15) As this hods for a s S, the assertion foows b) Let s S We write X ) t s for X n) t = s, n = 1,, N and S for the event X m ) s for a m = 0,, t 1 Then, as sampes are iid, P X ) t s, t = 0, 1, = P X ) 0 s V16) P X ) t s X ) m s for m = 0,, t 1 = P X 1) 0 = s N P X 1) t = s S N We have PX 1) 0 = s) = Q Π0 s) for the first factor Using Lemma 9 a), we obtain for the other factors in V16) with s i := s 1,, s i ) P X 1) t = s S = P X 1) t 1) = s 1 S L i=2 P X 1) t i) = s i X 1) t 1,, i 1) = s i 1, S = E Q Πt s 1 ; 1, ) S L E Q Πt s i ; i, s i 1 ) X 1) t 1,, i 1) = s i 1, S i=2 L E Q Π t s i ; i, s i 1 ) ) S, i=1 with s 0 = Under the condition S, the reative frequencies ws i ; i, X m ) are a equa to 1 for a m = 0,, t 1, hence we may use Lemma 11 d) beow see the remark foowing Lemma 11 and V36)) to obtain Q Π t s i ; i, s i 1 ) 1 1 ϱ m ) for t 1 and i = 1,, L Now, from V16), we obtain using Lemma 8 c) P X ) t s, t = 0, 1, V17) Q Π0 s) N 1 h t 1 ϱ m ) )) LN Proof of Theorem 1: a) From Lemma 7 a), we see that with r t := t 1 ϱ m) L and c := δs ), the assumption t 1 ϱ m) L = impies 1 δs ) 1 ϱ m ) L) = 0 Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

8 8 Now Pτ < ) = 1 foows from Lemma 10 b) Let s S S with Q Π0 s) > 0 Note that such an s must exist, otherwise we coud ony sampe optima soutions If Pτ < ) = 1, the sampes cannot be identica to s for a times, hence from Lemma 10 b) we must have 0 = PX ) t s, t = 0, 1, ) Q Π0 s) N 1 h t 1 ϱ m ) )) LN, and 1 h t 1 ϱ m) )) = 0 must hod From Lemma 7 a) we see that this impies h ) 1 ϱ m ) = The assertion now foows from Lemma 8 d) c) Assume ϱ t ϱ > 0, we have t 1 ϱ) <, hence by part b) of this Theorem, Pτ < ) < 1 Note that the infinite product in V15) in this case is smaer than 1 and continuous in ϱ, so V15) approaches 1 for either N or ϱ 0 C Proof of Theorem 2 Proof: With A = {0, 1}, we write π t i) := Π t 1; i) As we consider the unrestricted case here, the sampe X t = X 1) t,, X N) t ) can be viewed as a random matrix with mutuay independent entries and soution X n) t as n-th row We denote by X t = X 1 t,, X N ) t this matrix with rows ordered according to increasing cost function vaues : fx 1 t ) fx N t ) Then is the sub-matrix of the first rows of X and the empirica distributions from may be written as w1; i, X t ) = 1 Nb n=1 Xn t i) =: wi, X t ) and wx t ) := w1, X t ),, wl, X t )) For the proof, we et wx t ), t = 0, 1,, move over a sequence of increasing eves wt These eves are defined in such a way that fx t ) is stricty decreasing and at the same time the update of π t can be controed such that we are abe to give a ower bound on the probabiity for this to happen For t = 0,, L 1 et w t := w t 1),, w t L) ) := 1,, 1, α t,, α t ) V18) with t + 1 entries 1 and some α t 0, 1) defined in V21) We write wx t ) wt if and ony if wi, X t ) = 1 for i = 1,, t+1 and wi, X t ) α t for i = t+2,, L For t = L 1, wx t ) wt means that consists of the optima soution s = 1,, 1) ony Hence we have Pτ < L) PwX L 1 ) wl 1) P wx 0 ) w0,, wx L 1 ) wl 1 ) = PwX 0 ) w0) L 1 V19) P wx t ) w t wx m ) w m, m = 0,, t 1 We show beow that there are constants a, b, c > 0 such that for a L arge enough and for a N P wx t ) w t wx m ) w m, m = 0,, t 1 1 e an ) 1 e b N L 2 +c) L t 1 V20) With N = L 2+ɛ for some ɛ > 0 we may then concude that Pτ < L) 1 e al2+ɛ ) L 1 e blɛ +c ) L 2 L)/2 Now this ast bound can be shown to converge to 1 for L using standard methods from anaysis Hence the proof of Theorem 2 is compete, once we have shown V20) for t = 0,, L 1 In the first step we show how the eves w t infuence π t We abbreviate the event wx m ) w m, m = 0,, t 1 by W t, W 0 indicating the empty condition Let α t := L )t+1, t = 0,, L 1, and α 1 := 1 2 V21) Conditioned on W t, { 1 1 ϱ) π t i) t i+1 for 1 i t for t < i L α t 1 V22) for t = 0,, L 1 The proof is by induction on t using the basic recursion II7) and the property wi, X m ) = 1 for a m = i 1, i,, t 1, if i t, wi, X m ) α m for a m = 0, 1,, t 1, if i > t, which foows from the definition of wt under W t Let v t+1 = P X n) t 1) = = X n) t t+1) = 1 W t be the probabiity to sampe a soution that has t+1 eading 1s, then from V22) we obtain t+1 v t+1 = π t i) ϱ) m ) =: κϱ) V23) 3e i=1 for L arge enough, as then α t 1/3e) Now we want to determine simpe conditions on X t that impy wx t ) wt To do so, we ook at the matrix X t coumnwise observing the independence of its entries Let M t+1) be the number of rows with at east t+1 eading 1s, then wi, X t ) = 1, i = 1,, t+1, requires that M t+1) These M t+1) rows are aso the first rows in X t Next, wi, X t ) α t requires for the number of 1s in coumn i = t + 2 Y i) := M i 1) n=1 X n t i) α t V24) Here we have to restrict the number of rows to the present candidate rows 1,, M i 1) from which the set is seected After ooking at coumn i in this way, we define M i) := max{, Y i) } V25) and repeat V24), V25) for i = t + 3,, L We then obtain P wx t ) w t W t V26) P M t+1), Y i) α t, i = t + 2,, L W t = P M t+1) W t L i=t+2 P Y i) α t Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

9 9 Y ) α t, = t+2,, i 1, M t+1), W t To derive the desired ower bounds for these expressions we need the Chernoff bound see eg 14, Theorem 42) in the foowing form: et Z 1,, Z m be iid 0-1-distributed with success probabiity p, then for any 0 < r < mp we have P m i=1 Z i r ) e r mp )2 mp V27) Conditioned on W t, M t+1) is distributed as the number of successes in a row of N iid experiments, each with success probabiity p := v t+1 We obtain for β < κϱ) v t+1 that = βn βn < v t+1 N Hence for t = 0,, L 1 P M t+1) W t = 1 P M t+1) < W t 1 E e v t+1 N )2 v t+1n Wt 1 e β κϱ) )2κϱ)N, V28) where we used V27) Hence, in V20) we may define a := β κϱ) )2 κϱ) Simiary, Y t+2) is distributed as the number of 1s in M t+1) iid trias each with success probabiity π t t + 2) From V22) and the definition of α t we see that under the condition used in V26) we have π t t + 2)M t+1) α t 1 M t+1) > α t Using the Chernoff bound we therefore obtain, for L arge enough, P Y t+2) α t W t, M t+1) V29) = 1 P Y t+2) < α t W t, M t+1) 1 e α t α ) 2 t 1 3e = 1 e 6eL 2 1 e βn 1 6eL 2, where we used αt α t 1 = 1 1 L A competey anaogous derivation hods for the other factors in V26) with i = t+3,, L Hence, with b := β 6e, c := 1 6e, we see from V29), V28) and V26) that V20) hods for L arge enough D Proof of Theorem 5 The proof of Theorem 5 about convergent sampes uses an inductive argument The induction hypothesis assumes, that the first i positions of a sampes have aready converged against a fixed vaue y R i and we show that then aso the samped vaue on the i + 1st position wi become constant in finite time More precisey, we assume as induction hypothesis that for a fixed i I the foowing hods with probabiity one im Xn) t 1,, i) = y for a n = 1,, N and some y R i As R i, the set of possibe vaues of X m n) 1,, i) is finite, this condition is equivaent to requiring that there are random variabes y and T taking on vaues in R i and N, defined on the common probabiity space, such that the foowing event has probabiity one: m 0 n = 1,, N X n) T +m 1,, i) = y V30) Lemmas 11 and 12 beow contain the crucia induction step to prove convergence of sampes Lemma 11 shows that under condition V30) we may appy the recursion of Lemma 7 c) to Q Π t We use the foowing definitions for p PA) L, i I and y R i G i y, p) := pa ; i + 1) and V31) ϱ y t := a C iy) ϱ t G i y, Π t ) for t 1 From V5) we see that aways G i y, Π t ) > 0 V32) Lemma 11 Let i I be fixed and assume that V30) hods with probabiity one for random variabes y, T as above Then the foowing hods amost surey for a m 1 a) G i y, Π T +m ) = 1 1 G i y, Π T ) ) m 1 ϱ T + ), and G i y, Π T +m ) is an increasing function of m b) 0 < ϱ T +m ϱ y T +m < 1 c) For a a C i y) we have Q Π T +m a; i + 1, y) = 1 ϱ y T +m )Q Π T +m 1 a; i + 1, y) + ϱ y T +m wa; i + 1, X T +m 1) V33) Thus the impications of Lemma 7 c) hod with q t := Q Π T +t a; i + 1, y), r t := ϱ y T +t and w t := wa; i + 1, X T +t ) with the restriction that the strict right hand bound in V3) ony hods if C i y) > 1 d) If wa; i+1, X T + ) = w 0, 1 for a = 0,, m 1, then Q Π T +m a; i + 1, y) = w w Q Π T a; i+1, y) ) m 1 ϱ y ) V34) w w Q Π T a; i+1, y) ) m 1 ϱ ) V35) In fact, the foowing proof shows that, for the assertions of Lemma 11 to hod for a fixed m 1, instead of V30) the foowing weaker condition is sufficient = 0,, m 1 n = 1,, N X n) T + 1,, i) = y V36) Proof of Lemma 11: a) From the basic recursion II7), we have for any m 1 G i y, Π T +m ) = Π T +m a ; i + 1) V37) a C iy) = 1 ϱ T +m )G i y, Π T +m 1 ) + ϱ T +m, as a C iy) wa ; i + 1, X T +m 1 ) = 1 Hence, q t := G i y, Π T +t ) fufis the condition V1) of Lemma 7 with w m 1 Now V4) shows that G i y, Π T +m ) = 1 1 G i y, Π T ) ) m 1 ϱ T + ) Aso, from V37) we have G i y, Π T +m+1 ) G i y, Π T +m ) b) We have 0 < ϱ T +m ϱ y T +m and G iy, Π T +m 1 ) > 0 by V5) From V37) we now see ϱ T +m < G i y, Π T +m ), hence ϱ y t < 1 c) From V37) we obtain 1 ϱ T +m )G i y, Π T +m 1 ) = G i y, Π T +m ) ϱ T +m Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

10 10 This together with the basic recursion for Π t fufis the recursion Q Π T +m shows that ConvW) There is a random variabe) a 0 as above such that amost surey Q Π T +m a; i + 1, y) = Π T +ma; i + 1) G i y, Π T +m ) = 1 ϱ T +m)π T +m 1 a; i + 1) G i y, Π T +m ) + ϱ T +m wa; i + 1, X T +m 1 ) G i y, Π T +m ) = 1 ϱ y T +m )Q Π T +m 1 a; i + 1, y) + ϱ y T +m wa; i + 1, X T +m 1) For the appication of Lemma 7 c), we ony have to check that 0 < r t = ϱ y t < 1, which was shown in part b), and 0 < q 0 = Q ΠT a; i + 1, y) < 1 for C i y) > 1, which was shown in Lemma 9 b) d) From part c) and Lemma 7 V4) with q 0 = Q Π T a; i+ 1, y) we obtain Q Π T +m a; i+1, y) = w w Q Π T a; i+1, y) ) m 1 ϱ y ) such that the assertion foows from part b) Lemma 12 contains the induction step for the proof of Theorem 5 Lemma 12 Assume that for some ϱ > 0 we have ϱ t ϱ > 0 for a t 1, V38) and assume, as in Lemma 11, that there are i I and random variabes y, T such that V30) hods with probabiity one Then the foowing hods amost surey: there are T 0 and a 0 A with X n) T +m i + 1) = a 0 for a n = 1,, N and a m 0 V39) This Lemma says that, if a sampes have a common eading partia soution y of ength i after a finite number of iterations, then the sampes wi finay aso coincide on the next position i + 1, if ϱ t is bounded away from 0 Note that the condition is aways fufied for i := 0, y := Thus iterated appication of this Lemma aows to prove Theorem 5 beow Proof of Lemma 12: Part of this proof extends the approach used in 3 to prove convergence of densities and coses a sma gap in that proof As the proof is quite ong, we separate it into severa abeed steps Let i I be fixed and et y and T be such that V30) hods with probabiity one Note that if C i y) = 1, the concusion triviay hods, so we may assume C i y) > 1 To show the fina assertion of the Lemma we have to prove: ConvQ) There is a random variabe) a 0 taking on vaues in A such that amost surey im Q Π t a 0 ; i + 1, y) = im Q Π t a 0 ; i + 1, y) = 1 ConvQ) wi foow, after we have shown convergence of the reative frequencies: im wa; i + 1, X t) = { 1 if a = a0 0 if a a 0 for a a A ConvW) in turn foows, if we have shown that wa; i+1, X t ) finay becomes monotone: MonW) For a a C i y), it hods amost surey that Za; m) := wa; i + 1, X T +m ) wa; i + 1, X T +m 1 ) has ony finitey many sign changes as function of m We now prove these three steps in reverse order Reca that under the condition of the Lemma, amost a soutions from X T +m and X T +m 1 coincide in their first i positions To prove MonW) et M k a) be the k-th m such that Za; m + 1)Za; m) < 0 As A is finite, MonW) hods if for any fixed a C i y) and M k := M k a) Now, observe that P k N M k < ) = PM 1 < ) P k N M k < ) = 0 V40) P M k < M k 1 < k=2 1 P Mk = M k 1 < ) k=2 V41) We are going to show that P M k = M k 1 < κ > 0 for a constant ower bound κ > 0, this proves V40) We have P M k = M k 1 < = P M k = M k 1 = d P M k 1 = d M k 1 < d=k 1 If M k 1 = d then Za; d + 1) 0, hence P M k = M k 1 = d = P M k = M k 1 = d, Za; d + 1) > 0 P Za; d + 1) > 0 M k 1 = d + P M k = M k 1 = d, Za; d + 1) < 0 P Za; d + 1) < 0 M k 1 = d V42) We use the abbreviation W m := wa; i + 1, X T +m ), M for the event M k 1 = d, Za; d + 1) > 0, and W m for W = 1, = d + 2,, m 1 Conditioned on M, the event M k = is impied by W m = 1, m d+2 We can therefore give the foowing rough bound for the first term in V42) P M k = M k 1 = d, Za; d + 1) > 0 P W m = 1, m d + 2 M = P W d+2 = 1 M m=d+3 QΠT E a; i + 1, y) N +d+2 M P W m = 1 W m, M V43) Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

11 11 m=d+3 E QΠT +m a; i + 1, y) N Wm, M From Lemma 9 a) and V44) beow, we get for the first factor in V43) QΠT E a; i + 1, y) N +d+2 M E Q a; i + 1, y)) N ΠT +d+2 M ϱ ) N, where we used the fact that M impies Za; d + 1) > 0 Therefore wa; i + 1, X T +d+1 ) > wa; i + 1, X T +d ) 0 and wa; i + 1, X T +d+1 ) 1 Lemma 11 c) now shows Q Π T +d+2 a; i + 1, y) ϱy T +d+2 ϱ > 0 V44) For the second factor in V43), we see from Lemma 11 d) that, under the condition W = 1, = d + 2,, m 1, we have Q Π T +m a; i + 1, y) 1 1 ϱ) m d 2 Hence, in the second term of V43) the integrand can be bounded for a m d + 3 in the foowing way QΠT +m a; i + 1, y) N Q ΠT +m a; i + 1, y)) N 1 1 ϱ) m d 2) N = 1 h 1 ϱ) m d 2) N Coecting the equations above, we now obtain for a k, d N P M k = M k 1 = d, Za; d + 1) > 0 V45) ϱ ) N 1 h 1 ϱ) m d 2) N = ϱ ) N m=d+3 1 h 1 ϱ) m) N =: κ From Lemma 8 d), we see that 1 ϱ)m < impies h 1 ϱ) m) <, hence κ > 0 In a competey anaogous manner, using the events W m = 0, m d + 2, we can aso show that P M k = M k 1 = d, Za; d + 1) < 0 κ, which now proves V40) and MonW) We are now going to prove ConvW) From MonW) we know that m wa; i + 1, X T +m ) must eventuay be monotonic and therefore converge to one of the vaues in 1 W := {0,,, 1, 1} remember that w ) is a reative frequency from a sampe of size ) Hence, there is a random variabe T T, and for each a A a random variabe V a taking on vaues in W, such that with probabiity 1, wa; i + 1, X T +m) = V a for a m 0 >From the recursion V33) and Lemma 7 c) and b), we see that, under our conditions for a m 0 and a C i y), Q Π T +m a; i + 1, y) = Q Π T a; i + 1, y) V46) 1 ϱ y T + ) + V m a 1 1 ϱ y T + )) From V38) and ϱ y T + ϱ T + > ϱ, we have ϱy T + = and hence 1 ϱy T + ) = 0 Therefore, V46) impies that, amost surey, im Q Π t a; i + 1, y) = V a for a a C i y) V47) The bounds from Lemma 9 a) and the continuity of the bounding functions h and see Lemma 8 a)) ead us to concude that for a a C i y) V a ) im inf Q Π t a; i + 1, y) V48) im sup Q Πt a; i + 1, y) hv a ) We want to show next, that the imit V a can take on vaues in {0, 1} ony, more precisey: a 0 C i y) V a0 = 1 and a C i y) {a 0 } ) V a = 0 V49) hods amost surey To prove V49), we first show that for a a C i y) we have PV a 0, 1)) = 0 We again use the abbreviation W m = wa; i + 1, X T +m ) and W 0 := W 0, 1) = { 1, 2,, 1 1 } We then have PV a 0, 1)) = P im W k W 0 ) V50) = im P ) W k W 0 P W m W 0 W W 0, = k,, m 1 ) im m=k+1 m=k+1 P W m W 0 W W 0, = k,, m 1 Writing W for W W 0, = k,, m 1, we have for m > k PW m W 0 W W 0, = k,, m 1 = 1 E 1 Q ΠT +m a; i + 1, y) ) N W 1 1 h 1 ϱ) m k ) ) N, V51) where we used the fact that under W we have W 1 1, = k,, m 1 Hence we can see as in V46), that for m > k Q Π T +m a; i + 1, y) Q Π T +k a; i + 1, y) V52) 1 ϱ y T + ) ) 1 1 ϱ y T + )) =k+1 =k+1 ϱ =k+11 y T + ) ϱ) m k But then, for m arge enough, the upper bound of the probabiity V51) is smaer than 1, and the infinite product in V50) vanishes Therefore, we have shown P V a {0, 1} for a a C i y) ) = 1 We aso know that, P-amost surey, 1 = W m = wa; i + 1, X T +m ) a C iy) a C iy) V53) Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg

12 12 for a m 0 Hence, this must aso hod for the imits: P a C iy) V a = 1 ) = 1 This together with V53) proves V49) and hence ConvW) Now ConvQ) immediatey foows: from V47) we see that there is a random variabe a 0 such that P im Q Π t a 0 ; i + 1, y) = 1 ) = 1 V54) From h1) = 1) = 1 see Lemma 8 b)) and V48) we see that the same must hod for Q Πt a 0 ; i + 1, y) Thus we have proved ConvQ) and we are now going to show that this more precisey V54)) impies the assertion of the Lemma, ie X m n) i + 1) = a 0, n = 1,, N, for arge enough m, P-amost surey In other words, the next position i + 1 aso becomes identica in a sampes after finitey many iterations We write X ) t a 0, n = 1,, N that a sampes have the same i + 1-st i+1) a 0 for the event X n) t i+1) = component We then have ) P k N m k X m ) i+1) a 0 = im P X ) k i+1) a 0) m=k+1 V55) P X ) m i+1) a 0 X ) i+1) a 0, = k,, m 1 ) Now from V54), using bounded convergence, we see that for the first factor im P X ) k i+1) a 0) = im E Q Πk a 0 ; i+1, y) N) = E QΠk a 0 ; i+1, y) ) ) N = 1 V56) im For the second expression in V55), we obtain for m k T P X m ) i+1) a 0 ) X i+1) a 0, = k,, m 1 QΠm = E a 0 ; i+1, y) ) N X ) i+1) a 0, = k,, m 1 V57) We want to bound this ast expression from beow using Q Π m The condition X ) i+1) a 0, = k,, m 1, impies wa 0, i+1, X ) = 1 for = k,, m 1 Therefore, we have from Lemma 11 d) Q Π m a 0 ; i+1, y) 1 1 Q Π k a 0 ; i+1, y))1 ϱ) m k Using the bounds from Lemma 9 a) and Lemma 8 for m k T, we arrive at QΠm a 0 ; i+1, y) N Q Πm a 0 ; i+1, y) ) N 1 1 h Q a Πk 0; i+1, y) ) 1 ϱ) m k) N =: 1 H k,m k) N V58) where we introduced the abbreviation H k,m k) Note that H k,m k) is independent of the condition X ) i+1) a 0, = k,, m 1 Combining V57) and V58) we may bound the second term in V55) from beow by im E 1 H k,m k) ) N ) V59) m=k+1 im = ) N 1 EH k,m) 1 E im H k,m) ) ) N = 1, as im H k,m) = 0 amost surey by V54) To see that im and product may be interchanged, we use ogarithms and then bounded convergence with 0 n1 EH k,m ) n1 h1 ϱ) m ) for a k From V59), V56) and V55), we now see that there is a random variabe a 0 such that with probabiity one k N m k X ) m i+1) a 0, but this is the assertion of the Lemma We can now prove Theorem 5 about the convergence of the sampes Proof of Theorem 5: a) For i I, et Γ i denote the event y R i n = 1,, N and et γ i be the event a 0 A n = 1,, N im X n) t 1,, i) = y, im X t i) n) = a 0 In Lemma 12, we have shown that, if PΓ i ) = 1, then aso Pγ i+1 ) = 1 But, as Γ i γ i+1 Γ i+1, we see that PΓ i ) = 1 impies PΓ i+1 ) = 1 As a soutions have the empty substring as partia soution, we have PΓ 0 ) = 1, as was noted before But then we see from repeated appication of Lemma 12 that PΓ L ) = 1, ie convergence of the compete sampes hods with probabiity one b) We first show that convergence of sampes impies 1 ϱ ) < V60) Assume that convergence of sampes hods, then there must be at east one s = s 1,, s L ) S with 0 < P im X n) t = s for n = 1,, N ) P im X n) t 1) = s 1 for n = 1,, N ) = im P X ) k 1) s 1) m=k+1 P X m ) 1) s 1 X ) 1) s 1, = k,, m 1 im E h Q Π m s 1 ; 1, ) )) N m=k+1 ws 1 ; 1, X ) = 1, = k,, m 1 where we used the bound of Lemma 9 a) From Lemma 11 c) and V3) of Lemma 7, we see that E h Q Π m s 1 ; 1, ) )) N ws1 ; 1, X ) = 1, = k,, m 1 Copyright c) 2014 IEEE Persona use is permitted For any other purposes, permission must be obtained from the IEEE by emaiing pubs-permissions@ieeeorg