HITTING TIMES FOR RANDOM WALKS WITH RESTARTS

Transcription

1 HITTING TIMES FOR RANDOM WALKS WITH RESTARTS SVANTE JANSON AND YUVAL PERES Abstract. The time it takes a ranom walker in a lattice to reach the origin from another vertex x, has infinite mean. If the walker can restart the walk at x at will, then the minimum expecte hitting time γ(x, ) (minimize over restarting strategies) is finite; it was calle the grae of x by Dumitriu, Tetali an Winkler. They showe that, in a more general setting, the grae (a variant of the Gittins inex ) plays a crucial role in control problems involving several Markov chains. Here we establish several conjectures of Dumitriu et al on the asymptotics of the grae in Eucliean lattices. In particular, we show that in the planar square lattice, γ(x, ) is asymptotic to x log x as x. The proof hinges on the local variance of the potential kernel being almost constant on the level sets of h. We also show how the same metho yiels precise secon orer asymptotics for hitting times of a ranom walk (without restarts) in a lattice isk. 1. Introuction Consier a Markov chain (X n ) on a (countable) state space V, with transition probabilities ( p(x, y) ) x,y V. We use P x an E x to enote probability an expectation in the chain with initial state X = x. We assume that the chain is irreucible, i.e. that each state can be reache from any other state. Dumitriu, Tetali an Winkler [1] efine a function γ(x, z) for pairs of states x, z V. This function is a version of the Gittins inex an is calle the grae; it can be efine as follows [1, Theorem 6.1]: Consier a player that starts at x with the goal of reaching z as quickly as possible. Each time the player moves, the state changes ranomly accoring to the transition matrix p of the Markov chain; however, the player then has the option (if she fins the new state to be too ba) to restart by an instantaneous jump back to x. The grae γ(x, z) then is the minimum, over all strategies for restarting, of the expecte number of moves until z is reache. (Thus, a restart is not counte as a separate move, but the moves alreay performe are inclue in the total count. Note that a restart always Date: May 5, 1. Mathematics Subject Classification. Primary: 6J15; Seconary: 6J1, 6J45, 6J65. Key wors an phrases. ranom walk, hitting time, Gittins inex, harmonic function, potential kernel. 1

2 SVANTE JANSON AND YUVAL PERES moves back to the original starting state x.) For other equivalent efinitions, an applications of the grae to other games, see [1]. Remark 1.1. Once the grae is compute for all starting positions, then the optimal strategies for the game above (with initial state x) can all be escribe as follows: If the current state is y, then restart if γ(y, z) > γ(x, z), but not if γ(y, z) < γ(x, z); if γ(y, z) = γ(x, z), then it oes not matter whether we restart or not. Remark 1.. The setting in [1] is more general than the one presente here, since that paper allows for a cost of each move that may epen on the present state, while we consier here only the case of constant cost, so that the total cost is the total time. The purpose of this paper is to answer questions raise by Dumitriu et al [1], on the asymptotics of the grae in Z,. (The case = 1 is simple; as shown in [1], γ(x, ) = x ( x + 1) for Z.) By translation invariance it clearly suffices to consier z =. Denote the Eucliean norm of x by x. Theorem 1.3. For simple ranom walk on Z, γ(x, ) = x log x + (γ e + 3 log 1) x + O( x log x ), x, where γ e := lim n ( n j=1 1 j log n) is Euler s constant. Theorem 1.4. For simple ranom walk on Z, 3, γ(x, ) = ω p x + O( x 1 ), where ω = π / /Γ(/ + 1) is the volume of the unit ball in R, an p is the escape probability of the simple ranom walk in Z, i.e., the probability that the walk never returns to its starting point. The leaing terms were conjecture by Dumitriu, Tetali an Winkler in the preprint version of [1]. Base on the heuristic argument that the lattice structure shoul be unimportant on large scales, they suggeste that a nearoptimal strategy might be: Restart if the current state has a larger Eucliean istance to the target than the starting state. The expecte hitting time for this strategy can then be estimate using electrical network theory. Theorems 1.3 an 1.4 together with Remark 1.1 imply the following corollary, which shows that the optimal restarting strategy is inee as outline above, except possibly at some borer-line cases. Corollary 1.5. For simple ranom walk on Z, with target z =, there exists a constant C = C(), inepenent of the starting position x, such that every optimal strategy restarts from every position y with y > x + C, but never when y < x C.

3 RANDOM WALKS WITH RESTARTS 3 For intermeiate cases with x C y x + C, we cannot prescribe explicitly the optimal startegy; numerical calculations inicate that the simple heuristic strategy is not always optimal, i.e. we cannot take C = in the corollary. (Peter Winkler, personal communication). To prove the theorems above, we state an prove in Section a result, Theorem.1, yieling bouns on the grae for general Markov chains. This theorem is applie to Z in Sections 3 an 4. (We separate the recurrent case = from the transient case 3 since the etails are somewhat ifferent.) In Section 5 we present analogous results for a continuous version of the problem, with the ranom walk replace by Brownian motion in R. In this case we obtain exact results, analogous to the asymptotic results in Theorems 1.3 an 1.4. In the final section we show how our metho yiels precise secon orer asymptotics for hitting times of a ranom walk (without restarts) in a lattice isk.. A general estimate We state a theorem yieling upper an lower bouns on the grae. The theorem applies in principle to any Markov chain, but its usefulness epens on the existence of a suitable harmonic function for the Markov chain. Recall that a function h : V R is harmonic at x V if E x h(x 1 ) = h(x), i.e. if y p(x, y)h(y) = h(x). For a function h : V R an x V, efine the local variance, V h (x) := E x h(x 1 ) h(x) = y p(x, y) h(y) h(x). Theorem.1. Let z V an suppose that h : V [, ) is a non-negative function that is harmonic on V \ {z} an satisfies h(z) =. Suppose that g +, g are positive functions efine on [, sup h), such that for every x, y V with p(x, y) >, an every real number ξ between h(x) an h(y), the local variance satisfies Then, for every x V, h(x) s s γ(x, z) g + (s) where g (ξ) V h (x) g + (ξ). (.1) h (x) s s, (.) g (s) h (x) = sup{h(y) : p(w, y) > for some w V with h(w) h(x)}. Proof. Fix a starting position x V. To prove the lower boun in (.), efine a function F = F + : [, ) [, ) by F (s) := s h(x ) h(x ) t u t. (.3) g + (u)

4 4 SVANTE JANSON AND YUVAL PERES Thus F () =, an by Fubini s theorem, F (h(x )) = t u = g + (u) For all s, Moreover, an, a.e., <t<u<h(x ) F (s) = F (s) = h(x ) u u. (.4) g + (u) F (s) F (h(x )). (.5) { h(x ) s g + (u) u, s h(x ),, s h(x ), { g + (u), s h(x ),, s > h(x ). (.6) Let X n, n =, 1,..., be the process obtaine by starting at X = x, choosing successive states by running the Markov chain an restarting accoring to some non-anticipating strategy Λ. (Formally, Λ is a {, 1}-value function on finite sequences of states.) That is, suppose that a step of the Markov chain takes X n to X # n+1. If Λ( X 1,..., X n, X # n+1 ) =, then we let X n+1 = X # n+1, while if Λ( X 1,..., X n, X # n+1 ) = 1, then we let X n+1 = x. We claim that Y n := F ( h( X n ) ) + n is a submartingale for any choice of restarting strategy Λ. To see this, start by observing that F ( h( X n+1 ) ) F ( h(x # n+1 )), since F attains its maximum at h(x ). Hence, enoting X n = x, we fin that E(Y n+1 X 1,..., X n ) E ( F (h(x # n+1 )) + n + 1 X 1,..., X n ) = E x F (h(x 1 )) + n + 1. (.7) Denote Z = h(x 1 ) h(x). A Taylor expansion of F (with error in integral form), followe by an application of (.6) an (.1), yiels F ( h(x 1 ) ) = F ( h(x) + Z ) = F ( h(x) ) + ZF ( h(x) ) + 1 (1 t)f ( h(x) + tz ) Z t F ( h(x) ) + ZF ( h(x) ) 1 Z (1 t) ( ) t g + h(x) + tz F ( h(x) ) + ZF ( h(x) ) 1 Z (1 t) V h (x) t F ( h(x) ) + ZF ( h(x) ) Z V h (x).

5 RANDOM WALKS WITH RESTARTS 5 If x z, then h is harmonic at x, so E x Z = an E x Z = V h (x). Therefore, taking the expectation in the last isplaye inequality, we fin that E x F ( h(x 1 ) ) F ( h(x) ) 1. This also hols, trivially, when x = z. Thus by (.7), E(Y n+1 X 1,..., X n ) F ( h(x) ) + n = Y n, which proves that (Y n ) is a submartingale. We stop this submartingale at τ := inf{n : X n = z}. (.8) Note that Y τ = F ( h(z) ) + τ = τ. Moreover, by (.5), sup n τ Y n = sup Y n F ( h(x ) ) + τ. n τ Hence, if E τ <, the stoppe submartingale is uniformly integrable, an thus by the optional sampling theorem E τ = E Y τ E Y = F ( h(x ) ). This is trivially true if E τ = too. In other wors, for any restarting strategy, the expecte hitting time of z by ( X n ) is at least F ( h(x ) ), i.e. γ(x, z) F ( h(x ) ), an the left han sie of (.) follows by (.4), since x is arbitrary. Next, we prove the upper boun in (.). We enote the initial state by x, an use the simple restarting strategy: Restart to x from all points y = X # n with h(y) > h(x ). Denote the resulting process by ( X n ) an observe that Consier h( X n ) h(x ) an h( X n+1 ) h(x # n+1 ) h (x ) for all n. F (s) = F (s) := s h (x ) h (x ) By an argument similar to the one above, we fin that E ( F (h( X n+1 )) X 1,..., X n ) F ( h( X n ) ) 1. t u t. (.9) g (u) Denote Y n = F ( h( X n ) ) + n an let τ be efine by (.8). Then (Y n τ ) n is a positive supermartingale, whence by the optional sampling theorem, γ(x, z) E τ = E Y τ E Y = F ( h(x ) ) F ( h (x ) ). Remark.. The proof above suggests that a reasonable strategy is to restart from every state y with h(y) > h(x), as in the secon part of the proof. For Z,, with h as escribe in Sections 3 an 4, this is close (but not ientical) to the strategy base on Eucliean istance, an Corollary 1.5 shows that it is, in some sense, close to optimal.

6 6 SVANTE JANSON AND YUVAL PERES Remark.3. To obtain matching upper an lower bouns from Theorem.1, we want g g +. It is thus essential that we can fin a harmonic function h such that V h (x) is approximately a function of h(x), i.e. such that V h (x) is roughly constant in sets where h(x) is. Remark.4. The applications of Theorem.1 below follow a common pattern, here given as a heuristic guie to later precise calculations. Suppose that r(x) is a function on V such that h(x) ϕ(r(x)) an V h (x) ψ(r(x)) for some ϕ an ψ with ϕ increasing an ifferentiable. Suppose further that h(x) h(y) is sufficiently small when p(x, y) >. We then can take g ± (s) ψ(ϕ 1 (s)) an obtain γ(x, z) ϕ(r(x)) s ψ(ϕ 1 (s)) s = r(x) ϕ 1 () ϕ(t)ϕ (t) t. ψ(t) 3. Two imensions: Proof of Theorem 1.3 In this section the unerlying Markov chain (X n ) is simple ranom walk on Z. We choose h(x) = π a(x), where [ ] a(x) := P (X n = ) P (X n = x) n= is the potential kernel stuie in [8, 7, 4, ]. A complete asymptotic expansion of a(x) is presente in [, 3]; here we only quote the secon orer expansion given, e.g. in [8, ] an [4, Section 1.6]: h(x) = π a(x) = log x + b + O( x ), (3.1) where b = γ e + 3 log ; moreover, if e is a unit vector along one of the coorinate axis, then a(x + e) a(x) = e ( π log x ) + O( x ) an thus V h (x) = 1 (log x ) + O( x 3 ) = 1 x + O( x 3 ). If p(x, y) >, then x y = 1 an thus by (3.1) h(y) = h(x) + O( x 1 ) = log x + b + O( x 1 ). (3.) It is now easily seen that (.1) is satisfie with g ± (s) = 1 e (s b) (1 ± Ce s ) (3.3) if C is a sufficiently large constant. For small s this coul make g (s), but we reefine g (s) to be a small positive constant in these cases. We then have 1 g ± (s) = e(s b)( 1 + O(e s ) ) as s. (3.4) Furthermore, (3.) implies that h (x) = log x + b + O( x 1 ).

7 Hence Theorem.1 yiels, for x, γ(x, ) = RANDOM WALKS WITH RESTARTS 7 log x +b+o( x 1 ) 4se (s b)( 1 + O(e s ) ) s = [ se (s b) e (s b) + O(se s + e s ) ] log x +b+o( x 1 ) = (log x + b) x x + O( x log x ). 4. Transient case: Proof of Theorem 1.4 For simple ranom walk on Z, 3, we employ the Green function G(x) := G(x, ) = [ ] n= P x (X n = ). We have [4, Section 1.5] where a = ( )ω, an Let G(x) = a x + O( x ), V G (x) = 1 (a x ) + O( x 1 ). h(x) := a 1 ( ) G() G(x) = a 1 G() x + O( x ) an write h( ) = a 1 G(). Thus an h(y) = h( ) x + O( x 1 ), p(x, y) >, h (x) = h( ) x + O( x 1 ). Moreover, V h (x) = a V G(x) = 1 ( x ) + O( x 1 ) ( ) = x ( 1 + O( x 1 ) ). Hence we can take, for some large constant C an with a moification for small s to keep the values positive, ( ) ( ) g ± (s) = ( h( ) s 1 ± C(h( ) s) 1 ). Consequently, Theorem.1 yiels γ(x, ) = eh(x) s ( ) ( h( ) s ) ( 1 + O ( h( ) s ) 1 ) s, (4.1) where h(x) = h( ) x + O( x 1 ). (Recall that h(x) is h(x) in the lower boun for γ(x, ), an h (x) in the upper boun.)

8 8 SVANTE JANSON AND YUVAL PERES Next, we change variables to u = u(s) := (h( ) s) 1/( ). Observe that u( h(x)) = x + O(1) an s = ( )u 1 u. If we enote u = h( ) 1/( ), then x +O(1) ( h( ) u ) γ(x, ) = u ( ) u ( 1 + O(u 1 ) ) ( )u 1 u = h( ) x +O(1) u = h( ) x + O( x 1 ). ( u 1 + O(u ) ) u The result follows because G() = 1/p an thus h( ) = G() ( )a = ω p. 5. Brownian motion In this section we consier a continuous analogue of the problem stuie above. We consier Brownian motion in R, starting at some given x R, an again we are allowe to restart at x at any given time. Since, when, the Brownian motion a.s. never will hit, we now let our target be a small ball B r = {y : y r }, where r > is some arbitrary fixe number. (For = 1 we coul take r = too.) The grae then is efine as in the iscrete case, by taking the infimum of the expecte hitting time over all restarting strategies. Let x r, = 1, h(x) := log ( ) x /r, =, (5.1) r x, 3. Then h is harmonic an positive in the complement of B r, with h(x) = when x = r. Moreover, 1 = 1, h(x) = x, =, (5.) ( ) x, 3 is now exactly a function of h(x), say g(h(x)). Let the starting point be x an enote the process, using some nonanticipating restarting rule, by X t. Let further τ := inf{t : X t = r }. If F is efine by (.3) (with g + = g), we see as in the proof of Theorem.1, now using Itô s formula instea of a Taylor expansion, that F ( h( X t ) ) + t is a local submartingale an, again by the optional sampling theorem, that E τ F ( h(x ) ). Since this hols for any restarting strategy, γ(x, B r ) F ( h(x ) ).

9 RANDOM WALKS WITH RESTARTS 9 Conversely, using the strategy restart when h( X t ) h(x ) + ε for some ε >, we fin that if F is efine by (.9) with h (x ) = h(x ) + ε an g = g, then E T F ( h (x ) ). Letting ε, this an the lower boun above show, together with (.4), that the grae is given by γ(x, B r ) = F ( h(x ) ) = h(x ) s s. (5.3) g(s) Remark 5.1. We see that the optimal strategy is to restart whenever the current position is more istant from the origin than the starting point x, which is the intuitively obvious strategy. Some care has to be taken interpreting this, however, since this a.s. entails infinitely many restarts in any interval (, δ). The resulting process can be obtaine by taking a limit as in the proof of (5.3) above, or by utilizing a reflecte Brownian motion (for the raial part). Write h(x) = ϕ( x ) an h(x) = ψ(x), so that g(s) = ψ(ϕ 1 (s)), We obtain, cf. Remark.4, that ϕ( x ) s x γ(x, B r ) = ψ(ϕ 1 (s)) s = ϕ(r)ϕ (r) r. r ψ(r) Taking ϕ an ψ from (5.1) an (5.), we easily evaluate this integral an euce the following result. Theorem 5.. For Brownian motion in R, if x r >, ( x r ), = 1, γ(x, B r ) = x log x x ( 1 + log r ) + 1 r, =, r ( ) x 1 x + 1 r, 3. It is instructive to compare these exact results for R an the asymptotic results for Z in Theorems 1.3 an 1.4. Note first that the time scales iffer by a factor, since in the simple ranom walk, each coorinate of a step has variance 1/. With this ajustment we see that we obtain the same leaing term for Z an R when ; in this case, the choice of r affects only lower orer terms. When 3, however, we obtain the same x rate, but the constant for Brownian motion epens on the choice of r, an there is no reasonable way to obtain the right constant for Z from the continuous limit. This reflects the fact that the constant for Z involves the escape probability p, which epens on the local lattice structure near that is lost in the continuous limit. 6. Hitting times for ranom walk in a isk The metho above can also be use to estimate expecte hitting times in reversible Markov chains. For simplicity, we consier only simple ranom walk on a graph (V, E) with vertex set V. We thus assume p(x, y) = 1/ eg V (x) when x y (an otherwise), where eg V (x) := {y V : y x}.

10 1 SVANTE JANSON AND YUVAL PERES Theorem 6.1. Let z, h, g + an g be as in Theorem.1, for simple ranom walk on a graph (V, E), an let D be a finite connecte subset of V with z D. Define D := {x D : x y for some y / D}, D := D {x D : x y for some y D}, h 1 := min{h(x) : x D}, B := {x D : x y for some y with h(y) h 1 }. Let {X n } n= be a simple ranom walk on D. Let τ := min{n : X n = z}. Then, for any X = x D, h1 (u h(x )) g + (u) u E x τ where := E x #{n τ : X n B}. h1 (u h(x )) g (u) u +, (6.1) Note that h is harmonic on all of V, while X n is efine on D with transition probabilities p D (x, y) := 1/ eg D (x) when x y an x, y D. The error term can be estimate in several ways. One of them is to boun τ = τ z by the time τ that it takes the ranom walk to visit z an return to x. Then (see, e.g., Lemma 1.5 an Proposition 1.6 in [5]) E x #{n τ : X n B} = µ(b) µ(d) E x (τ ) = µ(b)r(x z), (6.) where µ(b) = x B eg D(x) an R(x z) is the resistance between x an z in D, regare as an electrical network. Proof. We efine, in analogy with (.3) an (.9), F ± (s) := s h1 h1 t g ± (u) ut = h1 (s u) u. (6.3) g ± (u) We thus integrate only up to h 1, an we may reefine g + (u) = for u > h 1. The right han inequality in (.1) then still hols for all x V, an we obtain from x D\ D, exactly as in Section, E(F + (h(x n+1 )) X n = x) = E x F + (h(x 1 )) F + (h(x)) 1. (6.4) On the other han, if X n = x D, then X n+1 D, an thus h(x n ), h(x n+1 ) h 1 an F + (h(x n+1 )) = F + (h(x n )), so (6.4) hols in this case too. Consequently, Y n := F + (h(x n )) + n is a submartingale, an as in Section E x τ = E x Y τ E x Y = F + (h(x )), which is the left han inequality of (6.1). For an upper boun, we assume that x z. The argument in Section works for x D\B, an we obtain E x F (h(x 1 )) F (h(x)) 1, x D\B. (6.5)

11 RANDOM WALKS WITH RESTARTS 11 For x B\ D, the same argument yiels only, using F, E x F (h(x 1 )) F (h(x)), x B\ D. (6.6) Finally, if x D, then h(x), h(x 1 ) h 1 for every X 1 x, an F (h(x 1 )) = F (h(x)), x D. (6.7) We efine N n := #{k < n : X k B} an fin from (6.5) (6.7) that if Y n := F (h(x n )) + n N n, then Y n τ, n, is a supermartingale an thus E x τ E x N τ = E x Yτ which completes the proof of (6.1). Y = F (h(x )), 6.1. Application. Take V = Z with eges between vertices at istance 1. Consier simple ranom walk on the isk D = {x Z : x R}. Let z = D an take h an g ± as in (3.1) an (3.3). Then by (3.4) an (6.3), F ± (x ) = = h1 h1 (u h(x )) u g ± (u) 4(u h(x ))(e (u b) ± O(e u ))u = [ (u h(x ))e (u b)] h 1 h(x ) h 1 ( e (u b) h 1 ) + O h(x )e u u = (h(x ) h 1 )e (h 1 b) e (h(x ) h 1 b) O(h(x )e h 1 ), where b = γ e + 3 log. By (3.1), Thus h 1 = log(r + O(1)) + b + O(R ) = log R + b + O(R 1 ), h(x ) = log x + b + O( x ). F ± (x ) = ( h(x ) + O(R 1 ) ) e log R+O(R 1) e log x +O( x +R 1) O(R log x ) = R h(x ) + O(R log R) x. Further, it is easily seen that x B implies x = R O(1), an thus µ(b) = O(R). Also, it is easy to see that for x in D we have R(x ) = O(log R). (This follows, e.g., from the metho of ranom paths [6] by picking a uniform point u on the chor bisecting the segment x z, an consiering the lattice path closest to the union of the segments x u an uz.) Thus (6.) yiels = O(E x τ /R) = O(R R(x )) = O(R log R). Consequently, Theorem 6.1 yiels for the hitting time τ of the origin, that E x τ = R h(x ) + O(R log R) x. For x R 1/ say, we can write this as E x τ = R log x + br x + O(R log R).

12 1 SVANTE JANSON AND YUVAL PERES Acknowlegement. This research was mostly performe uring the 3r IES workshop in Harsa an at Institut Mittag-Leffler, Djursholm, Sween, August 1. We thank Peter Winkler an other participants for helpful iscussions. The work was complete uring a visit by SJ to Microsoft, Remon, USA, May 1. References [1] I. Dumitriu, P. Tetali an P. Winkler (3), On playing golf with two balls. SIAM J. Discrete Math. 16, [] Y. Fukai an K. Uchiyama (1996), Potential kernel for two-imensional ranom walk. Ann. Probab. 4, [3] G. Kozma an E. Schreiber (4), An asymptotic expansion for the iscrete harmonic potential. Electron. J. Probab. 9, [4] G. Lawler (1991), Intersections of Ranom Walks. Birkhäuser, Boston. [5] D. A. Levin, Y. Peres an E. Wilmer (9), Markov Chains an Mixing Times. Amer. Math. Soc., Provience, RI. [6] Y. Peres (1999), Probability on trees: an introuctory climb. In Lectures on probability theory an statistics (Saint-Flour, 1997), vol of Lecture Notes in Math. Springer, Berlin, pp [7] F. Spitzer (1964), Principles of Ranom Walk. Van Nostran, Princeton. [8] A. Stöhr (195), Über einige lineare partielle Differenzengleichungen mit konstanten Koeffizienten. III. Zweites Beispiel: Der Operator Φ(y 1, y ) = Φ(y 1 + 1, y ) + Φ(y 1 1, y ) + Φ(y 1, y + 1) + Φ(y 1, y 1) 4Φ(y 1, y ). Math. Nachr. 3, Department of Mathematics, Uppsala University, PO Box 48, S Uppsala, Sween aress: svante.janson@math.uu.se URL: Microsoft Research, One Microsoft Way, Remon, WA , USA. aress: peres@microsoft.com URL: