ON DEVIATIONS FROM THE MAXIMUM IN A STOCHASTIC PROCESS. Catherine A. Macken. Howard M. Taylor. June 23, Cornell University

Transcription

1 ON DEVIATIONS FROM THE MAXIMUM IN A STOCHASTIC PROCESS By Catherine A. Macken Howard M. Tayor June 23, 1975 Corne University Ithaca, New York Revised - October 21, 1975

2 ABSTRACT Let T be the first time a stochastic process {X(t)} drops a units beow its maximum to date. We determine the transform of the joint distribution of T X(T) for integer vaued stochastic processes that are continuous in the sense that ony jumps of ± 1 occur. We concentrate on spatiay homogeneous processes, that is, when {X(t)} has stationary independent increments. In this case, the first two moments of T X(T) are derived, X(T) + a is shown to have a geometric distribution, the parameter of which is given. Appications in queuing theory to two armed bit probems are sketched. AMS Cassification: 60 G 40 KEYWORDS: STOCHASTIC PROCESSES, STOPPING TIMES, RANDOM WALK, POISSON PROCESSES, QUEUING THEORY.

3 For a fixed vaue a et T be the first time a stochastic process X(t) drops a units beow its maximum to date. Tnat is, setting M(t) = max{x(u); 0 < u ~ t}, et T = inf{t ~ 0; M(t)- X(t) ~a}. In [3] the transform of the joint distribution of T X(T) was determined for {X(t); t ~ 0} a Brownian motion having arbitrary drift diffusion, appications of this formua in economics quaity contro were discussed. Later Lehoczky [1] extended the approach obtained the transform for more genera diffusion processes. Here we derive the anaogous formua when (i) {X(t); t = 0,1,2,. } is a rom wak for which Pr{X(t+) = i + JX(t) = i} = pi' Pr{X(t+) = ijx(t) = i} = r i' Pr{X(t+) = i- JX(t) = i} = q.' where p. + r. + q. = 1, when (ii) X(t) s the minima process associated with the infinitesima parameters Pr{X(t+L't) = i + JX(t) = i} = }J.L't + o(m), Pr{X(t+L't) = i - JX(t) = i} = v.l't + o(t), Pr{JX(t+L't) - ij> JX(t) = i} = o(l't). -1-

4 In both cases the state space is the set of a integers the process starts from X(O) = 0. The key to the derivation is a simpe observation: In order that M(T) equa k, the process must first reach state 1 before hitting state - a, then, starting afresh from state 1, it must next reach state 2 before hitting state - a + 1, so on unti finay, again starting afresh from state k, the process must reach state k - a before hitting state k + 1. We want to derive the joint transform E[eaX(T)-ST], whie the factor -8T e compicates the anaysis somewhat, nevertheess, the task reduces to soving a series of gamber's ruin type probems as just indicated. Expicit resuts are obtained when the processes are spatiay homogeneous, that is, when pi = p for a i, ]1. =.]1 for a Then -aa e i, etc. /.~a (A a - e ) which hods for 8 > 0 where (:_; 2) in Case (i), 4 } /2 - V]. (1. 3) in Case (ii). -2-.

5 In both cases, when the processes are spatiay homogeneous, M(T) has a geometric distribution, Pr{M(T) > k} the parameter e is identified. The formua can be used to determine the moments of T X(T) some asymptotic distributions as was done in [3]. Appications of some of these resuts in queuing theory to the ~wo-armed bit probem are sketched in a ater section. 2. ~~ To begin that part of the derivation which is common to both cases, et X(t) be an integer-vaued Markov process continuous in the sense that ony jumps of ± are permitted. Fix a positive integer a. Because the process is continuous, X(T) = M(T) - a the quantity we seek differs from E[eaM(T)-ST], which we wi study first, by the constant factor a a e Confine k to integer vaues et T(k) = inf{t > 0; X(t) = k} be the hitting time to k. Introduce the notation E[ ; A]= JA Pr{dw} for expectation restricted to an event A. The suggested decomposition into gamber's ruin type probems motivates us to write E[eaM(T)-ST] co I ak = e E[e-ST; M(T) = k] k=o co t. i1 = I eak i~o E[e-8T(1+1); T(i+) < T(i-a) xco) = k=o x E[e-S (k-a\ T(k-a) < <(k+) IXCO) = k]. -3-

6 Set ;; ;; (B) E[ -BT(i+) (" ) jx(o).] u. = u. = e ; T + < T(i-a) = ( ) E[ -BT(i-a) c ) I ( ) "] y. = y. B = e ; T -a < T(i+) X 0 = so that The connection with gamber's ruin probems becomes cearer if we bring in a rom ifetime 1;;, independent of the X( ) process having the exponentia distribution Pr{;; > t} = e -Bt t ~ o, write A = (A) '\, '\, for the indicator rom variabe associated with an event A. Then, in view.of the assumed independence oi = E[e-ST(i+) ~{T(i+) < T(i-a) }jx(o) = i] = E[~{T(i+) < T(i-a) }E[e-ST(i+)jX( )]jx(o) = i] = E[{T(i+) < T(i-a) }E[{T(i+) < ;;}jx( )JjX(O) = i] '\, '\, = Pr{T(i+) < T(i-a) A 1;; jx(o) = i} where a~ b = min{a, b}, simiary y. = Pr{T(i-a) < T (i+) A ~;;jx(o) = i}. -4-.

7 Thus o. ( y. ) may be derived by soving a system of inear equations derived from a "first step anaysis." Fix a state i set u(j) = u.(j) = Pr{T(i+)< T(i-a)A six(o) = j}. Then o. = u. ( i). i - a ~ j ~ i + 1 is determined by In Case (i), u.(j) for u(i-a) = 0; u(i+) = 1 whie u(j) = e-b(p.u(j+) + r.u(j) + q.u(j-1)), J J J for i - a < j < i + 1. Simiar equations determine y. in Case (i). In Case (ii), again u(i-a) = 0; u(i+) = 1 whie conditioning on the first jump yieds u( j) = {].u(j+) + v.u(j-1)}, for ].+v.+b J J J J i - a < j < i + 1, again, simiar equations determine y. 3. It is easiy seen from the foregoing anaysis that k-1 Pr{M(T) = k} = ( IT j=o oj)yk where 0. = o.(f3=0) = 1- y J J J Expicity, y. = Pr{T(j-a) < T (j+)ix(o) = j} J a = 1/C I i=o p. ) J -5-

8 where = 1 p. = J p. p. 1 J J- q. q. 1 J J- X X X X for 0 < i ~a, in Case (i) whie in Case (ii), p. J = ]J. ]J. 1 X X ]J. 1 J J- ]- + v. v. 1 X X v.. J ]- J-+ 4. ~~~~~ By far the most interesting resuts appear when the parameters pi = p, J.i = ]J, etc. do not depend on the state. From now on, we consider ony this case. Under this assumption, 0. = 0 are aso constant so E[eaM(T)-STJ = 00 I eak ok y k=o (4.1) for a < n(/o). In both of our cases we wi determine distinct vaues A = A± for which Z( t) = t < ; = 0 t > ; is a martingae, use this to evauate y 6. Invoking the martingae optiona stopping theorem at the Markov time T ~ T() = min{t, T()} we -6-

9 have 1 = E[Z(T() ~ T)] = E[Z(T()); T() ~ T] + E[Z(T); T < T()] -a = Ao + A y. Using the distinct vaues A = A+ A = A we may sove for o y to obtain A - A y = / A. -a - A A -a = ~-a -a - A+ A ~-a - A A -a

10 Finay (4.2) which hods for S > 0 a < tn(/6). It remains ony to prescribe Case (i): X(t) = s st' t = 1,2,. where s' S2 are independent share the common distribution Pr{s n k} p for k, = r for k 0, q for k -1, with p + q + r. In this case z (t) t < z;;, 0 t > z;;, is a martingae provided -s - re S 2-2S 112 ± {(- re- ) - 4pqe } (4.3) because these A sove Ak = e-s[pak+ + rak + qak-] which is the computation that checks the martingae condition -8-

11 Case (ii): X(t) = U(t) - V(t) where U(t) V(t) are homogeneous Poisson processes having rates ~ v respectivey. In this case Z(t) AX(t) I t < s = 0 t > s is a martingae when 8+v+~ 2~ 4 } 1/2 - v~ (4.4) oltain the Lapace transform ( 5.1) We wi do nothing further with this now. Later, in connection with an appication., we wi deveop the asymptotic distribution of T as a + oo in Case (ii). The margina distribution of X(T) has a nice expicit description. We have X(T) = M(T) - a M(T) has a geometric distribution in which Pr{M(T) ~ k} = 8k, k = 0,1, 8 = oj 8 =0 = Pr{T() ~ T} is 4 a gamber's ruin probabiity. To see this, correspond s oo with 8 = 0 write k (o8=0). = Pr{T (k) ~ T} = Pr{M (T) ~ k} -9-

12 To obtain e, note that T() ~ T if ony if the process reaches state 1 before state -a. The we know gamber's ruin probabiities are: Case (i): e a/ (1 + a) if p q -a 1 - (g/)2) if p -I q, (q/p) - (q/p) -a ( 5. 2) Case (ii) : e a/(+a) if \) f.! a (V/]J) - 1 (V/JJ)a+ - 1 if \) -I ]J (5.3) The mean of the geometricay distributed M(T) is E[M(T)] 8/(1-8) whence E [X (T) ] a ( 5.4) with 8 given in (5.2) (5.3). We aso obtain the variances Var [X (T)] Var [M (T)] {5.5) Finay we may use Wad's equation E[X(T)] E[T]E(X()], to get E(T] 1 p-q ( 1~8 - a) in Case (i) with p -I q, 1 ]J-\) ( a) in Case (ii) with ]J -I \). (5.6) -10-

13 - 2 get 2 When p q or \) we use E[X(T) ] E[T]E[X() ] to 8 E [T] Case ( i), p q, (-r) (-8) 2 8 Case ( ii), \). 2 2 v(-8) 6. ~- Here are brief sketches of two appications. Queuing theory. In Case (ii), Y(t) = M(t) - X(t) evoves ike the queue ength in an M/M/1 queue system having arriva rate v service rate ].1. Then T is the first time the queue hod~ a customers. Referring to (5.3) E [T] ( 5. 6), ":" t~)a we see the mean is - (V/)a _ )_ - (V/].1) ) When the service rate ].1 exceeds the arriva rate v, for arge a the dominant term is 2 a [].1/ (f.-v) ] (f.!/v) In genera, we have the Lapace transform y/(-6) - i! with A± given by (4.4). We wi derive the asymptotic distribution of T as a ~ oo when the service rate ].1 exceeds the arriva rate v. Indeed, we wi show that asymptoticay U a = (V/].)a T is exponentiay distributed with parameter 2 (].1-V) /].1. To do this we need ony show the Lapace transforms -11-

14 corresponding to U U have the imit a To begin the proof, stipuating that ~ > v using eementary cacuus, derive the Tayor series expansions A U3) + + s + 0 {t3) ~-\) A (f3) \) ~ ( - 1 f3) + 0 (f3) ~-\) ;, for A± { S+~+\J ±. 2 [ ( S+~+\J) - 4~\J] /2}/2~. when ~ > \). Then im E[e-suJ a-+«> a im E[e-S(\Jj~) T] a-+«> = im a-+«> 1 - (\!jj) - (\!jj) S/(~-v) (\!/~) as caimed. -12-

15 In summary, roughy speaking, the time it takes to competey fi a arge waiting room hoding a customers in an M/M/1 queueing system is approximatey exponentiay distributed with mean 2 a [].1/ (].1-V) ) (].1/V) Two armed bit probems. A pay on a one armed bit wins a doar with probabiity p oses a doar with probabiity q = 1 - p. Suppose we face two such machines having unknown win probabiities p 1 respectivey. We begin with machine 1 continue to pay it as ong as wins are secured. At the first oss, we switch to machine 2, now pay it as ong as we win. When we first ose on machine 2 we revert to machine 1 repeat the cyce. This strategy is caed the Pay-the-winner rue. In the notation of this artice, it corresponds to paying a machine unti time T = T for a a = 1. That is, we continue on a machine as ong as our fortune increases new maxima are reached on each pay. At the first oss, when our fortune first drops one unit beow its maximum to date, we stop pay switch to the other machine. It is an obvious question to ook at the behavior of the strategies T T for a > in this context. a denote the expectations under p 1 respectivey. An easy argument shows that the ong run average gain per pay is G (a) I I I (p-q)e[ta] + (p2-q2)e2[ta] E [Ta] + E2[Ta] -13.:

16 e. Let us suppose that at east one of the games is favorabe, that is, at east one win probabiity exceeds one-haf. To be definite, suppose p 1 = max{p 1,p 2 } > /2. As a increases, the dominant term in is That is, E 1 [Ta] grows geometricay with a at rate p 1 /q 1. The same argument shows that E 2 [Ta] can grow at most geometricay at rate a if p 2 < 1/2. In either case we have.asymptoticay grows ineary in im G (a) a-+= -14-

17 the gain rate if machine 1 were payed continuousy! That is, as a -+ oo the strategy T a 1s asymptoticay optima, in particuar, does significanty better than the Pay-the-winner rue. Woud the same improvement over Pay-the-winner appear if criteria other than average gain were used? In particuar, what about criteria more appropriate to the medica trias probem [2]? These are provocative open questions. REFERENCES 1. Lehoczky, J.P. (1975). Formuas for Stopped Diffusion Processes with Stopping Times Based on the Maximum. (Submitted for pubication Ann. Probabiity.) 2. Sobe, M. Weiss, G. (1970). Pay-the-Winner Samping for Seecting the Better of Two Binomia Popuations. Biometrika J? _. Tayor, H.M. (1975). A Stopped Brownian Motion Formua. Ann _Probabiity "' -15-