arxiv: v1 [math.pr] 22 Apr 2014

Transcription

1 Renewal Processes wih Coss and Rewards Maria Vlasiou Ocober 5, 28 arxiv:44.56v [mah.pr] 22 Apr 24 Absrac We review he heory of renewal reward processes, which describes renewal processes ha have some cos or reward associaed wih each cycle. We presen a new simplified proof of he renewal reward heorem ha mimics he proof of he elemenary renewal heorem and avoids he echnicaliies in he proof ha is presened in mos exbooks. Moreover, we menion briefly he exension of he heory o parial rewards, where i is assumed ha rewards are no accrued only a renewal epochs bu also during he renewal cycle. For his case, we presen a counerexample which indicaes ha he sandard condiions for he renewal reward heorem are no sufficien; addiional regulariy assumpions are necessary. We presen a few examples o indicae he usefulness of his heory, where we prove he inspecion paradox and Lile s law hrough he renewal reward heorem. Basic noions and resuls Many applicaions of he renewal heory involve rewards or coss (which can be simply seen as a negaive reward). For example, consider he classical example of a renewal process: a machine componen ges replaced upon failure or upon having operaed for T ime unis. Then he ime he n-h componen is in service is given by Y n = min{x n, T}, where X n is he life of he componen. In his example, one migh be ineresed in he rae of he number of replacemens in he long run. An exension of his basic seup is as follows. A componen ha has failed will be replaced a a cos c f, while a componen ha is replaced while sill being operaional (and hus a ime T) coss only c < c f. In his case, one migh be ineresed in choosing he opimal ime T ha minimises he long-run operaional coss. The soluion o his problem involves he analysis of renewal processes wih coss and rewards. Moivaed by he above, le {N(), } be a renewal process wih inerarrival imes X n, n, and denoe he ime of he n-h renewal by S n = X + + X n. Now suppose ha a he ime of each renewal a reward is received; we denoe by R n he reward received a he end of he n-h cycle. We furher assume ha (R n, X n ) is a sequence of i.i.d. random variables, which allows for R n o depend on X n. For example, N() migh coun he number of rides a axi ges up o ime. In his case, X n is he lengh of each rip and one reasonably expecs he fare R n o depend on X n. As usual, we denoe by (R, X) he generic bivariae random variable ha are disribued idenically o he sequence of rewards and inerarrival imes (R n, X n ). In he analysis, ogeher wih he sandard assumpion in renewal processes ha he inerarrival imes have a finie expecaion E[X] = τ, we will furher assume ha E[ R ] <. The cumulaive reward up o ime is given by R() = N() n= R n, where he sum is aken o be equal o zero in he even ha N() =. Depending Dep. of Mahemaics & Compuer Science, Eindhoven Universiy of Technology, P.O. Box 53, 56 MB Eindhoven, The Neherlands, m.vlasiou@ue.nl

2 on wheher he reward is colleced a he beginning or a he end of he renewal period, one migh wish o adap his definiion by aking R() = N()+ n= R n. The imporance of renewal processes wih coss and rewards is evidenced by heir applicaion o Markovian processes and models. Mos queuing processes in which cusomers arrive according o a renewal process are regeneraive processes (see he aricle on Regeneraive Processes) wih cycles beginning each ime an arrival finds he sysem empy. Moreover, for every regeneraive process Y() we may define a reward srucure as follows: R n = S n Y() d, where all R S n, n, are i.i.d., excep possibly for R ha migh n follow a differen disribuion. The basic ools ha are used are he compuaion of he reward per uni of ime and he rae of he expeced value of he reward. These wo resuls are summarised in he following heorem. Theorem. Le N() be a renewal reward process generaed by (R, X), n. Assume ha E[ R ] < and le r = E[R], τ = E[X] <. Then lim R() lim E[R()] = r, wih probabiliy, and () τ = r τ. (2) These resuls do no depend on wheher rewards are colleced a he beginning of he renewal cycle or a is end, or if hey are colleced in a more complicaed fashion (e.g. coninuously or in a non-monoone way during he cycle), as long as (R n, X n ), n, is a sequence of i.i.d. bivariae random variables and minor regulariy condiions hold; see he following secion. Moreover, he resuls remain valid also for delayed renewal reward processes, i.e. when he firs cycle migh follow a differen disribuion. Noe ha (2) does no follow from () since almos sure convergence does no imply he convergence of he expeced values. Think of he disribuion n/(n + ), x =, P[Y n = x] = /(n + ), x = n +. Since for all n, E[Y n ]=, we have lim n E[Y n ] =. Moreover, Y n goes almos surely o zero, i.e. P[lim n Y n = ] =, which means ha he value of he random variable and is expecaion differ in he limi. Proof of Theorem. To prove (), wrie R() = N() n= R n N() N(). By he srong law of large numbers we have ha lim N() n= R n N() = E[R] = r, since N() goes o infiniy almos surely, and by he srong law for renewal processes we know ha N() lim = E[X] = τ. 2

3 The proof of (2) is a bi more involved. The classical proof applies Wald s equaion [5] and requires E[R he proof ha lim N()+ ] = by consrucing a renewal-ype equaion (see he aricle on Renewal Funcion and Renewal-Type Equaions). Here we presen an alernaive proof ha follows he same seps of he classical proof for he Elemenary Renewal Theorem (see e.g. []) bu seems o be new in he seing of renewal reward processes. We firs consruc a lower bound. From Faou s lemma we have ha lim inf E[R()] E[lim inf R() R() ] = E[lim from (). For he upper bound we use runcaion. Take M < and se R M n We hen have ha E[R()] E[RM ()] N() N() R() = R n n= = E[ N()+ n= R M n R M N()+ ] n= R M n de f = R M (). ] = r τ = E[N() + ]E[RM ] E[RM N()+ ] = max{r n, M}. Noe ha (3) E[N()] + E[R M ]+ M. The las equaliy follows from Wald s equaion since N() + is a sopping ime and for he las inequaliy we simply observe ha we can bound E[R M n ] from above wih M (by consrucion). Thus, lim sup E[R()] ( E[N()] + lim sup E[R M ] + M ) = E[RM ] τ by he Elemenary Renewal Theorem. Since we can choose any M < in our consrucion, by monoone convergence we have ha E[R M] E[R ] = r as M. E[R The advanage of his proof is ha i avoids he compuaion of lim N()+ ] and ha i is in essenials idenical o he classical proof for he corresponding heorem in renewal processes wihou rewards. Parial Rewards In some applicaions rewards migh no be given a he beginning or an end of a cycle, bu migh be earned in a coninuous (maybe non-monoone) fashion during a cycle. The simples case is when he reward during a cycle is given a a consan rae during he cycle, or only a porion hereof. The porion of he reward up o ime associaed wih a cycle saring a X N() is called a parial reward. Recall ha we have assumed ha E[X] < and E[ R ] <. Under furher assumpions, Theorem sill holds. In he case of parial rewards, we need o assume some form of good behaviour as nohing so far prevens he rewards from escaping o plus or minus infiniy while sill having he average reward per cycle being finie. As an example, o avoid such erraic behaviour consider he resricive assumpion ha he absolue value of rewards is uniformly bounded by some number; hen Theorem holds. This is also he case when rewards are non-negaive or if hey accumulae in a monoone fashion over any inerval. Wolff [6] assumes ha he parial reward during a cycle is he difference of wo differen monoone non-negaive processes, which could be inerpreed as income and coss over he renewal cycle. I seems o be a sandard misconcepion ha only having E[ R ] < is sufficien for Theorem o hold also for parial rewards. In lack of necessary and sufficien condiions in he lieraure for for his resul, or 3

4 a leas of a counerexample where E[ R ] < bu where Theorem does no hold for parial rewards, we presen here a counerexample ha illusraes he needs for addiional assumpions. In his case, he average reward over a cycle is finie (and we will consruc i o be equal o zero), bu he average reward for every finie is no defined, and hus also no is limi over ; see (2). Take a renewal reward process ha is defined as follows: X i = and R i = for all i. During a cycle, rewards build up and are depleed in he following fashion. For cycle j choose a Cauchy random variable C j and ake he auxiliary funcion f (x) = x for x [, /2] and f (x) = x for x [/2, ]. Define he reward for any ime during cycle j as C() = C j f ( []). Then we see ha he cumulaive reward up o ime, for all > is given by R() = C(). Thus, since a Cauchy random variable has no a well-defined firs momen, we see ha he average cumulaive reward in his example canno be defined, and hus Theorem canno be exended for his case, despie he fac ha a he end of a cycle (and hus a all ineger imes) he cumulaive reward is equal o zero. In he following, we will see wo examples where parial rewards are assumed. Examples We proceed wih a few applicaions of renewal processes wih coss and rewards ha exhibi he usefulness of hese processes. The firs wo show how (cyclic) renewal processes can be seen as a special case of renewal processes wih coss and rewards, while he las wo make use of rewards earned coninuously over ime. Naurally, an abundance of examples of renewal (reward) processes migh be found in Markov processes and chains. Example (Sandard renewal processes). Observe ha he sandard renewal process is simply a special case of a renewal reward process, where we assume ha he reward a each cycle is equal o. In his case we see for example ha (2) is simply he Elemenary Renewal Theorem. Example 2 (Alernaing renewal processes). For an alernaing renewal process (see he aricle on Alernaing Renewal Processes), i.e. a renewal process ha can be in one of wo phases (say ON and OFF) suppose ha we earn a he end of a cycle an amoun equal o he ime he sysem was ON during ha cycle. Alernaively, one may assume ha we earn a a rae of one per uni of ime when he sysem is ON, bu one need no consider coninuous rewards a his poin. Then he oal reward earned in he inerval [, ] is equal o he oal ON ime in ha inerval, and hus by () we have ha as ON ime in[, ] E[Y ON ] E[Y ON ] + E[Y OFF ], where Y ON is a generic ON ime in a cycle and equivalenly for he OFF imes. Thus, for non-laice disribuions, he limiing probabiliy of he sysem being ON is equal o he long-run proporion of ime i is ON. Naurally, hese resuls exend o general cyclic renewal processes, i.e. a process wih more han wo phases []. Example 3 (Average ime of age and excess). Le A() denoe he age a ime of a renewal process generaed by X n X and suppose ha we are ineresed in compuing he average value of he age. Wih a sligh abuse of erminology, define his o be equal o lim A(s) ds. 4

5 Suppose now ha we are obaining a reward coninuously a a rae equal o he age of he renewal process a ha ime. Thus, A(s) ds represens he oal earnings by ime. Moreover, since he age of a renewal process a ime s since he las renewal is simply equal o s we have ha he reward during a renewal cycle of lengh X is equal o X s ds = X2 /2. Then by () we have ha wih probabiliy, A(s) ds lim = E[X2 /2]. E[X] We can apply he same logic o he residual life a ime, B(), and we will end up a he same resul. Since he renewal epoch covering ime is equal o X N()+ = A() + B() we have ha lim X N()+ ds = E[X2 ] E[X] E[X] where we have equaliy only when Var(X) =. Thus, we rerieve he inspecion paradox. Example 4 (Lile s law). Consider a GI/GI/ sable queue wih he inerarrival imes X i generaing a nonlaice renewal process. Suppose ha we sar observing he sysem upon he arrival of a cusomer. Denoe a generic inerarrival ime by X and le E[X] = /λ. Also denoe by n() he number of cusomers in he sysem a ime. Consider he average number of cusomers in he sysem in he long run, which we denoe by L. Then, wih a sligh abuse of noaion we have ha L = lim n(y) dy. Define a renewal reward process wih cycles C saring each ime an arrival finds he sysem empy and suppose ha we earn a reward a ime y wih a rae n(y). Then from (2) we have ha L = E[reward during a cycle] E[cycle ime] = E[ C n(y) dy] E[C]. (4) Now le N be he number of cusomers served during a cycle and define he long-run average sojourn ime of cusomers as T = lim n (T + + T n )/n, where T i is he ime cusomer i spen in he sysem. Suppose now ha in each cycle of lengh C where N cusomers were served. We can see hen ha he reward we have received in ha cycle, which was defined o be equal o a rae of n(y) a each ime y, can also be seen as having each cusomer pay a a rae for each ime uni he is in he sysem. In oher words, in a cycle wih N cusomers, he reward is equal o T + + T N, i.e. he oal ime all cusomers in ha cycle spen in he sysem. Then T is he average reward per uni ime (observe ha since he second definiion of he reward depends on he number of cusomers in a cycle, hen he duraion of he cycle is also defined in erms of cusomers) and again by (2) we have ha S = E[reward during a cycle] E[cycle ime] = E[ N T i ]. (5) E[N] We can now prove Lile s law, one of he fundamenal relaions in queuing heory, which saes ha L = λt. To see his, observe ha N C = X i, 5

6 and since N is a sopping ime for he sequence {X i } we have from Wald s equaion ha Thus, by (4) and (5) we see ha E[C] = E[N]E[X] = E[N]/λ. L = λt E[ C n(y) dy] E[. N T i ] However, as we have seen, he fracion is equal o, since boh he numeraor and he denominaor describe he reward earned during a cycle. Thus we rerieve Lile s law. Furher reading Almos all books on sochasic processes and inroducory probabiliy have a secion on renewal heory. Here we only refer o he ones ha have a separae menion of renewal processes wih coss and rewards. Cox [2] is one of he few manuscrips exclusively devoed o renewal processes. There, renewal processes wih coss and rewards are found under he erm cumulaive processes. Anoher erm ha has been used o describe hese processes is compound renewal processes. The general concep of a cumulaive process and he asympoic resuls relaed o hem are due o Smih [, 2]. Mercer and Smih [6] invesigae cumulaive processes acclimaed wih a Poisson process, in connecion wih a sudy of he wear of conveyor beling. Brief menions of renewal reward processes can also be found in Taylor and Karlin [3] as well as in Heyman and Sobel [3] in a conribuion by Serfozo. In Moder and Elmaghraby [7] one may find a shor reamen of cumulaive processes ha also includes asympoic resuls no only for he expecaion of R() as given by (2), bu also for is variance. As noed in he ex here, here are mulivariae versions of hese resuls as well. The following books offer a horough reamen of his opic and muliude of examples; moreover, hey are approachable o a wide audience wih only undergraduae knowledge of probabiliy heory [4, 5, 8, 9, 4, 6]. References [] Asmussen, S. (23). Applied Probabiliy and Queues. Springer-Verlag, New York. [2] Cox, D. R. (962). Renewal heory. Mehuen & Co. Ld., London. [3] Heyman, D. P. and Sobel, M. J., Eds. (99). Sochasic Models vol. 2 of Handbooks in Operaions Research and Managemen Science. Norh-Holland Publishing Co., Amserdam. [4] Kulkarni, V. G. (995). Modeling and Analysis of Sochasic Sysems. Texs in Saisical Science Series. Chapman and Hall Ld., London. [5] Medhi, J. (994). Sochasic Processes second ed. John Wiley & Sons Inc., New York. [6] Mercer, A. and Smih, C. S. (959). A random walk in which he seps occur randomly in ime. Biomerika 46, [7] Moder, J. J. and Elmaghraby, S. E., Eds. (978). Handbook of operaions research. Van Nosrand Reinhold Co., New York. Foundaions and fundamenals. 6

7 [8] Resnick, S. (992). Advenures in Sochasic Processes. Birkhäuser Boson Inc., Boson, MA. [9] Ross, S. M. (996). Sochasic Processes second ed. Wiley Series in Probabiliy and Saisics: Probabiliy and Saisics. John Wiley & Sons Inc., New York. [] Serfozo, R. (29). Basics of Applied Sochasic Processes. Probabiliy and is Applicaions (New York). Springer-Verlag, Berlin. [] Smih, W. L. (955). Regeneraive sochasic processes. Proceedings of he Royal Sociey. London. Series A. Mahemaical, Physical and Engineering Sciences 232, 6 3. [2] Smih, W. L. (958). Renewal heory and is ramificaions. Journal of he Royal Saisical Sociey. Series B. Mehodological 2, [3] Taylor, H. M. and Karlin, S. (998). An Inroducion o Sochasic Modeling hird ed. Academic Press Inc., San Diego, CA. [4] Tijms, H. C. (986). Sochasic Modelling and Analysis. A Compuaional Approach. Wiley Series in Probabiliy and Mahemaical Saisics: Applied Probabiliy and Saisics. John Wiley & Sons Ld., Chicheser. [5] Wald, A. (944). On cumulaive sums of random variables. Annals of Mahemaical Saisics 5, [6] Wolff, R. W. (989). Sochasic Modeling and he Theory of Queues. Prenice Hall Inernaional Series in Indusrial and Sysems Engineering. Prenice Hall Inc., Englewood Cliffs, NJ. 7