Bounds on the bias terms for the Markov reward approach

Transcription

1 Bounds on the bas terms for the Markov reward approach Xnwe Ba 1 and Jasper Goselng 1 arxv: v1 [math.pr] 3 Jan Department of Appled Mathematcs, Unversty of Twente, P.O. Box 217, 7500 AE Enschede, the Netherlands Emal: x.ba@utwente.nl; j.goselng@utwente.nl Abstract An mportant step n the Markov reward approach to error bounds on statonary performance measures of Markov chans s to bound the bas terms. Affne functons have been successfully used for these bounds for varous models, but there are also models for whch t has not been possble to establsh such bounds. So far, no theoretcal results have been avalable that guarantee bounds on the bas terms. We consder random walks n the postve orthant and provde suffcent condtons under whch quadratc and/or geometrc functons can be used to bound the bas terms. In addton, we provde a lnear programmng framework that establshes the quadratc bounds as well as the resultng bound on the statonary performance. Keywords: Markov reward approach, bas terms, quadratc bounds, geometrc bounds, lnear programmng 1 Introducton Ths paper deals wth the Markov reward approach for error bounds [13]. The am of ths approach s to provde bounds on the statonary performance of a Markov chan R for whch the statonary probablty dstrbuton π s unknown. These bounds are establshed through a perturbed random walk R wth known statonary probablty dstrbuton π. Ths gst of the approach, startng from a statonary performance measure F = n π(n)f (n) for some non-negatve F (n), s to nterpret F (n) as the one-step reward for beng n state n and to consder the expected cumulatve reward up to tme t f 1

2 R starts from n at tme 0, denoted by F t (n). The basc result, see, for nstance, [13] s that f we can fnd functons F and G that satsfy F (n) F (n) ( P (n, n ) P (n, n ) ) ( F t (n ) F t (n) ) G(n), (1) n for all n and all t 0, then F F n π(n)g(n). (2) In the above, P (n, n ) and P (n, n ) denote the transton probablty from n to n n R and R, respectvely. Also, F = n π(n) F (n). Terms of the form F t (n ) F t (n) are called bas terms and an essental step n applcaton of the above result s to bound the bas terms unformly n t. In most of the exstng lterature, for nstance, [3, 11 16], such bounds are essentally establshed through tral and error wth a verfcaton provded through nducton n t. The dffculty n ths s that the verfcaton s tedous and often requres qute some nsght nto the behavor of the Markov chan at hand. In [6] a general framework has been ntroduced for establshng error bounds for a specfc class of Markov chans, more specfcally, for random walks n the quarter plane. Ths framework allevates the need to manually establsh bounds on the bas terms. In partcular, a general lnear program s presented n whch the values of the transton probabltes and the functon F (n) enter as smple parameters. The advantage of ths method s that can provde bounds for any Markov chan that s a random walk n the quarter plane wthout the need to manually establsh bounds on the bas terms. The common aspect n both the manual methods of [3, 11 16] and the lnear programmng approach of [6] s that the bas terms are bounded usng affne functons. The dscusson n [13] as well as the numercal results n [6] ndcate that t mght not always be possble to establsh such bounds. More precsely, [6] contans examples for whch ths has not been successful. It s, however, not clear f ths s due to the approach that s taken, or f ths s an nherent lmtaton mposed by boundng wth affne functons. More generally, not much s known about the behavor of the bas terms. In [13], t s shown that the bas terms are bounded by the mean frst passage tme between two states. Ths establshes exstence of bounds on the bas terms, but does not gve an ndcaton on the type of bound that one can hope to establsh, snce closed-form expressons for mean frst passage tme are by themselves dffcult to obtan. 2

3 In ths paper two classes of functons are used to bound the bas terms. In partcular, we consder quadratc and geometrc functons. Suffcent condtons under whch the bas terms can be bounded by these functons are found. Also, the lnear programmng framework of [6] s extended to work wth quadratc boundng functon. More precsely, the contrbutons of ths paper are: 1. We present a boundng functon on the bas terms that s geometrcally ncreasng n the coordnate of the state f R has negatve drft. We gve an explct expresson for the boundng functon. 2. We show that f R has negatve drft, there exsts a boundng functon that s quadratcally ncreasng n the coordnate of the state. The explct expresson for the quadratc bound s dffcult to obtan. Hence, we have formulated a lnear program to obtan bounds on F based on quadratc bounds on the bas terms. 3. We compare numercal results obtaned by consderng varous bounds on bas terms. We see that the geometrc bounds are often not tght. By consderng quadratc and lnear bounds on the bas terms, we obtan relatvely tght bounds on F. Moreover, by consderng quadratc bounds we can obtan error bounds n cases where error bounds are not avalable consderng lnear bounds. We can also get tghter bounds by consderng quadratc bounds. The class of Markov chans that we study n ths paper s as follows. We consder a dscrete-tme random walk R n the M-dmensonal postve orthant,.e., on state space S = {0, 1,... } M. The state space s parttoned nto a fnte number of components such that the transton probabltes are homogeneous wth each component. As demonstrated n [1] ths enables us to model, for nstance, queueng networks wth break-downs, overflows and fnte buffers. In [1] the lnear programmng framework of [6] has been generalzed to ths class of models. Note, that n order to apply the Markov reward approach one requres a perturbed random walk R wth a known statonary probablty dstrbuton π. In ths paper, our focus s not on constructng R or π. Instead, we use the results from [1] (see also, [2, 4, 5]) to apply the Markov reward approach for specfc examples. The remander of the paper s structured as follows. In Secton 2 we defne the model and notaton consdered n ths paper. Then, n Secton 3 we revew the results on the Markov reward approach, geometrc ergodcty and µ-ergodcty. In Secton 4 we use these results to fnd geometrc and quadratc boundng functons on the bas terms. Next, n Secton 5, we formulate 3

4 a lnear program for obtanng the bounds based on quadratc bounds on the bas terms. Fnally, n Secton 6, we mplement the lnear program n numercal examples, where we consder varous performance measures for the upper and lower bounds. 2 Model descrpton Let R be a dscrete-tme random walk n S = {0, 1,... } M. Moreover, let P : S S [0, 1] be the transton matrx of R. In ths paper only transtons between the nearest neghbors are allowed,.e., P (n, n u) > 0 only f u N(n), where N(n) denotes the set of possble transtons from n,.e., N(n) = { u { 1, 0, 1} M n u S }. (3) For a fnte ndex set K, we defne a partton of S as follows. Defnton 2.1. C = {C k } k K s called a partton of S f 1. S = k K C k. 2. For all j, k K and j k, C j C k =. 3. For any k K, N(n) = N(n ), n, n C k. The thrd condton, whch s non-standard for a partton, ensures that all the states n a component have the same set of possble transtons. Wth ths condton, we are able to defne homogeneous transton probabltes wthn a component, meanng that the transton probabltes are the same everywhere n a component. Denote by c(n) the ndex of the component of partton C that n s located n. We call c : S K the ndex ndcatng functon of partton C. In ths paper, we restrct our attenton to an R that s homogeneous wth respect to a partton C of the state space,.e., P (n, nu) depends on n only through the component ndex c(n). Therefore, we denote by N c(n) and p c(n),u the set of possble transtons from n and transton probablty P (n, n u), respectvely. To llustrate the notaton, we present the followng example. Example 2.2. Consder S = {0, 1,... } 2. Let C consst of C 1 = {0} {0}, C 2 = {1, 2, 3, 4} {0}, C 3 = {5, 6,... } {0}, C 4 = {0} {1, 2,... }, C 5 = {1, 2, 3, 4} {1, 2,... }, C 6 = {5, 6,... } {1, 2,... }. The components and ther sets of possble transtons are shown n Fgure 1. 4

5 n 2 p 5,u p 6,u p 4,u C 4 C 5 C 6 p 1,u p 2,u p 3,u C 1 C 2 C 3 n 1 Fgure 1: A fnte partton of S = {0, 1,... } 2 and the sets of possble transtons for ts components. Based on a partton, we now defne a component-wse lnear functon. Defnton 2.3. Let C be a partton of S. A functon H : S [0, ) s called C-lnear f H(n) = ( ) M 1 (n C k ) h k,0 h k, n. (4) k K In ths paper, we consder an F (n) that s C-lnear. 3 Prelmnares 3.1 The Markov reward approach Suppose that we have obtaned an R for whch π s known explctly. The Markov reward approach can be used to obtan upper and lower bounds on F n terms of F. An ntroducton to the approach s gven n [13]. In ths secton, we gve a revew on ths approach and defne the bas terms. In the Markov reward approach, F (n) s consdered as a reward f R stays n n for one tme step. Let F t (n) be the expected cumulatve reward up to tme t f R starts from n at tme 0, F t (n) = t 1 P k (n, m)f (m), (5) k=0 m S 5

6 where P k (n, m) s the k-step transton probablty from n to m. Then, snce R s ergodc and F exsts, for any n S, F t (n) lm t t = F, (6).e., F s the average reward ganed by the random walk ndependent of the startng state. Moreover, based on the defnton of F t, t can be verfed that the followng recursve equaton holds, F 0 (n) = 0, F t1 (n) = F (n) p c(n),u F t (n u). (7) For any n S, u N c(n) and t = 0, 1,..., the bas terms are defned as D t u(n) = F t (n u) F t (n). (8) We present the man result of the Markov reward approach below. Theorem 3.1 (Result n [13]). Suppose that F : S [0, ) and G : S [0, ) satsfy F (n) F (n) ) ( pc(n),u p c(n),u D t u (n) G(n), (9) for all n S, t 0. Then F F n S π(n)g(n). In addton to the bound on F F, the followng theorem s gven n [13] as well, whch s called the comparson result and can sometmes provde a better upper bound. Theorem 3.2 (Result n [13]). Suppose that F : S [0, ) satsfes F (n) F (n) n S ( P (n, n ) P (n, n ) ) D t (n, n ) 0, (10) for all n S, t 0. Then, F F. Smlarly, f the LHS of (10) s non-postve, then F F. 6

7 3.2 Geometrc ergodcty and µ-ergodcty In ths secton, we revew some defntons and results on geometrc ergodcty and µ-ergodcty that are gven n [10]. Frst, we gve the followng defntons. Defnton 3.3 (µ-norm). Let µ : S [1, ). Then, for h : S R, the µ-norm of h s defned as h(n) h µ = sup n S µ(n). (11) Defnton 3.4 (µ-total varaton norm). Let µ : S [1, ). Then, for h : S R, the µ-total varaton norm of h s gven by h µ = sup h(n)g(n). (12) g: g µ 1 Defnton 3.5 (Geometrc ergodcty). A random walk R s geometrcally ergodc f there exst functon V : S [1, ), constant b > 0, ε > 0 and fnte set B S such that p c(n),u V (n u) V (n) εv (n) b1 B (n), n S. (13) Defnton 3.6 (µ-ergodcty). A random walk R s µ-ergodc f there exst functons µ : S [1, ) and V : S [0, ), constant b > 0 and fnte set B S such that p c(n),u V (n u) V (n) µ(n) b1 B (n), n S. (14) If (13) holds, by takng µ = V, V = ε 1 V, b = ε 1 b, (14) holds for µ, V, b and B. Therefore, geometrc ergodcty mples µ-ergodcty. In the followng lemmas, we present results from [9] and [10] for geometrcally ergodc and µ-ergodc random walks. Lemma 3.7 ([9, Theorem 2.3]). Suppose that R s rreducble and aperodc. If (13) holds for V : S [1, ), ε > 0, b > 0 and fnte set B S, and δ := mn n B n S : nu B p c(n),u > 0, (15) 7

8 then, Pn k Pn k V (1 γ)ρ(1 ρ) 1 (ρ ϑ) 1 [V (n) V (n )], (16) k=0 for any ρ > ϑ = 1 M 1 B, where M B = (1 λ) [1 2 λ ˆb ˆb 2 η(ˆb(1 λ) ˆb ] 2 ), γ = δ 2 [4b 2δ(1 ε)v B ], λ = (1 ε γ)/(1 γ), ˆb = vb γ, v B = max n B V (n), η = δ 5 (4 δ 2 )ε 2 b 2. Lemma 3.8 ([10, Theorem ]). Suppose that R s rreducble and aperodc. If (14) holds for V : S [0, ), µ : S [1, ), b > 0 and fnte set B S, then there exsts b 0 < such that for any n, n S, Pn k Pn k µ V (n) V (n ) b 0. (17) k=0 In both lemmas, bounds can be obtaned on the sum of µ-total varaton norms of Pn k Pn k over k. The dfference s that under the stronger geometrc ergodcty condton an explct bound can be obtaned, whle under the weaker µ-ergodcty condton, the constant b 0 of the bound s not known explctly. 4 Bounds on the bas terms In ths secton, we show that f R has negatve drft, then the bas terms are bounded by a geometrc functon as well as a quadratc functon. In Secton 4.1, we defne what we call a random walk wth negatve drft. Next, n Secton 4.2 we show that a random walk wth negatve drft s geometrcally ergodc. Hence, we can obtan a geometrc boundng functon on the bas terms. Then, n Secton 4.3 we show that a random walk wth negatve drft s also µ-ergodc. Thus, the bas terms can be bounded by a quadratc functon. 4.1 Random walks wth negatve drft We frst consder the followng defntons. For state n S, let I(n) denote the dmensons for whch n > 0,.e., I(n) = { = 1,..., M n > 0}. (18) 8

9 Moreover, for = 1,..., M, defne partal sums of the transton probabltes n the followng way, s (n) = p c(n),u, (19) s (n) = s (n) = :u =1 :u =0 :u = 1 p c(n),u, (20) p c(n),u. (21) Note that f / I(n), then n = 0 and s (n) = 0. Snce the transton probabltes sum up to one, for any = 1,..., M and n S, s (n) s (n) s (n) = 1. (22) Therefore, below we defne a random walk wth negatve drft. Defnton 4.1 (Random walk wth negatve drft). A random walk R s sad to have negatve drft f { s (n) s (n)} < 0. (23) sup n S, I(n) Intutvely, t means that n any dmenson, f n > 0, then the drft at state n n dmenson s strctly negatve. Snce R s homogeneous wth respect to a partton C, the supremum above can be obtaned. 4.2 Geometrc bounds on the bas terms In ths secton, we show that f a random walk has negatve drft, then t s geometrcally ergodc. Usng the results gven n the prevous secton, we can obtan a geometrc boundng functon on the bas terms. Notce that n Lemmas 3.7 and 4.5, bounds have been obtaned on the µ- total varaton norm of Pn k Pn k. Before we dve nto geometrc ergodcty, we buld a relaton between bounds on the bas terms and the µ-total varaton norm of Pn k Pn k n the followng lemma. Lemma 4.2. Consder a C-lnear functon F : S [0, ). Let µ : S [1, ) be a functon for whch F µ 1. Then, D t u (n) Pnu k Pn k µ. (24) k=0 9

10 Proof. Snce Du(n) t = D t (n, n u) = F t (n u) F t (n), usng (5) and the defnton of µ-total varaton norm, we have D u(n) t t 1 [ = P k nu (m)f (m) Pn k (m)f (m) ] (25) k=0 m S t 1 [ P k nu (m) Pn k (m) ] F (m) (26) k=0 m S Pnu k Pn k µ. (27) k=0 Thus, f we establsh an upper bound on k=0 P k nu P k n µ, we also obtan a boundng functon on D t u(n). In the next lemma, we show that a random walk R wth negatve drft s geometrcally ergodc. Lemma 4.3. Suppose that the random walk R s rreducble, aperodc, postve recurrent and has negatve drft. Let r, = 1,..., M satsfy { s 1 < r < nf (n) } n S: I(n) s (n). (28) Then, R s geometrcally ergodc, wth V (n) = v 0 M v r n, (29) for any v 0 1, v > 0, = 1,..., M, 0 < ε < ε, and b = εv 0 B = M ) { v (sup s (n)} (r 1) ε, (30) n S { n S n max { log(v 1 } } b) log(ε ε), 1, = 1,..., M, log r { where ε = mn n S, I(n) s (n)(1 r ) s (n)(1 r 1 ) }. (31) 10

11 Proof. Frst we show that V (n) s well defned. For any = 1,..., M, snce R s rreducble, there exsts at least one n 0 S for whch I(n 0 ) and s (n 0) > 0. Therefore, nf n S: I(n) { s (n) s (n) } s (n 0) s (n 0) <. (32) Due to negatve drft of R, t can be verfed that { s nf (n) } n S: I(n) s (n) > 1. (33) Thus, there exsts r satsfyng (28) and V (n) s well defned. Moreover, V (n) 1 snce v 0 1 and v 0. Next, for geometrc ergodcty, t s suffcent to verfy that (13) holds. We have p c(n),u V (n u) V (n) = ( ) M M v 0 v r n u v 0 v r n Snce p c(n),u M usng (22) we get p c(n),u = v r n u = p c(n),u V (n u) V (n) = M = p c(n),u M ( v s (n)rn 1 s (n)r n M I(n) M v ( s (n) s (n) s (n)) r n ( s (n)(r 1) s (n)(r 1 v r n u M ( v s (n)rn 1 s (n)r n ) s (n)rn 1 1) ) r n / I(n) v r n. (34) ) s (n)rn 1, v ( s (n)(r 1) ). (35) (36) (37) 11

12 Note that r n Thus, I(n) vanshes from the second term n (37) snce n = 0 and r n = 1. p c(n),u V (n u) V (n) εv (n) = ( v s (n)(r 1) s (n)(r 1 1) ε ) r n I(n) εv 0 / I(n) v ( s (n)(r 1) ε ) v ( ε ε) r n b, (38) where we use the defnton of ε and b for the nequalty. Snce 1 < r < s (n)/s (n), t can be checked that as a functon of r, s (n)(1 r ) s (n)(1 r 1 ) > 0 for any I(n). Hence, ε > 0 and ε ε < 0. Thus, the frst term n (38) s decreasng n n whle the second term s a constant n n. From the defnton of B t can be checked that for any n / B there exsts at least an 0 I(n) for whch Therefore, (13) holds. v 0 ( ε ε) r n 0 0 b 0. (39) Next we use the results of Lemmas 3.7, 4.2 and 4.3 to obtan a geometrc boundng functon on the bas terms, whch s one of the man results of ths chapter. Theorem 4.4. Suppose that random walk R s rreducble, aperodc, postve recurrent and has negatve drft. Then, for any C-lnear F : S [0, ), D t u(n) (1 γ)ρ(1 ρ) 1 (ρ ϑ) 1 [V (n) V (n u)], (40) 12

13 for any ρ > ϑ = 1 M 1 B, where V (n) = f 0 M 1 < r j < nf ε = n S: I(n) nf n S, I(n) b = εf 0 M f r n, f 0 = max { s (n) s (n) }, {f k,0, 1}, f = max k {f k,, 1}, k K log r r log r { s (n)(1 r ) s (n)(1 r 1 ) }, 0 < ε < ε, f ( sup n S ) { s (n)} (r 1) ε, M B = (1 λ) [1 2 λ ˆb ˆb 2 η(ˆb(1 λ) ˆb ] 2 ), δ = mn n B u N(n): nu B p c(n),u, v B = f 0 Mb ε ε, η = δ 5 (4 δ 2 )ε 2 b 2, γ = δ 2 [4b 2δ(1 ε)v B ], λ = (1 ε γ)/(1 γ), ˆb = vb γ, Proof. In the proof we only need to show that F V 1. Let f 0 = max k K {f k,0, 1}, f = max k K {f k,, 1}, = 1,..., M. (41) Then f > 0, for all {1,..., M}. Moreover, t can be verfed that { } f x f max = f x>0 r x log r r log r. (42) Hence, for any = 1,..., M, f n f r n and then F V 1. Moreover, snce B {0} and R has negatve drft, δ = mn n B u N(n): nu B p c(n),u > 0. (43) The result follows mmedately from Lemmas 3.7, 4.2 and 4.3. We have obtaned a geometrc boundng functon on the bas terms that can be computed based on the parameters of the random walk. However, as wll be seen n Secton 6, these bounds are often far from tght. Thus, we can not obtan reasonable error bounds on F usng these bounds. Therefore, next we show that a random walk wth negatve drft s also µ-ergodc. Thus, a quadratc boundng functon for the bas terms exsts. 13

14 4.3 Quadratc bounds on the bas terms In ths secton, we follow the same steps as the prevous secton. Frst, n the next lemma we show that a random walk wth negatve drft s µ-ergodc for any lnear functon µ. Lemma 4.5. Suppose that random walk R s rreducble, aperodc, postve recurrent and has negatve drft. Then, for any functon µ(n) = µ 0 M µ n, wth µ 0 1 and µ 1,..., µ M 0, R s µ-ergodc. More precsely, (14) holds wth V (n) = M v n 2, v = µ sup n S: I(n) { s M µ = max {µ }, b = µ 0 v, =0,...,M { B = n S n µ 0 } M v. µ (n) s (n)}, Proof. Ths proof follows the same approach as the proof of Lemma 4.3. Thus, some ntermedate verfcaton steps are omtted for brevty. Frst, we show that V (n) s well defned. Snce R has negatve drft, 0 < v <. Moreover, for = 1,..., M, there exsts at least one state n o for whch I(n 0 ). Then, v µ s (n 0) s (n 0) 0. (44) Therefore, v 0 for = 1,..., M and V (n) s non-negatve. Next, we verfy that (14) holds. Pluggng n the expresson for V (n), we have p c(n),u V (n u) V (n) = M p c(n),u v (n u ) 2 M v n 2. (45) 14

15 Usng (22), we have p c(n),u V (n u) V (n) = [ s (n)v (2n 1) s (n)v ( 2n 1) ] s (n)v. (46) I(n) Rewrtng the RHS gves that p c(n),u V (n u) V (n) = [ 2v s (n) s (n)] n I(n) I(n) 2µ n I(n) / I(n) v [ s (n) s (n)] / I(n) s (n)v M v, (47) where the nequalty follows from the defnton of v. Hence, p c(n),u V (n u) V (n) µ(n) µ n µ 0 I(n) M v, (48) Observe that RHS has lnearly decreasng terms n n and one constant part. Moreover, µ 1 from ts defnton. Therefore, t can be verfed that (14) holds for the specfed B and R s µ-ergodc. In the followng theorem we show that the bas terms can be bounded by a quadratc functon, whch s the other man result of ths secton. Theorem 4.6. Suppose that random walk R s rreducble, aperodc, postve recurrent and has negatve drft. Then, for any C-lnear F : S [0, ), there exsts b 0 < such that D t u(n) V (n) V (n u) b 0, where V (n) = f = v = M v n 2, (49) max k=1,...,k max k,, 1}, =0,...,M (50) f { sup n S: I(n) s (n) s (n)}. (51) 15

16 Proof. Let and take f 0 = max {f k,0}, f = max {f k,}, = 1,..., M, k=1,...,k k=1,...,k µ(n) = max{1, f 0 } M f n. Thus, t s clear that µ(n) 1 and F µ 1. The result follows mmedately from Lemmas 3.8, 4.2 and 4.5. From Theorems 4.4 and 4.6, we see that for a random walk wth negatve drft the bas terms can be bounded by a quadratc functon and by a geometrc functon. In general, the quadratc boundng functon s much tghter than the geometrc boundng functon. However, the constant b 0 of the quadratc functon s not known explctly. 5 Lnear programmng for error bounds based on quadratc bounds on the bas terms As s seen from Secton 4.3, the bas terms can be bounded by quadratc functons yet the constant b 0 s not known. In ths secton, we use deas from [6] and [1] to formulate a lnear program that gves bounds on F based on quadratc bounds on Du(n), t ncludng an explct value for the constant b 0. The dfference s that n ths paper we consder quadratc bounds on the bas terms whle n [6] and [1] lnear bounds are consdered. 5.1 Optmzaton problem for upper bound on F In ths secton, we revew some results from [1]. In partcular, we formulate the optmzaton problem for obtanng an upper bound on F. From the result of Theorem 3.1, the followng optmzaton problem comes up naturally to provde an upper bound on F. In the problem, the varables are F (n), G(n), Du(n) t and the parameters are π(n), F (n), p c(n),u and p c(n),u. 16

17 Problem 1 (Upper bound). mn [ ] F (n) G(n) π(n), n S s.t. F (n) F (n) ) ( pc(n),u p c(n),u D t u (n) G(n), n S, t 0, F (n) 0, G(n) 0, n S. Smlarly, replacng the objectve functon wth max [ ] n S F (n) G(n) π(n) we can obtan a lower bound on F, but we omt the detals here. Consder functons A u : S [0, ) and B u : S [0, ) for u N c(n), for whch (52) A u (n) D t u(n) B u (n), (53) for all t 0. Then, n Problem 1, replacng Du(n) t wth the boundng functons, we get rd of the tme-dependent terms and obtan the followng constrants that guarantee (52), F (n) F (n) max {( p ) c(n),u p c(n),u Bu (n), ( p ) c(n),u p c(n),u Au (n) } G(n), (54) F (n) F (n) max {( p ) c(n),u p c(n),u Au (n), ( p ) c(n),u p c(n),u Bu (n) } G(n). (55) Besdes the constrants gven above, addtonal constrants are necessary to guarantee that (53) holds. Before we present the constrants, we consder a refnement of the partton C. Snce we consder a C-lnear functon F, t s of nterest to consder A u and B u that have component-wse propertes. Then, partton C s not good enough n the sense that for dfferent n C k, nu can be located n dfferent components for any fxed u N k. Therefore, we defne a refnement of partton C as follows. Defnton 5.1. Gven a fnte partton C, Z = {Z j } j J s called a refnement of C f 1. Z s a fnte partton of S. 17

18 2. For any j J, any n Z j and any u N j, c(n u) depends only on j and u,.e., c(n u) = c(n u), n, n Z j. (56) Remark that the refnement of C s not unque. To gve more ntuton, n the followng example we gve a refnement of C that s gven n Example 2.2. Example 5.2. In ths example, consder the partton C gven n Example 2.2. A refnement of C s shown n Fgure 2. n 2 Z 13 Z 14 Z 15 Z 16 Z 17 Z 18 Z 7 Z 8 Z 9 Z 10 Z 11 Z 12 Z 1 Z 2 Z 3 Z 4 Z 5 Z 6 n 1 Fgure 2: A refnement of C as consdered n Example 2.2. In [1], t s shown that there exst constants φ z(n),u,d,v 0 such that Du t1 (n) = F (n u) F (n) φ z(n),u,d,v Dv(n t d). v N c(nu) d N c(n) un c(nu) (57) Moreover, a lnear program s formulated to obtan φ z(n),u,d,v. Then, the followng nequaltes are suffcent condtons for A u (n) and B u (n) to be a lower and upper bound on Du(n), t respectvely, F (n u) F (n) φ z(n),u,d,v B v (n d) B u (n), v N c(nu) F (n u) F (n) d N c(n) un c(nu) d N c(n) un c(nu) (58) v N c(nu) φ z(n),u,d,v A v (n d) A u (n). (59) 18

19 Therefore, summarzng the dscusson above, the followng problem gves an upper bound on F. In the problem, the varables are F (n), G(n), A u (n), B u (n) and the parameters are π(n), F (n), p c(n),u, p c(n),u, φ z(n),u,d,v. Problem 2. mn n S [ F (n) G(n) ] π(n), s.t. F (n) F (n) max {( p c(n),u p c(n),u ) Bu (n), ( p c(n),u p c(n),u ) Au (n) } G(n), (60) F (n) F (n) max {( p ) c(n),u p c(n),u Au (n), ( p ) c(n),u p c(n),u Bu (n) } G(n), (61) F (n u) F (n) φ z(n),u,d,v B v (n d) B u (n), v N c(nu) F (n) F (n u) d N c(n) un c(nu) d N c(n) un c(nu) (62) v N c(nu) φ z(n),u,d,v A v (n d) A u (n), A u (n) 0, B u (n) 0, F (n) 0, G(n) 0, for n S, u N c(n). (63) The problem has countably nfnte varables and constrants. In the next secton, we wll show that by consderng component-wse quadratc A u and B u, we can formulate a lnear program whose feasble set s a subset of the feasble set of Problem Lnear program for upper bounds on F Consder A u (n) and B u (n) that are component-wse quadratc wth respect to partton C,.e., ( K M ( ) ) A u (n) = 1 (n C k ) a c(n),u,0 ac(n),u, n α c(n),u, n 2, (64) B u (n) = k=1 ( K 1 (n C k ) b c(n),u,0 k=1 M ( ) ) bc(n),u, n β c(n),u, n 2. (65) Let Z = {Z j } j J be a refnement of partton C. Moreover, we consder a Z n whch all the bounded components have only one state. For nstance, the 19

20 partton n Example 5.2 s not such a partton snce Z 3 and Z 9 have two states. The reason for consderng ths type of refnement wll be dscussed after presentng Lemma 5.3. For any Z j and = 1,..., M defne L j,, U j,, I(Z j ) and Z j as L j, = mn n Z j n, U j, = sup n Z j n, (66) I(Z j ) = { = 1,..., M U j, = }, (67) Z j = {n Z j n = L j,, I(Z j ), n k {L j,k, U j,k }, k / I(Z j )} (68) Intutvely, I(Z j ) contans the dmenson n whch Z j s unbounded. Then, for any Z j, f / I(Z j ), then L j, = U j,. Hence, Z j = {n Z j n = L j,, = 1,..., M}. (69) Let c(j, u) denote the ndex of the component n partton C that n u s located n for any n Z j, and N j,u = N j ( u N c(j,u) ). (70) Next, we show that by restrctng our attenton to a C-lnear F and componentwse quadratc G, A u and B u, we can formulate a lnear problem wth a fnte number of varables and constrants, whose feasble set s a subset of the feasble set of Problem 2. We refer to the problem as the restrcted problem. Snce A u and B u are component-wse quadratc wth respect to partton C, we can verfy that the constrants n Problem 2 have the form H(n) 0 where H(n) s component-wse quadratc wth respect to partton Z. Therefore, we present the followng lemma, n whch we wrte suffcent condtons for H(n) 0 n terms of the coeffcents of H. Lemma 5.3. Suppose that H(n) = ( 1 (n Z j ) h z(n),0 j J M ( ) ) hz(n), n η z(n), n 2 (71) Then, H(n) 0 for all n Z j f η j, 0, 2L j, η j, h j, 0, I(Z j ), (72) H(n) 0, n Z j. (73) Proof. We can check that the constrans n (72) ensures that h z(n), n η z(n), n 2 s monotoncally decreasng n n on [L j,, ) for all I(Z j ). Therefore, H(n) 0 at all the corners n guarantees that H(n) 0 for all n Z j. 20

21 Suppose that a bounded component Z j has more than one state,.e., there exsts {1,..., M} such that L j, < M j,. Then, to obtan suffcent constrants such that H(n) 0 for all n Z j, we need to fnd suffcent constrants on h z(n),, η z(n), such that h z(n), n η z(n), n 2 0 for n [L j,, M j, ]. We can requre h z(n), n η z(n), n 2 to be monotoncally decreasng and nonpostve at n = L j,. However, these constrants are often too strong. Therefore, we suppose that Z j has only one state and hence H(n) 0 for all n Z j can be reduced to only one constrant. It s easy to check that the objectve functon of Problem 2 can be reduced to a lnear functon n terms of the coeffcents of F and G when π s productform. Therefore, we gve the man result of ths secton n the followng theorem. Theorem 5.4. Suppose that F s C-lnear and G, A u and B u are componentwse quadratc wth respect to partton C. Then, a restrcted lnear problem for Problem 2 can be formulated, whch provdes an upper bound on F and has a fnte number of varables and constrants. Proof. From Lemma 5.3, we see that a restrcted problem can be formulated, n whch the constrants and objectve functon are all lnear n the varables,.e., the coeffcents of F, F, G, Au and B u. Moreover, the feasble set of the restrcted problem s a subset of the feasble set of Problem 2. Hence, the restrcted problem gves an upper bound on F. Next, we show that the restrcted problem has a fnte number of varables and constrants. Snce F s C-lnear, G, A u and B u are component-wse quadratc wth respect to C, the total number of coeffcents s at most 2 K (3 M 1)(2M1). Moreover, for each component Z j, there s only one state n Z j and at most M unbounded dmensons. Hence, for each constrant n Problem 2, we formulate at most 2 J (3 M 1)(2M 1) constrants n the restrcted problem. Thus, the number of constrants s fnte. In ths secton, we have formulated a lnear program to obtan the error bound based on quadratc A u and B u. Comparng the result n ths secton and that n [1], we see that f we consder F, G, A u and B u to be C-lnear, we can formulate suffcent and necessary condtons for the constrants n Problem 2. If we consder the functons to be component-wse quadratc, we can only formulate suffcent condtons for the constrants. A fnal remark s that the restrcted lnear program we formulate does not requre that R has negatve drft. 21

22 6 Numercal experments From the results of Sectons 4 and 5, for a random walk wth negatve drft, on one hand, the bas terms are bounded by explct geometrc functons. On the other hand, quadratc bounds exst but we can only obtan them numercally n some cases. In ths secton, we mplement the lnear program based on quadratc bounds n Python and consder two examples. We compare the obtaned bounds on F based on lnear, quadratc and geometrc bounds on the bas terms. 6.1 Two-node tandem system wth boundary speed-up or slow-down In the frst example, we consder a random walk wth negatve drft and compare bounds on F obtaned through geometrc and quadratc bounds on the bas terms. Consder a tandem system contanng two nodes. Every job arrves at Node 1 and then goes to Node 2 to complete ts servce. In every node jobs are served by the Frst-In-Frst-Out dscplne. In the end, a job leaves the system through Node 2. Let λ denote the arrval rate. Suppose that Node 1 and Node 2 have servce rates µ 1 and µ 2, respectvely. For Node 1, the servce rate changes to µ 1 f Node 2 becomes empty. Remark that the job n the server s also ncluded for the number of jobs n a node. The dagram of the system s gven n Fgure 3. λ µ 1 (µ 1) µ 2 Fgure 3: Dagram of the tandem queueng system In ths example, we have S = {0, 1,... } 2 wth the partton C 1 = {1, 2,... } {0}, C 2 = {0} {1, 2,... }, C 2 = {0} {0} and C 1 = {1, 2,... } {1, 2,... }. Note that the tandem system we use s a contnuous-tme system. Then, the unformzaton method ntroduced n [7] can be used to transfer the contnuous-tme tandem system nto a dscrete-tme R. Wthout loss of generalty, assume that λ max{µ 1, µ } µ 2 1 and we use the unformzaton constant γ = 1. Thus, a dscrete-tme random walk R can be obtaned wth 22

23 the non-zero transton probabltes p k,(1,0) = λ, p k,(0, 1) = 1 ((0, 1) N k ) µ 2, k = 1, 2, 3, 4, { µ, f k = 1, p k,( 1,1) = µ 1, f k = 4, p k,(0,0) = 1 p k,u. u N k :u (0,0) For the stablty of the system, assume that λ/µ < 1 for = 1, 2. Assume that λ < µ and µ 1 < µ 2. Thus, R has negatve drft. The perturbed random walk The non-zero transton probabltes of the perturbed random walk are p k,(1,0) = λ, p k,(0, 1) = 1 ((0, 1) N k ) µ 2, p k,( 1,1) = 1 (( 1, 1) N k ) µ 1, Hence, p k,(0,0) = 1 u N k :u (0,0) π(n) = (1 ρ 1 )(1 ρ 2 )ρ n 1 1 ρ n 2 2, p k,u, k = 1, 2, 3, 4. where ρ = λ/µ for = 1, 2. In Fgure 4, the transton structures of the orgnal and the perturbed random walks are shown. The perturbed transton s marked wth dashed lnes. n 2 n 2 µ 1 µ 1 µ 2 λ µ λ µ 2 λ µ λ λ µ 1 λ n 1 λ µ 1 λ n 1 (a) Transton structure of R (b) Transton structure of R Fgure 4: Transton structures of the orgnal and perturbed random walks. Let F (n) = n 1. Take µ 1 = µ 2 = 2µ 1 and consder varous values for λ/µ 1. 23

24 For the explct geometrc bounds on the bas terms, we use the result n Theorem 4.4 and we have 1 < r 1 < max {µ 1/λ, µ 1 /λ}, 1 < r 2 < {µ 2 /µ 1 }, f 1 = f 2 = 1/2, ε = mn { λ(1 r 1 ) µ 1 (1 r 1 1 ), λ(1 r 1) µ 1(1 r 1 1 ), µ 1(1 r 2 ) µ 2 (1 r 1 2 )}, V (n) = rn 1 1 rn 2 2 2, B = {(0, 0), (1, 0), (0, 1), (1, 1)}, b = λ/2(r 1 1) ε. Unfortunately, no matter how we choose r 1, r 2 and ε, we get the bounds D t u (n) C [V (n) V (n u)], where C Thus, the resultng error bound s too large to gve meanngful nformaton. For the quadratc boundng functon on the bas terms, we mplement the restrcted lnear program gven n Secton 5 to obtan upper and lower bounds on F. In Fgure 5, these bounds are shown for varous λ/µ 1. We use F u (q) and F (q) l to denote the upper and lower bounds gven by the restrcted problem, respectvely. Moreover, we nclude upper and lower bounds gven by the lnear program from [1] by consderng lnear bounds on Du(n), t denoted by F u and F l respectvely. Moreover, snce n ths case µ 1 µ 1, usng the comparson result n Theorem 3.2 we see that F provdes an upper bound for F. On the contrary, f µ 1 < µ 1 then F provdes a lower bound for F. The comparson upper or lower bound s denoted by F u (c) or F (c) l F (q) u F (q) l F (c) u F u F l F λ/µ 1 Fgure 5: Bounds on F for varous λ/µ 1 : F (n) = n 1, µ 1 = µ 2 = 2µ 1. 24

25 From Fgure 5, we see that the upper bounds are always F. For the lower bounds, when the λ/µ 1 0.6, consderng quadratc bounds or lnear bounds on Du(n) t does not affect much to the lower bound. However, when λ/µ 1 > 0.6, by consderng quadratc bounds on the bas terms, we can sgnfcantly mprove the lower bound on F. Next, fx λ/µ 1 = 0.8, µ 2 = 2µ 1 and let µ 1 = η µ 1. In Fgure 6, we show the upper and lower bounds on F for varous η. F F (q) u F (q) l F (c) u F (c) l F u F l Fgure 6: Bounds on F for varous η: F (n) = n 1, λ/µ 1 = 0.8, µ 2 = 2µ 1. We see that the smaller perturbaton we make, the tghter upper and lower bounds we obtan. When η > 1, all the upper bounds are consstent. Snce λ/µ 1 > 0.6, we observe that when η 1, the lower bound gets tghter by consderng quadratc bounds on the bas terms. 6.2 Three-node tandem system wth boundary speedup Next, we consder the three-node tandem system that has been consdered n [1] and we see that by consderng quadratc bounds on the bas terms, bounds on F can be obtaned for more cases. The three-node tandem system does not have negatve drft. Thus, we show that the numercal program s also applcable when R does not have negatve drft. Consder the tandem system where every job arrves at Node 1 and goes through all the nodes to complete ts servce. In the end, a job leaves the system through Node 3. Assume that the arrval rate s λ. Moreover, we η 25

26 assume that each server follows the Frst-In-Frst-Out dscplne and has the servce rates µ when there are jobs n the queues. For Server 1, the servce rate changes to µ f both Queue 2 and Queue 3 become empty. The orgnal random walk In ths example, we have S = {0, 1,... } 3. Notce that the tandem system descrbed above s a contnuous-tme system. Therefore, we use the unformzaton method to transform the contnuous-tme tandem system nto a dscretetme R. Wthout loss of generalty, assume that λ max{µ, µ } 2µ 1. Hence, we take the unformzaton constant 1. Then, the transton probabltes of the dscrete-tme R are gven below. P (n, n e 1 ) = λ, P (n, n d 2 ) = 1 (n d 2 S) µ, (74) P (n, n e 3 ) = 1 (n e 3 S) µ, (75) { µ, f n 2 = n 3 = 0, P (n, n d 1 ) = (76) µ, otherwse, P (n, n) = 1 P (n, n u), (77) u {e 1,d 1,d 2,d 3 } for all n S, wth e 1 = (1, 0, 0), d 1 = ( 1, 1, 0), d 2 = (0, 1, 1) and e 3 = (0, 0, 1). The perturbed random walk For the perturbed random walks R, we take P (n, n e 1 ) = λ, P (n, n d2 ) = 1 (n d 2 S) µ, (78) P (n, n e 3 ) = 1 (n e 3 S) µ, P (n, n d1 ) = 1 (n d 1 S) µ, (79) P (n, n) = 1 P (n, n u). (80) u {e 1,d 1,d 2,d 3 } We know from [8] that the statonary dstrbuton of R s, π(n) = (1 ρ) 3 ρ n 1n 2 n 3, (81) where ρ = λ/µ. Let F (n) = n 1 and let µ = 1.5µ. We consder varous values for λ/µ. In Fgure 7, we plot the upper and lower bounds for F. The bounds obtaned by consderng quadratc bounds on the bas terms are denoted by F u (q) and 26

27 6 F (q) u F (q) l F u F l 4 F Fgure 7: Bounds on F for varous λ/µ: F (n) = n 1, µ 1 = 1.5µ. λ/µ F (q) l. We also nclude the upper and lower bounds obtaned n [1], whch are denoted by F u and F l respectvely. In Fgure 7, we see that F (q) u and F (q) l are slghtly better than F u and F l. Moreover, by allowng quadratc bounds on the bas terms, results can be obtaned for cases where the lnear problem n [1] s nfeasble. However, for random walks wth very heavy load, the lnear program based on quadratc bounds on the bas terms becomes nfeasble as well. 7 Conclusons In ths paper, we have dscussed bounds on the bas terms. Moreover, we have shown that f functon F s C-lnear and the random walk has negatve drft, a geometrc boundng functon as well as a quadratc boundng functon can be found. The geometrc boundng functon has an explct expresson, but t often provdes bounds that are far from tght. The quadratc boundng functon s relatvely tght. Nevertheless, we are not able to obtan a closedform expresson n general. We have formulated a lnear program to fnd bounds on the statonary performance based on quadratc bounds on the bas terms. Indeed, the lnear program can provde qute tght bounds on the statonary performance, even when the random walk does not have negatve drft. However, the lnear problem s only feasble n some cases. Therefore, one drecton for future research s to explore technques that enable us to apply the lnear program to more general cases. 27

28 We see from numercal results that when the load of the system s heavy and the lnear program s feasble, the bounds on F obtaned based on quadratc bounds can be tghter than those based on lnear bounds. Thus, t s promsng that by consderng polynomal bounds of hgher order, we can mprove the bounds on F. It wll be of nterest to develop mplementaton technques for obtanng performance bounds based on hgher-order polynomal bounds on the bas terms. References [1] X. Ba and J. Goselng. A lnear programmng approach to markov reward error bounds for queueng networks. arxv preprnt arxv: , [2] N. Bayer and R. J. Bouchere. On the structure of the space of geometrc product-form models. Probablty n the Engneerng and Informatonal Scences, 16(2): , [3] R. J. Bouchere and N. M. van Djk. Monotoncty and error bounds for networks of erlang loss queues. Queueng Systems, 62(1-2): , [4] Y. Chen, R. J. Bouchere, and J. Goselng. The nvarant measure of random walks n the quarter-plane: representaton n geometrc terms. Probablty n the Engneerng and Informatonal Scences, 29(2): , [5] Y. Chen, R. J. Bouchere, and J. Goselng. Invarant measures and error bounds for random walks n the quarter-plane based on sums of geometrc terms. Queueng Systems, 84(1-2):21 48, [6] J. Goselng, R. J. Bouchere, and J. C. W. van Ommeren. A lnear programmng approach to error bounds for random walks n the quarterplane. Kybernetka, 52(5): , [7] W. K. Grassmann. Transent solutons n markovan queueng systems. Computers & Operatons Research, 4(1):47 53, [8] J. R. Jackson. Networks of watng lnes. Operatons Research, 5(4): ,

29 [9] S. P. Meyn and R. L. Tweede. Computable bounds for geometrc convergence rates of markov chans. The Annals of Appled Probablty, pages , [10] S. P. Meyn and R. L. Tweede. Markov Chans and Stochastc Stablty. Sprnger Scence & Busness Meda, [11] N. M. van Djk. Perturbaton theory for unbounded markov reward processes wth applcatons to queueng. Advances n Appled Probablty, 20(1):99 111, [12] N. M. van Djk. Bounds and error bounds for queueng networks. Annals of Operatons Research, 79: , [13] N. M. van Djk. Error bounds and comparson results: The markov reward approach for queueng networks. In R. J. Bouchere and N. M. van Djk, edtors, Queueng Networks: A Fundamental Approach, volume 154 of Internatonal Seres n Operatons Research & Management Scence. Sprnger, [14] N. M. van Djk and B. F. Lamond. Smple bounds for fnte sngle-server exponental tandem queues. Operatons Research, pages , [15] N. M. van Djk and M. Myazawa. Error bounds for perturbng nonexponental queues. Mathematcs of Operatons Research, 29(3): , [16] N. M. van Djk and J. van der Wal. Smple bounds and monotoncty results for fnte mult-server exponental tandem queues. Queueng Systems, 4(1):1 15,