Stochastic Optimal Controls for Parallel-Server Channels with Zero Waiting Buffer Capacity and Multi-Class Customers

Transcription

1 Advances n Systems Scence and Applcatons (009), Vol.9, No Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng Buffer Capacty and Mult-Class Customers Wanyang Da and Youy Feng Department of Mathematcs and State Key Laboratory for Novel Software Technology, Nanjng Unversty, Nanjng 0093, Chna. Emal: nan5lu8@netra.nju.edu.cn Department of Systems Engneerng and Engneerng Management, The Chnese Unversty of Hong Kong, Shatn, NT, Hong Kong Abstract We study the stochastc optmal admsson and routng controls for a system consstng of two parallel-server channels wth zero watng buffer capacty and mult-class customers va Markov decson processes movng n dscrete trangles. In each channel, there are two classes of customers to be served, whch are varyng n ther arrval rates, payments and penalty costs. The class of structured value functons s found and justfed to guarantee the exstence of optmal value functon and optmal admsson control polcy. The polcy for each class of customers s characterzed by a state-dependent threshold functon that s non-ncreasng n the number of the other class of customers beng served n the channel and decreases at most one unt when the number of the other class of customers ncreases a unt. Between channels, some classes of customers can be selected to be served n ether of the channels accordng to certan routng probabltes, whch are obtaned by combnng the optmal value functons wth smulaton. Numercal and applcaton examples are presented to llustrate the usage and effcency of our algorthm and optmal polcy. Keywords Dynamc programmng Optmal control Markov decson process State-dependent threshold polcy Queue Optmzaton Parallel-server system Zero watng buffer capacty Mult-class customer. Introducton Parallel-server systems have become a common tool n the studes of many communcaton networks, manufacturng and servce systems, see, for examples, Da [4] and Gans et al. [7]. Motvated from some practcal communcaton project, n ths paper, we consder a system that conssts of two parallel-server channels wth zero watng buffer capacty, whch dffers from wdely studed parallel-server systems wth nfnte watng buffer capacty (see, e.g., Harrson [8], Bell and Wllams [] ). In each channel, there are two classes of customers to be served, whch are varyng n ther arrval rates, payments and penalty costs. Payments are random and depend on the lengths of servce tmes, whch are dfferent from assumptons n exstng studes such as Ormec et al [] and Ormec and Wal [3], where constant payments are assumed. Between channels, some classes of customers can be selected to be served n ether of the channels accordng to certan routng probabltes. Snce the watng buffer capacty s zero for both channels, optmal admsson and routng controls for such a system are of partcular nterests. The purpose of the controller s to determne how the system should route an ncomng customer to a partcular channel and whether the channel should accept or reject the customer nto servce n order to maxmze the expected nfnte horzon dscounted proft for the servce provder. We wll restrct our study to Markovan control polces snce the optmal polcy belongs to ths class (see, e.g., Sennott [6] ). Some related dscussons about network optmzaton and control can be found n Cheng et al [3], Da and Jang [5], Dng et al [6] and Ma et al []. Optmal control problems for queueng systems arse n dfferent contexts such as Lppman [0]. The unfed approach developed n [0] allows one to transform a contnuous tme ISSN Internatonal Insttute for General Systems Studes, Inc.

2 68 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng Markov decson process nto an equvalent dscrete process and to establsh assocated optmalty equatons. Moreover, followng the approach, one can nvestgate the structure of an optmal control polcy by dentfyng a class of structured value functons that s preserved under certan sense and fnd a state-dependent threshold polcy that s easy to mplement, see, e.g., Benjaafar and ElHafs [], and etc. for a number of recent applcatons n producton and nventory controls. However, the classes of structured value functons are varyng for dfferent applcatons and they are dffcult to be dentfed for the correspondng decson processes movng n mult-dmensonal domans. In addton, the justfcaton of preservaton of propertes for each applcaton s complcated and nvolves heavy calculaton. In ths paper, frstly, we adopt the approach to study the optmal admsson control problem for the above parallel-server system n each channel. The class of structured value functons s found to satsfy the concave and certan submodular propertes and the preservaton of these propertes under some polcy related transformaton s justfed to guarantee the exstence of optmal value functon and optmal admsson control polcy. The obtaned optmal polcy for each class of customers s characterzed by a state-dependent threshold functon that s non-ncreasng n the number of the other class of customers beng served n the channel and decreases at most one unt when the number of the other class of customers ncreases a unt. Because the watng buffer capacty s zero and there are multple classes of customers n the system, our Markov decson process s confned to move wthn a -dmensonal dscrete trangle that s not a good shape to our study. Due to ths reason, our functon class s dfferent from those such as n [] and the justfcaton of the structure preservaton for our problem s partcular complex along the boundary of the trangle, moreover, due to the dfference of model formulatons, our admsson polcy and assocated analyss are dfferent from those as n [] and [3]. Secondly, by combnng the optmal value functons n dfferent channels wth smulaton, we obtan the optmal routng parameters. Thrdly, we mplement a number of numercal examples to support our theoretcal fndngs. From these examples, we see that our algorthm stably converges and s effcent, and furthermore, our polcy s better than some ntutvely reasonable polcy that s based on two -dmensonal Markov decson processes and s desgned for the purpose of comparson. The rest of the paper s organzed as follows. In Secton, we descrbe our model formulaton. In Secton 3, we derve the optmalty equatons and study the characterzaton of the optmal polcy. Numercal examples are gven n Secton 4. In Secton 5, we provde the techncal and lengthy proofs for three lemmas. Concludng remarks and future research are presented n Secton 6.. Model Formulaton We consder a communcaton system consstng of two channels as pctured n the followng Fgure. There are n ndependent and dentcal servers n Channel A and n servers n Channel A. Both channels have no addtonal bufferng capablty for an ncomng customer except beng served mmedately. There are two classes of external arrvng customers to Channel A and the arrval stream for each class {,} follows a Posson process wth rate λ. Smlarly, there are two classes of external arrvng customers to Channel A and the arrval stream for each class {,} follows a Posson process wth rate λ. For each {, K, m} wth some gven nteger m, the arrval streams correspondng to λ and λ are splt from a common Posson process wth rate β, that s, λ λ β t wll be dscussed later about how to determne the optmal values of + =, and moreover, λ and λ for a gven β accordng to routng decson. For each class {,}, an arrval customer to Channel A s

3 Advances n Systems Scence and Applcatons (009), Vol.9, No.4 69 ether rejected nto servce wth penalty cost l or accepted nto servce and experences exponentally dstrbuted amount of servce tme wth rate µ accordng to some admsson control polcy, where we assume that servce tmes among dfferent customers are ndependent. Smlarly, for each class {,}, an arrval customer to Channel A s ether rejected nto servce wth penalty cost l or accepted nto servce and experences exponentally dstrbuted amount of servce tme wth rate µ. Furthermore, for each {, K, m}, we suppose that l = l. Fgure A communcaton system consstng of two channels wth parallel servers s under admsson and routng controls. For each {,}, let X () t denote the number of class customers beng served n Channel A at tme t, whch takes values n the state space S = {0,, K, n}, and moreover, let X() t ( X (), t X ()) t, whch takes values n the followng -dmensonal dscrete trangle S ( x, x): x nx, S,,. () Let hx ( ()) t denote the payment functon defned on S, whch satsfes certan condtons that wll be elaborated later. Assocated wth Channel A, there s a fxed cost c per unt of tme, and let N () t denote the total number of rejected class customers at Channel A by tme t. Then we can wrte the value functon (expected nfnte horzon dscounted proft) for a gven dscount factor α, a gven ntal state x S and a gven admsson control polcy π as follows, π π ( ) t ( ()) t α α α v x E () t x e hxt dt e ldn t e cdt () Smlarly, for each {,}, let X () t denote the number of class customers beng served n Channel A at tme t, whch takes values n the state space S = {0,, K, n}, and

4 630 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng moreover, let X() t = ( X(), t X()) t, whch takes values n the followng -dmensonal dscrete trangle S ( x, x): x nx, S,,. (3) Let h( Xt ()) denote the payment functon defned on S. Assocated wth Channel A, there s a fxed cost c per unt of tme, and let N() t denote the total number of rejected class customers at Channel A by tme t. Then we can wrte the value functon for the gven dscount factor α, a gven ntal state x S and a gven admsson control polcy π as follows, π π αt αt αt v ( x) Ex e h( Xt ()) dt e ldn () t e cdt An admsson control polcy π (or π ) specfes at each tme nstant f an arrved. (4) customer to Channel A (or Channel A) s accepted nto servce or s rejected, and furthermore, we restrct our analyss to Markovan polces snce the optmal polcy belongs to ths class (e.g., Sennott [6] ). Our objectve s to look for optmal polces π and π such that π π π π v ( x, λ) = max v ( x, λ), v ( x, λ) = max v ( x, λ) (5) π Π where λ = ( λ, K, λm), λ = ( λ, K, λm) wth m {,}, and the sets Π and Π of Markov decson polces are assocated wth some acton spaces whch wll be elaborated later. Moreover, we are to fnd the optmal values λ and λ, for a gven vector β = ( β, K, β m ) such that ( λλ, ) Λ π Π ( ) π π π π v ( x, λ ) + v ( x, λ ) = max v ( x, λ) + v ( x, λ) (6) where the set Λ s defned to be Λ {( λλ, ): λ + λ = β, =, K, m}. Moreover, we remark that the determnatons of λ and λ are essentally the choces of the optmal routng probabltes to two channels for the common Posson nput streams correspondng to the rate vector β. 3. The Optmalty Equatons and Characterzaton of the Optmal Polcy π We frst focus our dscussons on Channel A. Let η ( x) be the control assocated wth a Markovan polcy π when the system s n state x S. It follows from Lemma 5. n Secton 5 that there s at most one customer arrvng at the channel at a tme pont. Thus we take π π π η ( x) = η ( x) when the arrved customer belongs to class, where η ( x) corresponds to the admsson acton such that π 0 f the acton s to accept the arrved class customer nto servce, η ( x) f the acton s to reject the arrved class customer nto servce. Thus the acton space can be taken as {0, } that s ndependent of state varable x. Let γ λ + nµ = sup λ( x, η( x)). (7) ( x, η( x)) S {0,} where η ( x) s an acton and λ( x, η ( x)) s the rate of changng of the system when t s n state x. Then by the studes n Lppman [0], we can transform the contnuous tme decson

5 Advances n Systems Scence and Applcatons (009), Vol.9, No.4 63 process nto an equvalent dscrete tme decson process. Moreover, defne v ( x) v ( x) and wthout loss of generalty, suppose that α + γ =, then we have the followng lemma. Lemma 3. For each x S, the followng optmalty equaton holds v ( x) = Tv ( x) (8) where the operator T s defned as follows, Tv( x) = ( hx ( ) c) + λ Tvx ( ) + µ xvx ( e) + n x vx ( ), e s the -dmensonal unt vector whose th component takes value one and all other components take values 0, and furthermore, max{ vx ( + e), vx ( ) l} f x, < n = Tvx ( ) vx ( ) l f x. = n = The proof of ths lemma can be found n Secton 5. To characterze the optmal polcy, for each {,}, we defne vx ( ) vx ( + e) vx ( ). (9) Next, suppose that v(x) s a functon satsfyng the followng condtons, that s, for each fxed {,} and a j {,}, vx ( ) vx ( + e) vx ( ) 0, (0) vx ( ) vx ( + e ) vx ( ) 0 for any j, () j j vx ( ) vx ( + e ) vx ( + e ) 0 for any j. () j j Remark 3. Condton (0) ndcates that v s non-ncreasng n the component x or equvalently v s concave n x ; Condton () states that v s submodular n the drecton of e and e j ; We call that v s submodular n the drecton of e and e -e j under condton () snce ths condton s equvalent to vx ( + e e ) vx ( ) 0 for 0 < x S. When x n j=, j j j =, the concavty property of v n terms of x s always true snce there are only two ponts nvolved; When one dmensonal problem s concerned, the only requred condton s the concavty. The followng examples are gven to show the exstence of the class of functons that satsfy condtons (0)-(). Example 3. vx ( ) = f ( ) x for x S, where f ( ) = x s a concave functon of x for each {,}. For examples, () v(x) s lnear n x, that s, vx ( ) = rx where r > 0 s a constant for each ; () vx ( ) = ( α ( )) βg x = where α and β are postve constants, rx g (x ) s convex n x, e.g., g( x) = e ± for some constant r > 0, or g( x) = ( x r) and etc. Example 3. Let v be a concave functon on R and defne f( x) x. Notce that f s an affne mappng from R to R (see, e.g., page 3 of Hrart-Urruty and Lemarechal [9] ). Then t s easy to check that v(f(x)) satsfes condton (). Moreover, the condtons (0) and () are reduced to the only concavty condton (0) and ths property can be easly shown by the smlar π

6 63 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng method used n Proposton..4 n page 88 of [9]. Here we remark that more such examples can be constructed, for example, by applyng Proposton..7 n page 88 of [9]. So, basng on the above dscussons, we can ntroduce the followng set of functons A {( vx): vx ( )satsfes thecondtons(0) (), x S}. (3) If assocated wth the value functon of a polcy, the condtons (0)-() mply some partcular propertes for that polcy. Concretely, for each {,} and each x S wth x n, let xˆ (, ) = xj j v A, we can defne =, moreover let x ˆ fxed and x vary, then for each a ( xˆ ) max{ x : vx ( ) + l > 0}. (4) From (4), we see that a ( ˆ x ) depends only on x ˆ as long as v and l s gven. In the sequel, we also use a (x) to denote a ( x ˆ) for convenence. Then we have the followng lemma. Lemma 3. If v A, then Proof. If x a ( xˆ ) f and only f vx ( ) + l > 0. x a ( xˆ ), then by the defnton of a ( x ˆ) n (4), we have that vx ( ) + l > 0. Conversely, let x= ( x, ˆ x) S satsfyng vx ( ) + l > 0 (here we take wthout loss of generalty). Due to the concavty property (0) of v, we have that vx ( ) + l vx ( ) + l > 0wth x = ( x, ˆ x) and x x. Then by the defnton gven n (4), we can conclude that x ( ˆ a x). Remark 3. Lemma 3. mples that the admsson control assocated wth a polcy π that has value functon v π A can be descrbed wth -dmensonal swtchng surfaces. More specfcally, the polcy can be expressed by state-dependent thresholds a (x) for all {,} such that t s optmal to accept a class customer nto servce f x a( x) (whch means that acceptng one s to get the bgger proft than rejectng one) and otherwse to reject. Lemma 3.3 For a fxed {,}, a j {,}( j ), a v A and a x S wth x < n, we have = a( x), a( x+ ej) a( x). Proof. Wthout loss of generalty, we take =. Then let x% = ( x, a ( x+ e )). Due to the j submodular property () of v n the drecton of e and e j and the defnton of a ( x+ e ), we have that vx (%) vx (% + e ) > l (5) j Thus, by (5) and the defnton of a( x% ), we can conclude that a ( x+ e ) = a ( x% + e ) a ( x% ) = a ( x). j j Secondly, f a (x) = 0, t s obvous that a( x+ ej) > a( x). If a( x), we take x% = ( x, a( x)), then by usng x% e to replace x n condton (), we have vx ( % e + e ) vx ( %) > l. Hence we have j j

7 Advances n Systems Scence and Applcatons (009), Vol.9, No a ( x) = a ( x %) a ( x % + e ) = a ( x+ e ). j j Therefore t follows from the above dscussons that the lemma s true. The followng lemma shows that T defned n Theorem 3. preserves the propertes of v and hence preserves the swtchng surface structure of the assocated control polcy. Lemma 3.4 If hv A,, then Tv A, where the operator T s defned n Proposton 3. and vx ( + e) f x, ( ), < nx a x Tvx ( ) vx ( ) l f x, ( ), < nx > a x = vx ( ) l f x. = n = The lengthy proof of the above lemma wll be gven n Secton 5. Instead, we present our man result as follows. Theorem 3. There s a unque optmal value functon v whch belongs to A f h A. Consequently, the optmal polcy s a threshold admsson control polcy wth state-dependent thresholds for each class. Specfcally, the polcy has the followng propertes.. For each xˆ = ( x, j ), there s a correspondng threshold a ( xˆ ) such that t s j optmal to accept a class customer nto servce f x a ( xˆ ) and to reject otherwse.. The threshold a ( xˆ ) s non-ncreasng n each of the varables x ( j ) wth a ( x) a ( x+ e ) a ( x) where one can use the same explanaton n (4) for a ( x). j Proof. It follows from Lemma 3.4, Theorem 5. of Porteus [4] and Theorem of Puterman [5] m that v = lm T v A and t s the unque soluton of v = Tv, where T m m refers to m compostons of operator T. Therefore, v satsfes condtons (0)-(). Hence the reman results of the theorem follow from Lemmas 3. and Numercal Examples In ths secton, we use numercal examples to llustrate the usage and effcency of our optmal threshold polcy. Frstly, we present several examples wth dfferent payment functons to show the dynamc evolvng of the optmal thresholds n terms of the number of the other class of customers n a channel. Secondly, we conduct the numercal comparson between our optmal value functon and another ntutvely reasonable one to exhbt the optmalty of our polcy. Thrdly, we obtan optmal routng parameters by ntegratng our optmal polcy wth smulaton. Fourthly, we extend our system to a more general one where dfferent classes of customers may have dfferent servce rates and we show that our optmal polcy s stll reasonably good by a numercal example. All of the numercal results gven here are obtaned by solvng the dynamc programs teratvely and the value teraton algorthm s run enough tmes n order that at least four-dgt accuracy s acheved for each problem nstance. Example 4. Here we take the payment functon h(x) to be a lnear functon of x, that s, for some constants q and q, h(x) = q x +q x. The correspondng numercal results about the evolvng of thresholds are descrbed n Fgure and Fgure 3. j

8 634 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng Fgure Optmal thresholds wth n = 9, α = 0., λ = 0.5, c = 000, q = 000 and q = 000. λ = 0.5, µ = 0.5/9, l = 00, l = 400, Fgure 3 Optmal thresholds wth n = 9, α = 0., λ = 0.03, λ = 0.37, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000 and q = 000. Example 4. For some constant q, we take the payment functon hx ( ) = q(500 6( x 4) 0( x ) ). The correspondng numercal results about the evolvng of thresholds are shown n Fgure 4.

9 Advances n Systems Scence and Applcatons (009), Vol.9, No Fgure 4 Optmal thresholds wth n = 9, α = 0., λ = 0.5, c = 000 and q = 000. Example 4.3 For some constants q, p and p, we take the payment functon λ = 0.5, µ = 0.5/9, l = 00, l = 400, px px hx ( ) = q( e e ). The correspondng numercal results about the evolvng of thresholds are dsplayed n Fgure 5. Fgure 5 Optmal thresholds wth n = 9, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, p = 0. and p = 0.3. Example 4.4 In ths example, frstly, we take the payment functon h(x) = q x + q x for some constants q and q and use our algorthm to obtan the values of the optmal value functon

10 636 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng defned n (8) at all possble ntal states. Secondly, we separate the servers nto two groups, each s assgned to serve only for a partcular customer class. Then we use two -dmensonal Markov decson processes on a fnte set to generate a -dmensonal optmal value functon. Concretely, suppose that there are b number of servers assgned for class one customers and (nb) number of servers assgned for class two customers. Then the teraton algorthms for the assocated -dmensonal Markov decson processes are gven by Tv( x) = ( h( x) c+ λ Tv ( x) + µ ( xv ( x ) + ( b x) v( x)) ) α + γ Tv( x) = ( h( x) c + λ Tv ( x) + µ ( xv ( x ) + ( n b x) v( x)) ) α + γ where γ = λ + bµ and γ = λ + ( n b) µ, h( x) = qx and h( x) = qx for some constants q and q, c = cb/n and c = c-c, moreover, v( x+ ) f x < bx, a, Tv ( x) = v( x) l f x < bx, > a, v( x) l f x = b, v( x + ) f x < n bx, a, Tv ( x) = v( x) l f x < bx, > a, v( x) l f x = n b and a = max{ x : v ( x ) + l > 0} wth v ( x ) = v ( x + ) v ( x ) for =,. Then we can obtan the value dfferences v ( x, x) ( v( x) + v( x)) for 0 x b and 0 x n b. In our numercal mplementaton, for any b {0,, K, n}, the correspondng value dfferences are postve and see for one partcular example gven n Fgure 6 where b= 3. Fgure 6 Comparson of value functons wth n = 9, b = 3,α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l

11 Advances n Systems Scence and Applcatons (009), Vol.9, No = 00, l = 400, c = 000, q = 000, q = 000. Example 4.5 In ths example, we consder two channels A and A. Note that all notatons related to Channel A wll be put a bar above the correspondng notatons for Channel A. The payment functons h(x) and h( x ) are taken to be lnear n terms of x and x respectvely,.e., hx ( ) = qx + qx and h( x) = qx + qx for some constants q, q, q and q. Basng on the teratve algorthm gven n Proposton 3. for Markov decson process, we add one more smulaton step to obtan the optmal routng parameters λ and λ that satsfy λ + λ = 0.3 and solve the optmzaton problem (6). In the smulaton, we take λ = 0.3( / N) for = 0,, K, N. Frstly, we handle the case that Channel A s confgured exactly the same as Channel A wth n = n = 6. Due to the symmetry property, the optmal routng parameters should concde for the ntal states (,,,) and (,,,) wth = 0,, K 4, whch are shown n Fgure 7 and Fgure 8. In Fgure 8, we see some dfference between optmal parameters at ponts (,,, ) and (,,, ). Ths phenomenon can be nterpreted as follows: when N takes odd number (for example N = 49), λ can never take the real optmal value 0.5, and moreover, due to the fact that λ+ λ = 0.3, the dfference appears. Secondly, we handle the asymmetry case wth n = 7 and n = 5. The assocated numercal results about the optmal routng parameters are gven n Fgure 9. Fgure 7 Optmal routng parameters wth n = 6, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000; n = 6, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000, N = 50.

12 638 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng Fgure 8 Optmal routng parameters wth n = 6, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000; n = 6, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000, N = 49. Fgure 9 Optmal routng parameters wth n = 7, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000; n = 5, α = 0., λ = 0.5, λ = 0.5, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000, N = 49. Example 4.6 In ths example, we extend the teratve algorthm presented n Proposton 3. to allow dfferent servce rates µ ( =,) for dfferent customer classes. Concretely, the new algorthm can be desgned as follows, Tv( x) = ( hx ( ) c) + λ Tvx ( ) + µ xvx ( e) + nmax{ µ, µ } µ x vx ( ) where the threshold polcy related to T v(x) for =, s the same as the one gven n

13 Advances n Systems Scence and Applcatons (009), Vol.9, No Proposton 3.4. In our numercal mplementaton, we see that the new algorthm stll converges and the lmtng thresholds keep the propertes as stated n Theorem 3., see for an example, px px numercal results n Fgure 0 where hx ( ) = q( e e ) for some constants q, p and p. Moreover, let v ( x, x) denote the lmtng value functon of our new algorthm. Then as n Example 4.4, we can obtan the value dfferences v ( x, x) ( v( x) + v( x)) for 0 x b and 0 x n b, and for any b {0,, K, n}, the correspondng value dfferences are postve as shown, for a partcular example, n Fgure. From these numercal results, we reach a reasonable conjecture that the lmtng value functon v ( x) s also an optmal value functon n certan functon class. Fgure 0 Optmal thresholds wth n = 9, α= 0., λ= 0.5, λ = 0.5, µ = 0.3/9, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, p = 0. and p = 0.3. Fgure : Comparson of value functons wth n = 9, b = 3, α= 0., λ= 0.5, λ = 0.5, µ = 0.3/9, µ = 0.5/9, l = 00, l = 400, c = 000, q = 000, q = 000.

14 640 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng 5. Justfcatons of Some Prevously Gven Lemmas 5. Lemma 5. and Its Proof Lemma 5. Let A () for each {,} be a Posson process wth rate λ and let I() denote the process satsfyng that I() t = f there s an arrval of class customer at the tme pont t and I() t = 0 otherwse. Moreover, let I () I (), then PIt {() > for allt [0, )} = 0. (6) Proof. Defne At () A () t for each t 0, then the superposed process A() s stll a Posson process wth rate λ = λ. Thus, due to the ndependent ncrement property and by usng the Posson dstrbuton, we have PIt {() > } PAt { ( + t) At () > } = o( t). Let t 0, we have that, for each t [0, ), PIt {() > } = 0. (7) Notce that [0, ) s an uncountable set, therefore we take a sequence of tme ponts as 0< t < t < L< tm < Lsatsfyng t m when m, then the assocated sequence of sets {() It > forall t [0, t m ]} decreases to the set {() It > forall t [0, )} as pq, q m. Moreover, for each t m, let tm = t p m for postve ntegers p and q that satsfy p q = 0,, K,. Then t follows from the rght contnuous property of I () and equaton (7) that PIt {() > forall t [0, )} pq, lm P U {( Itm ) } m p= pq, lm PIt {( m ) } m p = = lm PIt {() > forall t [0, t ]} m = > > = 0. Hence we complete the proof of the lemma. Remark 5. One can employ the same dea to prove that almost surely along any path, there s at most one customer who fnshes servce at a tme pont n [0, ) snce the servce tme dstrbutons are assumed to be exponental. 5. Proof of Lemma 3. The proof of the lemma s the evaluaton of equatons () and () n Lppman [0]. Frst, we calculate the followng expected α -dscounted reward earned durng one transton when startng from state x and choosng acton a(x) through formula () n [0]. Followng equaton (3) and ts expalnatons n [0], we substtute γ gven n equaton (7) nto equaton () n [0]. Then we have m

15 Advances n Systems Scence and Applcatons (009), Vol.9, No.4 64 t ατ γt { τ γ 0 0 } t ατ γt S{ τ γ 0 0 } S r ( xax, ( )) = e d(( hx ( ) c), t/ xax, ( ), x ) e dt dq ( x / xax, ( )) α = ( hx ( ) c) e d e dq( x / xax, ( )) ατ t γt = ( hx ( ) c) e e dt dq ( x / xax, ( )) S γ 0 α 0 ατ t γt = ( hx ( ) c) e γe dt 0 α 0 = ( hx ( ) c ) α + γ = hx ( ) c Next, notce that assocated wth a class customer s arrval, the transton probablty qx ( / xax, ( )) appeared n equaton (3) of [0] s λ / λ xa, ( x), and assocated wth a class customer s departure, the probablty s xµ / λ xa, ( x). Therefore, by substtutng γ n equaton (7), the above equaton (8) and equaton (3) n [0] nto equaton () n [0], we can conclude that the lemma s true. 5.3 Proof of Lemma 3.4 Step One. We prove the concave property (0) for Tv(x). In dong so, we frst show that Tvx ( ) 0 for all, j {,}. (9) j j As a matter of fact, for j =, we have vx ( + e) 0 f Σ x < n, x < a( x) ( vx ( + e) + l) < 0 f Σ x < n, x = a( x) vx ( + e) + l 0 f Σ x < n, x = a( x) Tvx ( ) vx ( ) 0 f Σ x < n, x > a( x) ( vx ( + e) + l) < 0 f Σ x = n, x a( x) vx ( + e) + l 0 f Σ x = n, x = a( x) vx ( ) 0 f Σ x = n, x > a( x). Then, t follows from Lemma 3.3 that a ( x) a ( x+ e ) a ( x) and a ( x+ e ) a ( x+ e ) a ( x+ e ). Thus, for j, we have j j j j j jvx ( + e) 0 f Σ x < n, x a( x) j jvx ( + e) ( vx ( + ej) + l) < 0 f Σ x < n, x = a( x), j jtvx ( ) a( x+ ej) = a( x) or a( x+ ej) = a( x), a( x+ ej) = a( x) (8)

16 64 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng j jvx ( + e) 0 f Σ x < n, x = a( x) a( x+ ej) = a( x) or a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jvx ( + e) + ( vx ( + ej) + l) 0 f Σ x < n, x = a( x), a( x+ ej) = a( x) j jvx ( + e) ( vx ( + ej) + l) < 0 f Σ x < n, x = a( x), a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jtvx ( ) j jvx ( + e) 0 f Σ x < n, x = a( x), a( x) = a( x+ ej) = a( x+ ej) j jvx ( ) 0 f Σ x < n, x > a( x), a( x) a( x+ ej) a( x+ ej) j jvx ( + e) ( vx ( + ej) + l) < 0 f Σ x = n, x a( x) j jvx ( + e) + ( vx ( + ej) + l) < 0 f Σ x = n, x = a( x), a( x+ ej) = a( x) j jvx ( + e) ( vx ( + ej) + l) < 0 f Σ x = n, x = a( x), a( x) = a( x+ ej) j jvx ( ) 0 f Σ x = n, x > a( x). Step Two. We prove submodular property () for Tv(x) by showng the followng clam, Tvx ( ) = Tvx ( ) 0 for j (0) j j As a matter of fact, j vx ( + e) 0 f x < n, x < a( x) ( vx ( + e) + l) < 0 f x < n, x = a( x), a( x+ e) = a( x) j vx ( + e) 0 f x < n, x = a( x) a( x+ ej) = a( x) j Tvx ( ) vx ( + ej) + l 0 f x < n, x = a( x), a( x+ ej) = a( x) 0 f x < n, x = a( x), a( x+ ej) = a( x) j vx ( ) 0 f x < n, x > a( x) ( vx ( + e) + l) < 0 f x = n, x a( x) vx ( + ej) + l 0 f x = n, x = a( x), j Tvx ( ) a( x+ ej) = a( x)

17 Advances n Systems Scence and Applcatons (009), Vol.9, No f x = n, x = a( x), j Tvx ( ) a( x+ ej) = a( x) j vx ( ) 0, f x = n, x > a( x). Step Three. We prove property () for Tv(x). We frstly show the followng clam, Tvx ( ) 0 for j. () j As a matter of fact, we have j vx ( + e) 0 f x < n, x a( x) 0 f x < n, x a( x), a( x+ ej) = a( x) ( vx ( + e + ej) + l) < 0 f x < n, x = a( x), a( x+ ej) = a( x) j vx ( ) 0 f x < n, x = a( x), a( x+ ej) = a( x) vx ( + e) + l 0 f x < n, x = a( x) a( x+ ej) = a( x) j vx ( ) 0 f x < n, x > a( x), j Tvx ( ) 0 f x = n, x a( x), 0 f x = n, x = a( x), a( x+ ej) = a( x) 0 f x = n, x = a( x), a( x+ ej) = a( x) j v( x) 0 f x = n, x = a( x) a( x+ ej) = a( x) vx ( + e) + l 0 f x = n, x = a( x), a( x+ ej) = a( x) j vx ( ) 0 f x = n, x > a( x). We secondly show that As a matter of fact, Tvx ( ) j j Tvx ( ) 0for j. () j j j jvx ( + e) 0 f x < n, x a( x) j jvx ( ) + j vx ( ) 0 f x < n, x = a( x) a( x+ ej) = a( x), a( x+ ej) = a( x+ ej)

18 644 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng j jvx ( + e) + ( vx ( + e + ej) + l) 0f x < n, x = a( x), a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jvx ( + e) 0 f x < n, x = a( x), a( x+ ej) = a( x) a( x+ ej) = a( x+ ej) j jvx ( ) 0 f x < n, x = a( x), a( x+ ej) = a( x ), a( x+ ej) = a( x+ ej) j jvx ( ) 0 f x < n, x = a( x), a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jvx ( ) ( vx ( + ej) + l) < 0 f x < n, x = a( x), a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jvx ( + e) 0 f x < n, x = a( x), a( x+ ej) = a( x), a( x+ ej) = a( x+ ej) j jvx ( ) 0 f x < n, x > a( x) j jvx ( ) + j vx ( ) 0 f x = n, x a( x) j jvx ( ) 0 f x = n, x = a( x), a( x+ ej) = a( x) j jvx ( ) ( vx ( + ej) + l) < 0 f x = n, x = a( x), a( x+ ej) = a( x) j jvx ( ) 0 f x = n, x > a( x). Step Four. Let f( x) = xvx ( e ) + ( n x )( vx) (3) Then we have the clam that f(x) satsfes (0)-() for (0) s satsfed. As a matter of fact, for, j {,}, f( x) j = ( xvx ( e )) + n vx ( ) ( xvx ( )) j j j j j j x. n Frstly, we show that = ( x vx ( e ) + x vx ( e )) + ( n x x ) vx ( )( for j) 0. j (4) j j j j j j j j j

19 Advances n Systems Scence and Applcatons (009), Vol.9, No Secondly, we show that () s satsfed. As a matter of fact, for jk, {,} and j k, f( x) Thrdly, we show that () s satsfed. As a matter of fact, for jk, {,} and j k, jk f( x) = ( xvx ( e)) + n vx ( ) ( xvx ( )) jk j jk j jk j = ( x vx ( e ) + x vx ( e )) + ( n x x ) vx ( ) = ( xvx ( e )) + n vx ( ) ( xvx ( )) j k j k j k = ( x vx ( e ) + x vx ( e )) + ( n x x ) vx ( ) j k j k j k k j j k j k j j k (6) k jk j k j jk j j k j jk j Fnally, by steps one to four and by the fact that h(x) satsfes condtons (0)-(), we know that Tv(x) satsfes condtons (0)-(). Therefore we complete the proof of the lemma. 6. Concludng Remarks and Future Research In ths paper, we studed the stochastc optmal admsson and routng controls for a system consstng of two parallel-server channels wth zero watng buffer capacty and mult-class customers va Markov decson processes movng n dscrete trangles. Our fndngs can possbly be extended to more general cases n two drectons. Frstly, n the current paper, the classes of customers are classfed accordng to the arrval rates, payments and penalty costs. We conjecture that our results should be true when the servce rates for dfferent classes are also dfferent. Ths conjecture s partally justfed by the numercal mplementaton summarzed n Example 4.6. Secondly, when there are more than two classes of customers who are served n each channel, the correspondng Markov decson processes move n hgh-dmensonal dscrete trhedrons. The research to fnd the sutable class of structured value functons that s preserved under certan sense s also nterestng. 7. Acknowledgements Wanyang Da s supported by Natonal Natural Scence Foundaton of Chna under Contract The man part of the project was done whle the frst author was vstng the Department of Systems Engneerng and Engneerng Management at The Chnese Unversty of Hong Kong n 007. The author expresses sncere thanks for the knd nvtaton and fnancal support from the department and the unversty. Youy Feng s supported by a grant from the Research Councl of the Hong Kong Specal Admnstratve Regon (Project No. CUHK 430/0E). References [] Bell, S.L., Wllams, R.J. Dynamc schedulng of a system wth two parallel servers n heavy traffc wth resource poolng: asymptotc optmalty of a threshold polcy. Annals of Appled probablty, 00, (3): [] Benjaafar, S., ElHafs, M. Producton and nventory control of a sngle product Assemble-to-Order system wth multple customer classes. Management Scence, 006, 5 (5)

20 646 Da: Stochastc Optmal Controls for Parallel-Server Channels wth Zero Watng (): [3] Cheng, C., Huang, S., Peng, S., Kang, J. The optmzaton of manufacturng system based on materal flow and captal flow. Advances n Systems Scence and Applcatons, 005, 5(): [4] Da, W. Product-form solutons for ntegrated servces packet networks. Proceedngs of MICAI 06. IEEE Computer Socety Press, Los Alamtos. 006: [5] Da, W., Jang, Q. Stochastc optmal control of ATO systems wth batch arrvals va dffuson approxmaton. Probablty n the Engneerng and Informatonal Scences, 007, (3): [6] Dng, L.Y., Zhou, Y., Luo, H.B., Gu, X. Study on constructon cost control system based on multple dmenson data model. Advances n Systems Scence and Applcatons, 007, 7(4): [7] Gans, N., Koole, G., Mandelbaum, A. Telephone call centers: Tutoral, revew and research prospects. Manufacturng and Servce Operatons Management, 003, 5(): [8] Harrson, J.M. Heavy traffc analyss of a system wth parallel servers: asymptotc optmalty of dscrete-revew polces. Annals of Appled Probablty, 998, 8(3): [9] Hrart-Urruty, J., Lemarechal, C. Fundamentals of Convex Analyss. Sprnger, Berln. 00: [0] Lppman, S. Applyng a new devce n the optmzaton of exponental queueng systems. Operatons Research, 975, 3(4): [] Ma, C., Fang, H., Kong, D., Huo, Z. Stochastc optmal control of networked control systems wth mult-step delay. Advances n Systems Scence and Applcatons, 005, 5(4): [] Ormec, E.L. Burnetas, A., Wal, J.V.D. Admsson polces for a two class loss system. Stochastc Models, 00, 7(4): [3] Ormec, E.L., Wal, J.V.D. Admsson polces for a two class loss system wth general nterarrval tmes. Stochastc Models, 006, (): [4] Porteus, E.L. Condtons for characterzng the structure of optmal strateges n nfnte horzon dynamc programs. Journal of Optmzaton Theory and Applcatons, 98, 36(4): [5] Puterman, M. Markov Decson Processes: Dscrete Stochastc Dynamc Programmng (Chapetr 6). John Wley and Sons, Inc. New York. 994: [6] Sennott, L. Average reward optmzaton theory for denumerable state spaces. In Handbook of Markov Decson Processes Methods and Applcatons: Internatonal Seres n Operatons Research and Management Scence. Edtors: E.A. Fenberg and A. Schwartz. Kluwer Academc. 00: 53-7.