The 7 th Balkan Conference on Operational Research BACOR 05 Constanta, May 2005, Romania STEADY-STATE SOLUTIONS OF MARKOV CHAINS

Transcription

1 The 7 th Blkn Conference on Opertonl Reserch BACOR 5 Constnt My 25 Romn STEADY-STATE SOLUTIONS OF MARKOV CHAINS DIMITAR RADEV Deprtment of Communcton Technque & Technologes Unversty of Rousse Bulgr VLADIMIR DENCHEV ELENA RASHKOVA Deprtment of Communcton Technque & Technologes Unversty of Rousse Bulgr Abstrct The pper s devoted on methods nd lgorthms for stedy-stte nlyss of Mrkov chns. Bsc drect nd tertve methods for stedy-stte nlyss of Mrkov chns re concerned where Gussn Elmnton method nd Grssmn method s well s Power Jcob s nd Guss-Sedel s methods re mplemented. Algorthms for computton of stedy-stte probblty vector for fnte Mrkov chns re developed. Comprson of numercl solutons to ect equlbrum soluton for locl-blnce equton of Dscrete-Tme Mrkov Chn s gven. Emple nd numercl results for feedbck networks of Mrkovn queues re shown. Keywords: Stedy-Stte Probbltes Queung Theory Dscrete-Tme Mrkov Chns Numercl Methods Appromton Technques. INTRODUCTION Mrkov processes provde very fleble powerful nd effcent mens for descrpton nd nlyss of dynmc (communcton computer) system propertes. Performnce nd dependblty mesures for communcton networks cn be derved nd evluted wth stedy-stte nlyss of Dscrete-Tme Mrkov Chns (DTMC) nd Contnuous-Tme Mrkov Chns (CTMC). Drect methods nd tertve methods cn be used for numercl soluton stedy-stte nlyss of Mrkov chns []. Drect methods operte nd modfy the prmeter mtr nd use fed mount of computton tme

2 ndependent of the prmeter vlues [4] but re subect to ccumulton of round-off errors nd hve dffcultes wth sprse storge [2]. Itertve methods re bsed on the property of successve convergence to the desred soluton. The evluton cn be termnted f tertes re suffcently close to the ect vlue. The mn dvntge of tertve methods compred wth drect methods s tht they preserve the sprsty of the prmeter mtr [3] becuse effcent sprse storge schemes nd effcent sprsty-preservng lgorthms cn be used. Other dsdvntge of tertve methods s tht convergence s not lwys gurnteed nd depends on the method. The rte of convergence s hghly senstve to the vlues of entres n the prmeter mtr [5]. One of mportnt tsks here s recevng numercl solutons to ect equlbrum soluton for locl-blnce equton of dscrete-tme Mrkov chn. Ths s very nterestng for modelng of computer nd communcton networks especlly wth hevy tled trffc. These trffc processes re descrbng wth dscrete-tme Mrkov chns contnuous-tme Mrkov chns nd ergodc Mrkov chns. Tht s why n ths reserch s workng out lgorthms for numercl soluton of equlbrum for locl-blnce equton of dscrete-tme Mrkov chn. On the bse of numercl soluton methods s suggestng procedure for stedy stte probblty vector. 2. STEADY-STATE ANALYSIS OF MARKOV CHAINS For computton of stedy-stte probblty vector of ergodc Mrkov chns most often s usng the followng model. Settng í íp nd ðq cn be wrtten (2.). í( P I) (2.) Therefore both for dscrete-tme nd contnuous-tme Mrkov chns lner system (2.2) need to be solved: A (2.2) Due to ts type of entres representng the prmeters of Mrkov chn mtr A s sngulr nd t cn be shown tht A s of rnk n- for ny Mrkov chn of sze S n. It follows mmedtely tht the resultng set of equtons s not lnerly ndependent nd tht one of the equtons s redundnt. To yeld unque postve soluton normlzton condton hve to be ppled on the soluton of equton A. We drectly mpose the normlzton condton nto the (2.2) wth (2.3). (2.3) Ths cn be regrded s substtutng one of the columns (sy the lst column) of mtr A by the unt vector. The resultng lner system of non-homogeneous equtons s shown n (2.4). b A b [...] (2.4) For ny gven ergodc contnuous-tme Mrkov chns dscrete-tme Mrkov chns cn be constructed whch yelds n dentcl stedy-stte probblty vector s for the CTMC. Consder the genertor mtr Q q ] of contnuous-tme Mrkov chns where s formulted (2.5) P Q / q I (2.5) where q s chosen such tht [ q m S q. Settng m S q q should be voded

3 n order to ssure perodcty of the resultng DTMC [2]. The resultng mtr P cn be used to determne the stedy-stte probblty vector ð í by solvng í íp nd í. Ths method s used to reduce CTMC to DTMC nd s clled rndomzton or sometmes unformzton n the lterture [3]. On the other hnd trnston probblty mtr P of n ergodc DTMC s gven nd genertor mtr Q of CTMC cn be defned ccordng to (2.6). Q P I (2.6) By solvng ðq under the condton ð the desred stedy-stte probblty vector ð í cn be obtned. To determne the stedy-stte probbltes of fnte Mrkov chns dfferent pproches for the soluton of lner system of the form A re used. In ths cse both drect nd tertve numercl methods nd technques cn led to closed-form results. Whle drect methods yeld ect results tertve methods re generlly more effcent both n tme nd spce. Dsdvntges of tertve methods re tht for some of them no gurntee convergence gven n generl. Snce tertve methods re consderbly more effcent n solvng Mrkov chns they re commonly used for lrger models. For smller models wth less thn few thousnd sttes drect methods re relble nd ccurte. Though closed-form results re hghly desrble they cn be obtned for only smll clss of models tht hve some structure n ther mtr. 3. DIRECT METHODS FOR NUMERICAL SOLUTION The closed-form soluton methods re pplcble when Mrkov chns possess specl structures. For Mrkov chns wth more generl structure we need to refer to numercl methods. There re two brod clsses of numercl methods to solve the lner systems of equtons: drect methods nd tertve methods. Drect methods operte nd modfy the prmeter mtr. They use fed mount of computton tme ndependent of the prmeter vlues nd we don t m to rech convergence. The use of sprse storge s dffcult snce orgnl zero entres cn become non-zeros. Drect methods re lso subect to ccumulton of round-off errors. There re mny drect methods for the soluton of system of lner equtons. Some of them re restrcted to certn regulr structures of the prmeter mtr tht re of less mportnce for Mrkov chns snce these structures generlly cnnot be ssumed n the cse of Mrkov chn. Among the most commonly ppled technques re the Gussn elmnton lgorthm nd dervtve of t - Grssmnn's lgorthm. The orgnl verson of the lgorthm s usully referred to lgorthms of Grssmnn Tksr nd Heymn (GTH) whch re bsed on renewl rgument [5]. There s newer vrnt where smple relton to the Gussn elmnton lgorthm s done. The Gussn elmnton lgorthm suffers sometmes from numercl dffcultes creted by subtrctons of nerly equl numbers. It s ectly ths property tht s voded by the GTH lgorthms nd ts vrnt through reformultons relyng on regenertve propertes of Mrkov chns. Cncellton errors re convenently voded n ths wy.

4 3. GAUSSIAN ELIMINATION The de of the lgorthm s to trnsform the system of equtons (3.) nto n equvlent one by pplyng elementry opertons on the prmeter mtr tht preserve the rnk of the mtr. n n b n n b (3.) b. n n n n n n As result n equvlent system of lner equtons specfed by (3.2) wth trngulr mtr structure s derved from whch the desred soluton whch s dentcl to the soluton of the orgnl system cn be obtned: ( n) ( n) b ( n2) ( n2) ( n2) b (3.2) () () () b. n n n n n n If the system of lner equtons hs been trnsformed nto trngulr structure the fnl results cn be obtned by mens of strghtforwrd substtuton process. To rrve t system (3.2) n elmnton procedure frst needs to be performed on the orgnl system (3.). Informlly the lgorthm cn be descrbed s follows; frst the n -th equton of (3.) s solved for n nd then n s elmnted from ll other n- equtons. Net the (n-) th equton s used to solve for n2 nd gn n2 s elmnted from the remnng n-2 equtons nd so forth. Fnlly (3.2) results where denotes the coeffcent of n the (+) -th equton obtned fter the k -th elmnton step. The Gussn elmnton procedure tkes dvntge of elementry mtr opertons tht preserve the rnk of the mtr. Such elementry opertons correspond to nterchngng of equtons multplcton of equtons by rel-vlued constnt nd ddton of multple of n equton to nother equton. In mtr terms the essentl prt of Gussn elmnton s provded by the fctorzton of the prmeter mtr A nto the components of n upper trngulr mtr U nd lower trngulr mtr L. As result of the fctorzton of the prmeter mtr A the computton of the result vector cn splt nto two smpler steps: b A UL yl. 3.. (3.3) Now the Gussn elmnton lgorthm cn be summrzed s follows from Fg.

5 STEP : Construct the prmeter mtr A nd the rght-sde vector b ccordng to: b A b [...] ; STEP 2: Crry out elmnton steps or pply the stndrd lgorthm to splt the prmeter mtr A nto upper trngulr mtr U nd lower trngulr mtr L such tht A UL holds. Note tht the prmeters of U cn be computed wth: n k n k 2... n n 2... n k nk nk otherwse ( k) nk nk nd the computton of L cn be delbertely voded. STEP 3: Compute the ntermedte results y ccordng to yl b or compute the ntermedte results wth the result from ( n) ( n2) U b b... b ) ccordng to: ( n ( k) nk b b bnk nk nk where n k n... STEP 4: Perform the substtuton to yeld the fnl result ccordng to U y by pplyng the formule: b b ( n ) ( n ) ( n ) ( n ) k ( n ) k ( n ) 2... n. k Fg. 3. Algorthm for Gussn Elmnton 3.2 THE GRASSMANN ALGORITHM Grssmnn's lgorthm s numerclly stble vrnt of the Gussn elmnton procedure. The lgorthm completely vods subtrctons nd t s therefore less senstve to roundng nd cncellton errors cused by the subtrcton of nerly equl numbers. Grssmnn's lgorthm ws orgnlly ntroduced for the nlyss of ergodc dscretetme Mrkov chns X = {Xn; n =...} nd ws bsed on rguments from the theory of regenertve processes [3]. The trnston rtes of new Mrkov chn hvng one stte less thn the orgnl one re defned. Ths elmnton step.e. the computton of q s cheved merely by ddng non-negtve qunttes to orgnlly non-negtve vlues q. Only the dgonl elements q nd q re negtve. The elmnton procedure s tertvely ppled to the genertor mtr wth entres q of stepwse reduced stte spces untl n upper trngulr mtr results where q denotes the mtr entres fter hvng ppled elmnton step k k n. Fnlly ech element q on the mn dgonl s equl to -. ( n) The elmnton s followed by substtuton process to epress the reltons of the stte probbltes to ech other. To yeld the fnl stte probblty vector the normlzton condton must be ppled. Grssmnn's lgorthm s presented n terms of CTMC genertor mtr nd the prmeter mtr must ntlly be properly defned:

6 STEP : A Q P I forctmc fordtmc STEP 2: For l n n 2... : ( nl ) ( nl ) l ( nl) l m m ( nl) ( n ( n l ) l l l ( nl l m m l l l. l ) ) l l l STEP 3: For l 2... n : l l ( nl ) l STEP 4: For l... n :. n. Fg. 3.2 The Grssmnn lgorthm In mtr notton the prmeter mtr A s decomposed nto fctors of n upper trngulr mtr U nd lower trngulr mtr L such tht the followng equtons hold. A UL. (3.4) Of course ny soluton of U s lso soluton of the orgnl equton A. Therefore there s no need to represent L eplctly. Although cncellton errors re beng voded wth Grssmnn's lgorthm roundng errors cn stll occur propgte nd ccumulte durng the computton. Therefore pplcblty of the lgorthm s lso lmted to medum sze (round 5 sttes) Mrkov models. 4. ITERATIVE METHODS FOR NUMERICAL SOLUTION The mn dvntge of tertve methods over drect methods s tht they preserve the sprsty of the prmeter mtr nd effcent sprsty-preservng lgorthms nd sprse storge schemes cn be used. A good ntl estmte cn speed up the computton consderbly. The evluton cn be termnted f the tertes re suffcently close to the ect vlue.e. prespecfed tolernce s reched. Fnlly becuse the prmeter mtr s not chnged n the terton process tertve methods re not subect to ccumulton of round-off errors. The mn dsdvntge of tertve methods s tht convergence s not lwys gurnteed nd dependng on the method the rte of convergence s hghly senstve to the vlues of entres n the prmeter mtr. 4.. CONVERGENCE OF ITERATIVE METHODS Convergence s very mportnt ssue for tertve methods tht must be delt conscously. A heurstc pproch cn be ppled for choosng pproprte technques for decsons on convergence but there re no generl lgorthms for the selecton of such technque. Becuse the desred soluton vector s not known n estmte of the error must be used to determne convergence. A tolernce level must be specfed to provde mesure of (k ) how close the current terton vector s to the desred soluton vector. New York Some (k ) dstnce mesures re often used to evlute the current terton vector n relton to some ( erler terton vectors l ) l k. If the current terton vector s "close enough" to erler ones wth respect to then ths condton s tken s n ndctor of convergence to the fnl result. If s too smll convergence could become very slow or not tke plce t ll. If s too lrge ccurcy requrements could be volted or worse convergence could be wrongly ssumed. Some pproprte norm functons hve to be ppled n order to compre dfferent

7 terton vectors. Sze nd type of the prmeter mtr should be tken nto consderton for the rght choce of such norm functon. Concernng the rght choce of nd the norm functon we cn sy tht components of the soluton vector cn dffer sgnfcntly from ech other. 4.2 POWER METHOD The Power method s relble tertve method for the computton of the stedy-stte probblty vector of fnte ergodc Mrkov chns. It sometmes tends to converge slowly nd the solely condton needed for convergence s the trnston probblty mtr P to be perodc nd then rreducblty s not necessry. The power method follows the trnsent behvor of the underlyng dscrete-tme Mrkov chns untl some sttonry not necessrly stedy-stte convergence s reched. Therefore t cn lso be used s method for computng the trnsent stte probblty vector í (n) of DTMC. Equton í íp suggests strtng wth n ntl guess of some probblty vector nd repetedly multplyng t by the trnston probblty mtr P untl convergence to v s ( ) reched wth lm í í. Snce ergodcty or t lest perodcty of the underlyng Mrkov chn re ssumed ths procedure s gurnteed to converge to the desred fed pont of the unque stedy-stte probblty vector. A sngle terton step s s follows from (4.). ( ) ( ) í í P. (4.) The relton between the terton vector t step nd the ntl probblty vector cn be presented s (4.2). ( ) () í í P. (4.2) To yeld the fnl result of the stedy-stte probblty vector v only renormlzton remns to be performed. The speed of convergence of the power method depends on the reltve szes of the egenvlues. The closer non-domnnt egenvlues re equls to whch slower the convergence. The lgorthm of the power method s shown on Fg. 3.. () í STEP : Select q pproprtely: A Q P / q I; () () () () í í í... í n. Select convergence crteron nd let n. Defne some vector norm functon ( n) ( l ) f í í n. l Set convergence = flse. STEP 2:Repet untl convergence: STEP 2.: í í ( n) ( n) A ; ( n) ( l ) STEP 2.2: If f í í l n THEN convergence = true; STEP 2.3: n n l l. ð STEP 3: í (n) í. Fg. 4. The power method lgorthm 4.3. JACOBI S METHOD Let defne the system of lner equtons (4.3). b A. (4.3) The normlzton condton my or my not be ncorported n (4.3). The prmeters of both DTMC nd CTMC re gven by the entres of the mtr A ]. The soluton vector [

8 wll contn the uncondtonl stte probbltes. If the normlzton s ncorported we hve b [...] nd b otherwse. Consder the -th equton from the system (4.3) s (4.4). b. (4.4) S Solvng (4.4) for b leds to (4.5).. (4.5) Any gven ppromte soluton [ ˆ ˆ... ˆ ] cn be nserted for the vrbles ˆ n on the rght sde of (4.5). From these ntermedte vlues better estmtes of the on the left sde of the equton my be obtned. The tertve method requres pplyng ths k procedure repetedly nd n prllel for ll n equtons. The vlues of the k -th terton step re computed from vlues obtned from the (k - l) -st step for ech equton ndependently s (4.6). b S. (4.6) The terton my be strted wth n rbtrry ntl vector. Note tht the equtons cn be evluted n prllel fct tht cn be used s mens for computtonl speed-up. The method s clled method of smultneous dsplcement or smply the Jcob method. Snce the method s qute smple t suffers from poor convergence nd hence s rrely ppled n ts rw form. The lgorthm s presented on Fg STEP : Defne prmeter mtr A nd b properly from genertor mtr Q or trnston probblty mtr P. f Choose ntl vector () ; Choose convergence crteron. Choose some norm functon ( l) k l. Splt prmeter mtr A=D-L-U. convergence=flse nd k l. STEP 2: Repet untl convergence: ( k) STEP 2.: b ( U L) D ( k l) STEP 2.2: If f Then convergence =true; Else k k nd l... k ð STEP 3: í. n ( k) ( k). ; Fg. 4.2 The Jcob s method lgorthm Splttng the mtr A=D-L-U nto ts consttuents of the dgonl mtr D the strctly lower-trngulr mtr L nd the strctly upper-trngulr mtr U provde wy to present the mn computton step of the Jcob s method n mtr notton s s shown n (4.7). b ( U L) D. (4.7) The Jcob s method s of less prctcl mportnce due to ts slow pttern of convergence. But technques hve been derved to speed up ts convergence resultng n wellknown lgorthms such s Guss-Sedel terton.

9 4.4 GAUSS-SEIDEL METHOD To mprove convergence gven method often needs to be chnged only slghtly. We cn serlze the procedure from (4.6) nd tke dvntge of the lredy updted new estmtes n ech step. Assumng the computtons to be rrnged n order... n where S n t mmedtely follows tht for clculton of the estmtes (k ) ll prevously computed ( estmtes k ) cn be used n the computton. Tkng dvntge of the more up-to-dte nformton we cn sgnfcntly speed up the convergence. The resultng method s clled the Guss-Sedel terton nd t mn prncple s presented n (4.8). n b S. (4.8) (k ) Note tht the order n whch the estmtes re clculted n ech terton step cn hve crucl mpct on the speed of convergence. Most often the mtrces of Mrkov chns re sprse nd the nterdependences between the equtons re lmted to certn degree nd prllel evluton mght stll be possble even f the most up-to-dte nformton s ncorported n ech computton step. The equtons cn delbertely be rrnged so tht the nterdependences become more or less effectve for the convergence process. Apprently trde-off ests between the pttern of convergence nd possble speedup due to prllelsm. In mtr notton the Guss-Sedel terton step s wrtten s (4.9). b L ( D U) k. (4.9) Reflectng the Guss-Sedel step more obvously we cn rewrte hs pproch s (4.). b U LD k. (4.) 5. COMPARISON OF NUMERICAL SOLUTION METHODS 5.. EXAMPLE We consder n rbtrry connected three-node network wth four customers. The stte trnston rte dgrm s shown n Fg. 5.. In ths dgrm re represented possble trnstons between nodes of Mrkov chn. The trnston rtes between the sttes re tken equl to The numerton of the sttes represents the totl number of customers n ech node. Consder recevng of locl blnce for two emples p p2 2 3 p p3 p 2 p 3 2 Fg. 5. Stte-trnston rte dgrm showng locl blnce for ) emple nd b) emple2

10 Frstly we wrte down the locl blnce equtons nd then we fnd the soluton by substtuton process from where we get the ect stedy-stte probbltes s ndcted n Tble 5.. Net we follow the lgorthm n Fg 3. nd cheve the results for the stedy stte probbltes from the Gussn elmnton lgorthm. Comprng the results wth the ect ones we my sy tht the Gussn elmnton lgorthm gves precse results (to the eght decml) nd only n sttes 9 nd mstkes re found. Stte Number Ect Vlue Computed Vlue Error (3) (4) (4) (3) (3) (22) (22) (3) (3) (4) (2) (3) (2) (22) (2) Tble 5. Comprson of numercl results for clculted stedy-stte probbltes usng ect method nd Gussn elmnton 5.2 EXAMPLE 2 Consder two customers crcultng mong three nodes. When customer hs receved servce of men durton / t the frst stton t queues wth probblty p 2 t stton two for servce of men durton / 2 or wth p3 t stton three wth men servce durton / 3. After completon of servces t sttons two or three customers return wth probblty bck to stton one. The stte trnston rte dgrm s shown n Fg. 5. (b). In ths dgrm we represent contnuous-tme Mrkov chn wth possble trnstons between nodes. The trnston rtes between the sttes re p2.4 p3. 6. The numerton of the sttes represents the totl number of customers n ech node. Frstly s computed the ect vlues for the stedy stte probbltes usng substtuton process. These vlues cn be for comprson wth the terton vector. Net were used the power method lgorthm to compute the stedy stte probbltes rechng 45 tertons form where ws receved ccurcy to the sth decml s s shown n Tble 5.2. Afterwrd from lgorthm on Fg. 4.2 were computed probbltes ccordng to Jcob s method. The obtned results were better but for ther recevng t s necessry to provde much more terton steps 3. The power method converges fster for ths network nd gves results whch re enough precse.

11 Stte Ect Power Jcob 45 Error 3 Error (2) () (2) () () (2) Tble 5.2 Comprson of numercl results for clculted stedy-stte probbltes usng ect method Power method nd Jcob s method CONCLUSIONS Bsc drect nd tertve methods for stedy-stte nlyss of Mrkov chns re emned by Gussn Elmnton method nd Grssmn method s well s Power Jcob s nd Guss-Sedel s method. Numercl results for two networks re compred usng ect methods Gussn elmnton Power nd Jcob s method. The Jcob s method s of less prctcl mportnce due to ts slow pttern of convergence. The Power method s relble tertve method though t sometmes tends to converge slowly. It cn lso be used s method for computng the trnsent stte probblty vector of DTMC. Applcblty of the Gussn elmnton lgorthm s lmted to medum sze (round 5 sttes) Mrkov models nd t suffers from round-off nd cncellton errors s well s numercl dffcultes creted by subtrctons of nerly equl numbers but gves good results. BIBLIOGRAPHY [] Bhl S. (22) Mrkov Decson Processes: the control of hgh-dmensonl systems Unversl Press Amsterdm The Netherlnds 42 pp; [2] Bolch G. Grener S. Meer H Trved K. (998) Queueng Networks nd Mrkov Chns: Modellng nd Performnce Evluton wth Computer Scene Applctons New York John Wley& Sons 726pp; [3] Bom O. Koole G. Lu Z. (994) Queueng-theoretc soluton methods for models of prllel nd dstrbuted systems Performnce Evluton of Prllel nd Dstrbuted Systems - Soluton Methods CWI Trct 5 & 6 CWI Amsterdm The Netherlnds pp. -24; [4] Rdev D. Denchev V. Rshkov E. (25) Appromtons Algorthms for Stedy-Stte Solutons of Mrkov Chns Proceedngs of the Interntonl Conference on Computer Systems nd Technologes CompSysTech 25 Vrn Bulgr; [5] Trved K. (2) Probblty nd Sttstcs wth Relblty Queung nd Computer Scence Applctons New York John Wlley & Sons 83pp.