Coninuous Time Markov Chain (Markov Process) The sae sace is a se of all non-negaive inegers The sysem can change is sae a any ime ( ) denoes he sae of he sysem a ime The random rocess ( ) forms a coninuous-ime Markov chain if P( ) ( ) e, ( ) e,, ( ) e P ( ) ( ) e n 2 2 n n n n n for any n, and for any ime sequence,,, and any as hisory e, e,, e n 2 n 2 Simly saed, he fuure deends on he resen sae only The as hisory is irrelevan Sae Residence Time Suose he sysem eners sae E and leaves he sae afer an amoun of ime is referred o as he sae residence ime Suose he sysem has sayed in sae E for a duraion of ime s Wha is he robabiliy ha he sysem will leave he sae E wihin ime? By he definiion of he Markove chain, he robabiliy does no deend on he as hisory, is memoryless P s s P Therefore, for any sae E, he sae residence ime is exonenially disribued
2 Theorem Memoryless Proery: roof P s s P for any s, is an exonenially disribued random variable P s and s P s s P s Tha imlies Ps s P s F ( s) F ( s) F ( ) F ( s) Noe F () Thus we have F ( s) F () s F () F () F () s F( s) F( s) F( ) F() F ( s ) aking he limi as, F () s F () F () s mus equal o P Define R ( s) F ( s) R s R R s R Then F ( s) R ( s) and F () R () Therefore we have ( ) () ( ) Define () Then R ( s) R ( s) Solving he above equaion wih R () F (), s R ( s) e s and finally F ( s) e
3 Sae Transiion Probabiliies and Raes in a Homogeneous Markov Process Assume P s k s does no vary wih ime s, and define he sae ransiion robabiliies k ( ) P s k s for any s Define he sae ransiion raes k k k ( ) lim is he rae a which he sysem changes is sae from sae E o E k Sae robabiliies Define () P ( ) is he robabiliy ha he sysem is in sae E a ime The seady-sae robabiliies are lim ( ) We will learn how o find he seady-sae robabiliies The seady-sae robabiliies are found from Global Balance Equaions,,2, ( e) k i i k i Eq e is referred o as he global balance equaion A flow is he sum of he corresonding sae ransiion raes The flow ou of he sae E is, while he flow ino he sae E is k i i k i The global balance equaion saes ha, for any sae, he flow ou of he sae mus equal he flow ino he sae
4 We mus relace one of he equaions in Eq e by 2 roof ( ) P i i P ip i () ( ) i i Dividing he sum ino wo erms, ( ) () ( ) () ( ) i i i Rearranging erms, () i( ) i() i( ) i i ( ) () () ( ) () ( ) i i i i i Dividing boh sides by, ( ) ( ) i ( ) i ( ) () i() ( e2) In he sae sae, we require ( ) ( ) lim As and, eq e2 becomes i i i i i i i
5 Markov Processs Examle: M/M/ queue wih one waiing roomm Cusomer Arrivals Cusomers arrive in a Poisson manner wih arrival rae The iner-arrival imes beween cusomers are exonenially disribued wih df a( x) e x x An arriving cusomer eners he server if f he server is idle If he server is busy, he cusomer eners he waiing room if here is a sace in he waiing room If he waiing room is full (already occuied by someone else inn his examle), hen he cusomer leaves he sysem No reservaion No reurn The cusomer is reeced Cusomer Service The service ime of each cusomer is a random variable exonenially disribued wih mean / The service ime has he df b( x) e x x Uon comleion of service, he server immediaely begins service of a cusomer waiing in he waiing room, if any If no cusomer waiing, he server becomes idle unil he nex cusomer arrives
6 Sae Sace Le ( ) denoe he number of cusomerss in he sysem a ime Since he waiing room can accommodaee a mos one cusomer, he sae sace is,, 2 A any ime, ( ), or 2 The sae sace of he sysem is comosed of hree saes Markov Process Le ( ) denoe he number of cusomerss in he sysem a ime Then () forms a coninuous-ime Markov chain since a sae change occurs if and only if eiher an arrivall occurs or he service of a cusomer ends Boh yes of he evens occurr in a memoryless manner The sae residencee imes are memoryless s Sae Transiion Raes Consider Sae ransiion occurs from sae o sae if an arrival occurss during, and Therefore The sae ransiionn rae is is he sae ransiionn robabiliyy from sae o sae in he inerval : P an arrival during from a Poisson source wih rae P s lim ( ) s for any s s
7 Nex consider Sae ransiion occurs from sae o sae when he service iss comleedd and no arrivals occur wihin is he sae ransiionn robabiliyy from sae o sae in he inerval : P service comleion wihin b x dx e P s s for any s P no arrivals during from a Poisson source wih w rae Therefore The sae ransiionn rae is lim ( ) Likewise, we can obain 2 2 2 2
8 Global Balance Equaion For sae, Flow Ou, Flow Ino The gbe is () e For sae, Flow Ou, Flow Ino 2 The gbe is 2 (2) e For sae 2, Flow Ou, Flow Ino The gbe is 2 2 (3) e Eq e-3 are linearly deenden Relace any one wih The soluion is 2 2 2 2 Sae Residence Times The sae residencee ime iss an exonenially disribued random variable For examle, he df of is e and he df of is, e
9 Birh deah Process and Local Balance Equaions A birh-deah rocess is secial case of he homogeneous Markov rocess in which for k k E E E k- E k E k+ The global balance equaion saes k k k k k k k I can be shown ha wih a birh-deah rocess, he flow equilibrium occurs and he local balance equaions are saified: and k k k k k k k k E k- E k E k+ Flow Equilibrium
B D Process Examle: M/M/ Queue Poisson Arrivals wih rae Infinie Waiing Room for Cusomers Exonenial Service Time wih mean M/M/ queue is a birh-deah rocess, and we can use he local balance equaions 2 3 4 o o o where 2 2 2 o o n n n nn o o 2 n n o 2 o o n n n n,,2,
Performance Measuress The fracion of ime he sysem is idle is Uilizaion, he fracion of ime he sysem is busy, is The average number of cusomers in he sysem is N, The average waiing ime, in FIFO, is n / n n W N The average sysem ime, he oal ime sen in he sysem, is T W Noe T N When a cusomer leaves he sysem, here are N cusomer rs waiing behind