Unit - III RANDOM PROCESSES. B. Thilaka Applied Mathematics

Transcription

1 Ui - III RANDOM PROCESSES B. Thilaka Applied Mahemaics

2 Radom Processes A family of radom variables {X,s εt, sεs} defied over a give probabiliy space ad idexed by he parameer, where varies over he idex se T, is kow as radom process/ chace process/ sochasic process. X: SxT R Xs,=x Noaio: X

3 Radom Processes The values assumed by he radom variables X are called he saes ad he se of all possible values is called he sae space of he radom process. Sice he radom process {X} is a fucio of boh s ad, we have he followig

4 Radom Processes Observaios:. s ad are fixed : X real umber. is fixed : X radom variable 3. s is fixed : X - fucio of ime called he sample fucio or realisaio of he process 4. s ad are varyig : X radom process

5 Classificaio Three ypes Type I Sae Space, Parameer Space: Discree/ Coiuous

6 Classificaio S Discree Coiuous T Discree Coiuous Discree radom sequece Discree radom process Coiuous radom sequece Coiuous radom process

7 Classificaio Predicable/ Deermiisic : Fuure values ca be prediced from pas values. eg: X= A cos w+θ, where ay oempy combiaio of A, w, θ may be radom variables Upredicable/ No- deermiisic: Fuure values cao be prediced from pas values. eg: Browia moio

8 Classificaio Saioariy Probabiliy for a radom process: For a fixed ime, X is a radom variable ha describes he sae of he process a ime. The firs order disribuio fucio of a radom process is defied as

9 Firs order Disribuio ad Desiy Fucio F X x ; F x PX x X The firs order desiy fucio of a radom process is defied as f x ; F x X x X ;

10 Firs order Disribuio Fucio Saisical Average The saisical average of he radom process {X} is defied as x E[ X ] x dx f X ; provided he quaiy o he RHS exiss.

11 Firs order saioary process The radom process {X} is said o be a firs order saioary process / saioary o order oe if he firs order desiy/ disribuio fucio is ivaria wih respec o a shif i he ime origi f x ; f x ; x,, X X or F X ;,, x F x x ; X

12 Firs order saioary process Resul: The saisical mea of a firs order saioary radom process is a cosa wih respec o ime. Proof: Le he radom process {X} be a firs order saioary process. The is firs order desiy fucio saisfies he propery ha

13 Firs order saioary process x ; f x ; x,, fx X The mea of he radom process {X} a ime is defied as ] X ; E[ X x f x dx

14 Firs order saioary process Cosider ] x f ; x E[ X dx Le = +. The ] x f ; x E[ X dx x x f ; dx

15 Firs order saioary process E[ X ] E[ X ] Hece E[X] is a cosa wih respec o ime.

16 Secod order Disribuio fucio The secod order joi disribuio fucio of a radom process {X} is defied as The secod order joi desiy fucio of he radom process {X} is defied as,, ;, x X x X P x x F X, ;,, ;, x x F x x x x f X X

17 Secod order saioary process A radom process {X} is said o be a secod order saioary process/ saioary o order wo if is secod order disribuio/ desiy fucio is ivaria wih respec o a shif i he ime origi. I oher words, FX x x;, ad / or f X F x, x ;, x, x,,,, X ;,, ;,,,,, x x f x x x x, X

18 Secod order Processes Auo-correlaio fucio: The auo-correlaio fucio of a radom process {X} is defied as, EX X R XX provided he quaiy o he RHS exiss.

19 Secod order Processes Sigificace of auo-correlaio fucio. I provides a measure of similariy bewee wo observaios of he radom process {X} a differe pois of ime ad.. I also defies how a sigal is similar o a ime-shifed versio of iself

20 Secod order Processes Auo-covariace fucio: The auo-covariace fucio of a radom process {X} is defied as ] [ ] [, X E X X E X E C XX ] [ ] [ ] [ ] [ X E X E X X E X E X X X E ] [ ] [ ] [ ] [ ] [ ] [ X E X E X E X E X E X E X X E ] [ ] [,, X E X E R C XX XX

21 Wide-sese Saioary Process A radom process {X} is said o be WSS/ weakly saioary/ covariace saioary process if i saisfies he followig codiios:. E[X] is a cosa wih respec o ime.. R XX, is a fucio of he legh of he ime differece. i.e. R XX, = R XX - 3. E[X ]<.

22 Wide-sese Saioary Process Remark: Sice C XX, R, E[ X ] E[ X XX ] C XX, = C XX -, he auo-covariace fucio C XX, is a fucio of he legh of he ime differece. Hece, a WSS process is also called a covariace saioary process.

23 Wide-sese Saioary Process Aleraely, a radom process secod order is said o be WSS/ weakly saioary process if i saisfies E[X] R, XX R XX Remark: A secod order saioary process is a WSS process, bu he coverse eed o be rue.

24 h order disribuio/ desiy fucio The h order joi disribuio fucio of a radom process {X} is defied as The h order joi desiy fucio of a radom process {X} is defied as,..,,,..,, ;,..,, X X x X x X P x x x F X X x x x F x x x x x x f,..,, ;,..,,...,..,, ;,..,,

25 h order saioary process A radom process {X} is said o be h order saioary/ saioary o order if he h order desiy/ disribuio fucio is ivaria wih respec o a shif i he ime origi i.e. ad / or,,..,,,,..,,,,.., ;,..,,..,, ;,.., X X x x x x x x f x x x f,,..,,,,..,,,,.., ;,..,,..,, ;,.., X X x x x x x x F x x x F

26 Sricly saioary Process A radom process is said o be sricly saioary SSS/ saioary i he sric sese if is saioary o all orders Aside: For a SSS- CDF/PDF h order is ivaria wih respec o a shif i he ime origi Mea-cosa Auo-correlaio, Auo-covariacefucios of legh of ime iervals.

27 Sricly saioary Process Remark : If a radom process is saioary o order, he i is saioary o all orders k. WHY?

28 Evoluioary Process Radom processes which are o saioary o ay order are called osaioary/ evoluioary processes

29 Furher Properies Cross correlaio fucio: The cross correlaio fucio of wo radom processes {X} ad {Y} is defied as R XY, =E[X Y ] Cross covariace fucio: The cross correlaio fucio of wo radom processes {X} ad {Y} is defied as C XY, =E{[X -EX ][Y -EY ]} =R XY, -E[X ]E[Y ]

30 Furher Properies Two radom processes {X} ad {Y} are said o be joily WSS if i boh {X} ad {Y} are each WSS ii he cross-correlaio fucio R XY, =E[X Y ] is exclusively a fucio of he legh of he ime ierval -

31 Idepede Radom Process A radom process {X} is said o be a idepede radom process if is h order joi disribuio fucio saisfies he propery ha A similar codiio holds for joi p.m.f./ p.d.f. X X X X x x x x F x F x F x x x F,..,,,,..,,, ;... ; ;,..,, ;,..,

32 Process wih idepede icremes A radom process {X} is defied o be a process wih idepede icremes if for all 0< < < < <, he radom variables X - X, X 3 - X,.., X - X - are idepede.

33 Time averages of a radom Prelude: process The ime average of a quaiy f ime is defied as T l A [ f ] f d T T T Time average of a radom process: The ime average of a radom process {X} is defied as

34 Time averages of a radom T l A [ X ] x d T T process Time auo-correlaio fucio : T The ime auo-correlaio fucio of a radom process {X} is defied as T l A[ X X ] x x d,, T T T

35 Ergodic Process A radom process is said o be ergodic if is ime averages are all equal o he correspodig saisical averages. Examples???

36 Radom Process Correlaio coefficie of a radom process : The correlaio coefficie ormalized auo-covariace fucio of a radom process {X} is defied as he correlaio coefficie bewee is radom variables X ad X for arbirary ad. I oher words, XX, Var[ X C XX ], Noe: Var[X] = C XX, Var[ X ]

37 Markov Processes A radom process {X} is called a Markov process if for all 0 < < < < <, he codiioal disribuio of X give he values of X 0, X, X,.., X depeds oly o X. P[X x X o x 0,X x,..., X x ] P X x X x P[a X b X o x 0,X x,..., X x ] P a X b X x

38 Markov Processes I oher words, a radom process {X} is called a Markov process if he fuure values of he process deped oly o he prese ad are idepede of he pas. Examples: Biomial process, Poisso process, Radom elegraph process

39 Markov Processes Noe: If he sae space of a Markov process is discree, he he Markov process is called a Markov chai.

40 Markov Processes Sae Space Discree Time Discree Discree- Time Markov Chai Coiuous Coiuous - Time Markov Chai Coiuous Discree- Time Markov Process Coiuous - Time Markov Process

41 Markov Processes

42 Markov Processes If i a Markov chai, he codiioal disribuio is ivaria wih respec o a shif i he ime origi, he Markov chai is said o be ime homogeeous. I a homogeeous Markov chai, he pas hisory of he process is compleely summarized i he curre sae. Hece, he disribuio of ime ha he process speds i he curre sae mus be memoryless.

43 Couig Process A radom process {X} is called a couig process if X represes he oal umber of eves ha have occurred i he ierval [0,.. X 0; X0=0- begis a ime =0. X is ieger valued 3. If s, he Xs X 4. X - Xs : umber of eves i [s,]

44 Couig Process Aleraely, he radom process {N} is called a couig process if i assumes oly ieger values ad i is a icreasig fucio of ime.

45 Types of Radom Processes Beroulli process : {X : }, X s are i.i.d. Beroulli variaes wih parameer p. Cosider a sequece of idepede ad ideical Beroulli rials. Le he radom variable Y i deoe he oucome of he i h rial so ha he eve {Y i =} idicaes success wih probabiliy p o he i h rial for all i ad he eve {Y i =0} idicaes failure wih probabiliy -p=q o he i h rial for all i.

46 Beroulli Process Hece he Y i s may be cosidered as idepedely ad ideically disribued radom variables. The radom process {Y } is called a Beroulli process wih P[Y =]=p ad P[Y =0]=-p, for all. Discussio: Classify Beroulli process 3 ypes

47 Biomial Process Biomial process : Cosider a Beroulli process {Y i } where he Y i s are idepedely ad ideically disribued Beroulli radom variables wih parameer p. Form aoher sochasic process {S } wih S = X +X +..+X The radom variable S =follows a Biomial disribuio wih parameers ad p.

48 Biomial Process The radom process {S : } is called a Biomial process. Discussio: Prelimiary classificaio of Biomial Process

49 Ca you classify Biomial process eve furher?? Firs order pmf: Biomial Process The firs order p.m.f. of he radom process {S } is give by k C p k p k, k 0,,..., P S E[ S ] p k Var[ S ] p p pq

50 Biomial Process Cosider S + = Y +Y + +Y + = Y +Y + +Y +Y + = S +X + Hece P[S + = k S = k] = P[Y + = 0]=-p P[S + = k+ S = k] = P[Y + = ]=p Ca you ow classify Biomial process eve furher?

51 Biomial Process The Biomial process is also called he Biomial couig process. Probabiliy geeraig fucio: Secod order joi p.m.f. S pz q z G l l k m m k p p k l m k m l S k S P l l m 0,,..,,,, 0,,..,,,

52 Observaio: Biomial Process The oal umber of rials T from he begiig of he process uil ad icludig he firs success is a geomeric radom variable. The umber of rials afer i- h success upo ad icludig he i h success will have he same disribuio as T Ca you geeralise?

53 Biomial Process If T is he umber of rials upo ad icludig he i h success, he T is he - fold covoluio of T wih iself ad follows a egaive biomial disribuio wih E[T ]=/p ad Var[T ]=/-p.

54 Biomial Process If i he Biomial process {S }, is large ad p is small such ha p is fiie, he he Biomial process approaches a Poisso process wih parameer λ=p.

55 Poisso Process Le E be ay radom eve ad {N} deoe he umber of occurreces of he eve E i a ierval of legh. Le p = P[N=]. The couig process {N} is called a Poisso process if i saisfies he followig posulaes:

56 Poisso Process. Idepedece : {N} is idepede of he umber of occurreces of E i a ierval prior o 0, i.e. fuure chages i N{} are idepede of he pas chages.. Homogeeiy i ime: p depeds oly o he legh of he ime ierval ad is idepede of where he ierval is siuaed i.e. p gives he probabiliy of he umber of occurreces of E i he ierval 0, 0 + for all 0 3. Regulariy: I a ierval of ifiiesimal legh h, he probabiliy of exacly oe eve occurrig is λh+oh, he probabiliy of zero eves occurrig is -λh+oh, he probabiliy of more ha oe eve occurrig is oh.

57 Poisso Process

58 Poisso Process Relax he posulaes: 3. Regulariy- Compoud Poisso Process muliple occurreces a ay isa. Homogeeiy i ime: λ is a fucio of ime λ- No-homogeeous. Idepedece: fuure depeds o prese- Markov process

59 Poisso Process Resul : The Poisso process defied above follows Poisso disribuio wih mea λ. i.e. e p P[N ], 0,,,...! Proof: Le {N} be a Poisso process wih parameer λ. We ow cosider

60 Poisso Processes- pmf Theorem o oal probabiliy homogeeiy ] [ N P p k k N P k N N P 0 ] [ ] / [ k k N P k N N P 0 ] [ ] [ k k N P k N P 0 ] [ ], [

61 Poisso Processes- pmf Assumig o be of ifiiesimal legh, we have ] [ ] [ 0 k N P k N P k ] [ ] [ ] [ ] [ 0] [ ] [ 0 o k N P k N P N P N P N P N P k 0 0 m k k k p p p p p p

62 Poisso Processes- pmf Regulariy Dividig hroughou by 0 k k o p o p o p p 0 o o p o p p p p k k o p p p p

63 Poisso Processes- pmf Takig limis as 0 o boh sides of he above equaio, dp p p, d A =0, we have p0 P[ N 0] P[ N 0/ N 0] P[ N 0]

64 Poisso Processes- pmf homogeeiy regulariy Takig limis as 0 o boh sides of he above equaio, 0] [ 0] [ N P N N P 0] [ 0], [ N P N P 0] [ 0] [ N P N P 0 p o o p p p 0 0 0

65 Poisso Processes- pmf dp d 0 p We ow solve he above sysem of differeial differece equaios ad subjec o he iiial codiios p00, p Cosider equaio amely,

66 dp Poisso Processes- pmf d dp p log subjec o p p p 0 0 d c p 0 e c p 0 Ke p 0 K A =0, p 0 e a

67 Poisso Processes- pmf Subsiuig = i equaio, we have dp p p subjec o p 0 0 d dp d dp d p p 0 e e, p 0 0 O solvig he above equaio, we have p p e e d e e e e d d c d c

68 Poisso Processes- pmf p A =0, e p c ce e 0 ce 0 p e b Subsiuig = i equaio, we have dp p p d

69 Poisso Processes- pmf. dp d p e, p 0 0 p e d e e d d c p e e e d c c

70 p A =0, Poisso Processes- pmf e p c p ce e 0 ce 0 e! c Proceedig i his maer, we have

71 Poisso Processes- pmf p e P[ N ]!, 0,,,... Hece, he umber of eves {N} i a ierval of legh follows a Poisso disribuio wih mea λ. Ca you classify Poisso process??

72 Poisso Processes Poisso process is a evoluioary process E[N]=λ ad Var[N]=λ. Furher, l N E, l Var N Hece λ is called he arrival rae of he process 0

73 Poisso Process Characerizaio: If {N} is a Poisso process wih mea λ, he he occurrece/ ier-arrival ime follows a expoeial disribuio wih mea / λ. Proof: Le {N} be a Poisso process wih parameer λ. The p e!, 0,,,...

74 Poisso Process If W deoes he ime bewee wo successive arrivals/ occurreces of he eve, he CDF of W is F W w P[W P[Nw 0] w] P[W w] F W w e w The above is he CDF of a expoeial variae wih mea /λ

75 Poisso Process If he ier-arrival imes are i.i.d. o ecessarily expoeial, he we have a reewal process.

76 Poisso Process- Properies Superposiio Decomposiio Markov Process Differece of wo Poisso processes is o Poisso.

77 Poisso Process- Properies Resul: The superposiio of idepede Poisso processes wih meas λ, λ,., λ, respecively is a Poisso process wih mea λ + λ +. +λ. Proof: Cosider idepede Poisso processes N, N,, σ, wih respecive meas λ, λ,.. λ.

78 Poisso Process- Properies The mome geeraig fuciom.g.f. of each N k is give by M E N k By propery of mome geeraig fucios, he mg.f. of he sum is give by e e N k k e N k N k

79 Poisso Process- Properies which is he mome geeraig fucio of a Poisso disribuio wih mea λ + λ +. +λ. k N N k M M k e k e k k e e

80 Poisso Process- Properies Hece by uiqueess propery, he sum of idepede Poisso processes wih meas λ, λ,., λ, respecively is a Poisso process wih mea λ=λ + λ +. +λ.

81 Poisso Process- Properies Decomposiio of Poisso process: A Poisso process N wih mea arrival rae λ ca be decomposed io muually idepede Poisso processes wih arrival raes p λ, p λ,.., p λ such ha p + p + + p = Proof: HW

82 Poisso Process Noe: If N is a Poisso process wih mea arrival rae λ, he he ime bewee k successive arrivals/ occurreces follows a k- Erlag disribuio. WHY??

83 Poisso Process Secod order joi p.m.f. : Give a Poisso process wih mea arrival rae λ, he secod order joi p.m.f. is obaied as follows:,,, P N N P N N N P,, N P N N P,,, P N N P

84 Poisso Process homogeeiy,, P N N P,,!! e e

85 Poisso Process Hece elsewhere e N N P 0,,,!!,

86 Poisso Process Similarly he hird order joi p.m.f. of he Poisso process wih mea arrival rae λ is give by

87 Poisso Process. elsewhere e N N N P 0,,,!!!,,

88 Poisso Process Auo-correlaio fucio : If N is a Poisso process wih mea arrival rae λ, he E[N] = λ Var[N] = λ E[N ] = Var[N] + E[N] = λ + λ

89 Poisso Process The auo-correlaio fucio of N is ow obaied as follows: ] by defiiio [, N N E R NN, N N N N E, N N N N E,, E N N N E

90 Poisso Process idepedece, E N N N E, E N N E N E, ] [, R NN,, mi, R NN

91 Poisso Process Auo-covariace fucio of a Poisso process : The auo-covariace fucio of a Poisso process N wih mea arrival rae λ is C NN, RNN, E[ N ] E[ N ] by defiiio mi, C NN, mi,

92 Poisso Process The correlaio fucio of a Poisso process N wih mea arrival rae λ is give by NN, Var[ N C mi mi NN, ],, Var[ N ]

93 Poisso Process Hece NN,,

94 Radom Processes Process wih saioary icremes: A radom process {X} is said o be a process wih saioary icremes if he disribuio of he icremes X+h-X depeds oly o he legh h of he ierval ad o o ed pois.

95 Radom Processes Wieer process/ Wieer-Eisei Process/ Browia Moio Process: A sochasic process {X} is said o be a Wieer Process wih drif µ ad varaice, if i X has idepede icremes ii every icreme X-Xs is ormally disribued wih mea µ-s ad variace -s

96 Poisso Process. For a Poisso process wih parameer ad for show ha Soluio : Cosider def k s s C N k s P N k k k,... 0,,, N k s N P N P N k s N P

97 Poisso Process by defiiio N P k s N k s N P N P k s P N k s N P!!! e k s e k s e k s k s

98 Poisso Process k k s k k s e s e e s e k k!!! k k k k s s C k k k k s s C

99 Poisso Process Hece he proof. k k k k s s C k k k s s C N k s N P k s s C k k k,... 0,,,

100 Poisso Process. Suppose ha cusomers arrive a a bak accordig o a Poisso process wih a mea rae of 3 per miue. Fid he probabiliy ha durig a ime ierval of miues a exacly 4 cusomers arrive b more ha 4 cusomers arrive. Soluio: Le N be a Poisso process wih mea arrival rae λ.

101 Poisso Process Give ha λ=3. k e Hece P N k, k 0,,,.., k! a P N 4 e ! e ! 0.33

102 Poisso Process b P N 4 PN 4 P P X 0 PX X P X 3 P X 4 e 6 6 0! 0 e 6 6! e 6 6! e 6 6 3! 3 e 6 6 4! 4 = 0.75

103 Poisso Process 3. If a cusomer arrives a a couer accordig o a Poisso process wih a mea rae of per miue, fid he probabiliy ha i 5 cusomers arrive i a 0 miue period ii he ierval bewee successive arrivals is a more ha miue b bewee ad miues c 4 miues or less iii he firs wo cusomers arrive wihi 3 miues iv he average umber of cusomers arrivig i hour.

104 Poisso Process Soluio: Le N deoe he umber of cusomers who arrive a he couer i a ierval of legh. We are give ha σ follows a Poisso process wih mea arrival rae λ= per miue. Therefore we have, e P[ N ]!, 0,,,...

105 Poisso Process i P N0 5 e 0 x0 5! e 5! ii Sice N is a Poisso process wih mea arrival rae λ=, he ier-arrival ime T bewee successive arrivals follows a expoeial disribuio wih p.d.f.

106 Poisso Process a 0, e f d e T P e 0 e 0.353

107 Poisso Process b P T e d e e 4 e e e

108 Poisso Process c PT 4 = -e -4 =-e -8 = iii Sice N is a Poisso process wih mea arrival rae λ, we kow ha he ierarrival ime follows a expoeial disribuio wih mea /λ. If T i deoes he ier arrival ime bewee he i- h cusomer ad he i h cusomer, he he ime ake for he firs

109 Poisso Process cusomers o arrive is give by T +T. Sice T adt are idepedely ad ideically disribued expoeial variaes wih parameer λ, T +T follows a secod order Erlag disribuio wih p.d.f. e, 0, 0 e, 0

110 Poisso Process =0.986 d e T P T ] [ e e e e 7 6 e

111 Poisso Process iv Sice E[N]= λ, he average umber of cusomers arrivig i oe hour is E[N60]= x60 = A radioacive source emis paricles a he rae of 5 per miue i accordace wih a Poisso process. Each paricle has a probabiliy of 0.6 of beig recorded. Fid he probabiliy ha 0 paricles are recorded i a 4 miue period.

112 Poisso Process Soluio: Give ha he emissio of paricles follows a Poisso process wih arrival rae λ=5 per miue. From he decomposiio propery of Poisso process, he umber of emied paricles N follows a Poisso process wih mea arrival rae λ = λp =5x0.6/ mi i.e. λ =3 per miue

113 Poisso Process Hece 0,,,...,! ] [ e N P 0! 0! e x e P N

114 Poisso Process 5. A machie goes ou of order wheever a compoe fails. The failure of his par follows a Poisso process wih a mea rae of per week. Fid he probabiliy ha weeks have elapsed sice las failure. If here are 5 spare pars of his compoe i a iveory ad ha he ex supply is o due i 0 weeks, fid he probabiliy ha he machie will o be ou of order i he ex 0 weeks.

115 Poisso Process Soluio: Le X deoe he umber of failures of he compoe i uis of ime. The X follows a Poisso process wih mea failure rae= mea umber of failures i a week = λ = P[ weeks have elapsed sice las failure] = P[ Zero failures i he weeks sice las failure] =P[X=0]

116 P[X Poisso Process =e - =e - =0.35 There are oly 5 spare pars ad he machie should o go ou of order i he ex 0 weeks. Hece 0] e 0! 0 P[ he machie will o be ou of order i he ex 0 weeks] = P[X0 5]

117 Poisso Process e e 0 e 0 e 0 e 0 e 0 0 0!!! 3! 4! 5! e P[X0 5]=0.068

118 Saioary Processes 6. Deermie he mea ad variace of he radom process {X} is give by P X a a a a,,3,.. Verify wheher {X} is saioary or o., 0

119 Soluio: Saioary Processes Cosider he radom process {X} defied by P X a a a a,,,3,.. 0 Sice he firs order p.m.f. of X is a fucio of, he radom process {X}

120 Saioary Processes is o a saioary process. I is a evoluioary process. Now, he mea of he process is give by E X 0 P[ X a a ] a by defiiio a a a 3 a a a 3... a a a...

121 Saioary Processes E[X]= We ow compue Var[X]... 3 a a a a a a a a a a a a a a a a

122 Saioary Processes Var [X]=E[X ]-{E[X]}. Cosider 0 ] [ X P X E 0 ] [ ] [ X P 0 0 ] [ ] [ X P X P a a x a a x a x a a a a a a a a a

123 Saioary Processes a 3 a a a 3 a a a 3 a a E[X ]=+a- =3a-

124 Saioary Processes Var[X]=a+- Var[X]=a. 7. Show ha he radom process {X} defied by. X Acos where A ad ω are cosas ad θ is a uiform radom variable over 0,π is wide sese saioary. Furher, deermie wheher {X} is mea ergodic, correlaio ergodic.

125 Saioary Processes Soluio : A radom process {X} is said o be WSS if i saisfies he followig codiios:. E[X] is a cosa wih respec o ime.. The auo-correlaio fucio R XX, is a fucio of he legh of he ime differece. i.e. R XX, = R XX - 3. E[X ]<.

126 Saioary Processes Cosider he radom process give by X Acos where A ad ω are cosas ad θ is uiform disribued i 0,π. Hece, he p.d.f. of θ is give by f,0 0, elsewhere Cosider E[ X ] EAcos

127 Saioary Processes AE cos A cos 0 A si A si d 0 si E[X]=0, a cosa wih respec o ime

128 Saioary Processes We ex cosider he auo-correlaio fucio R XX,+=E[XX+] } { cos cos A A E cos cos E A cos cos E A cos cos E A E A

129 A E Cosider Saioary Processes cos cos Subsiuig he above expressio i R XX,+, we have A cos cos d E si 4 si si

130 R XX, Saioary Processes A cos Cosider E[X ]= R XX,= R XX 0 A Hece, he give radom process is WSS. <. Now, he radom process {X} is said o be mea ergodic if E[X] = A[X], where T l A [ X ] x d T T T

131 Saioary Processes Cosider T l A[ X ] Acos d T T T T l A cos d T T T A T l T si T T A l T T si T si T = 0 sice siθ for all θ ----4

132 Saioary Processes From equaios ad 4, we see ha E[X] = A[X]. Hece {X} is mea ergodic. The radom process {X} is said o be correlaio ergodic if E[XX+] = A [XX+] where T l A[ X X ] x x d T T T

133 Saioary Processes Cosider T T d A A T T l X X A } { cos cos ] [ T T d T T l A cos cos T T T T d T T l A d T T l A cos cos T T T T d T T l A T T l A cos si

134 Saioary Processes A l sit 4 T A cos l T T si T T T T A cos l 0 si, T A XX, A cos From equaios ad 5, we see ha E[XX+] = A [XX+]

135 Saioary Processes Hece he give radom process {X} is correlaio ergodic. Noe: Sice he above process is WSS, i is firs order saioary. 8. Show ha he process X Acos Bsi where A ad B are ucorrelaed radom variables is wide sese saioary if

136 Soluio : HW E Saioary Processes A E B 0; E A E B Soluio : HW 9. Deermie he mea ad he variace of he radom process X Acos where A ad ω are cosas ad θ is uiformly disribued i 0,

137 Saioary Processes 0. Two radom processes X ad Y are defied by X Acos 0 Bsi 0 ad Y Bcos 0 Asi 0. Show ha X ad Y are joily wide sese saioary, if A ad B are ucorrelaed zero mea radom variables havig he same variace ad 0 is a cosa. Soluio: HW

138 Saioary Processes. If X is a WSS process wih auocorrelaio fucio R XX =Ae - where A is ay cosa, obai he secod order mome of he radom variable X8-X5. Soluio : HW. Give a radom variable Y wih characerisic fucio φω=e[e iyω ] ad a a radom process X=Cosλ+Y. Show

139 Saioary Processes ha X is WSS if φ=φ=0. Soluio : A radom process {X} is said o be WSS if i saisfies he followig codiios:. E[X] is a cosa wih respec o ime.. The auo-correlaio fucio R XX, is a fucio of he legh of he ime differece. i.e. R XX, = R XX - 3. E[X ]<.

140 Saioary Processes Sice φω is he characerisic fucio of he radom variable Y, we have φω=e[e iyω ] =E[cos Yω +i si Yω] Sice φ=0, we have E[cos Y +i si Y]=0 E[cos Y] + i E[si Y]=0 WHY?? This implies ha E[cos Y] =0 ad E[si Y] =0 WHY??

141 Saioary Processes Also, φ=0 yields E[cos Y +i si Y]=0 E[cos Y] + i E[si Y]=0 This implies ha E[cos Y] =0 ad E[si Y] = Cosider E[X]=E[Cosλ+Y] = E[cos λ cos Y si λ si Y] = cos λ E[cos Y] si λ E[si Y] = 0 from, a cosa.

142 Saioary Processes Now cosider R YY, +=E[XX+] Therefore, R XX, += E[Cosλ+Y Cosλ[+ ]+Y] =E[{Cos{λ+Y}+{ λ[+ ]+Y} +Cos{λ+Y }-{λ[+]+y}}/] = E[cosλ+ λ +Y]/+ E[cos- λ]/ why?? = E[cos λ+ λ cos Y- si λ+ λ si Y]/ + E[cosλ]/ why??? = cos λ+ λ E[cos Y]/ - si λ+ λ E[si Y]/+ E[cosλ]/

143 Saioary Processes Hece R XX, += E[cosλ]/ from, is exclusively a fucio of he legh of he ime ierval ad o he ed pois. Furher, E[X ]= R XX,= E[cosλx0]/ = / ]<. Hece he give radom process {X} is WSS

144 Saioary Processes HW 3. Verify wheher he radom process X=ACosω+θ is WSS give ha A ad ω are cosas ad θ is uiformly disribued i i -π, π ii 0, π iii 0, π/.

145 Saioary Processes 4. If X= Y cos ω + Z si ω, where Y ad Z are wo idepede ormal RVs wih E[X]=E[Y]=0,E[X ]=E[Y ]= ad ω is a cosa, prove ha {X} is a saioary process of order. Soluio: HW

146 Saioary Processes 5. Verify wheher he radom process X=Ycos ω where ω is a cosa ad Y is a uiformly disribued radom variable i 0, is a sric sese saioary process. Soluio : HW

147 Sie Wave Processes Defiiio: A radom process {X} of he form X = A cos ω+θ or X = A siω+θ where ay o-empy combiaio of A, ω, θ are radom variables is called a sie wave process. A is called he ampliude ω is called he frequecy θ is called he phase.

148 Radom Processes Orhogoal Processes: Two radom processes {X} ad {Y} are said o be orhogoal if heir cross correlaio fucio R XY, =0. Ucorrelaed Processes: Two radom processes {X} ad {Y} are said o be ucorrelaed if heir cross covariace fucio C XY, =0. i.e. R XY, =E[X ]E[Y ]

149 Radom Processes Noe: Two idepede radom processes are ucorrelaed bu he coverse eed o be rue. Ca you give a Couerexample??? Remark: If wo radom processes {X} ad {Y} are saisically idepede, he heir cross-correlaio fucio is give by R XY, =E[X ]E[Y ]

150 Radom Processes Remark: If wo radom processes {X} ad {Y} are a leas WSS, he R XY, =XY, a cosa wih respec o ime.

151 Normal/ Gaussia Process Defiiio: A radom process {X} is called a Normal/ Gaussia process if is h order joi desiy fucio is give by T X X X C X X X X e C x x x f ] [ ] [,..,, ;,..,,

152 Normal/ Gaussia Process where C X is he covariace marix give by T X X X X X X X X.. X C C C C C C C C C C

153 Normal/ Gaussia Process where C i,j =C XX i, j is he auo-covariace fucio of X. Also, X X T is he raspose of he marix where X i = E[X i ]. X X Remarks:. A Gaussia process is compleely deermied by is mea ad auocovariace fucios. WHY???

154 Normal/ Gaussia Process. If a Gaussia process is WSS, he i is sricly saioary. WHY??? Hi: If {X} is WSS, he is mea is a cosa wih respec o ime ad is auocovariace fucio is oly a fucio of he legh of he ime ierval ad o o he ed pois.

155 Normal/ Gaussia Process 6. If {X} is a Gaussia process wih mea µ=0 ad he auo-covariace fucio C XX, =6e - -. Fid he probabiliy ha X0 8 ad X0-X8 4. Soluio: Cosider he Gaussia process wih mea µ=0 or E[X]=0 ad he auo-covariace fucio C XX, =6e - -.

156 Normal/ Gaussia Process The radom variable X E[ X where i is ay fixed ime poi follows a sadard ormal disribuio. i C XX i, i i ] P X 0 8 X 0 E[ X 0] P C XX P Z ,0 6 P[ Z ]

157 Normal/ Gaussia Process Cosider he radom variable X0-X8. The E[X0-X8]=E[X0]-E[X8]=0. Var[X0-X8] =Var[X0]+Var[X8]-Cov[X8,X0] =6 + 6 x6e 8-0 = 3 3 e - = = 7.669

158 Normal/ Gaussia Process P 4 0 X 0 X 8 4 P Z P Z P[0 Z 0.76] = P[Z< ] = [ ] = 0.558

159 Radom Telegraph Process Le {X} be a radom process saisfyig he followig codiios:. {X} assumes oly oe of he possible levels + or - a ay ime.. X swiches back ad forh bewee is wo levels radomly wih ime. 3. The umber of level rasiios i ay ime ierval of legh is a Poisso radom variable. i.e. he probabiliy of

160 X is called a semi-radom elegraph sigal process. Radom Telegraph Process exacly k rasiios wih average rae of k rasiios λ is give by e 0,,, Trasiios occurrig i ay ime ierval are saisically idepede of he rasiios i ay oher ierval. 5. The levels a he sar of ay ierval are equally probable. k!, k

161 Radom Telegraph Process Aleraely, if N represes he umber of occurreces of a specified eve i 0,, N is Poisso wih parameer λ ad Y= - N, he he radom process Y is called a semi-radom elegraph sigal process. If Y is a semi-radom elegraph sigal process, A is a radom variable which is

162 Radom Telegraph Process idepede of Y ad assumes values + ad - wih equal probabiliy, he he rasom process {X} defied by X=AY is called a radom elegraph sigal process We ow show ha he radom elegraph process defied above is a WSS process. To show ha X is WSS, we eed o prove he followig:

163 Radom Telegraph Process. E[X] is a cosa wih respec o ime.. The auo-correlaio fucio R XX, is exclusively a fucio of he legh of he ime differece. i.e. R XX, = R XX - 3. E[X ]<. Cosider E[X] = E[AY]

164 Radom Telegraph Process E[X] = E[A]E[Y] A ad Y are idepede. Cosider E[A] = PA= + - PA=- = ½ + -½ = Now, E[A ] = PA= + - PA=- = ½ + ½ =

165 Radom Telegraph Process By defiiio of Y, Y akes he value + wheever he umber of level rasiios N i he ierval 0, is eve ad Y assumes he value - wheever he umber of level rasiios N i he ierval 0, is odd. Hece P[Y=] = P[N= eve] = P[N= 0] + P[N= ] + P[σ= 4] +..

166 Radom Telegraph Process e e! e 4! e...! 4! e e e P[Y=] = e λ cosh λ

167 Radom Telegraph Process Also, P[Y=-] = P[N= odd] = P[N= ] + P[N= 3] + P[σ= 5] +.. e! e 3! e...! 3! P[Y= -] = e λ sih λ

168 Radom Telegraph Process E[Y] = e λ cosh λ + - e λ sih λ e e e E[Y] = e λ e e e Subsiuig expressios ad 5 i E[X], we obai E[X] = 0, which is a cosa wih respec o ime.

169 Radom Telegraph Process Hece E[X] is a cosa wih respec o ime. We ex cosider he auo-correlaio fucio of X, viz, R XX,+= E[XX+] = E[AY AY+] = E[A YY+]

170 Radom Telegraph Process R XX,+= E[A ]E[YY+] A is idepede of Y = R YY,+ We ow compue R YY,+= E[YY+] If Y =, he Y+ =, if he umber of level rasiios i he ierval, + is eve.

171 Radom Telegraph Process Hece P[Y, + = Y = ] = P[ umber of level rasiios i, + is eve] P[Y =,Y, + = ] = e λ cosh λ = P[Y, + = Y = ] P[Y = ] = e λ cosh λ e λ cosh λ a

172 Radom Telegraph Process Similarly, P[Y =,Y, + = -] = e λ sih λ e λ cosh λ b P[Y = -,Y, + = ] = e λ sih λ e λ sih λ c P[Y = -,Y, + = -] = e λ cosh λ e λ sih λ d

173 Radom Telegraph Process Hece, R YY,+= e λ cosh λ e λ cosh λ + - e λ sih λ e λ cosh λ + - e λ sih λ e λ sih λ + -- cosh λ e λ sih λ = e λ e λ {cosh λ cosh λ - sih λ cosh λ - sih λ sih λ + cosh λ sih λ}

174 Radom Telegraph Process R YY,+ = e λ e λ {cosh λ [cosh λ + sih λ] - sih λ [cosh λ + sih λ]} = e λ e λ [cosh λ - sih λ] [cosh λ + sih λ] = e λ e λ e λ e λ R YY,+ = e λ Subsiuig equaio 8 i R XX,+ = R YY,+ we have

175 Radom Telegraph Process R XX,+ = e λ, which is exclusively a fucio of. Furher E[X ] = R XX, = <. Hece, he radom elegraph process X is a WSS process.

176 Radom Telegraph Process Remark: From equaios 3, 4, 5 ad 8 we see ha eve hough he auo-correlaio fucio of he semi-radom elegraph sigal {Y} is exclusively a fucio of, sice he firs order p.m.f is iself a fucio of, {Y} is o eve a firs order saioary process. I is a evoluioary process.

177 Markov Chais Discree-Time Markov Chais: Wihou loss of geeraliy, assume ha he parameer space T={0,,,,.}. Hece he sae of he sysem is observed a he ime pois 0,,, These observaios are deoed by X 0,X,X, If X =j, he he sae of he sysem a ime sep is said o be j. Le p j =P[X =j] deoe he probabiliy ha X is i sae j.

178 Markov Chais I oher words, p j deoes he p.m.f. of he radom variable X. The codiioal p.m.f. p jk m,= P[X =k X m =j] is called he rasiio p.m.f.

179 Markov Chais Discree-Time Markov Chais: Le {X } be a discree-ime ieger valued Markov chai sarig a =0 wih iiial PMF p j0 P[X 0 j], j 0,,,... The joi PMF for he firs + values of he process is X i,x i,.., X i P[X i / X i ]...PX i X i P[X i ] P

180 Discree-Time Markov Chai If he oe-sep sae rasiio probabiliies are fixed ad do o chage wih ime, i.e. P[X j/ X i] pij {X } is said o have homogeeous rasiio probabiliies. A Markov chai is said o be a homogeeous Markov chai if he rasiio p.m.f. p jk m, depeds oly o he differece -m.

181 Discree-Time Markov Chai The homogeeous sae rasiio probabiliy saisfies he followig codiios: 0 j p p ij ij,i,,.., sice he saes are muually exclusive ad collecively exhausive

182 Discree-Time Markov Chai The rasiio probabiliy marix P : The oe- sep rasiio probabiliies of a discree parameer Markov Chai are compleely specified i he form of a rasiio probabiliy marix.p.m. give by P=[p ij ] P p p. p 00 0 i0 p p p 0. i p p p 0. i The row sum is for each row of P Sochasic marix All he eries lie i [0,] The sochasic marix is said o be doubly sochasic iff all he colum eries add up o for every colum. A sochasic marix is said o be regular if all is eries are posiive.

183 Discree-Time Markov Chai The joi PMF of X, X -,, X 0 is give by X i, X i,.., X i p...p p 0 P 0 0 i,i i0,i, i 0 Thus {X } is compleely specified by he iiial PMF ad he marix of he oe-sep rasiio probabiliies P

184 Discree-Time MC Compuer Repair Example Two agig compuers are used for word processig. Whe boh are workig i morig, here is a 30% chace ha oe will fail by he eveig ad a 0% chace ha boh will fail. If oly oe compuer is workig a he begiig of he day, here is a 0% chace ha i will fail by he close of busiess. If eiher is workig i he morig, he office seds all work o a ypig service. Compuers ha fail durig he day are picked up he followig morig, repaired, ad he reured he ex morig. The sysem is observed afer he repaired compuers have bee reured ad before ay ew failures occur.

185 Saes for Compuer Repair Example Idex Sae Sae defiiios 0 s = 0 No compuer has failed. The office sars he day wih boh compuers fucioig properly. s = Oe compuer has failed. The office sars he day wih oe workig compuer ad he oher i he shop uil he ex morig. s = Boh compuers have failed. All work mus be se ou for he day.

186 Eves ad Probabiliies for Compuer Repair Idex Curre sae Eves Probabiliy Nex sae 0 s 0 = 0 Neiher compuer fails. 0.6 s' = 0 Oe compuer fails. 0.3 s' = Boh compuers fail. 0. s' = s = Remaiig compuer does o fail ad he oher is reured. Remaiig compuer fails ad he oher is reured. 0.8 s' = 0 0. s' = s = Boh compuers are reured..0 s' =

187 Sae-Trasiio Marix ad Nework The eves associaed wih a Markov chai ca be described by he m m marix: P = p ij. For compuer repair example, we have: P

188 Sae-Trasiio Marix ad Nework Sae-Trasiio Nework Node for each sae Arc from ode i o ode j if p ij > For compuer repair example:

189 Discree-Time Markov Chai The -sep rasiio probabiliies : Le P=[p ij ] be he marix of -sep rasiio probabiliies, where p ij =P[X +k =j/ X k =i] = P[X =j/x 0 =i] for all 0 ad k 0, sice he rasiio probabiliies do o deped o ime. The P=P

190 Discree-Time Markov Chai Cosider P p p i] X k / k]p[x X j/ P[X i] P[X i] i]p[x X k / k]p[x X j/ P[X i] P[X i] X k, X j, P[X i] X k / X j, P[X kj ik k kj ik ij j i, p p p

191 Discree-Time Markov Chai P=PP=P P=P-P =P-PP =P-P =P Hece he -sep rasiio probabiliy marix is he h power of he oe-sep rasiio probabiliy marix.

192 Discree-Time Markov Chai Chapma- Kolmogorov Equaios: p p rp r ij ik kj k Ierpreaio: RHS is he probabiliy of goig from i o k i r seps & he goig from k o j i he remaiig r seps, summed over all possible iermediae saes k.

193 Discree-Time Markov Chai Sae Probabiliies: The sae probabiliies a ime are give by he row vecor p j ={p j }. Now, p j i P[X j/ X Therefore p=p-p. i]p[x Similarly, p=p0p=p0p, =,, Hece he sae PMF a ime is obaied by muliplyig he iiial PMF by P. i] i p ij p i

194 Discree-Time Markov Chai Limiig Sae/ Seady-sae Probabiliies: Le {X } be a discree-ime Markov chai wih N saes P[X =j] - he probabiliy ha he process is i sae j a he ed of he firs rasiios, j=,,..,n. The P[ X j] p j N k P[ X 0 i] p ij

195 Discree-Time Markov Chai - Seady-sae Probabiliies As, he - sep rasiio probabiliy p ij does o deped o i, which meas ha P[X =j] approaches a cosa as. The limiig- sae probabiliies are defied as lim P[ X Sice, j], j k j,,..., N pij pik p kj lim p ij lim k p ik p kj k k p kj

196 Discree-Time Markov Chai - Seady-sae Probabiliies Defiig he seady-sae/ limiig sae probabiliy vecor, we have j P j k j k p kj The las equaio is due o he law of oal probabiliy. The probabiliy π j is ierpreed as he log proporio of ime ha he MC speds i sae j.,...,, N

197 Discree-Time Markov Chai Classificaio of Saes: A sae j is said o be accessible from sae i j ca be reached from i if, sarig from sae i, i is possible ha he process will ever eer sae j. p ij >0 for some >0. Two saes ha are accessible from each oher are said o commuicae wih each oher.

198 Discree-Time Markov Chai - Classificaio of Saes Commuicaio iduces a pariio of saes Saes ha commuicae belog o he same class All members of a class commuicae wih each oher. If a class is o accessible from ay sae ouside he class, he class is said o be a closed commuicaig class.

199 Discree-Time Markov Chai - Classificaio of Saes A Markov chai i which all he saes commuicae is called a irreducible Markov Chai. I a irreducible Markov chai, here is oly oe class.

200 Discree-Time Markov Chai - Classificaio of Saes Saes ha he process eers ifiiely ofe ad saes ha he process eers fiiely ofe. Process will be foud i hose saes ha i eers ifiiely ofe

201 Discree-Time Markov Chai - Classificaio of Saes Probabiliy of firs passage from sae i o sae j i rasiios - f ij The codiioal probabiliy ha give ha he process is i sae i, he firs ime he process eers sae j occurs i exacly rasiios.

202 Discree-Time Markov Chai - Classificaio of Saes Probabiliy of firs passage from sae i o sae j f ij f ij f ij Codiioal probabiliy ha he process will ever eer sae j give ha i was iiially i sae i

203 Discree-Time Markov Chai - Classificaio of Saes Clearly ad f ij p ij f p f ij l j il lj f ii deoes he probabiliy ha a process ha sars a sae i will ever reur o sae I If f ii =, he sae i is called a recurre sae. If f ii <, he sae i is called a rasie sae.

204 Discree-Time Markov Chai - Classificaio of Saes Sae j is called rasie o-recurre if here is a posiive probabiliy ha he process will ever reur o j agai if i leaves j recurre persise if wih probabiliy, he process will eveually reur o j afer i leaves j A se of recurre saes forms a sigle chai if every member of he se commuicaes wih all he members of he se.

205 Discree-Time Markov Chai - Classificaio of Saes Recurre sae j is called a periodic sae if here exiss a ieger d, d>, such ha p jj is zero for all values of oher ha d, d, 3d, ; d is called he period. If d=, j is called aperiodic. posiive recurre sae if, sarig a sae j he expeced ime uil he process reurs o sae j is fiie; oherwise i is called a ull-recurre sae.

206 Discree-Time Markov Chai - Classificaio of Saes Posiive recurre saes are called ergodic saes A chai cosisig of ergodic saes is called a ergodic chai. A sae j is called a absorbig rappig sae if p ij =. Thus, oce he process eers a absorbig/ rappig sae, i ever leaves he sae.

207 Discree-Time Markov Chai - Classificaio of Saes Saes Recurre/ persise Trasie/ No - recurre Posiive recurre Null Recurre Periodic Aperiodic Ergodic Ergodic

208 Discree-Time Markov Chai - Classificaio of Saes If a Markov chai is irreducible, all is saes are of he same ype. They are eiher all rasie or all ull persise or all o-ull persise. Furher, all he saes are eiher aperiodic or periodic wih he same period. If a Markov chai is fiie ad irreducible, he all is saes are o-ull persise.

209 Coiuous-Time Markov Chai A radom process {X/ 0} is a coiuous-ime Markov chai if, for all s, 0 ad oegaive iegers i,j,k, I a coiuous ime Markov chai, he codiioal probabiliy of he fuure sae a ime +s, give he prese sae a s ad all pas saes depeds oly o he prese sae ad o o he pas. i s X j s X P s u k u X i s X j s X P ],0, [

210 Coiuous-Time Markov Chai If i addiio, P[X+s=j/Xs=i] is idepede of s, he he process {X, 0} is said o be ime-homogeeous or have he ime-homogeeiy propery. Time-homogeeous Markov chais have saioary or homogeeous rasiio probabiliies. Le p ij PX s j Xs i p j P X j

211 Coiuous-Time Markov Chai I oher words, p ij is he probabiliy ha a MC presely i sae i will be i sae j afer a addiioal ime ad p j is he probabiliy ha a MC is i sae j a ime. The rasiio probabiliies saisfy 0 j p p ij ij Furher j p j

212 Coiuous-Time Markov Chai Chapma-Kolmogorov equaio p ij s p p s k ik ks

213 Markov Chais 7. A ma eiher drives a car or caches a rai o go o office each day. He ever goes days i a row by rai, bu if he drives oe day, he he ex day he is jus as likely o drive agai as he is o ravel by rai. Now suppose ha o he firs day of he week, he ma ossed a fair die ad drove o work if ad oly if a 6 appeared, fid i he probabiliy ha he akes a rai o he hird day ad ii he probabiliy ha he drives o work i he log ru.

214 Markov Chais The ravel paer forms a Markov chai, wih sae space = rai,car The TPM of he chai is give by The iiial sae probabiliy disribuio is give by 0 p 5 6 6, sice Pravelig by car=pgeig a 6 i he oss of he die= /6. Also, Pravelig by rai= 5/6 P 0

215 Markov Chais Now, p p P p 3 p P Therefore, Phe ma ravels by rai o he hird day = /4 Le π = π, π be he limiig form of he sae probabiliy disribuio or saioary

216 Markov Chais sae disribuio of he Markov chai. By propery of π, πp= π 0,, ad Equaios ad are he same. Alogwih he equaio π + π =, ---3

217 Markov Chais sice π is a probabiliy disribuio, we obai Hece π =/3, π = /3. Therefore P[ma ravels by car i he log ru]= /3

218 Markov Chais 8. Three boys A,B ad C are hrowig a ball o each oher. A always hrows he ball o B ad B always hrows he ball o C, bu C is jus as likely o hrow he ball o B as o A. Show ha he process is Markovia. Fid he rasiio probabiliy marix ad classify he saes. Soluio: Le A,B,C deoe he saes of he Markov chai.

219 Markov Chais The rasiio probabiliy marix of {X } is give by 0 P Sice, he saes of X deped oly o X - ad o o X -, X -3,, he process {X } is a Markov chai. We firs observe ha he chai is fiie. Draw he pm/ ework 0

220 Furher sice sae A is aperiodic WHY??? all he saes are aperiodic, ergodic Markov Chais We observe ha all he saes commuicae wih each oher Hece, he MC is irreducible Sice he MC is fiie, all he saes are posiive recurre.

221 0 Markov Chais 9. The rasiio probabiliy marix of a Markov chai {X } wih hree saes , ad 3 is ad he P iiial disribuio is. Fid i X 3 ii. Soluio: Cosider p ,0.,0. p ,0.,0. P PX, X 3, X 3, X 3 0

222 . P P [ p ] X Markov Chais P PX X ip X i i 3 X X P X PX 3 X P X PX 3 X P X 3 p P[ X 0 ] p3p[ X 0 ] p33p[ X ] 0

223 Markov Chais P[X =3] = =0.79 ii P P P X, X 3, X 3, X 3 0 X 3 X 3, X 3, X 0 PX 3, X 3, X 0 codiioal probabiliy X X 3P X p3. p33 p3. P X 0 X 3, X 0 P X 3, X Markov propery = 0.4x0.3x0.x0.=

224 Markov Chais 0. A gambler has Rs.. He bes Re. a a ime ad wis Re. wih probabiliy 0.5. He sops playig if he loses Rs. or wis Rs. 4. Wha is he rasiio probabiliy marix of he relaed Markov chai? Soluio : HW

225 Markov Chais. There are whie marbles i ur A ad 3 red marbles i ur B. A each sep of he process, a marble is seleced from each ur ad he marbles seleced are ierchaged. Le he sae ai of he sysem be he umber of red marbles i A afer i chages. Wha is he probabiliy ha here are red marbles i A afer 3 seps? I he log ru, wha is he probabiliy ha here are red marbles i ur A? Soluio : HW

226 Markov Chais.Fid he aure of he saes of he Markov chai wih he TPM, Soluio : HW P =

227 Thak You