Markov Model. Model representing the different resident states of a system, and the transitions between the different states

Save this PDF as:

Size: px
Start display at page:

Download "Markov Model. Model representing the different resident states of a system, and the transitions between the different states"


1 Markov Model Model representing the different resident states of a system, and the transitions between the different states (applicable to repairable, as well as non-repairable systems) System behavior that varies randomly with time and space is known as a stochastic process. A stochastic process that meets the following requirements is a Markovian process, otherwise non-markovian Requirements:. system states must be identifiable. lack of memory: future states are independent of all past states, except the present state 3. stationary: probability of transition between states is the same at all times Requirements & 3 are met by systems with probability distributions characterized by a constant hazard rate. Markov Approach: - discrete (time or space) Discrete Markov Chain - continuous (time) Continuous Markov Process

2 Discrete Markov Chain - State System 3/4 remaining in State /4 leaving State P[remaining in State ] + P[leaving State ] = ¾ + ¼ = The behavior of the system (probability of residing in a state after a number of time intervals) can be illustrated by a tree diagram. Probability of any branch - multiply the probability of each step in the branch Probability of residing in a state - sum of branch probabilities that lead to that state

3 State probabilities (time dependent) of the -state system:

4 Time interval State probability State State ½ = ½ = (½)(½) + (½)(¼) = (½)(½) + (½)(3/4) = Probability start in State start in State Number of Time Intervals State State As the # of time intervals increase, state probability tends to a constant (limiting) value limiting state probability Transient behavior (time-dependent state probability) depends on the initial condition Limiting state probability of ergodic system (or process) is independent of the initial condition.

5 Ergodic System: every state of a system can be reached from every other state directly or through intermediate states Systems with absorbing states are not ergodic. Absorbing State: a state once entered cannot be left e.g. a system failure state in a mission oriented system Evaluation Procedure using Markov Model: - develop Markov model for the component (or system) - evaluate state probability (time dependent or limiting state) using: o Tree diagram: impractical for large systems or a large number of time intervals o Stochastic Transitional Probability Matrix o Other techniques for continuous Markov process will be discussed later

6 Stochastic Transitional Probability Matrix Square matrix (order = number of states) Rows : from states Columns: to states Element : probability value from one state to another P ij = prob of transition from state i to state j from nodes P = to nodes.. n P P.. P n P P.. P n n P n P n.. P nn sum of probabilities in each row must be unity Transient behavior: State probabilities after n intervals is given by, P(n) = P(0).P n where P(0) is the initial probability vector (state probabilities at initial condition) Limiting State Probability: repeated multiplications of P until resulting P does not change with further multiplications. αp = α where α = limiting probability vector

7 Example: /4 Stochastic Transitional Probability Matrix, P = /4 3/4 If the system starts in State, Initial probability vector P(0) = [ 0] State probabilities after interval, P() = P(0).P = [ 0]. = [ ] /4 3/4 Limiting State Probabilities: α = [P P ] Using the equation, P = limiting probability of being in State P = limiting probability of being in State αp = α [P P ] = [P P ] () /4 3/4 P + P = () Solving () & (), P = and P = 0.667

8 Absorbing States System states when once entered, cannot be left until the system starts a new mission. e.g. failure states in mission oriented systems Need to evaluate: How many time intervals does the system operate on average before entering the absorbing state? Expected # of time intervals, where, N = [ I Q ] - I = identity matrix Q = truncated matrix created by deleting row(s) and column(s) associated with the absorbing states /4 /3 /3 3 absorbing state

9 Example: /4 3 absorbing state Stochastic Transitional Probability Matrix, P = 3 3/4 / Truncated Matrix (deleting Row 3 & Column 3 from P ) Q = 3/4 /4 0 Average number of time intervals spent in each state before entering the absorbing state, N = [ I Q ] /4 /4 0 /4 -/4 0 = { } - = = i.e. average no. of time intervals spent in state given that the system starts in state is 4.