Markov Model Model representing the different resident states of a system, and the transitions between the different states (applicable to repairable, as well as non-repairable systems) System behavior that varies randomly with time and space is known as a stochastic process. A stochastic process that meets the following requirements is a Markovian process, otherwise non-markovian Requirements:. system states must be identifiable. lack of memory: future states are independent of all past states, except the present state 3. stationary: probability of transition between states is the same at all times Requirements & 3 are met by systems with probability distributions characterized by a constant hazard rate. Markov Approach: - discrete (time or space) Discrete Markov Chain - continuous (time) Continuous Markov Process
Discrete Markov Chain - State System 3/4 remaining in State /4 leaving State P[remaining in State ] + P[leaving State ] = ¾ + ¼ = The behavior of the system (probability of residing in a state after a number of time intervals) can be illustrated by a tree diagram. Probability of any branch - multiply the probability of each step in the branch Probability of residing in a state - sum of branch probabilities that lead to that state
State probabilities (time dependent) of the -state system:
Time interval State probability State State 0.0 0.0 ½ = 0.500 ½ = 0.500 (½)(½) + (½)(¼) = 0.375 (½)(½) + (½)(3/4) = 0.65 3 0.344 0.656 4 0.336 0.664 5 0.334 0.666 Probability 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0. 0. 0 start in State start in State 0 3 4 5 6 Number of Time Intervals State State As the # of time intervals increase, state probability tends to a constant (limiting) value limiting state probability Transient behavior (time-dependent state probability) depends on the initial condition Limiting state probability of ergodic system (or process) is independent of the initial condition.
Ergodic System: every state of a system can be reached from every other state directly or through intermediate states Systems with absorbing states are not ergodic. Absorbing State: a state once entered cannot be left e.g. a system failure state in a mission oriented system Evaluation Procedure using Markov Model: - develop Markov model for the component (or system) - evaluate state probability (time dependent or limiting state) using: o Tree diagram: impractical for large systems or a large number of time intervals o Stochastic Transitional Probability Matrix o Other techniques for continuous Markov process will be discussed later
Stochastic Transitional Probability Matrix Square matrix (order = number of states) Rows : from states Columns: to states Element : probability value from one state to another P ij = prob of transition from state i to state j from nodes P = to nodes.. n P P.. P n P P.. P n.......... n P n P n.. P nn sum of probabilities in each row must be unity Transient behavior: State probabilities after n intervals is given by, P(n) = P(0).P n where P(0) is the initial probability vector (state probabilities at initial condition) Limiting State Probability: repeated multiplications of P until resulting P does not change with further multiplications. αp = α where α = limiting probability vector
Example: /4 Stochastic Transitional Probability Matrix, P = /4 3/4 If the system starts in State, Initial probability vector P(0) = [ 0] State probabilities after interval, P() = P(0).P = [ 0]. = [ 0.375 0.65 ] /4 3/4 Limiting State Probabilities: α = [P P ] Using the equation, P = limiting probability of being in State P = limiting probability of being in State αp = α [P P ] = [P P ] () /4 3/4 P + P = () Solving () & (), P = 0.333 and P = 0.667
Absorbing States System states when once entered, cannot be left until the system starts a new mission. e.g. failure states in mission oriented systems Need to evaluate: How many time intervals does the system operate on average before entering the absorbing state? Expected # of time intervals, where, N = [ I Q ] - I = identity matrix Q = truncated matrix created by deleting row(s) and column(s) associated with the absorbing states /4 /3 /3 3 absorbing state
Example: /4 3 absorbing state Stochastic Transitional Probability Matrix, P = 3 3/4 /4 0 0 3 0 0 Truncated Matrix (deleting Row 3 & Column 3 from P ) Q = 3/4 /4 0 Average number of time intervals spent in each state before entering the absorbing state, N = [ I Q ] - 0 0 3/4 /4 0 /4 -/4 0 = { } - = = - 4 0 i.e. average no. of time intervals spent in state given that the system starts in state is 4.