Advanced Computer Networks Lecture 2. Markov Processes

Size: px

Start display at page:

Download "Advanced Computer Networks Lecture 2. Markov Processes"

Vernon Phelps
6 years ago
Views:

1 Advanced Computer Networks Lecture 2. Markov Processes Husheng Li Min Kao Department of Electrical Engineering and Computer Science University of Tennessee, Knoxville Spring, /28

2 Outline 2/28 1 Definition and Motivation 2 Generators and Value Functions 3 Stability 4 Ergodicity 5 Continuous Time Markov Process

3 Discrete Time Markov Processes We first consider discrete time (hence Markov chain). Markov chain: P(X (t + 1) X (t),..., X (0)) = P(X (t + 1) X (t)), which means independence of the history except for X (t). Transition law: P(x, y ) = P(X (t + 1) = y X (t) = x). 3/28

4 A Simple Example Packet arrival 4/28 Packet departure

5 Almost Every Process is Markovian Why studying Markov process? Most processes can be approximated by Markov process. Consider a stationary stochastic process Z. Then we can define a Markov chain: Π(z0, z1 ) = P(Z (t) = z0, Z (t + 1) = z1 ). The Markov chain Π can approximate the one-step dynamics and steady-state of Z. 5/28

6 Key Issues in Markov Process Stability: will the Markov chain diverge (consider an exploding queue)? Cost: if we assign a cost of each state, how can we evaluate the sum cost over time? Control: if we can control the Markov chain, how can we make it stable and minimize its operation cost? 6/28

7 Outline 7/28 1 Definition and Motivation 2 Generators and Value Functions 3 Stability 4 Ergodicity 5 Continuous Time Markov Process

8 Discounted Cost Suppose that the cost value function for a Markov chain is hγ (x) = X (1 + γ) (t+1) Ex [c(x (t))], t=0 where x = X (0) is the initial state and c is a cost function dependent on the current state. According to dynamic programming, the cost function solves Dhγ = c + γhγ, where D = P I is called the generator (which is an operator). (Here P is the operator determined by the transition matrix). 8/28

9 Dynamic Programming Dynamic programming (DP) was proposed by R. Bellman in 1940s. It was rediscovered in the area of communications by A. Viterbi. It can solve any multi-stage optimization problem with a Markov structure (either deterministic or stochastic). Bellman s equation: h c(x) hγ (x) = min E 1+γ + 9/28 hγ (y) (1+γ)2 i.

10 Invariant Probability Measure A distribution π is called invariant if X π(x)d(x, y ) = 0, x X. y X An invariant distribution may not exist or may not be unique. For networking, we hope that the invariant distribution exists and is unique. 10/28

11 Irreducibility Define the resolvent by Rγ (x, y ) = X (1 + γ) t 1 P t (x, y), t=0 which can be considered as a matrix. We say that the Markov chain is x -irreducible if for one γ > 0 such that Rγ (x, x ) > 0, for each x X. The Markov chain is irreducible if it is x -irreducible for any x X. 11/28

12 Which is Irreducible? 12/28

13 Recurrence Consider Markov chain with finite states. For a subset A, consider the total number of returns to A: ηa = X I(X (t) A). t=0 The set A is called recurrent if Ex [ηa ] = for any x A, which means set A will be revisited for infinitely many times. Otherwise, A is called transient. 13/28

14 Dichotamy Consider an irreducible Markov chain with finite states. Then either all the subsets are transient or all are recurrent. So we can talk about the recurrence of a Markov chain, instead of individual subsets. For networking, we desire recurrence, which implies queue stability. Null recurrent: limt P t (x, y ) = 0, x, y. Positive recurrent: limt P t (x, y ) > 0, x, y. Harris recurrence: we say the Markov chain is Harris recurrent if Px (ηa = ) = 1. For finite state Markov chains, Harris recurrent is equivalent to recurrent. But this is not necessarily true for generic Markov chains. 14/28

15 First Return Time We define the first return time τx as the minimum time spacing between two visits to the state x, which is a random variable. Kac s Theorem: Suppose that X is x -irreducible. Then it is positive recurrent if and only if Ex [τx ] <. Denote by the stationary distribution of X by π, then we have π(x ) = 15/28 1. Ex [τx ]

16 16/28 First Entrance and First Return Times First entrance time: σx = min(t 0; X (t) = x ). First return time: τx = min(t 1; X (t) = x ). Consider an x X satisfying Ex [τx ] <. Then the following distribution is invariant: hp i τx 1 Ex 1(X (t) = x) t=0 π(x) =. Ex [τx ] (Let s prove it...)

17 Outline 17/28 1 Definition and Motivation 2 Generators and Value Functions 3 Stability 4 Ergodicity 5 Continuous Time Markov Process

Lyapunov Function We need a non-negative function V (the so called Lyapunov function), a finite subset S of the state alphabet X, and b <, which satisfy

18 Lyapunov Function We need a non-negative function V (the so called Lyapunov function), a finite subset S of the state alphabet X, and b <, which satisfy the following Foster s criterion: DV (x) 1 + bis (x), -1 means decreasing; bis (x) means that the Lyapunov function does not necessarily decrease in S. 18/28

19 Foster s Criterion The following are equivalent for an x -irreducible Markov chain: An invariant measure π exists. There is a finite set S X such that Ex [τs ] <. There exist V, S and b < such that the Foster s criterion holds. 19/28

20 Construction of V We define the drift vector field: (x) = E[X (t + 1) X (t) X (t) = x]. Suppose that a function V satisfies the following two conditions The chain is skip-free in the expectation, which means sup E[kX (t + 1) X (t)k X (t) = x] <. x There exist 0 > 0 and b0 < such that < (y), V > (1 + 0 ) + Then, V satisfies the Foster s criterion. 20/28 b0 kx y k. 1 + kxk

21 Comparison Theorem Comparison theorem is the most common approach to obtaining bounds on expectations involving stopping times. Suppose the nonnegative functions V, f and g satisfy DV f + g, then, for any stopping time τ, we have # "τ 1 # "τ 1 X X Ex f (X (t)) V (x) + Ex g(x (t)). t=0 21/28 t=0

22 Outline 22/28 1 Definition and Motivation 2 Generators and Value Functions 3 Stability 4 Ergodicity 5 Continuous Time Markov Process

23 Ergodicity Intuitively speaking, ergodicity means that the ensemble average is equal to the time average. The law of large numbers is an example of ergodicity. Consider two approaches of calculating the expectation of coin flipping: Use one coin to flip for times. Flip coins simultaneously. Some random processes are not ergodic (e.g., a random process with multiple invariant distributions). 23/28

24 Homework 1 deadline: Feb. 2, Problem 1. Prove that any irreducible Markov chain with finitely many states is always recurrent. Problem 2. Consider a two-state (0 and 1) Markov chain with transition probabilities: P(X (t + 1) = 0 X (t) = 1) = 0.7, P(X (t + 1) = 1 X (t) = 1) = 0.3, P(X (t + 1) = 0 X (t) = 0) = 0.4 and P(X (t + 1) = 1 X (t) = 0) = 0.6. Calculate the stationary (invariant) distribution of the states. Problem 3. Consider a single queue with discrete time. We assume that in each time slot, the probability of packet arrival is 0.7 and the probability of packet leaving is 0.9. The arrival and leaving are mutually independent. And at most one packet can arrive or leave. The arrivals and leaves are independent in the time. Then, describe the queue using a Markov chain, and calculate its stationary distribution (analytically or numerically). Problem 4. For the queue in Problem 3, use the Foster s criterion to prove that it is stable. 24/28

25 Mean Ergodic Theorem Using the technique of coupling, we can prove the following theorem: Under mild conditions, we have kp t (x, ) πkf 0 as t. There exists b0 < such that for each x, y X we have X kp t (x, ) P t (y, )kf 2[V (x) + V (y )] + b0. t=0 If π(v ) <, there exists a b1 < such that X t=0 25/28 kp t (x, ) πkf 2V (x) + b1.

26 Outline 26/28 1 Definition and Motivation 2 Generators and Value Functions 3 Stability 4 Ergodicity 5 Continuous Time Markov Process

27 Continuous Time Markov Process Xs is continuous in time. It is Markov if P(Xs+t A Fs ) = P t (Xs, A), where P t (x, A) = Px (Xt A). If the state is discrete, we have P(X (t + h) = j X (t) = i) = δij + qij h + o(h). 27/28

28 Transition Kernel P t Chapman-Kolmogorov equation (or the semi-group property): Z P t+s (x, A) = P t (y, A)P s (x, dy), which can also be written as P t+s = P t P s. This defines a semi-group. 28/28

STOCHASTIC PROCESSES Basic notions

J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving