MARKOV MODEL WITH COSTS In Markov models we are often interested in cost calculations.

Transcription

1 MARKOV MODEL WITH COSTS In Markov models we are often interested in cost calculations. inventory model: storage costs manpower planning model: salary costs machine reliability model: repair costs We will look at 3 types of cost calculations: 1. Expected total costs over a finite horizon 2. Long-run expected costs per period 3. Expected total costs over an infinite horizon (possible if you only have costs in the transient states) 1/25

2 Expected total costs over a finite horizon Assumption: every visit to state i will give you expected costs c(i). What are the expected total costs over the time span {0, 1,..., n}? Define g(i, n): expected total costs over time span {0, 1,..., n} when you start in state i. Then we have g(i, n) = N m i,j (n)c(j). j=1 2/25

3 Using the notation we have g(n) = g(1, n) g(2, n). g(n, n), c = g(n) = M(n) c. c(1) c(2). c(n) Conclusion: If we know the matrix of occupancy times M(n) and the cost vector c, we can calculate the expected total costs over a finite horizon. 3/25

4 Example: Inventory system (see Example 5.6) State space: S = {2, 3, 4, 5} Transition matrix: P = Assume storage costs in state i: 50 i Then expected total storage costs over time span {0, 1, 2,..., 10} g(10) = 2229, /25

5 Long-run expected costs per period If n, then often the total expected costs over the time span {0, 1,..., n} also tend to infinity. Hence, usually in long-run cost calculations we look at the expected costs per period: Theorem: g(i) = lim n g(i, n) n + 1. For an irreducible Markov chain with occupancy distribution ˆπ we have g(i) = g = N ˆπ j c(j). Remark that for an irreducible Markov chain the long-run expected costs per period do not depend on the initial state. j=1 5/25

6 Example: Manpower planning model (see Example 5.8). State space: S = {1, 2, 3, 4} Transition matrix: P = Assume salary costs in states 1,2,3,4: 400,600,800,1000 Then the occupancy distribution is given by ˆπ = [0.273, 0.455, 0.182, 0.091] and the long-run expected salary costs per week per employee are N ˆπ j c(j) = j=1 6/25

7 Expected total costs over an infinite horizon If there are only costs in transient states, then the expected total costs will not tend to infinity when n. How do we calculate in this case g(i) = lim n g(i, n)? Let A be the set of states in the end classes and hence S \ A the set of transient states. By using a first-step analysis we can show that g(i) = c(i) + p i,j g(j), i S \ A. j S\A This system of equations has a unique solution. 7/25

8 Example driver s license model State space:{t 1, T 2, P 1, P 2, P 3, P 4, S, F } Transition matrix: P = Transition diagram: see separate slide /25

9 What is the meaning of the different states? T 1, T 2 : theoretical exam (first + second time) P 1, P 2, P 3, P 4 : practical exam (1st,2nd,3rd, 4th time) S: Leaving with driver s license (Success) F : Leaving without driver s license (Failure) costs theoretical exam: 45 euro each time, costs practical exam: 90 euro each time. Question: What are the expected total costs over an infinite horizon? 9/25

10 Let g(i) be the expected total costs over an infinite horizon starting in state i. Then we have g(t 1 ) = g(t 2 ) g(p 1 ), g(t 2 ) = 45 + g(p 1 ), g(p 1 ) = g(p 2 ), g(p 2 ) = g(p 3 ), g(p 3 ) = g(p 4 ), g(p 4 ) = g(p 4 ). and hence g(p 4 ) = 120, g(p 3 ) = 156, g(p 2 ) = 207 g(p 1 ) = 276.3, g(t 2 ) = g(t 1 ) = /25

11 First passage times (Time until the Markov chain first enters a certain set of states) In the preceding slides we have seen twice that you can calculate a certain quantity by doing a so-called "first-step analysis": derive a system of equations by considering what can happen in the first period with the Markov chain. Calculation of the probability that a reducible Markov chain will end in a certain end class. Calculation of the expected total costs over an infinite horizon for a reducible Markov chain with only costs in the transient states. The same technique can be used to calculate expected first passage times of a Markov chain. 11/25

12 Let A be a subset of the state space S and define m i (A) as the expected time until the Markov chain first enters the subset A when it starts in state i. Then of course we have m i (A) = 0 if i A and furthermore m i (A) = 1 + p i,j m j (A), i / A. j S\A This, again, gives a system of equations from which we can obtain the quantities m i (A) for i / A. 12/25

13 Example: Manpower planning model Calculate the expected time an employee is working in the company. Markov model for 1 specific employee: State space S = {1, 2, 3, 4, left company} Transition matrix P = /25

14 Let A = {left company} and hence S \ A = {1, 2, 3, 4}. The quantities m i (A) satisfy the equations m 1 (A) = m 1 (A) m 2 (A) m 2 (A) = m 2 (A) m 3 (A) m 3 (A) = m 3 (A) m 4 (A) m 4 (A) = m 4 (A) The solution of this system of equations is given by m 4 (A) = 100, m 3 (A) = 60, m 2 (A) = 88.89, m 1 (A) = Hence, the expected time an employee is working in the company is equal to weeks (approximately 1.4 years). 14/25

15 COHORT MODELS Discrete time Markov chains are often used in the study of the behaviour of a group of persons or objects. These systems are often called Cohort models. An example of a cohort model is the manpower planning model. In the manpower planning model we assumed so far that the total number of employees is constant. Each time an employee leaves the company, he or she is instantaneously replaced by a new employee. Using the theory of Markov chains we were able to determine the shortterm and long-term behaviour of the number of employees in the different categories. 15/25

16 Example: Manpower planning model Transition matrix P = Assume we have 100 employees and in the beginning of week 1 we have that 50 employees belong to category 1, 25 to category 2, 15 to category 3 and 10 to category 4. How many employees do you expect in the different categories in the beginning of week 5, 11 and 100? How many employees do you expect in the different categories in the longrun? 16/25

17 We have and hence a (0) = [0.50, 0.25, 0.15, 0.10] a (4) = a (0) P 4 = [0.466, 0.289, 0.146, 0.099], a (10) = a (0) P 10 = [0.424, 0.336, 0.143, 0.098], a (99) = a (0) P 99 = [0.274, 0.461, 0.177, 0.088]. In this way we can calculate the expected number of employees in the different categories in the beginning of week 5, 11 and 100. The unique normalized solution of the system of equations π = πp is given by π = [0.273, 0.454, 0.182, 0.091]. Hence, in the long-run we expect approximately 27 employees in category 1, 45 in category 2, 18 in category 3 and 9 in category 4. 17/25

18 However, in many applications it is not realistic to assume that the number of persons in the group is constant over time. The departures of persons from the group on the one hand and the arrivals of new persons into the group can be independent processes. Example: The number of persons having a car insurance at a certain insurance company. (The different categories here represent the different levels in the no-claims bonus system). How do we calculate in such cases quantities like the expected number of persons in the group at a certain time instant and the division of the persons within the group over the different levels (short-term behaviour)? the expected number of persons in the group in the long-run and the division of the persons within the group over the different levels (longterm behaviour)? 18/25

19 Assume we have a group of persons, where the behaviour of each person can be described by a Markov chain with state space S = {0, 1, 2,..., N} and transition matrix P. State 0 represents the situation that the person has left the system. Q is the part of the transition matrix corresponding to transitions from states {1, 2,..., N} to states {1, 2,..., N}. Here, Q is a sub-stochastic matrix, i.e., a matrix with q i,j 0 for all i and j and N j=1 q i,j 1 for all i. 19/25

20 Example: Manpower planning model (model for 1 specific employee) State space S = {0, 1, 2, 3, 4} Transition matrix P = Q = /25

21 Notation: Short-term behaviour r (n) i : expected number of new persons, called recruits, entering the group from outside at time n in state i. s (n) i : expected total number of persons in the group at time n in state i. If we denote with r (n) and s (n) the transient vectors then we have r (n) = [r (n) 1, r (n) 2,..., r (n) N ], s (n) = r (n) + s (n 1) Q s (n) = [s (n) 1, s (n) 2,..., s (n) N ], Conclusion: If we know the beginvector s (0) and the vectors containing the expected numbers of recruits in the different states r (1), r (2), r (3),..., we can calculate the vectors s (1), s (2), s (3), /25

22 Example: Markov chain with state space S = {0, 1, 2, 3, 4} and transition matrix P = Furthermore assume that s (0) = [10, 10, 10, 10] and r (n) = [10, 0, 0, 0] for all n. s (1) = [10, 0, 0, 0] + [10, 10, 10, 10] = [16, 9, 9.5, 11] /25

23 s (2) = [10, 0, 0, 0] + [16, 9, 9.5, 11] = [19.6, 9.5, 8.9, 11.8]. s (3) = [10, 0, 0, 0] + [19.6, 9.5, 8.9, 11.8] = [21.76, 10.57, 8.61, 12.40] and so on... 23/25

24 Long-term behaviour In the case that the expected number of recruits is constant over time, i.e., r (n) = r for all n, we can also determine the long-run expected number of persons in the group in the different states. In this case we have that s = lim n s (n) satisfies and hence where I is the identity matrix. s = r + s Q, s = r (I Q) 1 Remark: The existence of the inverse of the matrix I Q follows form the fact that the matrix Q is sub-stochastic. 24/25

25 Example (continued): In the example we have r (n) = r = [10, 0, 0, 0] for all n and I Q = and hence = s = lim s (n) = [10, 0, 0, 0] n = [25, 16.67, 13.89, 27, 78] [25, 17, 14, 28]. 1 25/25