Analysis of Markov Reward Models with Partial Reward Loss Based on a Time Reverse Approach Gábor Horváth, Miklós Telek Technical University of Budapest, 1521 Budapest, Hungary {hgabor,telek}@webspn.hit.bme.hu M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 1/15
Markov Reward models with reward loss The difficulty of time forward approach The time reverse analysis approach Properties of the obtained solution s M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 2/15
Markov Reward models without reward loss Markov reward models (MRM) Markov Reward models without reward loss Markov Reward models with total reward loss Markov Reward models with partial reward loss a finite state CTMC, non negative reward rates (r i ), performance measures: reward accumulated up to time t, time to accumulate reward w. B(t) r i r k r j r k Z(t) k j i t t M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 3/15
Markov Reward models with total reward loss Markov Reward models without reward loss Markov Reward models with total reward loss Markov Reward models with partial reward loss We consider first order MRM (deterministic dependence on Z(t)), without impulse reward, but with potential reward loss at state transition. B(t) r i r k r j Z(t) k j r k t i t M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 4/15
Markov Reward models with partial reward los In case of partial reward loss: Markov Reward models without reward loss Markov Reward models with total reward loss Markov Reward models with partial reward loss α i remaining portion of reward when leaving state i, the lost reward is proportional to: total accumulated reward partial total loss, reward accumulated in the last state partial incremental loss. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 5/15
Markov Reward models with partial reward los Markov Reward models without reward loss Markov Reward models with total reward loss Markov Reward models with partial reward loss In case of partial reward loss: α i remaining portion of reward when leaving state i, the lost reward is proportional to: total accumulated reward partial total loss, reward accumulated in the last state partial incremental loss. B(t) Z(t) k r i r k B(T 1 )α i r j B(T 2 )α k r k B(t) B(T3 )α j r k r i r k α k r j α j α j [B(T3 ) B(T 2 )] r i α i B(T 2 ) t Z(t) t k r j r k j j i i T 1 T 2 T 3 t T 1 T 2 T 3 t M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 5/15
Time forward approach Time forward approach Time reverse approach Possible interpretation: Reduced (r i α i ) reward accumulation up to the last state transition, and total (r i ) reward accumulation in the last state without reward loss. B(t) r i r i α i r k r k α k r j r j α j r k α j [B(T3 ) B(T 2 )] B(T 2 ) Z(t) k t j i T 1 T 2 T 3 t M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 6/15
Time forward approach Time forward approach Time reverse approach Possible interpretation: Reduced (r i α i ) reward accumulation up to the last state transition, and total (r i ) reward accumulation in the last state without reward loss. B(t) r i r i α i r k r k α k r j r j α j r k α j [B(T3 ) B(T 2 )] B(T 2 ) Z(t) k t j i T 1 T 2 T 3 t Unfortunately, the last state transition before time T is not a stopping time. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 6/15
Time reverse approach Time forward approach Time reverse approach Behaviour of the time reverse process: Inhomogeneous CTMC with initial probability γ (0) = γ(t) and generator Q(τ) = { q ij (τ)}, where q ij (τ) = γ j (T τ) γ i (T τ) q ji if i j, γ k (T τ) γ i (T τ) q ki if i = j. k S,k i M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 7/15
Time reverse approach Time forward approach Time reverse approach Behaviour of the time reverse process: Inhomogeneous CTMC with initial probability γ (0) = γ(t) and generator Q(τ) = { q ij (τ)}, where q ij (τ) = γ j (T τ) γ i (T τ) q ji if i j, γ k (T τ) γ i (T τ) q ki if i = j. k S,k i Total (r i ) reward accumulation in the first state, and reduced (r i α i ) reward accumulation in all consecutive states without reward loss. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 7/15
Time reverse approach Potential model description: Time reverse approach duplicate the state space to describe the total reward accumulation in the first state (r i ), and the reduced reward accumulation in all further states (r i α i ). M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 8/15
Time reverse approach Potential model description: Time reverse approach duplicate the state space to describe the total reward accumulation in the first state (r i ), and the reduced reward accumulation in all further states (r i α i ). π (0) = [γ(t), 0], Q (τ) = Q D (τ) Q(τ) Q D (τ) 0 Q(τ), R = R 0 0 R α M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 8/15
Inhomogeneous differential Introducing Y i (τ, w) = Pr( B(τ) w, Z (τ) = i) Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward we can apply the analysis approach available for inhomogeneous MRMs. It is based on the solution of the inhomogeneous partial differential τ Y (τ, w) + Y (τ, w)r = Y (τ, w) Q(τ), w where Y (τ, w) = { Y i (τ, w)}. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 9/15
Inhomogeneous differential Introducing Y i (τ, w) = Pr( B(τ) w, Z (τ) = i) Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward we can apply the analysis approach available for inhomogeneous MRMs. It is based on the solution of the inhomogeneous partial differential τ Y (τ, w) + Y (τ, w)r = Y (τ, w) Q(τ), w where Y (τ, w) = { Y i (τ, w)}. But a drawback of this approach is that it requires the computation of Q(τ). M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 9/15
Homogeneous differential To overcome this drawback we introduce the conditional distribution of reward accumulated by the reverse process V i (τ, w) = Pr( B(τ) w Z (τ) = i) Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward and the row vector V (τ, w) = { V i (τ, w)}. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 10/15
Homogeneous differential To overcome this drawback we introduce the conditional distribution of reward accumulated by the reverse process V i (τ, w) = Pr( B(τ) w Z (τ) = i) Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward and the row vector V (τ, w) = { V i (τ, w)}. Using this performance measure we have to solve τ V (τ, w) + V (τ, w)r = V (τ, w)q T, w where Q T is the transpose of Q. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 10/15
Block structure of the differential Utilizing the special block structure of the Q (τ) and the R matrices (of size 2#S) we can obtain two homogeneous partial differential s of size #S: Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward and τ τ X1(τ, w) + X1(τ, w)r = X1(τ, w)q D, w X2(τ, w)+ X2(τ, w)r α = X1(τ, w)(q Q D ) T + X2(τ, w)q T, w M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 11/15
Moments of accumulated reward The analysis approach available for inhomogeneous MRMs allows to describe the moments of IMRMs with an inhomogeneous ordinary differential. Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward Similar to the reward distribution case, this approach is also applicable for our model, but it requires the the computation of Q(τ). M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 12/15
Moments of accumulated reward The analysis approach available for inhomogeneous MRMs allows to describe the moments of IMRMs with an inhomogeneous ordinary differential. Inhomogeneous differential Homogeneous differential Block structure of the differential Moments of accumulated reward Similar to the reward distribution case, this approach is also applicable for our model, but it requires the the computation of Q(τ). Using similar state dependent moment measures we obtain homogeneous ordinary differential s and d dτ M1 (n) (τ) = n M1 (n 1) (τ)r + M1 (n) (τ)q D, d dτ M2 (n) (τ) = n M2 (n 1) (τ)r α + M1 (n) (τ)(q Q D ) T + M2 (n) (τ)q M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 12/15
Randomization based numerical method The ordinary differential with constant coefficients allows to compose a randomization based numerical method. Randomization based numerical method and where M1 (n) (τ) = τ n e R n E D (τ), M2 (n) (τ) = n!d n k=0 λτ (λτ)k e k! D (n) (k), e (I A k D ) n = 0 D (n) (k) = 0 k n, n 1 D (n 1) (k 1)S α + D (n) (k 1)A+ ) e S n A k 1 n D (A A D ) k > n, n 1 ( k 1 n M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 13/15
Numerical Example α N =0.5 α N 1 =0.5 α N 2 =0.5 α 0 =1 r N =Nr r N 1 =(N 1)r r N 2 =(N 2)r r 0 =0 10 N N λ (N 1)λ N 1 N 2 λ 0 1 0.1 Numerical Example ρ σ σ σ M σ 0.01 0.001 r M =0 α M =1 Structure of the Markov chain 0.0001 1e-05 1st moment 2nd moment 3rd moment 4th moment 5th moment 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Moments of the accumulated reward With parameters N = 500000, λ = 0.000004, σ = 1.5, ρ = 0.1, r = 0.000002, α = 0.5, M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 14/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: non stopping time time reverse approach M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: non stopping time time reverse approach inhomogeneous differential proper performance measure, M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: non stopping time time reverse approach inhomogeneous differential proper performance measure, partial differential ordinary differential s, M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: non stopping time time reverse approach inhomogeneous differential proper performance measure, partial differential ordinary differential s, numerical stability, error control randomization based analysis. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15
The analysis of partial loss MRM is usually rather complex. We propose an analysis method with the following features: non stopping time time reverse approach inhomogeneous differential proper performance measure, partial differential ordinary differential s, numerical stability, error control randomization based analysis. Thanks for your attention. M Telek, Markov Anniversary Meeting, June 2006. Analysis of Markov Reward Models with Partial Reward Loss - p. 15/15