Stability and Asymptotic Optimality of h-maxweight Policies

Transcription

1 Stability and Asymptotic Optimality of h-maxweight Policies - A. Rybko, 2006 Sean Meyn Department of Electrical and Computer Engineering University of Illinois & the Coordinated Science Laboratory NSF support: ECS and DARPA ITMANET

2 Control Techniques for Complex Networks Draft copy April II Workload 158 Control Techniques for Complex Networks Draft copy April Workload & Scheduling Single server queue III Stability. & Performance Workload Outline for the CRW scheduling model Relaxations for the fluid model ODE methods Safety stocks and trajectory tracking Fluid-scale asymptotic optimality Control Techniques for Complex Networks Draft copy April I Modeling & Control 11 Simulation & Learning Control variates and shadow functions Scheduling E9E Estimating a value function Controlled random-walk model Notes N1N Fluid model Exercises Control techniques I for the fluid Models model &. Background III Stability & Performance rmance Optimization Optimality equations II h-maxweight. Policies 9.6 Optimization in networks I M. Modeling. o4d08e8ling & Control MaxWeight and MinDrift III 4.9 Perturbedr ed value function Heavy Traffic 4.10 Notes Exercises III Stability Conclusions & Performance rmance Foster-Lyapunov Techniques 319 IIII 8.1 Lyapunov functions a A Markov. Models Stability & Performance MaxWeight A.1 Every. process. s is (almost).s. Markov Ma Optimization 8.5 MaxWeight and the average-cost optimalitay optimality A.2e equation quga Generators Geti eno nen3 er t7o4rs and.a value functions.u nc34. tio8n ODE methods A.3 Equilibrium equations A.4 Criteria 436 for stability Safety stocks and trajectory tracking A.5 Ergodic theorems and coupling Fluid-scale asymptotic optimality A.6 Converse theorems List of Figures 572

3 Control Techniques for Complex Networks Draft copy April I Modeling & Control 34 4 Scheduling Controlled random-walk model Fluid model Control techniques for the fluid model III Stability & Performance Optimization Optimality equations Optimization in networks I Models & Background

4 Controlled Random-Walk Model µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Q(k+1) = Q(k)+B(k+1)U(k)+A(k+1), Q(0) = x Statistics & topology: B(k) = A(k) = S 1 (k) S 1 (k) S 2 (k) S 3 (k) S 3 (k) S 4 (k) A 1 (k) 0 A 3 (k) 0 Constituency constraints: CU(k) 1 U(k) 0 C = Lippman Henderson & M. 1997

5 Fluid Model & Workload µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 q( t) =x + Bz(t)+αt, t 0 q(0) = x Fluid model captures mean-flow: Workload and load parameters: B = E[B(k)] = α = E[A(k)] = ξ 1 = m 1 0 m 4 m 4 ξ2 = µ µ 1 µ µ µ 3 µ 4 0 α 3 0 m 2 m 2 m 3 0 ρ 1 = m 1 + m 4 α 3 ρ 2 = m 2 + m 3 α 3 with m i = µ 1 i - Newell 1982, - Vandergraft Perkins & Kumar Chen & Mandelbaum 1991, - Cruz 1991

6 Value Functions µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 q(t) =x + Bz(t)+αt Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) J(x) = 0 c(q(t; x)) dt h(x) = E[c(Q(t; x)) η] dt 0 alue function Relative value function η = c(x) π(dx) = average cost

7 Value Functions µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 q(t) =x + Bz(t)+αt Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) J(x) = 0 c(q(t; x)) dt h(x) = 0 E[c(Q(t; x)) η] dt alue function Relative value function η = c(x) π(dx) Large-state solidarity lim x J(x) h(x) =1 Holds for wide class of stabilizing policies, including average-cost optimal policy - M 96, 01,... following -Dai 95, - Dai & M 95

8 Myopic Policy: Fluid Model q(t) =x + Bz(t)+αt Constraints: X U(x) subset of R l + feasible values of d + dtq(t) =Bζ(t)+ ζ(t) when α x = q(t) X Given: Convex monotone cost function, c: R l + R +

9 Myopic Policy: Fluid Model d + dtq(t) =Bζ(t)+ α Constraints: X subset of R l + U(x) feasible values of ζ(t) when x = q(t) X Given: Convex monotone cost function, c: R l + R + arg min u U(x) d + dt c(q(t)) = arg min u U(x) c(x), Bu + α

10 Myopic Policy: CRW Model Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) Constraints: X subset of (lattice constraints, etc.) R l + U (x) feasible values of U(k) when x = Q(k) X Given: Convex monotone cost function, c: R l + R +

11 Myopic Policy: CRW Model Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) Constraints: X subset of (lattice constraints, etc.) R l + U (x) feasible values of U(k) when x = Q(k) X Given: Convex monotone cost function, c: R l + R + Myopic policy: arg min u U (x) E[c(Q(k +1)) Q(k) =x, U(k) =u]

12 Myopic Policy: CRW Model Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) Motivation: Average cost optimal policy is h-myopic, h: R l + R + is the relative value function, h(x) = inf U 0 E[c(Q(t; x)) η *] dt

13 Myopic Policy: CRW Model Q(k+1) Q(k) = B(k+1)U(k)+A(k+1) Motivation: Average cost optimal policy is h-myopic, h: R l + R + is the relative value function, h(x) = inf U 0 E[c(Q(t; x)) η *] dt Dynamic programming equation: min u U (x) E[h(Q(k +1)) Q(k) = x, U(k) =u] = h(x) c (x) + η *

14 Fluid Model & Myopia µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 q( t) =x + Bz(t)+αt, t 0 q(0) = x Given: Convex monotone cost function, c: R l + R + Myopic policy for fluid model is stabilizing: q(t) = 0 t T 0 - Chen & Yao 93 - M 01

15 Myopia & Instability µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Myopic policy may or may not be stabilizing Example: Two station model above with linear cost, c(x) =x 1 + x 2 + x 3 + x 4 Myopic policy for CRW model: Priority to exit buffers - Kumar & Seidman 89 - Rybko & Stolyar 93

16 Myopia & Instability µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Myopic policy may or may not be stabilizing Example: Two station model above with linear cost, c(x) =x 1 + x 2 + x 3 + x 4 Myopic policy for CRW model: Priority to exit buffers 800 Poisson Arrivals and Service 800 Fluid Arrivals and Service Periodic starvation creates instability - Kumar & Seidman 89 - Rybko & Stolyar 93

17 Myopia & Instability Quadratic Cost µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Myopic policy stabilizing for diagonal quadratic Example: Two station model above with, c(x) = 1 2 [x2 1 + x x x 2 4] Myopic policy: Approximated by linear switching curves - Tassiulas & Ephremides 92

18 Myopia & Instability Quadratic Cost µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Myopic policy stabilizing for diagonal quadratic Example: Two station model above with, c(x) = 1 2 [x2 1 + x x x 2 4] Myopic policy: Approximated by linear switching curves Condition (V3) holds with Lyapunov function For positive constants ε and η V = c PV (x):=e[v (Q(k + 1)) Q(k) =x] V (x) ε x + η - Tassiulas & Ephremides 92

19 MaxWeight Policy µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Tassiulas considers myopic policy for fluid model arg min u U (x) c(x), Bu+ α subject to lattice constraints where c(x) = 1 2 xt Dx, D =diag(d 1,...,d ) - Tassiulas & Ephremides 92

20 MaxWeight Policy µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Tassiulas considers myopic policy for fluid model arg min u U (x) c(x), Bu+ α subject to lattice constraints Obtains negative drift: For non-zero x, c(x), Bu+ α ε x Implies (V3) for MaxWeight policy

21 MaxWeight Policy µ 1 µ 2 µ 4 µ 3 Station 1 Station 2 α 3 Tassiulas considers myopic policy for fluid model arg min u U (x) c(x), Bu+ α subject to lattice constraints Obtains negative drift: For non-zero x, c(x), Bu+ α ε x Implies (V3) for MaxWeight policy Implies (V3) for myopic policy since myopic has minimum drift

22 Questions Since 1996 lim J(x) h(x) =1 x Value functions for fluid and stochastic models: Quadratic growth for linear cost with similar asymptotes; Policies are similar for large state-values

23 Questions Since 1996 lim J(x) h(x) =1 x Value functions for fluid and stochastic models: Quadratic growth for linear cost with similar asymptotes; Policies are similar for large state-values What is the gap between policies? What is the gap between value functions? How to translate policy for fluid model to cope with volatility? Connections with heavy traffic theory?

24 Questions Since 1996 lim J(x) h(x) =1 x Value functions for fluid and stochastic models: Quadratic growth for linear cost with similar asymptotes; Policies are similar for large state-values What is the gap between policies? What is the gap between value functions? How to translate policy for fluid model to cope with volatility? Connections with heavy traffic theory? Many positive answers in new monograph, as well as new applications for value function approximation Today s lecture focuses on third and fourth topics

25 Control Techniques for Complex Networks Draft copy April I Modeling & Control 34 (x) = α1 µ MaxWeight and MinDrift Perturbed value function Notes Exercises (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 II h-maxweight Policies III Stability & Performance Foster-Lyapunov Techniques Lyapunov functions MaxWeight MaxWeight and the average-cost optimality equation...348

26 Why Does MW Work? Geometric explanation Define drift vector field (for given policy) (x) = E[ Q(k +1) Q(k) Q(k) = x] = Bu+ α MaxWeight policy: arg min u U (x) c(x), (x) with c diagonal quadratic

27 Why Does MW Work? (x) =E[ Q(k +1) Q(k) Q(k) =x ] Example: Queues in tandem µ 1 µ 2 x 2 (x) = µ 2 MaxWeight policy: serve buffer 1 (x) = µ 1+ µ 2 +µ 1 (x) = µ 1+ µ 1 x 1

28 Why Does MW Work? (x) =E[ Q(k +1) Q(k) Q(k) =x ] Example: Queues in tandem µ 1 µ 2 x 2 (x) = µ 2 MaxWeight policy: serve buffer 1 (x) = µ 1+ µ 2 +µ 1 (x) = µ 1+ µ 1 x 1 Key observation: Boundaries of the state space are repelling

29 Why Does MW Work? (x) =E[ Q(k +1) Q(k) Q(k) =x ] Example: Queues in tandem µ 1 µ 2 x 2 (x) = µ 2 MaxWeight policy: serve buffer 1 (x) = µ 1+ µ 2 +µ 1 Level sets of c (diag quadratic) (x) = µ 1+ µ 1 x 1 Key observation: Boundaries of the state space are repelling Consequence of vanishing partial derivatives on boundary

30 h-maxweight Policy (x) = α1 µ 2 (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 Given: Convex monotone function h Boundary conditions x j h (x) =0 whenx j =0.

31 h-maxweight Policy (x) = α1 µ 2 (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 Given: Convex monotone function h Boundary conditions x j h (x) =0 whenx j =0. Economic interpretation: Marginal disutility vanishes for vanishingly small inventory

32 h-maxweight Policy (x) = α1 µ 2 (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 Given: Convex monotone function h Boundary conditions x j h (x) =0 whenx j =0. Economic interpretation: Marginal disutility vanishes for vanishingly small inventory Condition rarely holds, but we can fix that...

33 h-maxweight Policy (x) = α1 µ 2 (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 Given: Convex monotone function h 0 (perhaps violating condition) Introduce perturbation: For fixed θ 1 and any x R l + x i := x i + θ(e x i/θ 1), and x =( x 1,..., x l ) T R l +

34 h-maxweight Policy (x) = α1 µ 2 (x) = µ1+α1 µ 2+µ 1 (x) = µ1+α1 µ 1 Given: Convex monotone function h 0 (perhaps violating condition) Introduce perturbation: For fixed θ 1 and any x R l + x i := x i + θ(e x i/θ 1), and x =( x 1,..., x l ) T R l + Perturbed function: h(x) =h 0 ( x), x R l + Convex, monotone, and boundary conditions are satisfied

35 h-maxweight Policy Perturbed linear function µ 1 µ 2 h 0 linear: never satisfies condition h-myopic and h-maxweight polices stabilizing provided θ 1 is sufficiently large

36 h-maxweight Policy Perturbed linear function µ 1 µ 2 h 0 linear: never satisfies condition h-myopic and h-maxweight polices stabilizing provided θ 1 is sufficiently large x 2 Example: Tandem queues (x) = µ 2 h-maxweight policy: serve buffer 1 Level sets of h q 2 = θ log 1 c 1 c 2 ( ) q 2 (x) = µ 1+ µ 2 +µ 1 (x) = µ 1+ µ 1 x 1

37 h-maxweight Policy Perturbed value function µ 1 µ 2 h 0 minimal fluid value function, J(x) = inf 0 c(q(t; x)) dt h-myopic and h-maxweight polices stabilizing provided θ 1 is sufficiently large

38 h-maxweight Policy Perturbed value function µ 1 µ 2 h 0 minimal fluid value function, J(x) = inf 0 c(q(t; x)) dt h-myopic and h-maxweight polices stabilizing provided θ 1 is sufficiently large Resulting policy very similar to average-cost optimal policy: 30 x 2 Optimal policy: serve buffer x 1

39 Control Techniques for Complex Networks Draft copy April II Workload Workload & Scheduling Single server queue Workload for the CRW scheduling model Relaxations for the fluid model III Heavy Traffic III Stability & Performance Optimization ODE methods Safety stocks and trajectory tracking Fluid-scale asymptotic optimality...467

40 Relaxations & Asymptotic Optimality Single example for sake of illustration: Station 2 Station 1 Model of Dai & Wang

41 Relaxations & Asymptotic Optimality Single example for sake of illustration: Station 2 Station 1 Assume: Homogeneous model Service rate at Station i is µ i

42 Relaxations & Asymptotic Optimality Station 2 Station 1 Homogeneous CRW model: Q 1 (k +1) Q 1 (k) = S 1 (k +1)U 1 (k)+a 1 (k +1) Q 2 (k +1) Q 2 (k) = S 1 (k +1)U 2 (k)+s 1 (k +1)U 1 (k) Q 3 (k +1) Q 3 (k) = S 2 (k +1)U 3 (k)+s 2 (k +1)U 2 (k) Q 4 (k +1) Q 4 (k) = S 2 (k +1)U 4 (k)+s 2 (k +1)U 3 (k) Q 5 (k +1) Q 5 (k) = S 1 (k +1)U 5 (k)+s 2 (k +1)U 4 (k)

43 Relaxations & Asymptotic Optimality Station 2 Station 1 Homogeneous CRW model: Q 1 (k +1) Q 1 (k) = S 1 (k +1)U 1 (k)+a 1 (k +1) Q 2 (k +1) Q 2 (k) = S 1 (k +1)U 2 (k)+s 1 (k +1)U 1 (k) Q 3 (k +1) Q 3 (k) = S 2 (k +1)U 3 (k)+s 2 (k +1)U 2 (k) Q 4 (k +1) Q 4 (k) = S 2 (k +1)U 4 (k)+s 2 (k +1)U 3 (k) Q 5 (k +1) Q 5 (k) = S 1 (k +1)U 5 (k)+s 2 (k +1)U 4 (k) Constituency constraints: U i (k) {0,1} U 1 (k)+u 2 (k)+u 5 (k) 1 U 3 (k)+u 4 (k) 1

44 Relaxations & Asymptotic Optimality Station 2 Station 1 Workload (units of inventory) Y 1 (k) =3Q 1 (k)+2q 2 (k)+q 3 (k)+q 4 (k)+q 5 (k) Y 2 (k) =2(Q 1 (k)+q 2 (k)+q 3 (k)) + Q 4 (k)

45 Relaxations & Asymptotic Optimality Station 2 Station 1 Workload (units of inventory) Y 1 (k) =3Q 1 (k)+2q 2 (k)+q 3 (k)+q 4 (k)+q 5 (k) Y 2 (k) =2(Q 1 (k)+q 2 (k)+q 3 (k)) + Q 4 (k) Idleness processes: ι 1 (k) =1 (U 1 (k)+u 2 (k)+u 5 (k)) ι 2 (k) =1 (U 3 (k)+u 4 (k))

46 Relaxations & Asymptotic Optimality Station 2 Station 1 Workload (units of inventory) Y 1 (k) =3Q 1 (k)+2q 2 (k)+q 3 (k)+q 4 (k)+q 5 (k) Y 2 (k) =2(Q 1 (k)+q 2 (k)+q 3 (k)) + Q 4 (k) Idleness processes: ι 1 (k) =1 (U 1 (k)+u 2 (k)+u 5 (k)) ι 2 (k) =1 (U 3 (k)+u 4 (k)) Dynamics: Y 1 (k +1) Y 1 (k) = S 1 (k +1)+3A 1 (k +1) Y 2 (k +1) Y 2 (k) = S 2 (k +1)+2A 1 (k +1) + + S 1 (k +1)ι 1 (k) S 2 (k +1)ι 2 (k)

47 Relaxations & Asymptotic Optimality Station 2 Station 1 Workload Relaxation of N. Laws Y 1 (k +1) Y 1 (k) = S 1 (k +1)+3A 1 (k +1) + S 1 (k +1)ι 1 (k) with constraints on idleness process relaxed, ι 1 (k) {0, 1,2,...} - Laws 90 - Kelly & Laws 93

48 Relaxations & Asymptotic Optimality Station 2 Station 1 Workload Relaxation of N. Laws Y 1 (k +1) Y 1 (k) = S 1 (k +1)+3A 1 (k +1) + S 1 (k +1)ι 1 (k) with constraints on idleness process relaxed, ι 1 (k) {0, 1,2,...} Optimization based on the effective cost, c(y) =min c(x) s. t. 3x 1 +2x 2 + x 3 + x 4 + x 5 = y x Z 5 + (+ buffer constraints) - Laws 90 - Kelly & Laws 93 - Harrison, Kushner, Reiman, Williams, Dai, Bramson,...

49 Asymptotic Optimality Optimal policy is non-idling for one-dimensional relaxation Dynamic programing equation solved via Pollaczek-Khintchine formula

50 Asymptotic Optimality Heavy traffic assumptions Load is unity for nominal model Single bottleneck to define relaxation Cost is linear, and effective cost has a unique optimizer Model sequence: { A (n) A(k) with probability 1 n 1 (k) = 0 with probability n 1 Load less than unity for each n

51 Asymptotic Optimality h 0 (x) =ĥ (y)+ b 2 c( x) c(y) 2 ( ( h-maxweight policy asymptotically optimal, with logarithmic regret

52 Asymptotic Optimality h 0 (x) =ĥ (y)+ b 2 c( x) c(y) 2 ( ( h-maxweight policy asymptotically optimal, with logarithmic regret ˆη = O(n) optimal average cost for relaxation η average cost under h-mw policy

53 Asymptotic Optimality h 0 (x) =ĥ (y)+ b 2 c( x) c(y) 2 ( ( h-maxweight policy asymptotically optimal, with logarithmic regret ˆη = O(n) optimal average cost for relaxation η average cost under h-mw policy ˆη η ˆη + O(log(n))

54 Control Techniques for Complex Networks Draft copy April III Stability & Performance ODE methods Safety stocks and trajectory tracking Fluid-scale asymptotic optimality Simulation & Learning Control variates and shadow functions Estimating a value function Notes Exercises Conclusions A Markov Models 538 A.1 Every process is (almost) Markov A.2 Generators and value functions A.3 Equilibrium equations A.4 Criteria for stability A.5 Ergodic theorems and coupling A.6 Converse theorems List of Figures 572

55 Conclusions h-maxweight policy stabilizing under very general conds. General approach to policy translation. Resulting policy mirrors optimal policy in examples Asymptotically optimal, with logarithmic regret for model with single bottleneck

56 Conclusions h-maxweight policy stabilizing under very general conds. General approach to policy translation. Resulting policy mirrors optimal policy in examples Asymptotically optimal, with logarithmic regret for model with single bottleneck Future work Models with multiple bottlenecks? On-line learning for policy improvement?

57 Control Techniques for Complex Networks Draft copy April II Workload Control Tec 158 hniques for Complex Networks Draft copy Ap References 5 Workload & Scheduling Single server queue III Stability. & Performance Workload for the CRW scheduling model Relaxations for the fluid model ODE methods Safety stocks and trajectory tracking Fluid-scale asymptotic optimality Control Techniques N. Laws. for Complex Dynamic Networks routing in queueing networks. PhD thesis, Draft copy April Cambridge University, Cambridge, 11 Simulation UK, & Learning 485 I Modeling & Control Control variates and shadow functions Scheduling E9E Estimating a value function Controlled random-walk L. Tassiulas. model Adaptive. back-pressure congestion Notes N1N control based on 4.2 Fluid model Exercises Control techniques local for information. the fluid model (2): , III Stability & Performance rmance 318 Control Techniques for Complex Networks Draft copy April L. Tassiulas and A. Ephremides. Stability properties of constrained 9 Optimization Optimality equations queueing systems and scheduling policies for maximum throughput in 9.6 Optimization in networks I M. Modeling. o4d08e8ling & Control 34 multihop radio networks MaxWeight and MinDrift Perturbedr ed value function Notes Exercises S. P. Meyn. Sequencing and routing in multiclass queueing networks. Part II: Workload relaxations S. P. Meyn. Stability and asymptotic optimality of generalized MaxWeight policies. Submitted for publication, III Stability & Performance rmance Foster-Lyapunov A Techniques Models 538 IIII Stability & Performance Markov o 318 es Lyapunov A.1 Every functions process is (almost). Markov Optimization 8.4 MaxWeight A.2 Generators t7o4rs and value functions ODE methods MaxWeight Equilibrium S. P. Meyn. Control techniques for complex 8.5 A.3 and Criteria 436 the average-cost equations optimality equation A.4 d4 t3 networks. i for stability e-c oṗti To.. appear,.. uȧti. oṅ... Cambridge University Press, Safety stocks and trajectory tracking A E. Ergodic r.go. d4i46 t2heorems theorems and coupling Fluid-scale asymptotic optimality A C. Converse o.nv.. e46 46s67e7 theorems List of Figures 572