Stein s Method for Steady-State Approximations: A Toolbox of Techniques

Size: px

Start display at page:

Download "Stein s Method for Steady-State Approximations: A Toolbox of Techniques"

Grant Malone
5 years ago
Views:

1 Stein s Method for Steady-State Approximations: A Toolbox of Techniques Anton Braverman Based on joint work with Jim Dai (Cornell), Itai Gurvich (Cornell), and Junfei Huang (CUHK). November 17, 2017

2 Outline Toolbox for moment bounds: E X( ) C. Toolbox for gradient bounds: D 2 f h (x), D 3 f h (x). MDPs and control

3 Foster-Lyapunov Condition {X(t)} positive recurrent Markov chain V (x) R + is a Lyapunov function satisfying E x V (X(1)) V (x) cf (x) + b, where b, c, f (x) 0. Implies Ef (X( )) b/c. Problem: finding such a Lyapunov function is a bottleneck for many problems.

4 Join the Shortest Queue (JSQ)

5 JSQ Model Buffer exp(1) Poisson(nλ) arrival process Buffer Buffer exp(1) exp(1). Buffer exp(1) n servers, each has own buffer/queue, service rate µ = 1. Customer joins shortest queue, ties broken uniformly. Halfin-Whitt regime: λ = 1 β/ n.

6 State Space. X 4 (t) X 3 (t) X 2 (t) X 1 (t) X i (t) = number of servers with at least i customers at time t. {(X 1 (t), X 2 (t),...) Z } t 0 is a Markov chain.

7 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

8 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

9 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

10 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

11 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

12 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t)

13 Dynamics - Arrivals X 3 (t) X 2 (t) X 1 (t) General arrival transition: X i X i + 1 with rate nλ1(x 1 = X 2 =... = X i 1 = n, X i < n).

14 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t)

15 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t)

16 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t) X 1 X 1 1 with rate X 1 X 2 = 2.

17 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t) X 1 X 1 1 with rate X 1 X 2 = 2. X 2 X 2 1 with rate X 2 X 3 = 2.

18 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t) X 1 X 1 1 with rate X 1 X 2 = 2. X 2 X 2 1 with rate X 2 X 3 = 2. X 3 X 3 1 with rate X 3 X 4 = 1.

19 Dynamics - Departures X 3 (t) X 2 (t) X 1 (t) X 1 X 1 1 with rate X 1 X 2 = 2. X 2 X 2 1 with rate X 2 X 3 = 2. X 3 X 3 1 with rate X 3 X 4 = 1. In general: X i X i 1 with rate X i X i+1.

20 Known Results Let X(t) = ( X 1 (t), X 2 (t),...) R, where X 1 (t) = δ(x 1 (t) n), idle servers X i (t) = δx i (t), i 2. Gamarnik & Eschenfeldt (2017) establish: Functional Law of Large Numbers for δ = 1/n. Functional Central Limit Theorem for δ = 1/ n. Steady-state convergence not proved.

21 State Space Collapse (SSC) X 3 (t) X 2 (t) X 1 (t) Above is a picture of an unlikely state: For X 3 to increase, X 1 + X 2 must build up to 2n up rate nλ = n β n vs. down rate n when X 1 = n. In steady-state, expect fluid equilibrium + noise: X 1 ( ) n = β n + O( n), X 2 ( ) = 0 + O( n), X i ( ) = 0 + O(1), i 3. number of idle servers number of non-empty buffers

22 Applying Stein to JSQ

23 Markov Chain Generator Recall that X 1 (t) = δ(x 1 (t) n) and X i (t) = δx i (t) for i 2. Consider a function f (x 1, x 2 ) : R 2 R. For x 1 = δ(q 1 n), and x i = δq i, CTMC generator G X f (x) = nλ1(q 1 < n) ( f (x 1 + δ, x 2 ) f (x) ) + nλ1(q 1 = n, q 2 < n) ( f (x 1, x 2 + δ) f (x) ) + (q 1 q 2 ) ( f (x 1 δ, x 2 ) f (x) ) + (q 2 q 3 ) ( f (x 1, x 2 δ) f (x) ).

24 Markov Chain Generator Recall that X 1 (t) = δ(x 1 (t) n) and X i (t) = δx i (t) for i 2. Consider a function f (x 1, x 2 ) : R 2 R. For x 1 = δ(q 1 n), and x i = δq i, CTMC generator G X f (x) = nλ1(q 1 < n) ( f (x 1 + δ, x 2 ) f (x) ) + nλ1(q 1 = n, q 2 < n) ( f (x 1, x 2 + δ) f (x) ) + (q 1 q 2 ) ( f (x 1 δ, x 2 ) f (x) ) + (q 2 q 3 ) ( f (x 1, x 2 δ) f (x) ). Example third order error term: δ 2 x 3 1 f (x)δ(q 1 n) = δ 2 x 3 1 f (x)x 1. Suitable moment bounds needed on E X 1.

25 Moment Bounds via Fluid Approximation For diffusion approximation (δ = 1/ n), need moment bounds E(n X 1 ( )) n C. Equivalently, can we show that E(n X 1 ( )) n β n C/ n? β/ n is the fluid equilibrium of E(n X 1 ( ))/n.

26 Moment Bounds via Fluid Approximation For diffusion approximation (δ = 1/ n), need moment bounds E(n X 1 ( )) n C. Equivalently, can we show that E(n X 1 ( )) n β n C/ n? β/ n is the fluid equilibrium of E(n X 1 ( ))/n. Stein s method for fluid, or mean-field models (δ = 1/n). Stolyar (2015), Ying (2016), Gast (2017).

27 First Order Taylor Expansion Going forward, δ = 1/n: X 1 (t) = X 1(t) n n [ 1, 0], X 2 (t) = X 2(t) n ( X 1 (t), X 2 (t)) T = {y 1 [ 1, 0], 0 y 2 y 1 + 1}. Fluid equilibrium for ( X 1 ( ), X 2 ( )) is ( β/ n, 0). [0, 1], 1 1

28 The Partial Differential Equation (PDE) G X f (x) ( x 1 + x 2 β/ n ) x1 f (x) x 2 x2 f (x) + λ1(x 1 = 0) ( x1 f (x) + x2 f (x) ). For h(x) with h( β/ n, 0) = 0, consider the first-order PDE G F f h (x) = h(x), x T, x1 f h (x) x2 f h (x) = 0, x 1 = 0, where G F f (x) = ( x 1 + x 2 β/ n ) x1 f (x) x 2 x2 f (x).

29 Taylor Expansion Stein step: Eh( X( )) = EG F f h ( X( )) = EG X f h( X( )) EG F f h ( X( )) The expected generator difference equals 1 [ 2n E x1 x 1 f h (ξ (1) ) ( 1 + λ1( X 1 ( ) < 0) + X 1 ( ) + X 2 ( ) )] + 1 [ 2n E x2 x 2 f h (ξ (2) ) ( λ1( X 1 ( ) = 0, X 2 ( ) < 1) + X 2 ( ) )] λp( X 1 ( ) = 0, X 2 ( ) = 1) x2 f h (0, 1) ] + 1 n [X E 3 ( ) x2 f h (ξ (3) SSC error term. ) With fluid scaling, moment bounds simpler: X i ( ) 1. Still need gradient bounds...

30 Fluid Model for JSQ G F f (x) = ( x 1 + x 2 β/ n ) x1 f (x) x 2 x2 f (x). Reflection condition x1 f h (x) x2 f h (x) = 0 for x 1 = 0. Suggests dynamical system y(t) T, where ẏ(t) = F (y) R 2, and vector field (F 1 (y), F 2 (y)) equals ( y1 + y 2 β/ ) n, y(t) T \ {y y 1 = 0, y 2 [β/ n, 1]} 2 ( ) 0 β/, y(t) {y n 1 = 0, y 2 [β/ n, 1]}.

31 F (y) = ( y 1 + y 2 β/ n, y 2 ) 1 β/ n 1 β/ n

32 F (y) = ( y 1 + y 2 β/ n, y 2 ) 1 β/ n 1 β/ n

33 F (y) = ( y 1 + y 2 β/ n, y 2 ) 1 β/ n 1 β/ n

34 F (y) = (0, β/ n) 1 β/ n 1 β/ n

35 Gradient Bounds Recall the PDE Lemma G F f h (x) = h(x), x T, x1 f (x) x2 f (x) = 0, x 1 = 0. For nice h(x) : T R with h( β/ n, 0) = 0, f h (x) = 0 h(y(t))dt, where y(t) solves ẏ(t) = F (y) with initial condition y(0) = x. Perturbation theory: Ying(2016), Gast (2017) gradient bounds for f h (x) when vector field F (y) is continuous and y(t) exponentially stable.

36 Gradient Bounds X 1 ( ) Lemma Fix κ > β and let h(x) = min(x 1 + κ/ n, 0). Then n xi x i f h (x) κ β, x 2 f h (0, 1) = 0, x2 f h (x) 0, x T. 0 Eh( X 1 ( )) = 1 [ 2n E x1 x 1 f h (ξ (1) ) ( 1 + λ1( X 1 ( ) < 0) + X 1 ( ) + X 2 ( ) )] + 1 [ 2n E x2 x 2 f h (ξ (2) ) ( λ1( X 1 ( ) = 0, X 2 ( ) < 1) + X 2 ( ) )] + 1 ] [X n E 3 ( ) x2 f h (ξ (3) ) 2 n(κ β). As a consequence, E X 1 ( ) n / n C(β).

37 Gradient Bounds X 2 ( ) Conjecture (Work in Progress) Fix κ > β and let h(x) = max(x 2 κ/ n, 0). Then xi f h (x) nc(β), xi x j f h (x) nc(β), x T. Can then show that EX 2 ( )/ ( ) n C(β) 1 + E(X 3 ( )). Conjecture: EX 3 ( ) C(β), n 1.

38 Summary To prove diffusion scale moment bounds: 1. Perform first order Taylor expansion (fluid approximation). Fluid-scale moment bounds are much easier. 2. First order PDE solved by f h (x) = 0 h(y(t))dt, y(0) = x. 3. For gradient bounds, either 3.1 Verify conditions in Ying (2016) or Gast (2017). 3.2 brute-force track the behavior of fluid system like for JSQ; see Stolyar (2015) for another example.

39 Gradient Bounds for Multi-Dimensional Diffusion Approximations

40 The Poisson Equation Consider the Poisson equation G Y f (x) = Eh(Y ( )) h(x), x R d. Goal: bound Df (x), D 2 f (x), and D 3 f (x). Leverage solution form f h (x) = 0 ( Ex h(y (t)) Eh(Y ( )) ) dt. 1. Use elliptic PDE theory (Gilbarg and Trudinger) 2. Coupling arguments 3. x E x Y (t), xx E x Y (t), xxx E x Y (t)?

41 Gradient Bounds Diffusion Setting: {Y (t)} t 0 is a diffusion process on R d. Lipschitz drift (b 1 (x),..., b d (x))r d R d b(x) b(y) c b x y. Diffusion coefficient A = (a ij ) symmetric positive definite. Generator G Y f (x) = d i=1 b i (x) xi f (x) di,j=1 a ij xi x j f (x)

42 Schauder Interior Estimates For x R d, let B x = {y R d : y x 1 1+ x }. Lemma (Gilbarg & Trudinger (2001)) Suppose f h (x) solves G Y f h (x) = Eh(Y ( )) h(x). There exists a constant C depending only on A and c b, such that Df h (x) + D 2 D 2 f h (z) D 2 f h (y) f h (x) + sup y,z B x,y z z y ( C sup f h (y) + Eh(Y ( )) y B x ) h(z) h(y) + sup h(y) + sup (1 + x ) 3. y B x y,z B x,y z z y Gurvich (2014) was first to apply this idea.

43 Using Geometric Ergodicity How to bound f h (x)? Recall f h (x) = Suppose we know that 0 ( Ex h(y (t)) Eh(Y ( )) ) dt. E x h(y (t)) Eh(Y ( )) V (x)e ηt, η > 0. This condition is known as V -geometric ergodicity. Then E x h(y (t)) Eh(Y ( )) dt 0 0 V (x)e ηt dt CV (x). Can combine with Schauder estimates on previous slide.

44 Schauder Estimates Drawbacks Difficulties of using Schauder estimates: General result, not specialized to any system. Geometric ergodicity can be difficult to verify in practice. Non-smooth PDE domains Do we have an alternative?

45 One Idea Coupling For any ε > 0, xi f h (x) f h(x + εe (i) ) f h (x) ε = 1 ( Ex+εe (i)h(y (t)) E x h(y (t)) ) dt. ε 0 where e (i) R d equals 1 in the ith component, and zero elsewhere. Main idea: construct a coupling {Ȳ (t)} t 0 such that 1. Ȳ (0) = x + εe (i) 2. Ȳ (t) Y (t) = O(ε). 3. Mixing time of {Ȳ (t)} and {Y (t)} can be bounded.

46 Examples in the Literature Barbour 1988 Poisson process approximation Barbour 1990 Multivariate Normal approximation Mackey & Gorham 2016 Log-concave distributions Gan, Röllin, Ross 2017 Dirichlet approximation

47 Networks of Single Server Stations

48 A G/G/1 Queue Consider a single-server queue operating under first-come-first-serve discipline. A, A 1, A 2,... i.i.d. inter-arrival times with mean 1/λ > 1. S, S 1, S 2,... i.i.d. service times with mean 1. Traffic intensity ρ = λ < 1. Lindley recursion for waiting times: Lindley recursion for W n the nth customer s waiting time in queue: W n+1 = (W n + S n A n+1 ) +, x + := max(x, 0). A n, S n inter-arrival and service time of nth customer, respectively.

49 Generator Scaled steady-state customer waiting time Markov process generator where W ( ) = (1 ρ)w ( ). [ G W f (w) := E f ( (w + X) +)] f (w), w 0, X d = (1 ρ)(s A), EX = (1 ρ)2. ρ

50 Taylor Expansion Taylor expansion suggests diffusion generator bf (w) σ2 f (w) f (0)EX, where b = EX, and σ 2 = EX 2. Poisson equation G Y f h (w) = Eh(Y ( )) h(w), w 0, f h(0) = 0, where G Y f h (w) := bf (w) σ2 f (w).

51 Taylor Expansion Taylor expansion suggests diffusion generator bf (w) σ2 f (w) f (0)EX, where b = EX, and σ 2 = EX 2. Poisson equation G Y f h (w) = Eh(Y ( )) h(w), w 0, f h(0) = 0, where G Y f h (w) := bf (w) σ2 f (w). Gradient bounds via 1 ε 0 ( Ew+ε h(y (t)) E w h(y (t)) ) dt.

52 Reflected Brownian Motion Reflected Brownian motion on R + Y (t) = Y (0) bt + σb(t) + R(t). Regulator R(t) = sup 0 s t (Y (s)). BM( b, σ 2 ) 0 0 0

53 Coupling Recall f h (w) 1 ε Let 0 E w+ε h(y (t)) = E w h(ȳ ε (t)). Same source of randomness. ( Ew+ε h(y (t)) E w h(y (t)) ) dt. Y (t) =w bt + σb(t) + R(t), Ȳ ε (t) =w + ε bt + σb(t) + R(t) 0 0 0

54 Result Let τ w = inf{t 0 Y (t) = 0 given Y (0) = w}. Eτ w = w b. Standard result about Brownian motion. Lemma f h(w) = f h (w) = f h (w) = 0 + lim ] E w [h (Y (t))1(t τ w ) dt ] E w [h (Y (t))1(t τ w ) 0 1 ε 0 ε lim ε 0 ε dt ] E ε [h (Y (t))1(t τ ε ) dt ] E w [h (Y (t))1(t τ w ) dt 0 ] E ε [h (Y (t))1(t τ ε ) dt

55 Multidimensional SRBMs Consider the M/M/1 /M/1 tandem system, we are interested in the queue lengths. The approximating diffusion process is a two-dimensional semimartingale reflecting Brownian motion (SRBM) Y = {(Y 1 (t), Y 2 (t)) R 2 +, t 0}. See Williams (1995) for a review of SRBMs.

56 2-d SRBM ( ) ( ) Y1 (t) b1 = Y (0) + t + Σ Y 2 (t) b 2 b 1 ( ) ( B1 (t) + B 2 (t) b 1, b 2 > 0, {B(t)} is standard Brownian motion. Denote generator by G Y. R 1 (t) R 1 (t) + R 2 (t) )

57 f h (x) = 0 ( Ex h(y (t)) Eh(Y ( )) ) dt. xi f h (x) 1 ε 0 ( Ex+εe (i)h(y (t)) E x h(y (t)) ) dt.

58 Coupling Visualized x2 f h (x) = [ 0 E x x2 h(y (t))1(t τ x (2) ] (0)) dt. τ x (i) (s) = inf{t s Y i (t) = 0 given Y (0) = x}.

59 Coupling Visualized x1 f h (x) = [ ] E x x1 h(y (t))1(t τ x (1) (0)) dt E x [ x2 h(y (t))1(τ (1) x (0) t τ (2) x (τ (1) x )) ] dt.

60 First Derivatives Lemma x1 f h (x) = x2 f h (x) = [ ] E x x1 h(y (t))1(t τ x (1) (0)) dt E x [ x2 h(y (t))1(τ (1) x (0) t τ (2) x (τ (1) x )) E x [ x2 h(y (t))1(t τ (2) x (0)) ] dt. ] dt Expected hitting time bounds Dupuis & Williams (1994). Can t get explicit form of second derivative (can bound it though). To reach third derivative, one direction is to use higher order finite difference schemes: e.g. Barbour (1988), Gorham & Mackey (2016), Gan et al (2017).

61 Directional Derivatives of SRBM Lipshutz & Ramanan (2016) On directional derivatives of Skorokhod maps in convex polyhedral domains Characterize xi E x Y (t).

62 Conclusion Higher order derivatives tougher. One idea: apply first derivative bounds to Schauder estimates Df h (x) + D 2 D 2 f h (z) D 2 f h (y) f h (x) + sup y,z B x,y z z y ( C sup f h (y) + Eh(Y ( )) y B x ) h(z) h(y) + sup h(y) + sup (1 + x ) 3. y B x z y y,z B x,y z Need to be careful about existence/regularity of PDE solution.

63 Summary of Gradient Bound Techniques Obtaining multi-dimensional gradient bounds is difficult, and problem specific. Coupling-based bounds Schauder interior estimates needs bound on f h (x) Geometric ergodicity. Bounds on Df h (x) from coupling. Sensitivity analysis: x E x Y (t), xx E x Y (t), xxx E x Y (t).

64 Applications to Control: Taylor Expansions of Value Functions Based on joint work with Itai Gurvich and Junfei Huang

65 MDP Overview Controlled discrete time Markov chain {X(t) S} t Z+. S Z d, S < (think S = rectangle). r(x, u) reward for performing action u at state x. E x,u (X(1)) = E ( X(1) X(0) = x, u(x(0)) = u ). U(x) set of feasible actions at state x S. Policy π : S U(x). Inifinite horizon discounted expected reward: V π (x) = α t ( ) E x r(x(t), π(x(t)), x S, t=0 α < 1 is the discount factor.

66 Bellman Equation Goal is to find Bellman equation: V (x) = max π (x) arg max π(x), π (optimal policy) V (x) = max π(x), π (optimal value). u U(x) ( ) r(x, u) + αe x,u V (X(1)), ( 0 = max r(x, u) + α ( E x,u V (X(1)) V (x) ) ) (1 α)v (x) u U(x) Idea: perform Taylor expansion on E x,u V (X(1)) V (x).

67 Taylor Expansion Focus on 1-d case for notational convenience. E x,u V (X(1)) V (x) = y S P (u) x,y V (y) V (x) 2nd order V (x) y S = V (x) E x,u (X(1) x) }{{} b u(x) P x,y (u) (y x) V (x) P x,y (u) (y x) 2 y S V ( ) 2 (x) E x,u X(1) x. }{{} a u(x)

68 Taylored Control Equation Bellman vs. Taylored: vs. ( ) V (x) = max r(x, u) + αe x,u V (X(1)), u U(x) π (x) arg max u U(x) (1 α) V (x) = max u U(x) ˆπ (x) arg max u U(x) ( ) r(x, u) + αe x,u V (X(1)), (r(x, u) + α ( b u (x) V (x) a u(x) 2 V (x) )) ( r(x, u) + α ( b u (x) V (x) a u(x) 2 V (x) )).

69 Main Result Let d π = max π(x) P y x (maximal jump size). x,y >0 Theorem (B, Gurvich, Huang (2017)) Suppose there exists a solution V C 3 (R d ), that V (x) C(1 + x m ), and that dˆπ, d π <. Then, ( V ) (x) V (x) Vˆπ (x) V (x) d 3ˆπ d 3 [ π (E x,ˆπ α n D 3 V ] Xn±d ˆπ n=0 n=0 [ ]) + E x,π α n D 3 V Xn±d π, x S. First guarantees of this type appeared in Gurvich & Huang (2016) for a controlled M/G/1 + G system.

70 So What? Went from MDP on discrete, finite state space S to continuous space control problem.

71 So What? Went from MDP on discrete, finite state space S to continuous space control problem. Computational implications of Tayloring: Up to us to choose discretization level to solve Taylored problem useful when S is very large. Makes search over action space tractable.

72 Policy Iteration (PI) 1. Start with some initial policy π (0) (x). 2. For k = 0, 1, find V (k) (x) that satisfies V (k) (x) = r(x, π (k) (x)) + αe x,π (k) (x)[ V (k) (X(1)) ], x S. This is known as the policy evaluation step. 2.2 Set { π (k+1) (x) = arg max r(x, u) + αe x,u [V (k) (X(1)) ]}, x S. u U(x) This is known as the policy improvement step.

73 Taylored Policy Iteration (TPI) 1. Start with initial policy π (0) (x). 2. For k = 0, 1, For x R, find V (k) (x) satisfying (1 α) V (k) (x) = (r(x, π (k) (x)) + α ( b π (x)(x) V (k) (x) + 1 (k) 2 a π (x)(x) 2 V (k) (x) )) (k) 2.2 Set ˆπ (k+1) (x) equal to ( arg max r(x, u) + α ( b π (x)(x) V (k) (x) + 1 u U(x) (k) 2 a π (x)(x) 2 V (k) (x) )). (k) Not just a heuristic.

74 Dupuis and Kushner (2013) Coarse Grid Solution (1 α) V ( (x) = max r(x, u) + α ( b u (x) V (x) + 1 u U(x) 2 a u(x) 2 V (x) )). Use finite differences V (x) ( V (x + h) V (x h))/2h, which leads to new Bellman equation with discount β α,h = (1 + (1 α)h 2 ) 1 : ( V h (x) = max h 2 ( β α,h r(x, u) + β α,h P h,u V x,x+h h (x + h) u U(x) Choice of h is up to us! Sparse transition matrix + P h,u x,x h V h (x h) + P h,u x,x V h (x) )).

75 Example (based on Dai & Shi 2017) Author: Multi-pool model with overflow 4 Manufacturing & Service Operations Management 00(0), pp , c 0000 INFORMS 1(t) 2(t) 3(t) 4(t) 5(t) GeMed Surg Ortho Card OtMed GeMed Surg Ortho Card OtMed Figure 1 Illustration of the five-class, five-pool system. Pools 1 to 5 are the primary server pools for General Medicine (GeMed), Surgery (Surg), Orthopedic (Ortho), Cardiology (Card), and Other Medicine (OtMed), respectively; λ solid black lines represent the primary bed assignment, dashed red lines represent i = 0.8N i p i λ i = N i p i the preferred overflow assignment, and dashed blue lines represent the secondary overflow assignment. α h TPI rel. error α h TPI rel. error waiting patient to a non-primary ward, and (b) which ward to use 2 if multiple non-primary wards have available beds. A 1primary bed assignment, if possible, is made 1 immediately upon a patient arrival or bed release since 2 this is the most ideal situation and thus, 2 it is excluded from the decisions at each epoch. We model the overflow decisions as a Markov decision process (MDP), where the Table: J = decision 2, N maker aims to minimize the long-run average cost over (infinitely) many epochs. The total 1 = N 2 = 10, M = 10, (p 1, p 2 ) = (0.56, 0.56), (H 1, H 2 ) = (1, 4), and (B 12, B 21 ) = (5, 1). cost in each epoch includes: the holding cost associated with the number of waiting patients, which captures the system Relative congestion; errors displayed are sup x S V (x) Vˆπ (x) the overflow cost associated with each overflow patient, which captures the non-desirableness of placing a patient in a non-primary ward. These two costs represent the key tradeo in the overflow decision and provide us a tuning knob

76 References Barbour (1988), Stein s Method and Poisson Process Convergence, Journal of Applied Probability, Vol. 25, pp Barbour (1990), Stein s method for diffusion approximations, Probability Theory and Related Fields, September 1990, Volume 84, Issue 3, pp Mackey & Gorham (2016), Multivariate Stein Factors for a Class of Strongly Log-concave Distributions. Electronic Communications in Probability 2016, Vol. 21, paper no. 56, Gan, Röllin & Ross (2017), Dirichlet approximation of equilibrium distributions in Cannings models with mutation. Advances in Applied Probability, Volume 49, Issue 3, September 2017, pp

77 references Lipshutz & Ramanan (2016), On directional derivatives of Skorokhod maps in convex polyhedral domains. To appear The Annals of Applied Probability. Kushner & Dupuis (2013), Numerical Methods for Stochastic Control Problems in Continuous Time. Gurvich & Huang (2016), Beyond Heavy-Traffic Regimes: Universal Bounds and Controls for the Single-Server Queue. Dai & Shi (2017), Inpatient Bed Overflow: An Approximate Dynamic Programming Approach.

Stein s Method for Steady-State Approximations: Error Bounds and Engineering Solutions

Stein s Method for Steady-State Approximations: Error Bounds and Engineering Solutions Jim Dai Joint work with Anton Braverman and Jiekun Feng Cornell University Workshop on Congestion Games Institute