Management of demand-driven production systems

Transcription

1 Management of demand-driven production systems Mike Chen, Richard Dubrawski, and Sean Meyn November 4, 22 Abstract Control-synthesis techniques are developed for demand-driven production systems. The resulting policies are time-optimal for a deterministic model, and approximately time-optimal for a stochastic model. Moreover, they are easily adapted to take into account a range of issues that arise in a realistic, dynamic environment. These control synthesis approaches are based on the following development: (i) Workload models are investigated for demand-driven systems. (ii) The associated workload-relaxation is introduced as an approach to model-reduction. (iii) Optimal control synthesis approaches are developed for the workload-relaxation. (iv) The impact of hard constraints on performance, and on optimal control solutions is addressed via Lagrange multiplier techniques. These include deadlines and buffer constraints. (v) Control synthesis techniques are developed for models in which resources are temporarily unavailable. This may be due to failure, maintenance, or an unanticipated change in demand. (vi) Rules for choosing appropriate safety-stocks are constructed to ensure robustness of control solutions with respect to persistent disturbances, such as variability in demand and yield. Keywords Multiclass Queueing Networks, Pull Models, Routing, Scheduling, Optimal Control. AMS subject classifications Primary: 9B35, 68M2, 9B15. Secondary: 93E2. Work supported in part by grants DMI and ECS Department of Electrical and Computer Engineering and the Coordinated Sciences Laboratory, University of Illinois at Urbana-Champaign, Urbana, IL 6181, U.S.A. (s-meyn@uiuc.edu).

2 1 Introduction Together with finance and marketing, production management forms a major branch of modern scientific economic studies. This field has a long history, beginning with the combinatoric treatment of scheduling rules for a fixed number of jobs in job-shop and flow-shop production systems [54, 55, 27, 43, 35]. Over the past 2 years academic research has focused on dynamic, stochastic models [26, 16, 13, 4, 42, 3]. In this paper we consider control synthesis in a single production system, as opposed to a group of factories and their interactions. Even within this limited scope, there are numerous control decisions required for successful operation. These include, work release, routing, sequencing, lot sizing, batching, and preventive maintenance scheduling. These operating decisions are based on a range of performance metrics such as minimizing lead-time and inventory. An application of current interest in both academia and industry is the semiconductor fabrication system. A modern plant may cost upward of several billion dollars [18]. These systems are subject to high variations in demand, and frequent maintenance requirements result in significant internal plant variability [64]. Figure 1 shows a simplified model to illustrate some of the aforementioned issues. In this model there are two external demands with rates δ 1 and δ 2. Production management involves work release decisions, modeled by the parameters λ 1 and λ 2, as well as scheduling and routing decisions for the various resources. q λ 1 1 q 5 λ 2 q 2 q 6 q 3 Station 1 q 7 q 8 Station 2 q 13 δ 1 δ 2 q 14 q q 4 9 q 12 q 11 µ 1a q 1 µ 1b Station 5 Station 4 Station 3 Figure 1: A 16-buffer semiconductor production system. Much of the literature on production management is based on optimal control, using either combinatorics or dynamic programming techniques. It is well known that either approach is typically computationally infeasible in realistic models. Moreover, even in cases where an optimal solution is computable, the solution may not provide qualitative insights on closed-loop system behavior. The construction and analysis of policies to determine work release of raw material to a production system has been the subject of some research [44, 35]. The most common approach is the CONWIP (constant work-in-process) policy in which new material is released to the system just as completed jobs depart [9, 2, 61]. Hence, under the CONWIP policy the total customer population remains constant. Research has focused on the determination of an optimal value 1

3 for the overall customer population [34, 39, 53]. The policy synthesis techniques developed here are based upon simplified deterministic models, and relaxations based upon workload. We arrive at a geometric approach to policy synthesis for the deterministic fluid model, and translation to a model with variability is then performed using safety-stocks. Many of these concepts extend or synthesize previous approaches to policy synthesis. In particular, the application of fluid models in policy synthesis has become widespread in recent years (see e.g. [56, 12, 2, 14, 57, 3, 4, 66, 45, 46, 59, 5, 3]). In particular, fluid models have been used recently to construct policies that minimize makespan, which is closely related to the time-optimality constraint imposed here [49, 65, 21, 5]. The workload-relaxation described here is similar to the application of state-space collapse for networks in a heavy-traffic regime, where a stochastic model with Gaussian statistics is obtained via a Central Limit Theorem scaling [1, 31, 36, 5, 33, 51]. The present paper does not concern limiting regimes, but instead applies the workload relaxation as a first step in policy synthesis. Other approaches to model reduction through aggregation are developed in the monographs [19, 62]. It has been known for some time that optimal control solutions for a stochastic network model are typically defined by switching curves or threshold rules defined in the state space of buffer levels [52, 25, 28, 8, 49, 15]. The simplest instance involves safety-stock levels to avoid starvation. Appropriate safety-stock levels for network models in heavy-traffic are considered for a specific examples in [7], and general theory is developed in [47, 5, 5]. Buffer constraints have been treated previously, e.g., [58, 37, 41], where the focus is mainly on particular performance metrics such as blocking probabilities or throughput. Machine failures were investigated previously in [1, 63]. A list of contributions of the present paper is as follows: (i) Workload-models are described for demand-driven production systems, and structural properties of the associated workload-relaxations are investigated. (ii) The greedy-time-optimal (GTO) policy is introduced in which time-optimality is utilized as a constraint for the deterministic model. It provides a path-wise optimal solution whenever a path-wise optimal solution exists. Moreover, the policy is computable using a finite-dimensional linear program. (iii) The impact of buffer constraints is considered. The sensitivity to buffer constraints of both performance, and policy structure, is addressed using Lagrange-multiplier techniques. (iv) The GTO algorithm is adapted to account for scheduled maintenance, unanticipated breakdown, or transient exogenous demand. (v) The introduction of safety-stocks ensures that the policy is robust with respect to persistent, unpredictable disturbances. The remainder of the paper is organized as follows. In Section 2, we discuss various models of demand driven production systems. In particular, the workload formulation and relaxations are introduced. We focus on policy synthesis in Section 3 through consideration of the fluid model and its relaxations. The GTO policy arises naturally from an examination of optimal 2

4 policies for the workload relaxation. Section 4 concerns policy synthesis techniques for systems subject to breakdown, maintenance, and fluctuations in demand and yield. Conclusions and plans for future research are included in Section 5. 2 Network models We begin with a description of the network models to be considered, including stochastic and fluid network models, in addition to workload models and their relaxations. We also examine pull and push models, and describe how the former can be translated into the latter. Although most of the results discussed in this paper are derived through examination of deterministic models, our ultimate aim is to achieve desirable performance in the production system of interest. Physical production systems are most closely approximated by the continuous-time stochastic model (1) described next. 2.1 Continuous-time stochastic network model The most detailed of all network models considered in the literature is the continuous-time stochastic network model: Q(t; x ) = x S(Z(t; x )) + R(Z(t; x )) + A(t), t. (1) The state process Q evolves on the state space X R l +, where l is the number of buffers in the network, and x X is the initial condition. The inclusion X R l + may be strict when buffer constraints are imposed. The cumulative allocation process Z evolves on R lu +, for some integer l u 1. Its components are nondecreasing in t, with Z j (; x ) = for all 1 j l u, since Z j (t; x ) is the cumulative processing time up to time t allocated to activity j. The stochastic process A models exogenous arrival and departure of customers, S models service processes, and R models possibly random routing of customers to queues in the network. This network model has been analyzed in numerous references, e.g., [67, 22, 42, 13, 5], where the following constraints are imposed: (i) Q(t; x ) X, and Q i (t; x ) Z + for all 1 i l, t, x X. (ii) The cumulative allocation process Z is subject to the linear constraints: Z(t; x ) Z(s, x ) t s U, s < t <, (2) where U is the set of allowable allocation rates: For some integer l m, and a l m l u matrix C, U := {u R lu + : Cu 1}, (3) where 1 indicates a vector consisting of ones. (iii) The l m l u matrix C is called the constituency matrix. l m is equal to the number of resources in the network and l u is the number of activities (or controls variables). (iv) The process A is a R l +-valued stochastic process. Its components are non-decreasing functions of time. 3

5 (v) Each of the stochastic processes {S, R} is a random function from R lu + Rl +. The components are non-decreasing functions of time. The stochastic processes S and R are typically highly correlated. In this paper we assume that the constituency matrix has binary entries, with C ij = 1 if and only if activity j is performed by resource i. The condition Cu 1 indicates that resources are shared among activities, and that they are limited. While valuable in simulation and analysis, the stochastic model (1) has little value in policy synthesis when considered in this generality. Frequently, an effective policy may be generated through consideration of a much simpler model that retains important details. In this paper we consider primarily the fluid model (4). It is known that, under mild statistical assumptions on (1), it is possible to translate a stabilizing policy for the fluid model to obtain a stabilizing policy for the stochastic model [47, 5, 49, 5], and for a given policy, stability of the fluid limit model implies stability of (1) [22, 23, 38, 24, 49]. Moreover, an optimal policy for the stochastic model (1) is approximated by the optimal policy for the fluid model (4) [48, 49, 15]. We return to the stochastic model in Section 4.4 where we provide detailed statistical assumptions, and describe further details on the translation of policies from the fluid to the stochastic model. 2.2 Fluid network model The fluid model (4) is again a continuous time model. The state process is denoted q, which evolves on X R l +, the non-decreasing cumulative allocation process z evolves on R lu +, and x X is the initial condition. The components of q are again interpreted as the number of jobs at the buffers in the network, and for each 1 j l u, ζ j (t; x ) = d dt z j(t; x ) denotes the instantaneous percentage of time devoted to activity j at time t. The time-evolution equation is given by the linear system, q(t; x ) = x + Bz(t; x ) + αt, t, (4) where α R l + is a vector that models exogenous arrival and departure rates, and B is an l l u matrix that is defined by averaged routing and service rates, and network topology. The fluid model is subject to constraints analogous to those imposed on (1): (i) The state process trajectories are continuous and deterministic. (ii) The allocation process is subject to the constraint ζ(t; x ) U, where the set of allocation rates U is given in (3). (iii) The vector α R l + and the l l u matrix B are fixed. Throughout the paper we assume that the fluid model is stabilizable. That is, for any initial condition x X, one can find an allocation z and a time T such that q(t ; x ) = x + Bz(T ) + αt = θ, where θ R l is a vector of zeros. Characterizations of stabilizability are presented in [49], and some of these results are reviewed in Section 3. 4

6 The fluid model may be motivated through a scaling of the stochastic model (1) (see e.g. [12, 22]). Consider the averages of Q defined by q r (t; x ) = Q(rt; rx ) r z r (t; x ) = Z(rt; rx ), t, x X, r 1. r Assuming that the Law of Large Numbers holds, R(nζt) S(nζt) lim n n lim n A(tn) n = Bζt, ζ U = αt, a.s., t, it is not surprising that (q r, z r ) converge as r to a solution of the fluid-model equation (4) under mild conditions on Z, A, R, S. In this paper we focus on demand-driven models (also known as pull models). An example is the 16-buffer network shown in Figure 1 in which demand rates for the two products produced are indicated by {δ 1, δ 2 }. In contrast, the network shown in Figure 3 is a push model, in which arrivals are unregulated. A push model is not appropriate for the production systems considered here since we assume that demand is exogenous, while the arrival of new material is determined by the particular operating policy employed. In a production system, for each product produced, a virtual queue is used to model inventory: a negative value indicates deficit (unsatisfied demand), and a positive value indicates excess inventory. Ideally, a policy would regulate virtual queues to zero, though this is rarely feasible in practice. It is frequently desirable to maintain inventory through safety stocks to ensure that demand is satisfied with high probability. We avoid the use of negative buffer levels, and instead translate a given pull model into an equivalent push model with state space X R l +. We illustrate this construction using the network shown at left in Figure 2. This is a simple pull model with a single production resource, and a single virtual queue. At right is an equivalent push model, obtained by replacing the virtual queue by a single exit-resource with two buffers. Raw material is modeled as the supply resource with maximal rate λ. In the resulting push model, one buffer at the exitresource is fed by the production resource, and the other is fed by the demand process with instantaneous rate δ. These buffers are interpreted as surplus and deficit buffers, respectively. The fluid model equation for the push model is given by (4) with, B = λ µ µ µ d µ d, α = δ, C = (5) We assume that the policy at the exit-resource is non-idling, so that ζ 3 (t) = 1 when q 2 (t) > and q 3 (t) >. In this case, the two systems are equivalent if the service rate µ d of the exit-resource is sufficiently large. Any pull model may be transformed in this way to give a push model evolving on R l + for some integer l. 5

7 λ µ Surplus Defecit δ λ µ Surplus µ d δ Defecit Pull-model of inventory system Translation to equivalent push-model Figure 2: Example of the conversion of a pull model to a push model. If µ d µ then the two models shown are equivalent. 2.3 Workload formulations and relaxations Although considerably simpler than the stochastic network model, a fluid model may remain far too complex for exact analysis (although algorithms exist for numerical computation of policies [45]). In this section we introduce workload models and their relaxations. Frequently, these are of far lower complexity since the number of resources is typically far less than the number of buffers in a production system. We also define in this section a formulation of system load, which provides a characterization of stabilizability. The definitions given here are taken from [5] (related concepts are developed in [33, 1, 36]). The fluid model (4) is used to establish terminology since the transformation from buffercoordinates to workload-coordinates requires only the mean-flow properties of the network. The definition of system load is based on equilibrium allocations ζ ss U satisfying α+bζ ss = θ. In the classical scheduling problem, where no routing decisions are required, the matrix B is square since there is a unique activity associated with each buffer. Assuming that it is also full rank, it then follows that there is a unique equilibrium allocation given by ζ ss = B 1 α. In this special case, we define the vector load ρ by ρ = (ρ 1,..., ρ lm ) T = CB 1 α = Cζ ss, (6) and the system load ρ is the maximum, ρ = max i ρ i. When B is not square then the definition of load requires further effort. Consider first the velocity set V given by, V := {Bζ + α : ζ U}. (7) The fluid model may be described as a differential inclusion on X R l +, where the derivative of q is constrained to lie in V: d dt q(t; x ) V, q(t; x ) X, t. (8) Since V is a polyhedron, it can be expressed as the intersection of half-spaces: for some integer l v 1, vectors {ξ i : 1 i l v } R l, and constants {κ i : 1 i l v } R, V = {v : ξ i, v κ i, 1 i l v }. (9) It follows that for a given x X, the minimal draining time is given by T (x) = max 1 i l v ξ i, x κ i. 6

8 Stabilizability ensures that this is finite for any x X. Stabilizability also ensures that θ V, so that κ i for each i. It is shown in [5] that these parameters have the specific form, κ i = β i ξ i, α, 1 i l v, where the {β i } are binary-valued. We assume below that these are ordered so that, for some integer l r l v, β i = 1 for i l r and β i = for i > l r. We thus arrive at the following definitions: Workload and system load The vectors ξ i, 1 i l r, are called the workload vectors, and the index i is called a pooled resource. The vector load ρ R lr is defined by, ρ i := ξ i, α, 1 i l r, and the system load is defined as the maximum, ρ := max{ρ i }. For a given state trajectory q, the corresponding workload process is defined by w(t; w ) = Ξq(t; x ), (1) where Ξ denotes the l r l matrix with rows given by { ξ i : 1 i l r }, and w = Ξx. The following assumptions are imposed throughout the paper. The ordering in (A2) is for convenience, and may be assumed without any loss of generality. The expression (11) holds in the examples under consideration whenever the network load is sufficiently large. (A1) The given network is stabilizable. (A2) The vector load entries ρ i are non-increasing in i, and the system load satisfies ρ = ρ 1 < 1. (A3) There exists l r l r such that the workload vectors { ξ i : 1 i l r } are linearly independent, and the minimal draining time may be expressed, T (x) = ξ i, x max x X. (11) 1 i l r 1 ρ i The relationship between system load and stabilizability is illustrated in Proposition 2.1. We say that the ith pooled resource is a dynamic bottleneck at time t if T (q(t; x )) = ξi, q(t; x ) 1 ρ i, and we let I(x) denote the index set of dynamic bottlenecks when q(t; x ) = x. We obtain from these definitions the following proposition. For a proof, see [5, Proposition 2.5]. Proposition 2.1 Suppose that (A1-A3) hold. Then, for any condition x X, 7

9 (a) For any state x 1 X, provided this state is reachable from x, the minimum time to reach x 1 from x is given by T (x, x 1 ) := max 1 i l v ξ i, x 1 x κ i. (12) (b) For any allocation-state pair (z, q), the minimal draining time T (x ) is achieved if and only if d dt ξi, q(t; x ) = (1 ρ i ), i I(q(t; x )), t T (x ). The dynamics of the workload process are almost decoupled under (A3): Proposition 2.2 Under Assumptions (A1) (A3) the workload process w is a differential inclusion with state space W := {Ξx : x X}. For any t, the following lower bounds hold: d dt w i(t) (1 ρ i ) 1 i l r. In the relaxed model considered next this decoupling is complete. Workload relaxations For a given 1 n l r, the nth workload relaxation is defined as follows: (i) The state space is X R l +, and the velocity set is given by V = { v : ξ i, v (1 ρ i ), 1 i n }, (13) where ρ 1... ρ n. We denote by q any trajectory in X satisfying d dt + q(t) V, q(t) X, t, where d + dt denotes the right-derivative. (ii) The workload process for the relaxation is given by ŵ(t; w ) = Ξ q(t; x ), t, (14) where Ξ is the n l matrix with rows equal to { ξ i : 1 i n } and w = Ξx. The workload process evolves on the workload space, given by Ŵ := { Ξx : x X} We have the following analog of Proposition 2.2. The proof is immediate from the definitions. Proposition 2.3 Under Assumptions (A1) (A3), for each 1 n l r, the workload process ŵ for the the nth workload relaxation is a differential inclusion with state space Ŵ, and velocity constraints given by d dt w(t; w ) {φ R n : φ i (1 ρ i ), 1 i n}. (15) That is, the dynamics of the relaxed workload process ŵ(t; w ) are decoupled. 8

10 α 1 µ 1 µ 2 Station 1 Station 2 µ 3 Figure 3: Simple 2-station, 3-buffer network. λ 1 µ 1 µ 2 Station 2 Station 1 δ 1 µ 3 Figure 4: The 2-station model in a pull configuration. 2.4 Examples We conclude this section with several examples to illustrate these definitions. The simple pull-model The first two workload vectors for the network shown on the right hand side of Figure 2 are given by, ξ 1 = (, µ 1, µ 1 ) T ξ 2 = (,, µ 1 d )T, with respective loads given by ρ 1 = δµ 1, ρ 2 = δµ 1 d. For a given state trajectory q, the corresponding workload process is given by, ( w(t, w µ 1 ) (q ) = 3 (t) q 2 (t)) µ 1 d (q, w = Ξq(). (17) 3(t)) It may come as a surprise to find negative entries in the workload vectors shown in (16). However, this is physically reasonable given the expression (11) for the minimal draining time T. If one increases the quantity of material at buffer 2, then the processing time required to meet demand will be reduced proportionately, which implies that ξ 1 2 <. Similarly, ξi 1 =, i = 1, 2, since the addition of material at buffer one will not reduce the time required to meet demand. The 3-buffer model The network shown in Figure 3 is a push model, previously analyzed in [4, 66, 48, 15]. Its fluid model is given by (4), with µ 1 α 1 [ ] B = µ 1 µ 2, α = 1 1, C =, (18) 1 µ 2 µ 3 9 (16)

11 where {µ i } denotes service rates, and α 1 is the arrival rate to buffer 1. The vector load and workload vectors are given by, ρ = (α 1 (µ µ 1 3 ), α 1µ 1 2 )T ξ 1 = (µ µ 1 3, µ 1 3, µ 1 3 )T ξ 2 = (µ 1 2, µ 1 2, )T, (19) and for a given state trajectory q, the workload process is given by, ( w(t, w µ 1 ) = 1 q 1(t) + µ 1 3 (q ) 1(t) + q 2 (t) + q 3 (t)) µ 1 2 (q, 1(t) + q 2 (t)) t. (2) In this push model we see that the workload vectors have non-negative entries since increasing inventory cannot reduce the time to reach q = θ. The same model in the pull configuration is shown in Figure 4, with system parameters given as follows: λ µ 1 µ 1 µ 2 B = µ 2 µ 3 µ 3 µ d, α = 1, C = (21) 1 µ d d For a given demand rate δ, the vector load is given by, with corresponding workload vectors, ρ = (δ(µ µ 1 3 ), δµ 1 2, δµ 1 d )T, ξ 1 = (, µ 1 1, µ 1 1, (µ 1 ξ 2 = (,, µ 1 2, µ 1 2, µ 1 2 )T ξ 3 = (,,,, µ 1 d )T. 1 + µ 1 3 ), µ µ 1 3 )T (22) The vector ξ 3 corresponds to the exit-resource, and we assume the corresponding load ρ 3 is negligible. The workload vectors are related to those found in the push configuration, although some entries of ξ 1, ξ 2 are now negative. Routing model Figure 5 shows a simple routing model (related models are analyzed in [5, 36, 28, 29]). The fluid model is given by (4) with, B = µ 1 µ 3 µ 2 µ 3 µ 3 µ 3, α = α 3, C = (23) 1

12 α 3 µ 3 router µ 1 µ 2 Figure 5: Routing model. The vector load and workload vectors are given by, ρ = (α 3 (µ 1 + µ 2 ) 1, α 3 µ 1 3,, )T ξ 1 = (µ 1 + µ 2 ) 1 (1, 1, 1) T ξ 2 = (,, µ 1 3 )T ξ 3 = (µ 1 1,, )T ξ 4 = (, µ 1 2, )T. The form of the vector ξ 1 provides motivation for the terminology pooled resource. Positivity of the first and second entries of ξ 1 may be interpreted as resource pooling of the two down-stream resources [36]. 16-buffer production network In the network shown in Figure 1 there are five resources. We consider below several numerical examples. Suppose that the demand rate for each of the two products is equal to.2533, and the service rates are µ = (.8667, ,.8667, , 1, 2, 1, 2, 1,.3333,.5,.1, 1, 1) T. (25) The large values in entries 13 and 14 correspond to rates for the supply-resources. The rates for exit-resources were taken to be infinite, and we have ignored workload vectors corresponding to supply-resources and exit-resources in the calculations below. The first three workload vectors are given by: Ξ T (24), (26)

13 with corresponding loads, ρ i.95, i = 1, 2, 3. A three-dimensional relaxation is reasonable for this model since the first three (pooled) resources have relatively high loads. Just as in the previous pull models, we find that workload vectors have negative entries, and that entries corresponding to deficit buffers are non-negative. Now that we have specified the models to be considered, we turn to the control synthesis problem in this deterministic setting. 3 Policy synthesis The policies considered in this paper are based on the consideration of a cost function c: R l R +. We assume that c is a norm on R l, and that it is piecewise linear, of the specific form, c(x) = max( c i, x : i = 1,..., l c ) (27) with c i R l, i = 1,..., l c. For a given cost function we consider optimal control formulations for the fluid model and its relaxations. This requires a translation of cost in buffer coordinates to the effective cost on the space of workload configurations. 3.1 The effective cost For the nth relaxation, if two states x, y X are exchangeable, then we have T (x, y) = T (y, x) =, where T (x, x 1 ξ i, x 1 x ) := max, x, x 1 X. 1 i n 1 ρ i For the purposes of optimization, if q(t; x ) = y and there is an exchangeable state x satisfying c(y) > c(x), then there is no reason to remain at the state y. This motivates the following definitions for the nth relaxation: Effective cost Given any w Ŵ, the effective cost c(w) R + is the value of the linear program, c(w) = min η subject to c i, x η, 1 i l c Ξx = w x X (28) Effective state For any w Ŵ, the effective state X (w) is defined to be any vector x X that minimizes the linear program (28): ( X (w) = arg min c (x) : Ξx ) = w. (29) x X Optimal exchangeable state For any x X, the optimal exchangeable state P (x) is given by P (x) = X ( Ξx). (3) 12

14 Monotone region The monotone region Ŵ+ is the subset of Ŵ on which c Rn +. That is: { Ŵ + = w Ŵ : c(w) c(w ) whenever w w, and w Ŵ+}. (31) The effective cost function c : Ŵ R + is called monotone if Ŵ+ = Ŵ. In the nth relaxation we have decomposed the state space X into equivalent classes corresponding to each workload value. In the optimal control solutions considered below, for a given workload configuration w, we choose the representative state x = X (w) as the exchangeable state with the lowest cost. Since c is piece-wise linear it follows that this is also true for the effective cost, c(w) = max{ c i, w : 1 i l c }, (32) where { c i : i l c } R n are the extreme points obtained in the dual of (28). Up to now we have not considered in detail any structural properties of the state space. In practice, the state space is subject to constraints since there are upper bounds on buffer capacity. It is therefore important to investigate the impact of such constraints on performance and policies. Quantifying sensitivity of performance with respect to buffer constraints is of particular interest in network planning. Let b denote the l-dimensional vector of buffer constraints, satisfying < b i for 1 i l, and set X := {x R l + : x b}. The effective cost is found from the following linear program (the dual of (28)): max γ T w β T b subject to ΞT γ β c β θ γ not sign-constrained, (33) where γ R n and β R l +. The optimizers (γ, β ) to (33) are Lagrange multipliers. Consequently, γ provides sensitivity of the effective cost to workload, and β provides sensitivity with respect to buffer constraints. The following result is a consequence of this observation, and the fact that an optimal solution to (33) may be found among the basic feasible solutions for the constraint set in (33). We denote these by {(γ i, β i ) : 1 i l c } (note that the integer l c defined here is in general larger than the integer used in the unconstrained case with c given in (32)). Proposition 3.1 For the nth relaxation, and any w Ŵ, (a) The effective cost c(w) = c(w; b) is given by (b) Let i be the maximizing index in (34). Then, ( c(w) = max w T γ i b T β i). (34) 1 i l c w j c(w; b) = γ i j, 1 j n; b j c(w; b) = β i j, 1 j l. 13

15 1 1 w w 1 w 1 Figure 6: Two instances of the effective cost c for the two-dimensional relaxation of the network shown in Figure 1. On the left is shown contour plots of the effective cost when no buffer-constraints are imposed; at right is shown contour plots of the effective cost with bufferconstraints given in (36). To illustrate Proposition 3.1 we return to the 16-buffer model, with linear cost given by, c(x) = c, x, c = (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 5, 5, 1, 1) T. (35) That is, the holding-cost for each unit of completed inventory is 5; the cost for each unit of unsatisfied demand is 1; and the holding-cost for each unit of unfinished work is equal to 1. Consider a two-dimensional workload relaxation where Ξ = [ ξ 1T, ξ 2T]. The graph on the left hand side of Figure 6 shows the contour plot of the effective cost c(w) for the 16-buffer model under a two-dimensional workload relaxation when there are no buffer constraints. The figure shows that the monotone region Ŵ+ is a small subset of the entire feasible region Ŵ. The level sets of c change dramatically when buffer constraints are imposed. Consider the vector of constraints given by b = (1, 1, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 3, 3, 3, 3) T, (36) where the effective cost is now computed via (34). The level sets of c are shown on the right hand side of Figure 6. We have already noted that a three-dimensional relaxation may be more appropriate for this model - a two-dimensional relaxation is chosen for the purpose of illustration. Figure 7 shows contour plots of the effective cost for the three-dimensional relaxation, with w 3 fixed at the values w 3 = 3, w 3 = 6, and w 3 = 9, respectively. Based on this geometry we find that it is not difficult to devise an efficient policy for a workload model of moderate dimension. The question then arises, how can this be adapted to provide a policy for the original complex network of interest? 14

16 w w 3 = 3 w 8 3 = 6 w 8 3 = w 1 Figure 7: Contour plots of effective cost c for the three-dimensional relaxation of the 16-buffer model, with the buffer-constraints given in (36). The projections shown are based on fixing w 3 at the values 3, 6, and 9. This question is addressed in Section 4 of [5] for models in heavy traffic, but the same methods may be adapted to the present setting. Suppose that [ q (t; x ), ẑ (t; x )] is a piecewise linear, feasible solution to the nth relaxation. We assume that q (t; x ) = P ( q (t; x )) for all t >. Based on this solution for the relaxation, we define an allocation process ζ for the unrelaxed model as follows where, for each t, the rate ζ(t; x ) is a function of [ q(t; x ), ζ(t; x ), q(t; x )]. Given the current states y = q(t; x ), y = q (t; x ), let ζ(t; x ) be the optimizing value of the variable ζ in the following linear program. In words, this policy minimizes the instantaneous decrease in c(q(t)) at each time instant, subject to the constraint that w never exceed ŵ. min γ subject to γ c i, v, i I c (y) v i if y i = v i if y i = b i Bζ + α = v ζ U ξ i, (Bζ + α) ξ i, (B ζ + α), whenever 1 i n, and ξ i, y = ξ i, y. (37) In the first constraint we have used, I c (x) := {1 j l c : c(x) = c i, x }, x X. (38) The last constraint ensures that w i (t; x) ŵi (t; x) for all i n and all t. The policy is stabilizing under general conditions on the model and policy. In [5, Theorem 4.1] asymptotic optimality is established for networks in heavy traffic. Additional properties are developed in [5]. We now show how to generate polices for the relaxation through optimal control techniques. 15

17 3.2 Optimal control In the fluid model in which there is no variability we consider the transient control problem: given an initial condition x X, we wish to find an allocation z that drains the network in an economical manner. Below are three formulations of optimal control: Time-optimal control For each initial condition x X, find an allocation z that achieves the minimal emptying time, so that the resulting state trajectory q satisfies, q(t; x ) = θ, t T (x ). Infinite-horizon-optimal control For each initial condition x X, find an allocation z that minimizes the total cost, J(x ) = c(q(t; x ))dt. (39) We let J (x ) denote the optimal cost, i.e., the infimum over all policies when the initial condition is x. Pathwise-optimal control For each initial condition x X, find an allocation z such that, c(q(t; x )) = min { c(y) : y X {x + tv} }, for all t. (4) Under Assumption (A1), time-optimal and infinite-horizon-optimal control solutions exist for any initial condition, although the solution may not be unique. A path-wise optimal allocation minimizes c(q(t; x )) at every time instant. Consequently, it is simultaneously time-optimal and infinite-horizon-optimal. Note however that path-wise optimal allocations rarely exist in complex models. These control-criteria also extend to any workload relaxation. For the nth relaxation we may restrict attention to the workload process w by exploiting the exchangeability of states x, y X with a common workload value. The optimization criteria are then defined with respect to the workload space Ŵ Rn, and the cost function c: Ŵ R + is taken to be the effective cost. If the effective cost c(w) is monotone, then the infinite-horizon-optimal policy is work conserving, in the sense that dtŵ(t; d w ) = (1 ρ) when ŵ(t; w ) lies in the interior of Ŵ. When monotonicity fails, so that Ŵ+ is a proper subset of Ŵ, then from certain initial conditions one may have dtŵi(t; d w ) = + for some 1 i n [5]. For small values of n, an optimal policy can often be found through visual inspection (see [5, 15]). In particular, consider the case X = R l +. When n = 1, there is a path-wise optimal solution from any initial condition, regardless of the cost function. When n = 2, there is a path-wise optimal solution from each initial condition if and only if the two dimensional vector 1 ρ lies within the monotone region Ŵ+. An optimal solution satisfies ŵ (t; w ) Ŵ+ for all t >, and any initial condition w Ŵ. We now return to some of the examples to illustrate these definitions. 16

18 w 2 Infeasible Region W + c (w) < 1 w 1 Figure 8: An effective cost level curve for the 3-buffer push-model. Consider first the 3-buffer network shown in Figure 3 with c(x) = x 1 + x 2 + x 3. Applying (32), we find the effective cost c: Ŵ R + for the two-dimensional relaxation is given by, c(w) = max c i, w, (41) i=1,2 where c 1 = (, µ 2 ) T and c 2 = µ 3 (1, µ 1 1 µ 2) T. Contour plots for c are shown in Figure 8. The condition 1 ρ Ŵ+ may or may not be violated, depending on the relative service rates. When (1 ρ) Ŵ+, then a trade-off must be made between reducing the cost at time t = +, and draining the network efficiently. An infinite-horizon-optimal solution ŵ evolves for t > in a region R () satisfying Ŵ + R () Ŵ The boundaries of R () are interpreted as switching curves defining the optimal policy. For further details regarding the optimal policy see [5, 15]. We now consider a two-dimensional relaxation of the 16-buffer model. As shown in Figure 6, we again find that the effective cost is not monotone. The case where (1 ρ) Ŵ+ is illustrated in Figure 9, and an alternate situation is illustrated on the left hand side of Figure Policies based on time-optimality We are now prepared to introduce a class of policies that form the key building block for all of the policies considered below. Although the infinite-horizon-optimal policy for the fluid model is well motivated by the solidarity between optimal control solutions for fluid and stochastic models [48, 49, 5, 6, 15], in practice one may consider related policies with desirable properties that are more easily computed. As motivation, consider again the two-dimensional relaxation of the 16-buffer model. Suppose that the vector (1 ρ) lies below Ŵ+, as illustrated in the left hand side of Figure 1. Observe that the lower boundary of R () is defined by a linear switching curve s 1 (w 1) satisfying, bw 1 < s 1(w 1 ) < aw 1, w 1 >, (42) where b = (1 ρ 2 )/(1 ρ 1 ), and a is the slope of the lower boundary of the monotone region Ŵ+. Consequently, for initial conditions lying below the infinite-horizon-optimal switching curve with slope s 1, the infinite-horizon-optimal control is not time-optimal. 17

19 (1 ρ) 1 8 R w () 2 6 W + = w 1 Figure 9: The switching region R () = Ŵ+ defines the infinite-horizon-optimal policy in workload space for the model shown in Figure 1, only in the case where the vector (1 ρ) lies in Ŵ +. W + W w R () 8 6 R gto () Time-optimal switching curve 4 2 (1 ρ) (1 ρ) w 1 w 1 Figure 1: On the left is shown the switching region for the infinite-horizon-optimal policy, R (), and on the right is shown the switching region for the GTO policy, R gto (). In this numerical example, the vector (1 ρ) does not lie in Ŵ+, and it is seen that the region R () is strictly larger than the monotone region Ŵ+. 18

20 The greedy policy, considered in [14, 49], defines the allocation rate ζ(t) at time t so that d dt c( q(t; x )) = d dt c(ŵ(t; w )) is minimized. For a workload relaxation, for any initial w Ŵ, the workload trajectory satisfies ŵ(t; w ) Ŵ+, t >. In the example under consideration, for an initial condition lying below Ŵ+, the second resource is non-idling for t > : d dtŵ2(t; w ) = (1 ρ 2 ), < t < (1 ρ 2 ) 1 ŵ 2 (+; w ). One obtains a time-optimal allocation if the region R () is expanded to the set R gto () shown on the right hand side of Figure 1 since we then have (1 ρ) R gto (). This is one example of the greedy-time-optimal (GTO) policy described next. Recall the definition of the set of dynamic bottlenecks I(x) given below (11), the velocity set V given in (7), and the index set I c given in (38). GTO policy This is the state-feedback policy, defined for the nth relaxation as follows. Given the current state q(t; x ) = x at time t, the allocation rate ζ is defined to be any solution to the linear program, min η subject to c i, v η, if i I c (x), 1 i l c ξ i, v = (1 ρ i ), if i I(x), 1 i n v i, if x i =, 1 i l v i, if x i = b i, 1 i l Bζ + α = v ζ U (43) In words, the GTO policy minimizes the derivative d dtc(q(t)), subject to time-optimality, and state space constraints. The following properties of the GTO policy follow directly from Proposition 2.1 and the definitions. These conclusions hold for the queueing model with velocity space V, as well as any of its workload-relaxations. Proposition 3.2 Let q(t; x ) denote a trajectory defined by a GTO policy with initial condition x. Then, (a) The state trajectory q is time optimal. That is, q(t; x ) = θ, t T (x ). (b) If a unique path-wise optimal solution q exists starting from x, then q(t; x ) = q (t; x ) for all t. (c) c( q(t; x )) is convex and non-increasing in t. (d) I( q(t; x )) is non-decreasing in t. 19

21 q(t) q1 q2 q3 q4 q5 q6 q7 q8 q9 q1 q11 q12 q13 q t Figure 11: Buffer levels versus time for the 16-buffer model under the GTO policy. Figure 11 shows the buffer levels as a function of time for the 16-buffer model controlled using the GTO policy for the unrelaxed model (with V taken to be V). The initial condition in this simulation is, x = [ ] T. (44) The server rates are given in (25), and the cost function c is given in (35). Initially, the system behavior is dominated by the draining of internal buffers in the network. After a transient period, the deficit buffers drain slowly once all internal buffers have been emptied. This may be interpreted as an instance of state space collapse. In the second workload relaxation, one would observe this behavior instantaneously [5]. This fast initial draining is at the expense of starvation of bottleneck resources. This is explained through consideration of a two or three dimensional workload relaxation. Increasing some components of the workload vector will initially lower the effective cost c(w(t)) since the effective cost is not monotone. 4 Decision making in a dynamic environment We now develop extensions of the GTO policy to address a range of issues that arise in a dynamic environment. In particular, we find that the GTO policy may be adapted to provide effective policies in the following circumstances: (i) A transient demand is imposed, and is given priority over other products in the system. 2

22 (ii) Some component in the production network is temporarily in-operable, resulting in down-time for resources in the network. During repair it is still necessary to choose allocations at those resources in the network that are functioning. (iii) Some resources require preventative maintenance, so that some portion of the network is disabled for a period of time in the future. In this case, decisions regarding allocations prior to maintenance will be made subject to the knowledge of approaching down-time, and subsequent repair. (iv) Persistent, unpredictable disturbances, such as uncertainty in demand and yield. The impact of these disturbances can be reduced if the control synthesis problem is solved using all relevant information. This is especially true in the case of scheduled maintenance. Frequently, given prior knowledge of station down-time, the network can be positioned such that starvation of active resources is negligible during the maintenance period. We consider first control synthesis when a transient demand is imposed on the network. 4.1 Hot-lots In production parlance, a lot is a group of products in the system. A hot-lot is a lot (or group of lots) that is given high priority. It may be a prototype for a new product, in which case the (long-run) demand rate will be zero. We assume throughout that this is the case. Therefore, for the purpose of policy synthesis, a hot-lot is modeled in the initial condition of the network through the introduction of a corresponding virtual queue. Priorities may be hard or soft. We first consider a situation where the hot-lot has equal priority with other products. This may be desirable when the plant operator requires engineering samples during a production run. These samples may be used to evaluate the quality of some internal processes, and it may be desirable to maintain consistent priorities to minimize disturbances to the system. This is modeled as follows: Normal-priority hot-lot. The deficit buffer for the hot-lot is non-empty at time t = ; and the holding cost at this deficit buffer is commensurate with the holding cost at other deficit buffers in the network. The high-priority hot-lot is more common in practice. This is used to model a finite size customer order where, due to the premium paid by the customer, the system is realigned to meet the order quickly. High-priority hot-lot. The deficit buffer for the hot-lot is non-empty at time t = ; and the holding cost at this deficit buffer is far larger than the holding cost at other deficit buffers. The critical-priority hot-lot receives the highest priority possible. This is modeled as follows: Critical-priority hot-lot. The deficit buffer for the hot-lot is non-empty at time t = ; and the allocation is chosen to meet this demand in the shortest time possible. The GTO policy may be applied without modification in systems with normal-priority or highpriority hot-lots since in the definition (43) we do not require positive demand-rates. 21

23 Policy synthesis for a critical hot-lot is more complex. One must first obtain the minimal clearing time of the hot-lot. Based on this, one may compute the minimal draining time for the system, given that the critical hot-lot is cleared in minimal time. Each of these computations may be cast as a finite-dimensional linear program. The required modification in (43) is obvious and will not be presented here. To illustrate the impact of a hotlot on overall system performance we consider a version of the network model shown in Figure 1 in which the demand rate δ 2 is set to zero. However, the corresponding buffer q 16 is non-zero since we assume that there is a transient demand for this product. In the simulation below we take q 16 = 2, and the remaining initial buffer levels are given by, x = [ ] T. (45) We note that all of the buffers in the hot-lot path are initially empty. Consequently, to meet the hot-lot demand it is necessary to bring into the entrance buffers all of the required raw material. We first consider a simulation to establish a baseline for analysis. Figure 12 shows a plot of the system under examination with no hot-lots (q 14 () = q 16 () = ). This illustrates normal operation of the network when there is a single recurrent demand with rate δ 1, under the GTO policy. A simulation of the GTO policy when applied to the same system with a normal-priority hot-lot is shown in Figure 13. We see that in this case the hot-lot is not cleared until late into the time-horizon [, T (x )]. The emptying time T (x ) 16 in this simulation is approximately equal to the emptying time for the baseline system. Figure 14 shows a simulation of the network with high-priority hot-lots under the GTO policy in which the holding cost at buffer 16 is increased significantly. The initial condition (45) is the same as shown in the normal-priority case. The resulting state trajectory clears the hot-lot demand in approximately 6 time units as opposed to about 1 time units for the normal-priority hot-lot. Of course, because the GTO policy imposes a global time-optimality constraint, the system drains at time T (x ) 16, exactly as seen for the normal-priority hot-lot. Finally, we consider a critical-priority hot-lot. To determine the minimal time required to clear the hot-lot we examine the model shown in Figure 16, obtained by ignoring any buffers that are not on the path leading to the hot-lot deficit buffer. With the given initial condition, it is found that the minimal time required to clear the hot-lot deficit buffer is 4 time units. The minimal draining time for the network, subject to the constraint that the critical-priority deficit is cleared at time t = 4, is equal to approximately 325. This is approximately twice the clearing time seen for the normal or high-priority model. A simulation is shown in Figure Unanticipated breakdown Unscheduled down-time was ranked as the most significant cause of capacity loss in wafer fabs according to [6]. The unanticipated breakdown of resources presents a problem since, if a critical resource is lost without warning, large deficits can be generated while both upstream and downstream resources are forced into idleness. We introduce here an extension of the GTO policy for network regulation during a period in which some resource is unavailable. Our goal is to obtain allocations that minimize the impact 22

24 q(t) Buffer Level Time t Figure 12: Buffer levels versus time for normal transient operation under the GTO policy. 8 q(t) Hot-lot contribution Buffer Level Time t Figure 13: Buffer levels versus time for a normal-priority hot-lot of size 2. 23

25 q(t) 8 Hot-lot contribution buffer level Time t Figure 14: Plot of buffer levels for the network shown in Figure 1 with a high-priority hot-lot under the GTO policy. q(t) 8 Hot-lot contribution Buffer Level Time t Figure 15: Plot of buffer levels versus time under the GTO policy for the network shown in Figure 1 when a critical hot-lot is present. The system clearing-time is far longer than in previous simulations 24

26 λ q 5 1 q 1 δ 2 q 4 q 9 Station 1 Station 2 Figure 16: Subsystem serving the hot-lot. of these gross disturbances. We find in simulations that the proposed policy places the network in a position to return to its normal operating state quickly once the resource is again available. By normalization we may assume that the breakdown occurs at time t =, and take x as the state at this time. We assume that we are given the exact time to repair, denoted T MTTR. The acronym MTTR stands for mean time to repair, which reflects the fact that in practice only mean-values, and perhaps some higher order statistics will be available. Given that q(t; x ) = y at some time t < T MTTR, the minimum time required to empty the system is found using the following linear program: T (y ; T MTTR, t):= min T subject to y 1 = y + (B down ζ 1 + α)(t MTTR t) θ = y 1 + (Bζ 2 + α)(t T MTTR ) y 1 X ζ 1 U down ζ 2 U (46) where U down is the set of feasible control rates during the period [, T MTTR ) when a resource is disabled, and B down is the corresponding system matrix in (4). When t > T MTTF we may define T using (11). Given T (x; T MTTR, t) for all x X and t >, the greedy-time-optimal-with-breakdowns (GTO-B) policy is defined as follows: GTO-B policy Given t < T MTTF and q(t) = x, the allocation rate ζ(t) is an optimizer of the linear program: min η subject to c i, v η, if i I c (x), 1 i l c x T (x; T MTTR, t), v = 1 v i, if x i =, 1 i l v i, if x i = b i, 1 i l B down ζ + α = v ζ U down The GTO-B policy empties the system in minimal time due to the dynamic programming condition d dt T (q(t; x ); T MTTR, t) = 1 for all t T (x ; T MTTR, ). 25