Quadratic Approximation of Cost Functions in Lost Sales and Perishable Inventory Control Problems

Transcription

1 Quadratic Approximation of Cost Functions in Lost Sales and Perishable Inventory Control Problems Peng Sun, Kai Wang, Paul Zipkin Fuqua School of Business, Duke University, Durham, NC 27708, USA psun, kai.wang, We propose an approximation scheme for two important but difficult single-product inventory systems, specifically, the lost-sales system and the perishable product system. The approach is based on quadratic approximations of cost-to-go functions within the framework of Approximate Dynamic Programming via Linear Programming. We propose heuristics based on the approximation and evaluate their performance. Numerical results show promise in our approach. History : This version: March 22, Introduction 1.1. Overview Periodic-review inventory models with positive lead times can be hard to solve. The difficulty arises from the high number of state variables needed to represent the pipeline inventories of different ages. Dynamic programming for such a model therefore suffers from the so-called curse of dimensionality. An exception is the basic model with back orders. In this case, it is well-known that we can reduce the state to a single variable, the inventory position, and the optimal policy takes the simple form of base-stock structure. This lovely reduction does not work for many other systems, however, including systems with lost sales or a perishable product. This paper proposes an approximate solution framework for these two problems. In a system with lost sales, the state of the system is a vector of dimension equal to the lead time. The vector contains the on-hand inventory and also all the outstanding replenishment orders in the pipeline. The structure of the optimal policy has been partially characterized in Karlin and Scarf (1958), Morton (1969), and Zipkin (2008b). A numerical evaluation of existing plausible heuristics by Zipkin (2008a) has found room for improvement even for fairly short lead times. For example, one of the best performing policies, namely the myopic2 policy, is computationally expensive and its performance appears to deteriorate as the lead time increases. A perishable inventory system has a multidimensional state even with zero lead time. The state of the system contains information on the age distribution of the inventory. Most existing works on perishable inventory focus on this setting, including the analysis of the optimal policy structure and 1

2 2 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control development of heuristics. Even more challenging is a system with positive lead time. Even partial characterization of the optimal policy has been developed only recently by Chen et al. (2014b), and a heuristic policy that works well is yet to be proposed. These two inventory systems share an interesting structure: After a certain transformation of the state variables the, the cost-to-go functions are L -convex (Zipkin 2008b, Chen et al. 2014b). This property is a concept from discrete convex analysis, related to convexity, submodularity, and diagonal dominance. The property reveals structural characteristics of the optimal policy. In this paper, we propose an approximation scheme for these systems that exploits this structure. The overall framework of the approach is Approximate Dynamic Programming via Linear Programming (ADP-LP) (de Farias and Van Roy 2003b). This technique requires us to specify a convex cone of functions, and it approximates the cost-to-go function by an element of this cone. Here, we specify the cone so that all its functions are themselves L -convex. Thus, we aim to preserve this structural property of the true cost-to-go function. In particular, we focus on quadratic L -convex functions. There are several reasons for this restriction. First, as shown in Morton (1969) and Zipkin (2008b), the effective state space is a fairly small compact set, and casual examination of the exact cost-to-go function in several cases suggests that it is fairly smooth. Any smooth function is nearly quadratic over a small set. Second, this cone has a fairly small number of extreme rays, and so the resulting optimization model has a small number of variables. Third, the simplest policy obtained from such a function (called LP-greedy) is essentially a variant of a base-stock policy, where the inventory position is replaced by a (nearly) linear function of the state variables. This form is easy to understand and implement. For the lost-sales system, we propose several heuristic policies. A direct greedy policy (LPgreedy) derives the order quantity using the approximate cost-to-go function from ADP-LP as the continuation function. It is essentially a generalized base-stock policy, which depends on a twisted inventory position that is linear in all outstanding orders, plus a separable nonlinear function of the current inventory. There is also a second-degree greedy policy (LP-greedy-2), where we use the optimal cost of the greedy policy as the next period s cost-to-go. The linear-quadratic structure can be further exploited by applying value iterations to the quadratic approximate cost function obtained from the ADP-LP. This creates an entire class of order-up-to policies ranging from LPgreedy which is nearly linear in the state variables, to a policy we call T L that is nonlinear (but still fairly simple) in the state variables. We also have a one-step-further greedy policy, which we call T L+1, that searches for the best order quantity using the objective function from T L as the cost-to-go function. We then develop similar heuristics for the perishable inventory problem. Specifically, we have a class of policies ranging from LP-greedy to what we call T Λ, which can be also interpreted as

3 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 3 base-stock policies, using a twisted inventory position. We also have a perishable version of the T L+1 policy. We also introduce a myopic policy, which is a natural extension of the myopic heuristic introduced by Nahmias (1976a), but with lead times. To the best of our knowledge, these are the first heuristics proposed for the perishable inventory model with positive lead times. In the numerical study, we evaluate these heuristics against benchmark heuristics, namely, myopic policies. The results show that the heuristics perform better than the benchmarks in nearly all cases. The more complex heuristics are somewhat better than the simpler ones, but the differences are modest. For the lost-sales system we also test the modified myopic policy introduced recently in Brown and Smith (2014), which we call the B-S myopic policy. They find that this policy performs quite well in a few instances. We consider a larger range of problems. We observe that the policy performs as well or better than the other heuristics Literature Review The lost-sales model with positive lead time is first formulated by Karlin and Scarf (1958) and further explored by Morton (1969). The base-stock policy is found not optimal for such systems, and the optimal order quantity is partially characterized as decreasing in all pipeline inventories, with increasing but limited sensitivity from the most dated to the most recent order. Morton (1969) also derives bounds for the optimal policy and suggests such bounds could be used as a simple heuristic, later called the standard vector base-stock policy. Various other heuristics have been proposed. Morton (1971) studies a single-period myopic policy based on a modified accounting scheme, which is extended by Nahmias (1976b) to more general settings. Levi et al. (2008) introduce a dual-balancing policy that is guaranteed to yield costs no greater than twice the optimal. Asymptotic analysis has been done in both directions of lead time and penalty cost. The base-stock policies were found by Huh et al. (2009) to be asymptotically optimal as the penalty cost increases, while Goldberg et al. (2012) shows asymptotic optimality of constant-order policies proposed by Reiman (2004) for large lead times. Zipkin (2008a) evaluates the performance of various heuristics and compares them with the optimal policy for lead times up to four. He finds that base-stock policies perform poorly, and a two-period horizon version of Morton (1971) s myopic policy, namely myopic2, generally performs best among the heuristics studied. In a recent development, Brown and Smith (2014) propose a modified myopic policy with an adjustable parameter, the terminal cost. They demonstrate that, with a good choice of the terminal cost (found by simulation), this heuristic performs well. We contribute to the body of work on heuristics for the lost-sales model by introducing a class of well-performing policies that preserve the structural characterization of the optimal policy.

4 4 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control Most literature on perishable inventory control focuses on systems with zero lead time. Nahmias and Pierskalla (1973) studies a system with a single product with two periods of lifetime. Nahmias (1982) provides a review of early literature. More recent works are summarized in Nahmias (2011) and Karaesmen et al. (2011). Most recently, Chen et al. (2014b) partially characterize the optimal policies in a more general setting that allows positive lead times, either backlogging or lost-sales for unmet demand, and joint inventory and pricing decisions. As for heuristics, Nahmias (1976a) proposes a myopic base-stock policy for the zero lead-time case. It appears that the policies we propose here are the first for positive lead times. More recently, Chao et al. (2015) propose another heuristic, similar in spirit to the dual-balancing policy for the lost-sales model. The concept of L -convexity, developed by Murota (1998, 2003, 2005) in the area of discrete convex analysis, was first introduced to inventory management by Lu and Song (2005). Since then, Zipkin (2008b) reveals the L -convexity structure for lost-sales systems, which helps recover and extend earlier results by Karlin and Scarf (1958) and Morton (1969). Huh and Janakiraman (2010) extends the analysis to serial inventory systems. Pang et al. (2012) applies the approach to joint inventory-pricing problem for back order systems with leadtimes, and Chen et al. (2014b) further extend it to perishable inventories for both the backorders and lost-sales cases. For computation, Chen et al. (2014a) propose a scheme for finite-horizon problems with L -convex cost-to-go functions based on recursively applying a technique for extending an L -convex function to a multidimensional domain from a finite number of points. Our work is tailored for infinite-horizon problems. The approach is based on a parametric (specifically, quadratic) approximation of L - convex cost functions. We also derive intuitive and easy to implement heuristics based on the approximation. Also, we point out a simple but useful fact about L -convex functions under certain variable transformations. Our overall framework is Approximate Dynamic Programming using Linear Programming (ADP- LP). The method is studied in de Farias and Van Roy (2003b) for discounted costs and de Farias and Van Roy (2003a, 2006) for the average cost problem Organization and Notation The remainder of the paper is organized as follows. Section 2 lays out the model setup of the lost sales system with positive lead time. We then introduce the approximation scheme in Section 3. Section 4 presents heuristics. In Section 5, we extend the approach to the perishable inventory control problem. Numerical studies for both problems are presented in Section 6. Finally, we summarize and give concluding remarks in Section 7. Throughout the paper, for vectors a and b, we use a b and a b to represent the elementwise maximum and minimum of a and b, respectively. When a and b are scalars, they are simply a b = max(a, b) and a b = min(a, b). Also, a + = max(a, 0), and a = min(a, 0).

5 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 5 2. The Lost Sales Inventory Model 2.1. Formulation The model setup and notation closely follow Zipkin (2008b). Consider the standard, single-item inventory system in discrete time with lost sales. Denote L = order lead time. d t = demand in period t. z t = order at time t. x t = (x 0,t, x 1,t,, x L 1,t ) where x 0,t is the inventory level at time t and x 1,t = z t+1 L,, x L 1,t = z t 1 ρ t = x 0,t d t The demands d t are independent, identically distributed random variables. The state of the system is represented by the L-vector x t which includes the inventory on hand as well as orders of the past L 1 periods. Its dynamics follow x t+1 = ( [x 0,t d t + + x 1,t, x 2,t,, x L 1,t, z t ). For generic state variables stripped of time indices, we have x + = ( [x 0 d + + x 1, x 2,, x L 1, z ). We shall use such generic variables unless time indices are unavoidable. By default we treat such state vectors as column vectors in matrix operations. Next we introduce two different linear state transformations. First, we can represent the state by a vector s = (s 0,, s L 1 ), where s l = L 1 τ=l x τ, 0 l < L, are the partial sums of pipeline inventories. The dynamics of this new state vector are s + = ( [s 0 s 1 d + + s 1, s 2,, s L 1, 0 ) + ze, where e is the L-dimensional vector of 1s. Alternatively, the state can be represented by vector v = (v 0,, v L 1 ), where v l = l x τ, 0 l < L. The dynamics of v are v + = ( v 1,, v L 1, v L 1 + z ) (v 0 d)e, We shall mainly work with the v state, but refer to x and s at various points.

6 6 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control Let c = unit cost of procurement. h (+) = unit cost of holding inventory. h ( ) = unit cost penalty of lost sales. γ = discount factor. We assume that the procurement cost is paid only when the order arrives. There is no fixed order cost. Let q 0 (x 0 ) denote the expected holding-penalty cost at the end of the current period, starting with inventory x 0. Then, q 0 (x 0 ) = E [ h (+) ρ + + h ( ) ρ, recalling that ρ = x 0 d. Let f(v) be the optimal cost-to-go, as a function of the variables v. It satisfies the following Bellman equation: f(v) = min γ L cz + q 0 (v 0 ) + γe [ f(v + ) }. z 0 Let f(s) = f(s 0 s 1, s 0 s 2,, s 0 s L 1, s 0 ), (2.1) that is, the optimal cost with respect to the s variables. Definition 1 (L -Convexity). A function f : R L R is called L -convex if the function ψ(v, ζ) = f(v ζe) is submodular on R L+1. (This is one of several equivalent definitions. The same definition works for functions of integer variables.) Zipkin (2008b) shows that the function f(s) is L -convex. The following result implies that the function f(v) also enjoys this property. Proposition 1 (Preservation of L -Convexity by State Transformation). Let function f be any function of R L. Define f in terms of f, as in (2.1). Then, f is L -convex, if and only if f is. Our approximation procedure can be based on either the s state, or the v state. In order to be consistent with later developments, we focus on the v state. We shall also need the following quantities: ĥ (+) = h (+) γc, ˆq 0 (v 0 ) = cv 0 + E [ ĥ (+) ρ + + h ( ) ρ, ˆq(v, z) = γ L E [ˆq 0 (v 0,+L ). Thus, ˆq is the discounted expected purchase plus holding-penalty cost in period t + L (the first period where the current order affects such costs), as viewed from period t, assuming that the starting inventory in period t + L + 1 is valued at rate c.

7 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control Existing Heuristics Here we present some existing heuristics, namely, the standard vector base-stock (SVBS) policy, the myopic and myopic2 policies, and the Brown-Smith modified myopic policy (B-S myopic). These are either used as benchmarks later on or employed as part of our approximation scheme. SVBS is arguably the easiest to implement heuristic, and provides an upper bound for the optimal order quantity according to Morton (1969). Define s = (s, 0) = (s 0, s 1,, s L 1, 0), ϑ = (c + ĥ(+) )/(ĥ(+) + h ( ) ), and d [l,l = L k=l d +k for l = 0,, L. Now set s SVBS l = min s : P[d [l,l > s ϑ }, l = 0,, L. At state s, the order quantity is z(s) = [ min s SVBS l s l, l = 0,, L} +. The myopic policy selects the order quantity z to minimize the ˆq(v, z) above, and myopic2 is defined by z(v) = arg min ˆq(v, z) + γe [ˆq (v + ) }, z 0 where ˆq (v) = min z 0 ˆq(v, z) } is the minimal cost from myopic. Myopic and myopic2 are quite intuitive, and arguably the best performing heuristics studied in Zipkin (2008a). Although myopic2 always outperforms myopic, it is computationally more cumbersome and can be impractical for large L, when the set of states becomes large. Brown and Smith (2014) suggest a modified version of the myopic policy. The objective function of the original myopic policy is [ ˆq(v, z) = E cv 0,+L + ĥ(+) ρ + +L + h( ) ρ +L = E [cv 0,+L + h (+) ρ ++L + h( ) ρ +L + γ( cρ++l ). Brown and Smith (2014) propose adjusting the parameter c in the salvage value cρ + +L above. Their method conducts a line search to find the best such value, where each value is evaluated by direct simulation. In the few examples they present, the approach works quite well, though it is not clear why. 3. Approximation We now develop the Linear Programming approach to Approximate Dynamic Programming (ADP- LP), based on de Farias and Van Roy (2003b).

8 8 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 3.1. Linear Program for Approximate Dynamic Programming First we briefly describe the ADP-LP in a general framework. Consider a discrete-time infinite horizon dynamic program (DP) with state space V R L. For each state vector v V, an action vector a is chosen from the set of available actions A v R K, which incurs a single-stage cost of g(v, a). Let v + denote the next period s state, which follows the state transition v + = v + (v, a, ω), where ω is a random variable, whose distribution only depends on v and a. Letting γ denote the discount factor, the optimal discounted expected cost function f(v) satisfies the Bellman equation f(v) = min g(v, a) + γe [ f(v + ) }. a A v When the sets V and A v are finite, f(v) is the solution to the following linear program (LP): w v f(v) maximize f(v):v V} subject to v V [ f(v) g(v, a) + γe f ( v + (v, a, ω) ), v V, a A v. where w v } v V are some state-relevance weights. In this exact formulation, these weights can be any positive numbers. This LP has as many decision variables as the number of states in the original DP problem, and there is a constraint for each state-action pair. For a problem with a large state space, solving this exact LP is impractical. The idea of ADP-LP is to approximate f by a function in a relatively small set of functions. Specifically, consider a linearly parameterized class of functions f(v) = M m=1 r mf m (v). The f m are pre-defined basis functions, and the r m are their variable linear coefficients. This yields a Linear Program for Approximate Dynamic Programming (ADP-LP): M ( ) maximize r m w v f m (v) r m} M m=1 m=1 v V M [ M ( subject to r m f m (v) g(v, a) + γe r m f m v+ (v, a, ω) ) v V, a A v, m=1 m=1 which only has M decision variables. Some or all of the r m may also be required to be nonnegative. The number of constraints, however, remains large. Therefore, it is necessary to sample the constraints. Unlike the exact formulation, here the choice of state-relevance weights can affect the quality of the solution Quadratic Approximation for L -Convex Cost-to-Go functions There are three key components in the ADP-LP approach that we need to specify. First, we need a class of basis functions whose span is rich enough to mimic the shape of the cost-to-go function, but small enough that the number of decision variables is manageable. Second, the state-relevance weights need to be determined. Last, we need to decide how to sample the constraints, to keep the linear program tractable.

9 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control Basis functions For the first issue, we propose to approximate the cost-to-go function by an L -convex quadratic function. Such a function has the following form: Proposition 2 (Murota (2003)). A quadratic function f(v) is L -convex, if and only if L 1 f(v) = µ l v 2 l + l,k:l k L 1 µ lk (v l v k ) 2 + λ l v l + ν for some µ l } L 1 0, µ lk l k} 0, λ l } L 1 and ν, where µ lk = µ kl, l k. In matrix notation, define the vector λ and the matrix Q to be λ = (λ 0, λ 1,, λ L 1 ), and Q = l µ l E (l) + k,l:k l µ kl E (kl). Here, the matrix E (l) is an L L matrix with zero components except the lth diagonal component, which is 1, while E (kl) is a zero matrix except the kth and lth diagonal components taking value 1, and the k, lth and the l, kth components taking value 1. According to Proposition 2, a quadratic L -convex function can be expressed as v Qv + λ v + ν. Applying this approximation in the original DP yields the following ADP-LP formulation: maximize µ l } L 1 µ lk l k} λ l } L 1, ν subject to v V L 1 w v ( µ l v 2 l + L 1 µ l v 2 l + l,k:l<k γ L cz + ˆq 0 (y) + γe L 1 + l,k:l k L 1 µ lk (v l v k ) 2 + λ l v l + ν) L 1 µ lk (v l v k ) 2 + λ l v l + ν [ L 1 µ l v 2 l,+ + µ lk (v l,+ v k,+ ) 2 l,k:l<k λ l v l,+ + ν, v V, 0 z z SVBS (v) µ l 0, l = 0,, L 1 µ lk 0, l, k : l < k The number of decision variables is only (L + 1)(L + 2)/ State-relevance weights and constraint sampling Let u denote the optimal policy of the exact DP and π u the steady-state distribution of states under u. de Farias and Van Roy (2003b) suggest w v be as close to π u as possible for good approximations. Although π u is not available, we often can find a good heuristic policy û, and approximate its steady-state distribution πû. The hope is that the steady-state distributions πû and π u are close to each other. We propose to generate a sample of states V û by simulating a long trajectory of states under some easy to

10 10 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control implement and fairly well-performing heuristic policy û. We then use the sample state distribution ˆπû, an estimate of πû, as the state-relevance weights w v. In the lost-sales problem, for example, we can use an existing heuristic, such as SVBS or myopic. de Farias and Van Roy (2004) also discuss imposing only a sampled subset of the constraints, and conclude that a good sampling distribution should be close to π u as well. Thus a sample of states generated by a well-performing policy, call it V û, is a reasonable choice for the constraints. This is what we do. To summarize, the linear programming problem we ultimately solve is maximize µ l } L 1 µ lk l k} λ l } L 1, ν subject to v Vû L 1 w v ( µ l v 2 l + L 1 µ l v 2 l + γe [ L 1 l,k:l<k l,k:l k µ l v 2 l,+ + L 1 µ lk (v l v k ) 2 + λ l v l + ν) L 1 µ lk (v l v k ) 2 + λ l v l + ν l,k:l<k L 1 µ lk (v l,+ v k,+ ) 2 + λ l v l,+ + ν + γ L cz + ˆq 0 (y), v V û, 0 z z SVBS (v) µ l 0, l = 0,, L 1 µ lk 0, l, k : l < k (3.1) Let µ l } L 1, µ lk l k}, λ l } L 1, and ν be the solution to the linear program (3.1). We thus obtain the following quadratic approximation of the cost-to-go function f(v), represented in a more compact matrix form, f ADP (v) = v Qv + λ v + ν, where the vector λ and the matrix Q are defined as λ = ( λ 0, λ 1,, λ L 1 ), Q = µ l E (l) + µ kl E (kl). l k,l:k l 4. Heuristics for the Lost Sales System In this section, we develop several heuristics for the lost-sales model based on the approximation f ADP obtained as above. Since the SVBS policy is easy to compute and bounds the optimal order quantity from above, we use it as an upper bound for all of the following policies.

11 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control LP-Greedy Policy The approximate cost-to-go function f ADP immediately yields the following heuristic, in which the order quantity is determined by z LPg (v) = arg min γ L cz + q 0 (v 0 ) + γe [ f ADP (v + ) } 0 z z SVBS We call it the LP-greedy policy. Due to the linearity of both the single period cost and the state transition function in the decision variable z, the policy has a closed-form expression. For notational convenience, for a vector v, we denote v m:n to represent subvector (v m, v m+1,, v n ). Similarly, for a matrix Q, notation Q m:n,s:t represents its submatrix with rows m, m + 1,..., n and columns s, s + 1,..., t. We further define vector κ = (κ 0,, κ L 1 ) and scalar ς such that κ 0 = κ 1 = µ L 1 µ, κ l = µ L l 2 µ l,l 1, l = 2,, L 1, and ς = λ L γ L 1 c µ 2 µ, (4.1) in which the denominator L 2 µ = µ L µ l,l 1. (4.2) Proposition 3. The order quantity following the LP-greedy policy is z LPg = 0 ẑ LPg z SVBS (4.3) in which ẑ LPg (x) = ς ( κ 0 E [ [x 0 d + + κ 1:L 1x 1:L 1 ). Note that the order quantity ẑ LPg, without the upper and lower bounds, is almost linear in the state variables. More specifically, let ṽ = κ 0 E [ [x 0 d + + κ 1:L 1x 1:L 1, which is a κ 1:L 1 -weighted sum of vector ( E [ [x 0 d + + x 1, x 2,, x L 1 ). This vector, in turn, is the expected next period s pipeline inventories without the current period s order quantity. If we consider this ṽ a twisted inventory position, the LP-greedy policy can be interpreted as a generalized base-stock policy where the constant term ς serves as the order-up-to level. Since 0 κ 0 = κ 1 κ L 1 1 and 0 de [ [x 0 d + /dx 0 1, we have 1 dẑlpg dx L 1 dẑlpg dx 0 0, which echoes the monotone sensitivity result of Karlin and Scarf (1958), Morton (1969) and Zipkin (2008b).

12 12 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 4.2. LP-Greedy-2 Policy Following the LP-greedy policy, we define Given that z LPg f LPg (v) =γ L cz LPg (v) + q 0 (v 0 ) + γe[ f ADP ( v + (v, z LPg, d) ). has a closed-form expression, it is easy to compute f LPg (v). This suggests an extension of the LP-greedy policy by solving the following single dimensional optimization problem. min γ L cz + q 0 (v 0 ) + γe [ f LPg (v + ) }. 0 z z SVBS We call the solution to this problem LP-greedy-2 policy, since it essentially solves a two-period DP with f ADP as the terminal cost T L policy The quadratic form of f ADP can be further exploited if we temporarily put aside the non-negativity constraint and the upper bound z SVBS for the order quantity z, and recursively define functions f (l) for l = 0,, L 1 as f (0) (v) = f ADP (v), f (l) (v) = min γ L cz + q 0 (v 0 ) + γe [ f (l 1) (v + ) }, l = 1,..., L 1. z These functions are all quadratic in z. As a result, we can obtain an expression for z iteratively by computing functions f (1) to f (L 1). Formally, we have the following result. Proposition 4. Functions f (0)... f (L 1), as recursively defined in (4.4), can be expressed in the following manner: f (l) (v) = l [ E E[v 0:L (l+1),+l v +τ Q (l) τ E[v 0:L (l+1),+l v +τ l 1 + λ (l) E[v 0:L (l+1),+l + γ τ E[ˆq 0 (v 0,+τ ) + ν (l), where (L l) (L l) matrices Q (l) τ, 1 (L l) vectors λ (l), and scalar ν (l) are recursively defined as: Q (0) 0 = Q, λ (0) = λ, ν (0) = ν, and l = 0,..., L 2, and τ = 1,, l + 1, ( J L l 1 ( l Q(l) )( l τ,(1:l l,l l) Q(l) )J ) τ,(l l,1:l l) L l 1 Q (l+1) 0 = γ Q (l+1) τ = γj L l 1Q (l) τ 1J L l 1, λ (l+1) = γ (λ (l) (λ(l) + (L l) γl 1 c)( l ( ν (l+1) = γ ν (l) l Q(l) τ,(l l,l l) Q(l) ) τ,(l l,1:l l) l Q(l) τ,(l l,l l) ( λ (l) (L l) + γl 1 c ) 2 4 l Q(l) τ,(l l,l l) ), ) J L l 1,, (4.4) (4.5) (4.6)

13 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 13 in which matrix J l is an (l + 1) l matrix as follows, 1 1 J l =... 1, 1 and scalars ν (l) are constants that do not depend on the state v. Furthermore, function f (L 1) takes the following form: L 1 f (L 1) (v) = Q (L 1) τ E [E[v 0,+L 1 v +τ 2 Expression (4.6) provides a simple procedure to obtain L 2 + λ (L 1) E[v 0,+L 1 + γ τ E[q 0 (v 0,+τ ) + ν (L 1). Q (L 1) τ and λ (L 1) that are needed to compute f L 1. It is worth noting that this procedure is essentially the process of solving an (L 1)- period linear-quadratic control system. Now we employ f (L 1) as the approximate cost-to-go function, and solve minimize γ L cz + q 0 (v 0 ) + γe [ f (L 1) (v + ). 0 z z SVBS We call this the T L policy, where T refers to a dynamic programming operator as defined in the usual sense. Our procedure corresponds to applying the T operator L times to f ADP, with the first L 1 times unconstrained and the last one subject to constraint 0 z z SVBS. The order quantity, denoted by z T L, is the optimizer of the last iteration of operator T. Similar to LP-greedy, the T L policy is also quite simple. Note that z T L = 0 ẑ T L z SVBS, where ẑ T L (v) = arg min γ L cz + γ =ς T L E [ ρ + +L 1, ( L τ=1 [ Q (L 1) τ E E [ ρ + +L 1 + z v 2 +τ + λ (L 1) E [ ρ + +L 1 + z)} in which ς T L = (λ (L 1) + γ L 1 c)/(2 L τ=1 Q(L 1) τ ), and E [ ρ + +L 1 = E [ [ [x0 d x L 1 d +L 1 +. Note E [ ρ + +L 1 is the expected inventory level after a lead time of L periods right before the order quantity z arrives, and can be computed backward iteratively for l = L 1,, 1: E[ρ + +L 1 x 0,+L 1 = E [ (x 0,+L 1 d +L 1 ) + x 0,+L 1, [ E[ρ + +L 1 x 0,+l 1, x l,, x L 1 = E E [ ρ + +L 1 [x 0,+l 1 d +l x l, x l+1,, x L 1. Therefore this heuristic can be interpreted as another type of generalized base-stock policy that places an order to raise the expected inventory level after lead time, E[x 0,+L, to ς T L possible. whenever

14 14 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control The LP-greedy and T L heuristics form an interesting contrast. Both can be interpreted as orderup-to policies with respect to some twisted inventory positions. However, the sensitivity of the LP-greedy order quantity to the inventory state is governed by L 1 coefficients κ l } L 1 l=1 (note that κ 0 = κ 1 ), but the sensitivity of the T L order quantity is endogenously determined by the expression E [ ρ + +L T L+1 Policy The quadratic structure of the approximate cost function can be further exploited one step beyond the T L policy. Since the objective function of T L is still quadratic in z, we further define where f (L) (v) = min = z γ L cz + q 0 (v 0 ) + γe [ f (L 1) (v + ) } L [ Q (L) τ E E [ ρ + +L 1 v 2 +τ γ L ce [ L 1 ρ + +L 1 + γ τ E[q 0 (x 0,+τ ) + ν (L), Q (L) 0 = γ L τ=1 Q (L 1) τ, Q (L) τ = γq (L 1) τ 1 τ = 1,, L, and ( ( λ (L 1) + γ L 1 c ) 2 ) ν (L) =γ ν (L 1) 4. L 1 Q(L 1) τ We define the T L+1 policy by order quantity Note that E[q 0 (x 0,+L ), E [ ρ + +L z T L+1 = arg min γ L cz + q 0 (v 0 ) + γe [ f (L) (v + ) }. 0 z z SVBS and E [ ρ + +L v +τ 2 in f (L) (v + ) can all be evaluated through recursion in a similar fashion as E[ρ + +L 1 x 0,+L 1. The only difference is that they are now functions of z. Also we no longer have a closed-form expression for z T L+1 the solution. The computation cost is of the same magnitude as myopic. 5. The Perishable Product Model 5.1. Formulation and need to conduct a line search for Consider a perishable product inventory system. It takes Λ periods for an order to arrive. Upon arrival the product has M periods of lifetime, after which it must be discarded. The product is issued to satisfy demand according to a First In, First, Out (FIFO) policy. Unmet demand is backlogged. The state of this system is described by an L = Λ + M 1 dimensional vector x = (x 0,, x L 1 ). Here, x M 1+l for 1 l Λ is the order that will arrive in l periods. The x l for 0 l M 1 represent information related the inventories that have already arrived on hand

15 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 15 and also the current backorders. These are nonnegative, except for x 0, which may be negative. If x 0 0, then there are no backorders, and x l is the inventory with l periods of lifetime remaining. If x 0 0, then the actual backorders are [ M 1 x l, and the inventory with k periods or less of lifetime remaining is [ k x l +. This state definition makes the dynamics relatively simple: ( ) x + = x 1 [d x 0 +, x 2,, x L 1, z. Similar to the lost sales system, the state can be alternatively represented by the vector of partial inventory positions v, with v l = l x τ for 0 l < L 1. Let y denote the order-up-to level v L 1 + z, the dynamics of state v are v + = ( v 1,, v L 1, y ) (v 0 d)e. The disposal cost for expired inventory is θ per unit. The unit procurement cost and inventory holding cost are again denoted by c and h (+), respectively, and h ( ) now represents the unit backorder cost. Again, there is no fixed order cost. Let χ 0 (v) =E [ h (+) [v M 1 v 0 d + + h ( ) [v M 1 d + θ[v 0 d + =E [ h (+) [v M 1 d + + h ( ) [v M 1 d + (θ h (+) )[v 0 d +, be the expected inventory and disposal cost. The optimal cost-to-go function f(v) satisfies: f(v) = min γ Λ c(y v L 1 ) + χ 0 (v) + γe [ f(v + ) }. y v L 1 Chen et al. (2014b) show that f(v) is L -convex. Even with zero lead time (Λ = 0), the state space includes inventory levels of different remaining lifetimes, and therefore is multi-dimensional. A system with positive lead time is still more challenging Benchmark Heuristics As Karaesmen et al. (2011) point out, there has not been much work on heuristics for perishable inventory systems with lead time. Here we propose a myopic policy inspired by Nahmias (1976a) for zero lead time problems. This serves two purposes. First, this is a benchmark to compare to our ADP-LP based heuristics. Second, as discussed earlier, our approximation approach needs some easy-to-implement policy to generate relevance weights and a sample of states for the linear program. Define ˆχ(v, y) =γ Λ E [ h (+) [v M 1,+Λ d +Λ + + h ( ) [v M 1,+Λ d +Λ + γ L ˆθE [ [v0,+l d +L +.

16 16 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control This is the discounted expected purchase, holding-backorder and disposal costs, in the first periods, respectively, where the current order affects these quantities. A simple recursion can be used to compute the distributions of v M 1,+Λ and v 0,+L. The myopic policy sets the order-up-to level to 5.3. ADP-LP Based Heuristics y myopic (v) = arg min y v L 1 (1 γ)γ Λ cy + ˆχ(v, y) }. Now we present our ADP-LP based heuristics, many of which share similar ideas with the heuristics for the lost sales problem LP-greedy policy The LP-greedy policy for the perishable inventory model is very similar to one for the lost-sales model. Define vector ξ = (ξ 0,, ξ L 1 ) and scalar ς, such that ξ 0 = µ L 1 µ, ξ l = 2 µ l 1,L 1, for l = 1,, L 1, and ς = ( γ L M c + λ ) L 1, µ 2 µ in which µ is defined in (4.2). Based on the approximation result f ADP (v) = v Qv + λ v + ν, the LP-greedy policy sets the order-up-to levels to y LPg (v) = max v L 1, ŷ LPg}, where ŷ LPg (v) = arg min γ Λ c(y v L 1 ) + χ 0 (v) + γe [ v + Qv + + λ v + + ν } = arg min γ Q L,L y 2 + 2γ( Q L,1:L 1 v 1:L 1 e Q1:L,L E[v 0 d ) y + ( γ Λ c + γ λ L 1 ) y } =ξ 0 E[v 0 d + ξ 1:L 1v 1:L 1 + ς. Note that ξ 0 and L 1 ξ l = 1. Thus the order-up-to quantity under the LP-greedy policy is a weighted sum of (E[v 0 d, v 1,, v L 1 ) plus a constant. Again we look at the policy from the perspective of the order quantity, z, as a function of the original states x. Using κ = (κ 0,, κ L 1 ) as defined in (4.1), we have z LPg (x) = max 0, ẑ LPg (x) }, ẑ LPg (x) =ς ( κ 0 ( x0 E[x 0 d ) + κ 1:L 1x 1:L 1 ). Similar to the lost-sales model, we have 0 κ 0 κ 1 κ L 1 1 and 0 de[x 0 d/dx 0 1. Therefore, 1 dẑlpg dx L 1 dẑlpg dx 0 0, which is consistent with to the monotone sensitivity property of the optimal order quantity identified in Chen et al. (2014b).

17 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control T Λ policy With the quadratic approximation, we can apply the same operations used to compute the T L heuristic for the lost-sales system. In this problem, we can not apply the unconstrained dynamic programming operator in a linear-quadratic fashion for L times, but only for Λ = L M + 1 times. We take the solution of the Λth iteration and subject it to the additional nonnegativity constraint, and call the result the T Λ policy. Recursively define functions f (l) for l = 0,, L M as f (0) (v) = f ADP (v), f (l) (v) = min γ Λ cz + χ 0 (v) + γe [ f (l 1) (v + ) }, l = 1,..., L M. z The T Λ order quantity for perishable inventory system is defined as z T Λ = arg min γ Λ cz + χ 0 (v) + γe [ f L M (v + ) }. z 0 (5.1) Similar to Proposition 4, the order quantity has a closed-form expression. Proposition 5. The order quantity following the T Λ policy is [ z T Λ = ς T Λ ( ) κ T Λ 1 E[x 0,+Λ + κ T 2:M 1x Λ + Λ+1:L 1, for some appropriately defined scalar ς T Λ and vector κ T Λ 1:M 1 when M 2. When M = 1, z T Λ = [ ς T Λ + E[(x 0,+Λ 1 d) +. The general form shares some similarity to the LP-greedy order quantity. Note that the term κ T Λ 1 E[x 0,+Λ in z T Λ is similar to ( x 0 E[x 0 d ) in z LPg. The difference is, in T Λ, the first Λ elements of the state vector are combined in κ T Λ 1 E[x 0,+Λ in a nonlinear fashion. Heuristic T Λ can still be interpreted as an order-up-to policy, with base-stock level ς T Λ and an inventory position that is non-linear in the first Λ elements and linear in the last M 1 elements of the x state vector. In the case of M = 1, it is a special base-stock policy. The order quantity is the base-stock level ς T Λ plus a nonnegative term E[(x 0,+Λ 1 d), which is essentially the forecast of backlog Λ 1 periods later. Since there is only one period of life time, any excess inventory is outdated at the end of the period and does not contribute to future inventory levels. Therefore, the order quantity does not depend on the forecast of (x 0,+Λ 1 d) T Λ+1 policy Similar to the T L+1 policy for the lost-sales model, we have the following T Λ+1 policy for the perishable product problem. It does not have a closed-form expression and is solved by a one-dimensional search. The order quantity is z T Λ+1 = arg min z 0 γ Λ cz + χ 0 (v) + γe [ f Λ (v + ) }. We provide some further derivations for this policy in the Appendix.

18 18 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 5.4. Observations Before we present computational results, it is worth summarizing some general insights on our ADP-LP based heuristics for both inventory systems. The LP-greedy and T L /T Λ policies can be perceived as two extremes of a more general class of policies. For the lost sales system, the general rule takes the following form: [ z = ς ( κ 0 E[ρ + +l 1 + ) + κ 1:L lx l:l 1. for some positive base-stock level ς, and coefficients 0 κ 0 κ 1 κ L l 1 with l = 1,, L. A twisted inventory position x = κ 0 E[ρ + +l 1 + κ 1:L lx l:l 1 consists of E[ρ + +l 1, the forecast of inventory holding l 1 periods later, which is a nonlinear function of x 0:l 1, and a linear combination of x l:l 1. In this framework, LP-greedy corresponds to the case l = 1. The T L policy, on the other hand, corresponds to the case l = L, with κ 0 = 1. Similarly, for the perishable inventory system, the policy is: [ z = ς ( κ 0 E[ ρ +l 1 + ) + κ 1:L lx l:l 1. for l = 1,, Λ. LP-greedy corresponds to the case l = 1 and T Λ corresponds to the case l = Λ. The term E[ρ +l 1 is the forecast of backlog on the immediately outdating inventory l 1 periods later that needs to be subtracted from the linear combination of the rest of the pipeline inventory. This view of our heuristic policies may suggest good policies structures for other systems with L -convex structures. We leave such explorations to future research. 6. Computational Study Now we present computational results for the lost sales and perishable product problems Lost Sales Problem We take model parameters from Zipkin (2008a) by letting c = 0, h (+) = 1, and h ( ) take values from 4, 9, 19, 39}, and using Poisson and geometric distributions both with mean at 5 for demand. Zipkin (2008a) studied lead times from 1 to 4. To test how well our approach handles long lead times, we set L equal to 4, 6, 8, and 10. We use the myopic policy to generate a long trajectory of states for the state-relevance weights and the set of constraints. In particular, we simulate the system for 6000 periods under this policy, discarding the first 1000 periods to mitigate the initial transient effect. The remaining 5000 time periods states are used as our sample state set V û for the ADP-LP. (We also tried the SVBS and myopic2 policies as generating policies. We found that myopic produces better results than SVBS, and about the same as myopic2, which is far harder to compute.)

19 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 19 Then we solve the ADP-LP. Using the solution, we determine each ADP-LP-based heuristic policy (LP-greedy, LP-greedy-2, T L, and T L+1. We evaluate each policy by simulating the system for periods. Finally, we evaluate the myopic, myopic2, and B-S myopic policies, again by simulation. (These evaluation simulations are of course independent of those used to set up the ADP-LP.) Tables 1 and 2 present the average costs of the ADP-LP-based policies and the benchmark policies. We mark numbers bold to highlight cases where an ADP-LP-based policy outperforms the myopic2 policy. Table 1 Average Cost with Poisson Demand (Lost-Sales) L h ( ) LPg LPg2 T L T L+1 M1 M2 B-S Table 2 Average Cost with Geometric Demand (Lost-Sales) L h ( ) LPg LPg2 T L T L+1 M1 M2 B-S We observe that all the ADP-LP heuristics give reasonably good results, compared to the benchmarks. The T L+1 policy consistently outperforms myopic2. The LP-greedy policy has good and

20 20 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control stable performance. Although it does not yield the best results when L = 4, it does manage to outperform myopic2 in the majority of cases as the lead time increases. The performance of T L, however, can be poor for high penalty cost. (Wang (2014) contains a diagnosis of this erratic behavior.) It is worth noting that there are more bold case numbers in Table 2 than in Table 1, indicating that more variations of our ADP-LP-based heuristics outperform myopic2 when the demand distribution is geometric, than when it is Poisson. Now, Zipkin (2008a) observed that in almost all cases studied in that paper, the existing heuristics generally perform closer to optimality for Poisson demand than for geometric demand. Our result, on the other hand, suggests that our heuristics have better potential for cases that challenge traditional heuristics more. (We are not sure why this is so.) Observe also that the B-S myopic policy performs best among all policies in most cases. It remains a mystery why this should be so. (Recall that this method requires a line search for the best terminal value, with each step requiring a simulation.) The approximation scheme relies on linear programs generated from randomly simulated trajectories of states. It is therefore important to check if the performance comparisons in the study are robust. For each parameter setting, we repeat the same procedure to generate the ADP-LP and evaluate the heuristics 100 times, with 100 independently sampled demand trajectories. We pair up policies for comparison and take the difference in average costs for each demand trajectory, and check the t-statistic of such paired performance differences from the 100 repetitions. Table 3 presents the paired t-statistics computed for one-tailed tests on the performance differences between the ADP-LP-based heuristics and myopic-2. The t-statistic tables show that most of the corresponding performance orderings in Table 1 and Table 2 are statistically significant. In fact, 156 out of 160 instances are at least 90% significant (critical value of t-statistics at 1.29), among which 150 instances are at least 99.95% significant (critical value of t-statistics at 3.4). It is also worth investigating the stability of the ADP-LP-based policies, given uncertainties in the generated trajectories upon which linear programs are based. First, among the 10 out of 160 instances in Table 3 that are less than 99.95% significant, 5 correspond to L = 10 and 4 correspond to h ( ) = 39. Secondly, the performance of T L becomes significantly inferior to other policies for larger h ( ) s. In some cases with both L and h ( ) being high, average costs can be extraordinarily high. Generally speaking, higher L and h ( ) imply a larger state space. Fixing the sample size, we expect higher variation in the generated trajectories for constructing constraints and staterelevance weights for the linear program. Indeed, among the 100 repetitions for each parameter setting, we observe such a trend. Using sample trajectories of 5000 periods to generate the ADP-LPs for all instances, the standard deviation of average costs from any ADP-LP-based policy

21 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control 21 Table 3 Performance t-statistics for Lost-Sales Heuristics Poisson Geometric L h ( ) LPg LPg2 T L T L+1 LPg LPg2 T L T L almost always increases in L or h ( ). Furthermore, the standard deviation of the average costs from T L increases much faster in h ( ) than any other policies. In Table 4, we compare the LP-greedy and T L policies for some selected cases. In (L = 10, h ( ) = 39), the average costs of T L become highly unstable, compared with LPg. That is, when h ( ) is high, the same results and variation from linear programs yield much higher variation in the average costs of T L than LP-greedy or other policies. If we increase the length of LP-generating sample trajectories from 5000 to , we are able to shrink the standard deviation of T L from to except for two outliers. The clear conclusion is that more samples are needed to generate the LP for higher L and h ( ) values, especially for the T L heuristic. Table 4 Standard Deviation of Long-Run Average Costs, LP-greedy vs. T L, Poisson Demand L h ( ) LPg T L There are asymptotic results for the lost-sales system in both the lead time L and the lostsales penalty h ( ). On one hand, Goldberg et al. (2012) establish that a constant-order policy is asymptotically optimal as the lead time grows large. On the other hand, Huh et al. (2009) show that, as the lost-sales penalty increases, an order-up-to policy is asymptotically optimal. Although

22 22 Sun, Wang, and Zipkin: Approximation of Cost Functions in Inventory Control these asymptotic results do not necessarily imply that the optimal policy converges to a constantorder policy or an order-up-to policy when the corresponding parameter grows large, it is still of interest to check whether well-performing heuristic policies exhibit such tendencies. Under the LP-greedy policy, for example, such tendencies, if they exist, can be observed in the linear coefficients κ of the LP-greedy order quantity. Since the order quantity is almost affine in the state vector x, if all κ l } L 1 approach 0, the LP-greedy policy has a tendency towards a constant-order quantity; if all κ l } L 1 approach 1, on the other hand, the LP-greedy policy tends to behave like an order-up-to policy. Since the increases in L and h ( ) have competing impacts on the behavior of the order quantities, we choose (L = 4, h ( ) = 4) as the base case and increase one parameter L or h ( ) while keeping the other fixed. Tables 5 and 6 report the coefficients with increasing L and h ( ), respectively. To ease the comparison, we also present the average magnitude of LP-greedy coefficients, κ = ( L 1 κ l)/l, for each case. From the tables, we do observe decrease of κ l } L 1 and κ in L and increase of κ l} L 1 and κ in h( ). Table 5 Lost-Sales LP-greedy Coefficients with Increasing L Poisson Demand, h ( ) = 4 L κ κ 0 κ 1 κ 2 κ 3 κ 4 κ 5 κ 6 κ 7 κ 8 κ Geometric Demand, h ( ) = 4 L κ κ 0 κ 1 κ 2 κ 3 κ 4 κ 5 κ 6 κ 7 κ 8 κ Table 6 Lost-Sales LP-greedy Coefficients with Increasing h ( ) Poisson Demand, L = 4 h ( ) κ κ 0 κ 1 κ 2 κ Geometric Demand, L = 4 h ( ) κ κ 0 κ 1 κ 2 κ