Serial Inventory Systems with Markov-Modulated Demand: Derivative Bounds, Asymptotic Analysis, and Insights

Transcription

1 Serial Inventory Systems with Markov-Modulated Demand: Derivative Bounds, Asymptotic Analysis, and Insights Li Chen Samuel Curtis Johnson Graduate School of Management, Cornell University, Ithaca, NY Jing-Sheng Song Fuqua School of Business, Duke University, Durham, NC Yue Zhang Fuqua School of Business, Duke University, Durham, NC In this paper we consider the inventory control problem for serial supply chains with continuous, Markovmodulated demand MMD). Our goal is to simplify the computational complexity by resorting to certain approximation techniques, and, in doing so, to gain a deeper understanding of the problem. To this end, we analyze the problem in several new ways. We first perform a derivative analysis of the problem s optimality equations, and develop general, analytical solution bounds for the optimal policy. Based on the bound results, we derive a simple procedure for computing near-optimal heuristic solutions for the problem. These simple solutions reveal a closer relationship with the primitive model parameters. Second, we perform asymptotic analysis with long replenishment lead time and establish an MMD central limit theorem. We further show that the relative errors between our heuristics and the optimal solutions converge to zero as the lead time becomes sufficiently long, with the rate of convergence being the square root of the lead time. Our numerical results reveal that our heuristic solutions can achieve near-optimal performance even under relatively short lead times. Third, we show that, by leveraging the Laplace transformation, the optimal policy becomes computationally tractable under the gamma distribution family. This enables us to numerically compare various heuristic solutions with the optimal solution, and to demonstrate that our heuristic outperforms existing heuristics in most cases. Finally, we observe that the internal fill rate and demand variability propagation in an optimally controlled supply chain under MMD exhibit behaviors different from those under stationary demand. Key words : multi-echelon inventory systems, Markov-modulated demand, derivative, asymptotic analysis History : file version October 8, Introduction The increasingly open global economy has made it possible for companies to seek the best available resources and technologies worldwide and to serve new markets. As a result, many supply chains have been stretched long and thin, often consisting of multiple production and distribution stages across different countries and continents. This new supply chain configuration brings new challenges 1

2 2 to managers. For example, demand shocks, which could be political, economic, technological, or climatic, in one region can quickly propagate to other regions, causing shortages in some places but oversupplies in others. Thus, companies that traditionally operate their supply chains in a relatively stable environment, such as within one country, must now learn how to effectively rationalize inventory along the global supply chain to hedge against greater environmental uncertainties. To help managers to meet this new challenge, we consider in this paper a serial supply chain model that involves demand uncertainties driven by dynamic environmental factors. Specifically, we assume that customer demand in each period is influenced by a world state that evolves according to a discrete-time Markov chain. This demand process is known as the Markov-modulated demand MMD) process in the literature Iglehart and Karlin 1962 and Song and Zipkin 1993). The MMD process is useful and practical for its structured data requirement and modeling flexibility. To construct the Markov chain, one can first identify demand scenarios of different phases in a product development process or in a product life cycle, and then assess the likelihood of each scenario as well as the transition probabilities between the scenarios, much as in the construction of a decision tree Song and Zipkin 1996b). Chen and Song 2001) showed that a world-state-dependent echelon base-stock policy is optimal for serial inventory systems with MMD. Computing the optimal policy, however, is a nontrivial task. Chen and Song 2001) developed an iterative optimization algorithm, which requires solving a set of integral renewal equations in each iteration. Their algorithm is effective for the discrete demand case as the integral renewal equations reduce to linear equations), but remains computationally challenging for the continuous demand case. Muharremoglu and Tsitsiklis 2008) showed that the state-dependent echelon base-stock policy continues to be optimal for more general systems with exogenous and sequential) stochastic lead times and discrete demand processes including MMD). They decomposed the problem into a series of single-unit single-demand problems, and developed an efficient dynamic programming algorithm for computing the optimal policy. Due to the nature of their single-unit decomposition approach, their algorithm cannot be applied to the continuous demand case. In addition, while representing a significant improvement over the standard dynamic programming algorithms, these exact algorithms work somewhat like a black box numbers in and numbers out, lacking the transparency between the model parameters inputs) and the optimal inventory decisions outputs). In this paper we focus on the continuous demand case. Our goal is two-fold. First, we seek to develop new approximation techniques to simplify the computational complexity. Such solution techniques enable managers to make swift decisions in choosing appropriate inventory levels along the supply chain to hedge against environmental fluctuations. Second, we strive to make the black box more transparent by establishing a direct relationship between the primitive model parameters

3 3 and the simple solutions. Such relationship sheds light on both the qualitative and quantitative effects of the environmental uncertainties. It also helps guide managers to invest wisely to obtain the right information or problem parameters) and to react quickly in the right direction. To achieve the above goal, we analyze the problem in the following new ways. Derivative Bounds and Heuristics Our first approach is to perform a derivative analysis of the problem s optimality equations and develop derivative bounds. The solution bounds obtained from our derivative bounds can be computed without solving the integral renewal equations required in the algorithm of Chen and Song 2001). The derivative bound expressions also reveal a closer relationship with the primitive model parameters, which was not apparent in the algorithms of Chen and Song 2001) and Muharremoglu and Tsitsiklis 2008). From the derivative analysis, we obtain general, analytical solution bounds for serial inventory systems with MMD. The existing bounding approaches for problems with nonstationary demand all build on a simplifying assumption that requires the optimal state-dependent echelon base-stock level to be achievable in each period e.g., Dong and Lee 2003 and Shang 2012). This assumption, however, does not hold in general, and the lower bound obtained from these approaches may fail to bound the optimal solution under MMD. On the contrary, our derivative-based approach does not require such an assumption, and thus our solution bounds work in general. In addition, our solution bounds can be viewed as a generalization of those obtained by Shang and Song 2003) for the independent and identically-distributed i.i.d.) demand process. When the state space of the Markov chain degenerates to a singleton, the demand process becomes i.i.d. and our bounds reduce to theirs, whereas their bounding approach does not extend to the MMD case see Appendix D for a detailed discussion). Computing our derivative-based solution lower bound involves optimization over demand state permutations, which can be challenging if there is a large number of states. To simplify computation, we develop easy-to-compute heuristic solutions that only require evaluating a linear combination of derivatives of the newsvendor cost functions of each demand state. To our knowledge, this derivative-based heuristic is the first of its kind in approximating the optimal policy for both singlestage and serial inventory systems with MMD. Moreover, with this heuristic, we can establish a mapping between the underlying Markov chain transition probabilities and the linear weight of each demand state, making the solution process more transparent. Asymptotic Analysis with Long Lead Time Our second approach is to perform an asymptotic analysis of the problem with long lead time. In doing so, we first establish an MMD central limit theorem for the lead time demand i.e., demand

4 4 aggregated over the lead time periods). We find that, as the lead time becomes sufficiently long, the lead time demands with different initial states all converge to the same normal distribution. To prove this result, we use the concept of α-mixing to show that demands far apart in time under MMD are almost independent Billingsley 1995). This finding reveals an inherent averaging effect of aggregation under MMD: While the demand distributions of different states in a single period may be drastically different, the aggregated lead time) demand would become more resembling to that of an i.i.d. process as the aggregation period increases. Leveraging our derivative analysis and an MMD central limit theorem, we obtain asymptotic results for the solution bounds under long replenishment lead time. Specifically, we show that the relative errors between our solution upper and lower bounds converge to zero as the lead time increases, with the convergence rate being the square root of the lead time. Accordingly, heuristic solutions that fall between our solution bounds, such as our derivative-based heuristic and its simplified variants, are guaranteed to converge to the optimal solution at a minimum speed of square root of lead time. This observation suggests that, when the lead time is sufficiently long, the seemingly complex supply chain problem with MMD has surprisingly simple solutions. In proving the above result, we make a new methodological contribution to the literature one can apply our analysis approach to establish similar results for other inventory systems. Exact Evaluation and Observations While solving integral renewal equations is generally challenging for continuous demand, we discover an exception. When demand belongs to the gamma distribution family, we find that the exact algorithm of Chen and Song 2001) becomes tractable with the aid of Laplace transformation and its inverse see Appendix C for details). This finding enables us to numerically compare our derivative-based heuristic solutions with the optimal policy under relatively short lead times, complementing the asymptotic results established under long lead times. From our numerical studies, we observe that our heuristic achieves near-optimal performance in most cases. We further compare our derivative-based heuristic with the decoupling heuristic proposed by Abhyankar and Graves 2001). Their heuristic, designed for a two-stage system with MMD, is based on the following simplifying assumption. The upstream stage provides 100% internal fill rate, i.e., it can always fulfill orders from the downstream stage. As a result, the two-stage system is decoupled into two single-stage systems, with the downstream stage operating under a state-dependent installation base-stock policy, and the upstream stage operating under a static installation base-stock policy. Our numerical results show that our derivative-based heuristic outperforms their decoupling heuristic in most cases, with an average performance improvement of 4.6%. Besides the performance difference, we note that our derivative-based heuristic can be applied to systems with any number of stages, whereas their decoupling heuristic only works for two-stage systems.

5 5 It has been demonstrated in the literature that, in a serial inventory system with i.i.d. demand, an optimal internal fill rate is usually much lower than the fill rate at the customer-facing stage see Choi et al. 2004, Shang and Song 2006, and references therein). For systems with MMD, a natural question is whether the internal fill rate would continue to be low, or become much higher as assumed in the decoupling heuristic of Abhyankar and Graves 2001). To obtain insights into this question, we conduct numerical experiments to measure the optimal) internal fill rate under MMD. We find that, contrary to the observations documented for the i.i.d. demand case, when the lead time at the upstream stage is long, the internal fill rate can be high under MMD around 94-96%; the target fill rate at the customer-facing stage is around 94%), suggesting that the decoupling heuristic may yield a good approximation to the optimal policy in this case. On the other hand, when the lead time at the upstream stage is relatively short, the internal fill rate tends to be low around 84-93%, lower than the 94% target fill rate at the customer-facing stage). As a result, the decoupling heuristic yields a poor approximation; and our derivative-based heuristic has a clear advantage in this case, as it does not require the high internal fill rate assumption. In addition, we investigate numerically the demand variability propagation effect, or the bullwhip effect see Chen and Lee 2009, 2012 and the references therein). Interestingly, we observe that the bullwhip effect can be significantly dampened under the optimal policy, indicating that the statedependent inventory policy under MMD may smooth demand variability propagation in the supply chain see Appendix E for details). This contrasts the existing observations under autoregressive AR1) demand in the literature, suggesting interesting future research directions. Finally, we note that our derivative bounds are derived based on the optimality equations of the exact algorithm. Chen and Song 2001) showed that the exact algorithm can be extended to systems with a fixed setup cost at the upmost stage as well as the assembly systems. Thus, all our results can be extended accordingly to those more general systems. The rest of this paper is organized as follows. We provide a brief literature review in 2. We then analyze a single-stage inventory problem in 3. This lays the foundation for the analysis of serial inventory systems in 4. 5 presents the numerical studies, and 6 concludes the paper. All proofs are presented in Appendix A; and all other supplementary results are included in Appendices B-E. 2. Literature Review The serial system structure we consider in this paper is the classic multi-echelon inventory model, first developed and analyzed by Clark and Scarf 1960), and further studied by many researchers in various dimensions see Zipkin 2000 for a review). It serves as an important baseline model and a key building block for more complex supply chain structures. Much of the literature on this classic model assumes an i.i.d. demand process. Under this assumption, it has been shown

6 6 that a stationary echelon base-stock policy is optimal. Earlier work focused on the computational efficiency of the optimal policy; e.g., see Federgruen and Zipkin 1984) and Chen and Zheng 1994). More recently, researchers have developed simple bounds and heuristics that aim to increase the transparency of the factors that determine the optimal policy. See, for example, Gallego and Zipkin 1999), Dong and Lee 2003), Shang and Song 2003), Gallego and Ozer 2005), and Chao and Zhou 2007). Our work extends their efforts to systems with MMD. The MMD process, which extends the i.i.d. demand process to incorporate dynamic state evolution, was first introduced by Iglehart and Karlin 1962) to study a single-stage inventory model, but only gained popularity in the inventory literature in the last few decades; see, for example, Song and Zipkin 1993), Ozeciki and Parlar 1993), Beyer and Sethi 1997), and Sethi and Cheng 1997). An important special case of MMD is the cyclic demand model, which various authors explored; see, for example, Karlin 1960), Zipkin 1989), Aviv and Federgruen 1997), and Kapuscinsky and Tayur 1998). Several authors also adopted MMD in serial inventory systems, but focused on different aspects of the problem than we do. For example, Song and Zipkin 1992, 1996a, 2009) and Abhyankar and Graves 2001) analyzed specific types of policies, Parker and Kapukinski 2004) characterized optimal policy for a two-echelon capacitated system, Huh and Janakiraman 2008) used a sample-path approach to establish the optimality of the echelon base-stock policies, Angelus 2011) considered the complexity of incorporating secondary market sales, and Abouee-Mehrizi et al. 2014) conducted an exact analysis of capacitated two-echelon inventory systems with priorities. The most closely related works to ours are Chen and Song 2001) and Muharremoglu and Tsitsiklis 2008). These authors showed that, for a serial inventory system, a state-dependent echelon basestock policy is optimal. They also presented exact algorithms to compute the optimal policy. Using a decomposition approach similar to that in Muharremoglu and Tsitsiklis 2008), Janakiraman and Muckstadt 2009) developed efficient algorithms for computing the optimal policies for capacitated and lost-sales serial systems. We do not consider these system features in our paper. There has also been research in the literature on deriving solution bounds for serial inventory systems with nonstationary demand processes. For example, Dong and Lee 2003) developed lower bounds for optimal policies for serial systems with time-correlated demand. The time-series demand model requires past demand information, whereas the MMD model we consider here focuses on external factors. Shang 2012) extended the bounding approach of Shang and Song 2003) to derive solution bounds for serial inventory systems with independent but nonidentical demands which is a special case of MMD). As discussed earlier, the bounding approaches employed in these papers all build on a simplifying assumption that the optimal state-dependent echelon base-stock level is achievable in each period. This assumption does not hold in general; as a result, the lower bound obtained from these approaches may fail to bound the optimal solution under MMD. Our paper complements this literature by deriving solution bounds that work in general under MMD.

7 3. Single-Stage Inventory Systems For expositional purposes, we consider a single-stage inventory problem in this section. This allows us to clearly introduce the key ideas to be used later in the analysis for serial inventory systems. Specifically, the demand of a product is met with on-hand inventory in each period; when there is stockout, we assume the unmet demand is fully backlogged. Inventory is replenished from an external source with ample supply. The replenishment lead time is a constant of L periods. At the end of each period, unit inventory holding cost h and unit backlog penalty cost b are charged on the on-hand inventory and backorders, respectively. The planning horizon is infinite, and the objective is to minimize the long-run average cost of the inventory system. Because the linear ordering cost under the long-run average cost is a constant, we can assume the ordering cost is zero without loss of generality see Federgruen and Zipkin 1984). The demand process is nonstationary and has an embedded Markov chain structure i.e., the MMD process). Specifically, demand in each period is a nonnegative random variable denoted by D k, where k is the demand state that determines the continuous demand distribution with density function f k ) and the cumulative density function F k ). The demand state in period t follows a Markov chain W = {W t), t = 1, 2,...}, with a total of K states, denoted by {1, 2,..., K}. The Markov chain is time-homogeneous. Let P = p ij ) denote the transition probability matrix, where p ij is the one-step transition probability from state i to j. Without loss of generality, we assume the Markov chain is irreducible, which implies that all states communicate with each other. When the Markov chain is reducible, under the long-run average cost criterion, the problem is equivalent to one involving only the irreducible class of the chain.) Also let D k [t, t ] denote the total demand in periods t,..., t with the demand state being k in period t, and let D k [t, t ) denote the total demand in periods t,..., t 1. In each period, the sequence of events is as follows: 1) at the beginning of a period, the demand state k is observed; 2) an inventory replenishment order is placed with the supplier; 3) a shipment ordered L periods ago is received from the supplier; 4) demand arrives during the period; and 5) at the end of the period, inventory holding and backorder costs are assessed. 3.1 Preliminaries It has been shown that a state-dependent base-stock policy {s k), k = 1,..., K} is optimal for the problem see Iglehart and Karlin 1962, Beyer and Sethi 1997, Chen and Song 2001). The policy works as follows: When the demand state is k, if the inventory position is below the base-stock level s k), order up to this level; otherwise, do not order. Given the demand state k and inventory position y, the single-period expected cost is Gk, y) = h E { y D k [t, t + L]) +} + b E { D k [t, t + L] y) +}, 1) 7

8 8 where x) + = max{x, 0}. The above function is the well-known newsvendor cost function, which is convex in y. Its minimizer, denoted by sk) = arg min y Gk, y), is termed the myopic base-stock level. Specifically, sk) solves the following first-order condition: y Gk, y) = b + b + h)f k,l y) = 0, or F k,l y) = b b + h, 2) where we use y to denote the first-order derivative with respect to y, and F k,l ) is the cumulative distribution of the lead time demand D k [t, t + L]. We shall also refer to the function y Gk, y) as the newsvendor derivative function throughout the paper. For later usage, we define the smallest myopic base-stock level among all states and its corresponding demand state as s min = min 1 k K { sk)}, kmin = arg min 1 k K { sk)}. Although the myopic base-stock level sk) is easy to compute, it is not necessarily optimal. The demand state in the next period may result in a stochastically) much smaller demand distribution; therefore, stocking up to the myopic base-stock level in the current period may cause overstocking in the next period. More sophisticated methods are needed to determine the optimal policy. Exact Algorithm. Chen and Song 2001) developed the following K-iteration algorithm to compute the optimal policy. The algorithm starts with a partition of the state space of W. In each iteration i, let V i denote the set that contains the states for which we have not found the optimal solution yet, and U i the complementary set of V i. Specifically, in the first iteration i = 1, let V 1 = {1,..., K} and U 1 = φ. Define G 1 k, y) = Gk, y) as given in 1), solve the single-period problem, and let s 1 k) = sk) denote the solution for state k. Find the demand state that gives the smallest solution among V 1. Denote such a state as 1 = arg min k V 1 s 1 k) = arg min 1 k K { sk)}. Then, the optimal solution for state 1 is just s 1 ) = s 1 1 ) = s min. Next, update the state sets by V 2 = V 1 \{1 } and U 2 = U 1 {1 }, and proceed to iteration i = 2. In iteration i+1 1 i < K), for each k V i+1, solve the cost minimization problem min y {G i+1 k, y)}, where G i+1 k, y) = G i k, y) + p ku E{R i u, y D k )}, 3) u U i+1 and for any u U i+1, p uu E{R i u, y D u )} if u U i, R i u U i+1 u, y) = G i i, y) + 4) p u E{R i u, y D i )} if u = i, i u U i+1

9 with G i i, y) = G i i, max{y, s i i )}). It can be shown that G i+1 k, y) is convex in y. Let s i+1 k) = arg min y G i+1 k, y). Also let i + 1) = arg min k V i+1 s i+1 k), i.e., the demand state that gives the smallest s i+1 k) among V i+1. Then the optimal solution for state i + 1) is given by s i + 1) ) = s i+1 i + 1) ). Update the state sets by V i+2 = V i+1 \{i + 1) } and U i+2 = U i+1 {i + 1) }. Repeat the above procedure until reaching the final iteration K. In summary, the above algorithm finds the optimal base-stock level for each demand state by sorting the demand states based on the solutions to the cost minimization problems min y {G i+1 k, y)} in each iteration. Although the algorithm is simpler and more efficient than the standard dynamic programming algorithms, it remains quite complicated because each iteration builds on the results obtained in the previous iterations. The main difficulty lies in determining the function R i at each iteration, as given in 4). For the discrete demand case, computing R i 9 is relatively easy, as it involves solving a set of linear equations see Chen and Song 2001). For the continuous demand case, however, determining R i requires solving a set of integral renewal equations. To illustrate the last point more clearly and to facilitate our subsequent analysis, we introduce the following matrix notation. Let m = m 1,..., m K ) denote a permutation of the sequence of demand states {1,..., K}. Define the following sub-matrices of transition probabilities under m: for i = 1,..., K 1, P i) m = p m1 m 1. p mi m 1... p m1 m i )... p mi m i Let m = 1,..., K ) denote the optimal demand state sequence determined by the exact algorithm. Let D i) m x) be the following diagonal matrix involving demand densities for states 1,..., i : f 1 x)... 0 D i) m x) = ) 0... f i x) Also let R i y) = [R i 1, y), R i 2, y),..., R i i, y)] T. Then, we can rewrite 4) in a matrix form as follows: R i y) = where e i is the unit vector with the i-th element being one. 0 D i) m x) Pi) m Ri y x)dx + e i G i i, y), 7) Our Approach. Based on the exact algorithm, it is useful to let [k] denote the iteration in which the optimal solution s k) for state k is obtained. In other words, { s k) = arg min G [k] k, y) }. y 0 Because G [k] k, y) is convex in y, its derivative is increasing in y. If we can find bounding functions for its derivative, then the roots of the bounding functions become bounds for the optimal solution.

10 10 Thus, our approach is to develop bounding functions that involve only the primitive model parameters, so that the resulting solution bounds will depend only on the primitive model parameters. To this end, by repeatedly applying 3) and 4) and then taking derivatives, we observe that [k] 1 y G [k] k, y) = y Gk, y) + u U i+1 p ku y E{R i u, y D k )}. 8) The first term is simple, given by 2), so what remains is to find simple functions to bound y E{R i u, y D k )}. Note that all functions in the exact algorithm are continuously differentiable and all the expectations are finite. Therefore, we can interchange the derivative and expectation or integral) operators in our subsequent derivations, e.g., y E{R i u, y D k )} = E{ y R i u, y D k )}. 3.2 Derivative and Solution Bounds We now derive bounds for y G [k] k, y). Observe that y G i i, y) = y G i i, y)) +. Also, from equation 4), it is straightforward to show that y R i, y) = 0 for y s i i ). Using a standard result in renewal equations Sgibnev 2001), we obtain Lemma 1 For any u U i+1, R i u, y) is increasing convex in y. Moreover, for any k = 1,..., K, G i k, y) is convex in y and G i i, y) is increasing convex in y. According to the above Lemma, the second term of 8) is nonnegative, so we immediately have y G [k] k, y) y Gk, y) = b + b + h)f k,l y). Thus, y Gk, y) serves as a simple lower bound function for y G [k] k, y). We now proceed to derive an upper bound for y G [k] k, y). Because W is irreducible, it follows that I i) P i) m is invertible for any i = 1,..., K 1, where I i) is the i i identity matrix and P i) m is defined in 5). Thus, we can define the following parameters: for i = 1,..., K 1, I i 1) P i 1) m β i m) = I i) P m i), where is the matrix determinant and we adopt the convention that I 0) P 0) m = 1. Taking the derivative with respect to y on both sides of the equation 7), we have y R i y) = = 0 y 0 D i) m x) Pi) m yr i y x)dx + e i y G i i, y) D i) m x) Pi) m yr i y x)dx + e i y G i i, y) P i) m yr i y) + e i y G i i, y),

11 where the inequality follows from the fact that y R i, y) is increasing in y and the probability distribution is less than one. Therefore, I i) P i) m ) y R i y) e i y G i i, y). 9) Let e be the i-dimensional vector with all elements being one. With some additional argument, we obtain the following upper bound on y R i y): 11 Lemma 2 For i = 1,..., K 1, ) 1 y R i y) I i) P i) m ei y G i i, y) e β i m ) y G i i, y). Applying both Lemmas 1 and 2 to the second term of 8) yields [k] 1 u U i+1 p ku y E{R i u, y D k )} [k] 1 p ku β i m ) y G i i, y) u U i+1 K 1 1 p kk ) β i m ) y G i i, y), where the second inequality relaxes the range of the second summation to make it independent of the set U i. We can further show that see the detailed derivation in Appendix A) K 1 K 1 β i m ) y G i i, y) α i m ) y Gi, y)) + y), where α i m) = K 1 j=i+1 α j m) φ j,i m) + β i m), i = 1,..., K 1 10) with φ j,i m) = [p mj m 1, p mj m 2,..., p mj m i ] T I i) P i) m ) 1 ei, and K 1 y) = max α i m) y Gm i, y)) +, 11) m S with S being the set of all permutations of {1,..., K}. Thus, we have obtained an upper bound function for y G [k] k, y) based on a linear combination of simple newsvendor derivative functions. Note that both β i m) and α i m) depend only on the primitive model parameters the transition matrix P. The following proposition summarizes the above bound results: Proposition 1 For any state k = 1,..., K, the following holds: y Gk, y) y G [k] k, y) y Gk, y) + 1 p kk ) y).

12 12 When K = 1, y) 0, so the inequalities in the above proposition becomes binding and the result reduces to the derivative of the single-period cost function. This is intuitive because when K = 1, the demand process reduces to an i.i.d. process. With the derivative lower bound shown above, it follows that the myopic base-stock level sk) is an upper bound for the optimal solution s k). This result is implied by the exact algorithm of Chen and Song 2001). Symmetrically, from the derivative upper bound shown above, we can obtain a lower bound for the optimal solution s k). Specifically, let sk) be the solution to the following equation: y Gk, y) + 1 p kk ) y) = b + b + h)f k,l y) + 1 p kk ) y) = 0. 12) We obtain the following result: Corollary 1 For any demand state k = 1,..., K, the following holds: s min sk) s k) sk). where sk) and sk) are determined by 2) and 12), respectively. The inequalities become equalities when k = k min. While the bounds s min and sk) have appeared in the literature see Song and Zipkin 1993), the lower bound sk) is new to the literature. This new lower bound is tighter than the existing one, s min. Note that computing the new lower bound involves optimization over demand state permutations, which can be challenging if there is a large number of states. To further simplify computation, we develop easy-to-compute heuristic solutions in the next subsection. 3.3 Heuristics The insights gained from the derivation of the derivative bounds lead us next to construct heuristics that fall between the solution lower and upper bounds. To do so, we augment the derivative lower bound by a positive component that is less than 1 p kk ) y). Specifically, we first sort the myopic base-stock level sk) in an increasing order and let m = m 1,..., m K ) denote the resulting order sequence of the states. Suppose that k = m j. Based on Lemma 2, we can make the following approximation for the second term of y G [k] k, y) in 8): [k] 1 j 1 p ku y E{R i u, y D k )} p k mi y E{R i m i, y D k )} u U i+1 j 1 j 1 p k mi β i m) E { y G m i, y D k )) +} 1 p k mi β i m) F k y s m i )) y G m i, y)) +. 2

13 Here, in the first step we approximate m by m and replace U i+1 by { m i }. The second step follows from Lemma 2 with m replacing m. The last step is based on the following integral approximation: { E y G m i, y D k )) +} = y s mi 0 y G m i, y x)) + f k x)dx 1 2 F ky s m i )) y G m i, y)) +. Now let s a k) denote a heuristic solution determined by the following equation: y Gk, y) + 1 j 1 p k mi β i m) F k y s m i )) y G m i, y)) + = 0. 13) 2 Note that equation 13) involves only a linear combination of the newsvendor derivative functions of different demand states. Thus, solving s a k) only requires a direct evaluation of the lead time demand distributions of different demand states. With this heuristic, we also establish a direct relationship between the underlying Markov chain transition probabilities and the linear weights in equation 13), making the solution process more transparent. Moreover, we have Corollary 2 For any demand state k = 1,..., K, s min sk) s a k) sk). The inequalities become equalities when k = k min. The above Corollary shows that the heuristic solution obtained from 13) indeed falls between the solution lower and upper bounds. A detailed illustration of our bound and heuristic results is provided in Appendix B for a two-state example i.e., K = 2). In the example, we show that, when it is highly likely to transit from a high demand state to a low demand state, one should reduce the optimal inventory level for the high demand state, and our heuristic solution moves in the same direction as the optimal solution. In addition, our heuristic solution provides a closer approximation to the optimal solution when there is a high chance of transiting from a low demand state to a high demand state see Appendix B for details). Combining Corollaries 1 and 2, we observe that s k) and s a k) fall in the same interval [sk), sk)]. Therefore, if the interval is tight, the heuristic solution s a k) becomes a close approximation to the optimal solution s k). In the next subsection, we identify sufficient conditions to ensure that such an interval is tight. 3.4 Asymptotic Analysis with Long Lead Time Intuitively, when the replenishment lead time L increases, the lead time demands starting from different states may converge to the same distribution. As a result, the gap between s min and sk) will narrow. Below we formally prove this result by first establishing an MMD central limit theorem for the lead time demand. First, consider the case in which W has a stationary distribution π = [π1),..., πk)]. Let D π t) be the demand in period t under the stationary distribution, that is, D π t) follows the distribution 13

14 14 of D k with probability πk). Let µ k = E{D k } and σ 2 k = var{d k }. The mean and variance of D π t) are given by K K µ = E{D π t)} = µ k πk), var{d π t)} = σ 2 k + µ k) 2 πk), 14) where µ k = µ k µ is the relative difference between µ k and µ. k=1 Let D π [t, t + L] denote the lead time demand under the stationary distribution π, we have E{D π [t, t + L]} = L + 1)µ, cov{d π t), D π t + l)} = where p s) kl K k=1 K k=1 l=1 µ k µ l p s) kl πk), l = 1, 2,..., 15) is the s-step transition probability from state k to state l. We can further show see the proof of Lemma 3 in Appendix A) σ 2 var {D π [t, t + L]} = lim = var{d π t)} + 2 L L + 1 cov{d π t), D π t + l)}. 16) Thus, the variance limit σ 2 contains two parts: the single-period demand variance and the covariance across different periods. It is interesting to note that the covariance structure under MMD depends only on the demand mean of each state. Thus, besides the demand variance at each state, another major source of variability under MMD comes from the variation of the demand means. Let N 0, 1) denote the standard normal random variable with distribution function Φ ). We can establish the following result for the lead time demand distribution under MMD: l=1 Lemma 3 MMD Central Limit Theorem) Suppose that the Markov chain W has a stationary distribution and D k has finite moments for any demand state k. If σ 0, then as L, D k [t, t + L] L + 1)µ σ L + 1 where µ and σ are defined in 14) and 16), respectively. dist. N 0, 1), The above Lemma confirms the intuition that the lead time demands with different initial states converge to the same distribution as the lead time increases. To prove this result, we use the concept of α-mixing to show that demands far apart in time under MMD are almost independent Billingsley 1995). This finding reveals an inherent averaging effect of aggregation under MMD: While the demand distributions of different states in a single period may be drastically different, the aggregated lead time) demand would become more resembling to that of an i.i.d. process as the aggregation period increases. Based on the result, when L is sufficiently long, the lead time demand D k [t, t+l] can be approximated by a normal distribution with mean L + 1)µ and variance L + 1)σ 2, independent of the

15 15 initial demand state k. Consequently, the myopic base-stock level for state k can be approximated by the following formula: sk) L + 1)µ + z σ L + 1, k = 1,..., K, 17) where z = Φ 1 b/b + h)). In other words, all myopic base-stock levels sk) converge to the same value, so does s min. Next, consider a cyclic Markov chain W. In this case W does not have a stationary distribution. However, due to its cyclic nature, as the lead time increases, the lead time demands starting from different states share a growing common component. As a result, we can show that, for any two states k and k, the lead time demand distributions F k,l y) and F k,ly) converge to the same limit distribution for every y as L goes to infinity. Thus, it continues to hold that all myopic basestock levels sk) converge to the same value in this case. The following proposition summarizes our discussion above and further determines the convergence rate: Proposition 2 Suppose that the Markov chain W either has a stationary distribution or is cyclic, and that D k has finite moments for any demand state k. For any demand state k = 1,..., K, sk) s min lim L + 1 = 0. L s min The above proposition shows that the relative percentage error between sk) and s min converges to zero as L goes to infinity. The speed of convergence is at a rate of o 1/ L + 1 ). Thus, when the lead time is sufficiently long, we can use the closed-form expression 17) or the more sophisticated s a k) Corollary 2) to approximate s k). In proving the above convergence result, we leverage the fact that s min = min k {1,...,K} sk), which is the minimum of the myopic base-stock levels of different states. Since the myopic base-stock level of a given state depends only on the lead time demand distribution associated with that state, we can then apply the MMD central limit theorem to obtain the convergence rate result for our solution bounds. 4. Serial Inventory Systems In this section we present our main results for the general serial inventory system with N > 1 stages. Specifically, random customer demand arises in every period at Stage 1, Stage 1 orders from Stage 2,..., and Stage N orders from an external supplier with ample supply. We also call the external supplier Stage N + 1. If there is stockout at Stage 1, we assume the unmet demand is fully backlogged. The production-transportation lead time from Stage n + 1 to Stage n is a constant of L n periods. Let L n = n j=1 L j denote the total lead time from Stage n + 1 to Stage 1. At the end of each period, a unit installation) inventory holding cost H n is charged for on-hand

16 16 inventories at Stage n 1 n N) and a unit backlog penalty cost b is charged for backlogs at Stage 1. Following the convention in the literature, we define the echelon inventory holding cost at Stage n as h n = H n H n+1 > 0, with H N+1 = 0. The demand process at Stage 1 follows the MMD process as described in 3. We assume that all the replenishment activities in a period happen at the beginning of the period after observing the demand state. At Stage n n > 1), the sequence of events is as follows: an order from Stage n 1 is received, an order is placed with Stage n + 1, a shipment is received from Stage n + 1, and then a shipment is sent to downstream Stage n 1. For Stage 1, an order is placed at the beginning of the period, a shipment is received from Stage 2, and then the demand arrives during the period. The planning horizon is infinite and the objective is to minimize the long-run average cost of the serial inventory system. As in the single-stage problem, we assume the ordering cost is zero without loss of generality. In what follows, we shall assign a subscript n to the corresponding functions and variables for Stage n. 4.1 Preliminaries Let IL n t) denote the echelon inventory level at Stage n by the end of period t, which includes the total on-hand inventory at Stages 1,..., n, plus the total inventory in transit to Stages 1,..., n 1, minus the backorders at Stage 1. Also let Bt) denote the backorder level at Stage 1 by the end of period t. Then, the system inventory holding and penalty cost at the end of period t can be written as N N H n [IL n t) IL n 1 t)] + H 1 [IL 1 t) + Bt)] + bbt) = h n IL n t) + b + H 1 )Bt), n=2 n=1 where the right-hand side is a sum of a sequence of echelon-related costs from Echelon 1 to N. Consider first the Echelon 1 cost. Under the long-run average cost criterion, we can charge the cost of period t + L 1 to period t without affecting the cost assessment. Specifically, given the demand state k and inventory position y at the beginning of period t, the expected cost at the end of period t + L 1 is given by E{h 1 IL 1 t + L 1 ) + b + H 1 )Bt + L 1 )} = h 1 E { y D k [t, t + L 1 ]) +} + b + H 2 ) E { D k [t, t + L 1 ] y) +}. Thus, we can define the single-period expected cost function at Echelon 1 as G 1 1k, y) = h 1 E { y D k [t, t + L 1 ]) +} + b + H 2 ) E { D k [t, t + L 1 ] y) +}. 18) Recall that when N = 1, h 1 = H 1 and H 2 = 0. Therefore, in the single-stage system, G 1 1k, y) reduces to Gk, y) as defined in 1). Chen and Song 2001) showed that the echelon decomposition technique

17 17 of Clark and Scarf 1960) for the i.i.d. demand case can be extended to the MMD process. That is, the optimal state-dependent echelon base-stock policy can be determined sequentially from Echelons 1 to N, with the aid of an induced penalty function between the successive echelons. Exact Algorithm. Chen and Song 2001) developed the following algorithm for computing the optimal policy. Start with Stage 1. Apply the K-iteration algorithm described in 3.1 to G 1 1k, y) defined in 18), and obtain the optimal echelon base-stock levels s 1k) for Stage 1. For Stage n 2, first define the following induced penalty functions from stage n 1 to stage n: for k = 1,..., K, G n 1,n k, y) = G [k] n 1k, min{y, s n 1k)}) G [k] n 1k, s n 1k)), 19) where G [k] n 1k, ) is the function obtained from the K-iteration algorithm for Stage n 1 see 3.1). With the induced penalty functions, define the following cost functions for Stage n: for k = 1,..., K, G 1 nk, y) = E{h n y D k [t, t + L n ]) + G n 1,n W k t + L n ), y D k [t, t + L n ))}, 20) where W k t+l n ) is the demand state at the beginning of period t+l n given the state k in period t. Now apply the K-iteration algorithm again to G 1 nk, y), and obtain the optimal echelon base-stock levels s nk) for Stage n. Repeat the above procedure until reaching the final Stage N. Thus, the computation procedure for the N-stage system requires N K iterations. Besides the difficulty in determining the function R i n at each iteration as discussed in 3.1, determining the induced penalty function G n 1,n in the objective function 20) presents an additional challenge. Our Approach. As in 3.2, we seek to derive simple bounds for the optimal policy by performing a derivative analysis of the two key equations 19) and 20) in the exact algorithm. Analogous to the expression 8) in the single-stage system, we can show [k] 1 y G [k] n k, y) = y G 1 nk, y) + u U i+1 n p ku y E{R i nu, y D k )}. 21) For each Stage n and demand state k, we define the following newsvendor cost function parameterized by ζ h n ζ n h i): G n k, y ζ) = ζ E { y D k [t, t + L n]) +} + b + H n+1 ) E { D k [t, t + L n] y) +}. 22) Its derivative function y Gn k, y ζ) = b + H n+1 ) + b + H n+1 + ζ)f k,l n y) will play an important role in our bound and heuristic development. Note that when N = 1, ζ h 1 = h, H 2 = 0, and the above newsvendor cost function reduces to 1).

18 Derivative and Solution Bounds For illustrative purposes, consider the cost function for Stage 2. From 19) and Proposition 1, we have y G 1,2 k, y) = y G [k] 1 k, min{y, s 1k)}) y G 1 1k, min{y, s 1k)}) = min { 0, y G 1 1k, y) }. Because H 1 H 2, y G 1 1k, y) = b + H 2 ) + b + H 1 )F k,l1 y) b + H 2 ) + b + H 2 )F k,l1 y). Observe that the right-hand side of the above inequality is always negative. It follows that y G 1,2 k, y) b + H 2 ) + b + H 2 )F k,l1 y). With this inequality, we obtain y G 1 2k, y) = h 2 + E { y G 1,2 W k t + L 2 ), y D k [t, t + L 2 ))} h 2 + E { b + H 2 ) + b + H 2 )F Wk t+l 2 ),L 1 y D k [t, t + L 2 )) } = b + H 3 ) + b + H 2 )F k,l 2 y) = y G2 k, y h 2 ). where the second equality follows from E{F Wk t+l 2 ),L 1 y D k [t, t + L 2 ))} = F k,l 2 y). By an argument analogous to that used in Proposition 1, we have y G [k] 2 k, y) y G2 k, y h 2 ). 23) Repeating the above argument from Stage 2 to Stage n, we can obtain a lower bound for y G [k] n k, y). Developing the derivative upper bound for the serial system, however, is more complex, because we need to bound the induced penalty function G n 1,n in 20). Nevertheless, we can show the following result, which generalizes Proposition 1 to the serial system. Denote h [1,n] = n h i. Proposition 3 For any stage n = 1,..., N and any state k = 1,..., K, the following holds: a) y G [k] n k, y) y Gn k, y h n ), b) y G [k] n k, y) y Gn k, y h [1,n] ) + 1 p kk ) n y) + n 1 j=1 j,n k, y), where n y) and j,n k, y) are recursively defined as follows: for n = 1,..., N, K 1 [ n y) = max α i m) y Gn k, y h [1,n] ) + n 1 + m S j=1 j,n m i, y)], with j,n k, y) = E { j y Dk [t, t + L n L j) )} for 1 j n 1 and α i m) given by 10).

19 19 It is straightforward to verify that 1 y) = y) as defined in 11). Thus, the above result reduces to that of Proposition 1 when N = 1. When N > 1, there is an extra term n 1 j=1 j,n k, y) on the right-hand side of the inequality. This term captures the dependence of Echelon n on all the downstream echelon base-stock levels from Echelons 1 to n 1. Based on Proposition 3, let s n k) be the solution to the equation y Gn k, y h n ) = b + H n+1 ) + b + H n )F k,l n y) = 0, 24) and let s n k) be the solution to the equation: It follows that y Gn k, y h [1,n] ) + 1 p kk ) n y) + n 1 j=1 j,n k, y) = 0. 25) Corollary 3 For any stage n = 1,..., N and any state k = 1,..., K, s n k) s nk) s n k). The above result generalizes Corollary 1 to serial inventory systems. To our knowledge, Corollary 3 presents the first general, analytical solution bounds for serial inventory systems with MMD. The existing bounding approaches for problems with nonstationary demand all build on a simplifying assumption that the optimal state-dependent echelon base-stock level is achievable in each period e.g., Dong and Lee 2003, Theorem 6 and Shang 2012, Theorem 1). This assumption, however, does not hold in general, and the lower bound obtained from these approaches may fail to bound the optimal solution under MMD. On the contrary, our derivative-based bounding approach does not require such an assumption and accounts for the dependence on the downstream echelon base-stock levels through the terms in Proposition 3. Thus, our solution bounds work in general. To see the above point more clearly, observe that by ignoring the terms in the derivative upper bound in Proposition 3, we obtain a newsvendor derivative function. Let s v nk) be the root of this function, i.e., the solution to y Gn k, y h [1,n] ) = b + H n+1 ) + b + H 1 )F k,l n y) = 0. 26) Thus, s v nk) is the lower bound for the optimal solution s nk) under the assumption that the optimal state-dependent echelon base-stock level is achievable in each period e.g., Dong and Lee 2003 and Shang 2012). However, for Stage 1, we have h [1,1] = h 1, and thus, from Proposition 3 and Corollary 3, s v 1k) = s 1 k). Thus, s v 1k) is not a solution lower bound, but instead a solution upper bound under MMD. For Stage n > 1, it is also not guaranteed that s v nk) s nk) under MMD see Table 1 in 5 for the counterexamples marked with a sign). When the state space of the Markov chain degenerates to a singleton, i.e., K = 1, the demand process becomes i.i.d. In this case, the optimal state-dependent echelon base-stock level is always

20 20 achievable in each period, and the terms in Proposition 3 vanish. As a result, y G [k] n k, y) is bounded below and above by simple newsvendor derivative functions and s v nk) becomes a true lower bound in this case. It is interesting to note that under the i.i.d. demand process, the solution bounds obtained from our derivative bounds coincide with those obtained by Shang and Song 2003) through their single-stage approximation approach. However, their bounding approach does not extend to the more general MMD process, as it also requires the assumption that the optimal state-dependent echelon base-stock level is achievable in each period see Appendix C for a detailed discussion). Our solution bounds can be computed without solving the integral renewal equations required in the exact algorithm. However, computing the solution lower bound involves optimization over demand state permutations, which can be challenging if there is a large number of states. To further simplify computation, we develop easy-to-compute heuristic solutions in the next subsection. 4.3 Heuristics Based on the definition of s v nk) given in 26), by Proposition 3, it follows that s n k) s v nk) s n k). Hence, although s v nk) is not guaranteed to be a solution lower bound, it can still serve as a simple heuristic solution for our problem. We can further improve this heuristic based on the idea used in the single-stage problem in 3.3. Specifically, we can sort s v nk) in an increasing order and let m n = m n,1,..., m n,k ) denote the resulting order sequence of the states. Suppose that k = m n,j. Based on Lemma 2, we can make the following approximation parameterized by ζ) for y G [k] n k, y) in 21): j 1 y G [k] n k, y) y Gn k, y ζ) + p k mn,i y E{Rn i m n,i, y D k )} j 1 y Gn k, y ζ) + y Gn k, y ζ) j 1 { ) + } p k mn,i β i m n ) E y Gn m n,i, y D k ζ) + p k mn,i β i m n ) F k y s v n m n,i )) y Gn m n,i, y ζ)), where the parameter ζ takes value in h n ζ h [1,n], and the last step follows from the same integral approximation as in the single-stage problem in 3.3. Next, let s a nk ζ) denote a heuristic solution determined by the following equation: y Gn k, y ζ) j 1 + p k mn,i β i m n ) F k y s v n m n,i )) y Gn m n,i, y ζ)) = 0, 27) where the above equation involves only a linear combination of the newsvendor derivative functions of different demand states. Thus, solving the heuristic solution s a nk ζ) only requires a direct