Optimal Learning Policies for the Newsvendor Problem with Censored Demand and Unobservable Lost Sales

Transcription

1 Otimal Learning Policies for the Newsvendor Problem with Censored Demand and Unobservable Lost Sales Diana Negoescu Peter Frazier Warren Powell Abstract In this aer, we consider a version of the newsvendor roblem in which the demand for newsaers is unknown and the lost sales are unobservable. This roblem serves as a surrogate for more comlex suly chain roblems in which learning lays a role. We roose the knowledge gradient ) olicy, which considers how much is learned about the demand distribution from each order. This olicy combines comutability with good erformance. We analyze the behavior of this olicy under two demand distributions: an exonential distribution with a gamma rior, under which the otimal olicy is comutable, and a general demand distribution with finite suort and Dirichlet rior under which the otimal olicy is very hard to comute. In the exonential demand case we contrast the olicy with two naive olicies and the otimal olicy, and in the general demand case we contrast the olicy with a naive olicy. Our simulations show similar behavior to the otimal olicy in the exonential demand case, and good erformance in the general case. Introduction In suly chain management roblems, one often accounts for uncertainties in daily demand by assuming that the underlying robability distribution of demand is known. This aroach works well if the demand distribution is stable, and one has enough data from ast demands to accurately estimate it. There are many situations, however, when we are learning the demand while simultaneously managing our suly chain. This is articularly common when selling a new roduct, or when selling under new market conditions. Furthermore, our management of the suly chain often affects the amount of demand data we collect, since we often cannot observe demand for markets in which we do not sell, nor demand for out-of-stock roducts. We examine the effect of this tye of learning in the newsvendor roblem see, e.g., Scarf 959)), which is a simle and canonical suly chain roblem. This roblem has many immediate alications by itself, and is also seen as a comonent of most more comlex suly chain roblems. Consider the following examle newsvendor roblems: In IPO ricing, the shares of the comany need to be riced such that all shares are sold and rofits are maximized, without having an accurate estimate of the demand. A college admissions office needs to otimize the number of offers it makes to an alicant ool, in order to avoid having more students accet the offer than there are available rooms,

2 INTRODUCTION 2 or fewer students than the targeted class size, taking into account ossible demograhic changes that might occur from one year to another. A comany manufacturing ersonal jets needs to otimize the number of jets roduced in a year before knowing the demand for them, while keeing in mind that storing unsold lanes costs enormous amounts of money. A doctor administering blood thinners needs to administer enough of the drug to hel the atient, but not so much that it causes severe side effects, without knowing the atient s tolerance to the drug. An oil comany needs to otimize the number of holes it drills so that it finds good oil reserves without drilling too many unsuccessful holes, while taking into account that the cost of drilling is very high and the likelihood of success cannot be accurately araised. In these examles, demand distributions are initially unknown and observations are infrequent. For examle, the admissions office observes the demand only once er year, the airlane comany risks going out of business if it fails to learn the demand in a year or two, and the atient risks dying unless the doctor administers the right amount of drug in time for it to take effect. A crucial asect of the newsvendor roblem is therefore not just how well we can learn about the demand, but how fast we can learn. We are therefore most interested in analyzing how well the newsvendor can do in the first rounds of observations, not just whether we can eventually reach an otimal stocking olicy as the number of observations goes to infinity. In the standard newsvendor roblem a newsvendor tries to make a rofit by ordering the right number of newsaers at a fixed cost and selling them at a fixed market rice higher than that cost. In our version of the newsvendor roblem, the newsvendor s task is challenging because the demand is unknown, and he observes the actual demand only if he orders more newsaers than the demand, in which case he also incurs a loss due to the newsaers he bought but did not sell. Otherwise he only sees that the stock is exhausted. We assume that the newsvendor cannot sell the newsaers left from one eriod in a different eriod. In short, the demand is censored and the lost sales are unobservable. The newsvendor needs to udate his belief about the demand after each observation so as to make better stocking decisions in the next ordering rounds. The newsvendor roblem has been widely discussed in the inventory theory literature under various forms. However, only more recent work has focused on the case where the demand is unknown and censored. The two main aroaches have been the frequentist and the Bayesian aroaches, but other aroaches have also been studied. In the frequentist aroach, Nahmias & Smith 994) are the first ones to analyze the censored demand case with normally distributed demand. The work of Agrawal & Smith 996) considered the estimation of negative binomial demand in the resence of censoring. However, these studies address only the issue of estimating

3 INTRODUCTION 3 the demand without investigating the roblem of how to otimize the stocking level when stocking decisions affect the amount of information collected. Early Bayesian aroaches aear in Scarf 959), Clark & Scarf 96) and Azoury 985), but these studies assumed fully observed demand. Lariviere & Porteus 999) considers Bayesian udating with artially observed sales and comutes a dynamic rogramming exression for the otimal stocking olicy using the dimensionality reduction technique first resented by Azoury 985) in the case of Weibull demand with a gamma rior. Ding et al. 22) considers the censored demand setting for a erishable roduct and shows that, with continuous demand distributions, stocking levels in the censored case are higher than in the uncensored case. Bensoussan et al. 27) writes the dynamic rogramming equation, roves the existence of an otimal feedback solution and rovides the otimality equation for the inventory level. The authors then model the demand as a Markov Chain taking two values, high and low, and obtain aroximately otimal solutions numerically. Berk et al. 27) gives a more detailed literature review and develos exressions for the exact osteriors starting with conjugate riors for the negative binomial, gamma, Poisson, and normal distributions. Another method, different from the revious two aroaches, is a non-arametric algorithm introduced by Godfrey & Powell 2) called the Concave, Adative Value Estimation CAVE) algorithm which constructs a sequence of concave iecewise linear aroximations using samle gradients of the value function at different oints in the domain. The authors demonstrate near otimal behavior of the CAVE aroximation in the newsvendor roblem. Although an otimal solution to the newsvendor roblem exists in theory, in ractice it is very hard to comute in settings of interest because it requires solving an infinite-dimensional dynamic rogram. This motivates the develoment of comutable sub-otimal olicies that erform well in ractice. In this aer we roose the use of a new Bayesian algorithm first introduced by Frazier et al. 28) for ranking and selection roblems, called the knowledgegradient ) olicy. This olicy was extended by Ryzhov et al. 28) to roblems in which we learn while simultaneously acting in the world. We adat this algorithm to our learning roblem: we try to choose a stocking level to maximize the exected value of information, not just the immediate rofit. In other words, we choose our stocking level taking into account how much we learn from our actions. We test the olicy against two demand distributions, one continuous and the other discrete: exonential demand with a gamma rior and a general demand with finite suort and Dirichlet rior. In Section 2, we consider exonential demand distribution with a gamma rior. We derive the olicy for this demand distribution and we comare it against three other olicies, including the otimal olicy. We consider the case of exonential demand with a gamma rior because it allows us to derive simle udating formulas for the belief about the demand due to conjugacy roerties the rior and the osterior belong to the gamma distribution. Also,

4 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR 4 using this distribution enables us to use the result of Lariviere & Porteus 999) to comute the otimal olicy. Our simulations then show that the olicy erforms very close to the otimal olicy. In Section 3 we consider the general demand distribution with Dirichlet rior. We derive the olicy for this demand distribution and comare it to a myoic olicy that simly orders such as to maximize the immediate rofit. The general distribution can cature any shae of the demand density, and therefore it enable us to see how behaves in situations where we make no arametric assumtions about the demand distribution. This shows how the olicy can be used for realistic models of demand, for which comuting the otimal olicy is imossible. 2 Exonential demand with gamma rior We first discuss the exonential demand case. Section 2. models the roblem, Section 2.2 derives the order quantities for each olicy, and Section 2.3 resents and discusses the results obtained from imlementing the olicies derived in Section Model We model the demands as exonentially distributed with arameter λ. Starting with a Gammaa, b ) rior on λ, we decide what our stocking level should be, observe the number of roducts sold and udate our belief about λ. Our osterior belief will again be gamma-distributed, as shown below. We reeat this rocedure until the end of our horizon. As we collect more observations, our sequence of osterior distributions concentrate more around λ, reflecting what we have learned. Let P n be the robability measure conditioned on observations u to time n, so P n { } = P{ x k, minx k, ˆD k+ ), k < n}; a n, b n be arameters of the gamma distributed belief on λ after n time eriods. Thus, P n {λ } is a Gammaa n, b n ) distribution; x n be the olicy newsaers to be ordered) under the current belief about the demand a n and b n ); ˆD n+ be the demand after ordering newsaers according to the current belief a n, b n ; rofit = minx n, ˆD n+ ) cx n, where is the rice of a newsaer, and c is the cost of a newsaer. For simlicity, wherever ossible, we will dro the suerscrits. To udate the demand we start from a rior distribution λ Gammaa, b), and we comute

5 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR 5 our osterior distribution in the two cases. In the first case suose ˆD n+ < x n. Then, n P n λ du ˆD n+ = y) P n ˆD n+ dy λ = u)p n λ du) = bna n Γa n ) e ub +y) u an. This is the kernel of a Gammaa n +, b n + y) distribution, so the osterior belief P n λ ˆD n+ = y) is a Gammaa n+, b n+ ) distribution with a n+ a n +, ) b n+ b n + y. 2) In the second case, suose ˆD n+ x n. Then, ne P n λ du ˆD n+ x n ) P n D n+ x n λ = u)p n λ du) = bna bn +x n )u u an Γa n. ) This is the kernel of a Gammaa n, b n +x n ) distribution, so the osterior belief P n λ ˆD n+ x n ) is a Gammaa n+, b n+ ) distribution with a n+ a n, 3) b n+ b n + x n. 4) These udates do not deend on which of the four stocking olicies we use, and are articular to this choice of rior and demand distribution. 2.2 Comuting the stocking levels for each olicy given the current belief about the demand In this section, we describe four olicies for determining the stocking level x. The first olicy is a naive olicy in which we estimate λ and then find the decision that would be otimal if λ were actually equal to the estimate. We will refer to this olicy as the Point Estimate PE) olicy. The second olicy is a myoic olicy derived using the Gammaa n, b n ) distributed belief. This olicy would be otimal if we could not make any observations in the future. We will refer to this olicy as the Dist) olicy. The third olicy is the Knowledge Gradient ) olicy, which chooses the stocking level to maximize not just immediate rofit, but also the exected rofit on the next future run multilied by how many observations there are left. The fourth olicy is the otimal olicy obtained by solving the dynamic rogram as resented in Lariviere & Porteus 999) Point Estimate Policy We first comute the order quantity x that would be otimal if λ were equal to its Bayes estimate under the Gammaa n, b n ) belief. This olicy is the PE olicy and maximizes rofit under the assumtion that a n and b n are the true arameters of the gamma distribution from which λ was samled. This olicy is thus a ure exloitation olicy.

6 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR 6 At each iteration, the olicy first assumes that the Bayes estimate ˆλ = E n [λ] = an b n is the true value of λ. It then comutes the x n that would be otimal if λ really were equal to ˆλ and ˆD n+ Exonentialˆλ). Therefore, ] x P E,n = arg max [rofit λ E = an x n b n = arg max E x n [ minx n, ˆD n+ ) cx n λ = an For simlicity, let us dro the suerscrits. Letting F x, D) = minx, D) cx, { F x, D) c if x < D g = = x c if x > D By Lebesque s Bounded Convergence theorem see, e.g., Royden 988)), xe[f x, D)] = E[g] and E[g] = c)p D > x λ = a ) cp D x λ = a ) = e ˆλx c. b b Setting E[g] to and solving for x, we get from the concavity in x n of the function minx n, ˆD n+ ) that. x P E,n = ˆλ ) log = bn ) c a n log. c Substituting this decision into the objective function, the PE olicy achieves an exected rofit of b n ]).. E n [rofit] = E n [ minx, D) cx λ = ˆλ] = xpd > x λ = ˆλ) + E[D D < x, λ = ˆλ] PD < x λ = ˆλ) ) cx x ) = xe ˆλx + uˆλe ˆλu du cx = ˆλ ) e ˆλx cx. Since ˆλ = an b, the objective function becomes n E n [rofit] = bn a n e an x) b n cx Policy The second aroach, Dist, chooses the order x n that would be otimal if we could not observe future demands. This olicy i given by, x Dist,n = arg max x n E [rofit λ Gammaan, b n )]. Then, again droing the suerscrits, we comute the exected rofit by first comuting [ ] E[minx, D) λ Gammaa, b)] = E λ e λx ) = u e ux ) be bu bu) a n Γa n du = b ) ) a b. ) a b + x

7 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR 7 Thus, the objective function at the n th iteration for this aroach is: b n ) ) b n a n E n [rofit] = a n b n + x n cx n. as The goal is then to maximize F x, D) = E[minx, D)] cx. ) ) a b b b+x F x, D) = a cx. Setting the derivative of the above exression to and solving for x gives this olicy s decision x Dist,n = b n c ) /a n ). 5) Taking the limit of x n as a n, b n = λa n, we obtain λ log c ), the same value as that of the first olicy. This makes sense because in the limit the Bayes estimator is accurate, and therefore λ really is equal to ˆλ Knowledge Gradient Policy The third olicy is the knowledge gradient olicy. Under this olicy x is chosen to maximize not just the rofit at the next iteration, but the immediate rofit lus a future value of learning. The olicy chooses the action that would be otimal if we were to take one action, learn from it, but then not be allowed to learn afterwards. Intuitively, the order quantity is Formally, x,n = arg max x n E[ProfitNow + FutureValue an, b n, x n ] [ x,n = arg max E min ˆD n+, x n ) cx n + V a n+, b n+ ) a n, b n, x n], x n where V a n+, b n+ ) is the exected future value total rofit) from taking decision x,n, given that our current belief is a n, b n. The exected rofit for one ste in the future is comuted by taking into account how we will udate our belief after we make our order decision now and observe the demand once. That exectation may be comuted in the same way as it was in the olicy s derivation, since we are assuming that we sto learning after observing the next value of the demand and use the same order quantity for the remainder of the horizon. If there are N n time stes left until the end of the horizon, the value is [ V a n+, b n+ ) = E N n) min ˆD n+2, x Dist a n+, b n+ )) )] cx Dist a n+, b n+ ) λ Gammaa n+, b n+ ) [ = N n) max E min ˆD ] n+2, x n+ ) λ Gammaa n+, b n+ ) cx n+). x n+

8 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR 8 Figure : Units ordered on the first day under the olicy as a function of the cost/rice ratio, for horizons, 2 and 3 days and keeing a and b constant. The maximum on the last line came from the fact the x Dist maximizes exected rofit given the current belief at time n + and λ is assumed to be distributed according to the gamma distribution with that belief. The challenge in comuting the olicy is that a n+ and b n+ are not known at time n. This requires taking an exectation over the redictive distribution on a n+ and b n+. This calculation is given in the aendix. Letting r c, the order quantity under the olicy is then x = b r [ ] + N n) + ar a a a + )r a a+ a. To see the behavior of this olicy, we lot in Figure the number of units ordered as a function of the cost/rice ratio for different horizons N, assuming we are on the first day n = ). As exected, the olicy orders more when the cost/rice ratio is smaller because the loss from ordering too much is smaller. Also, as the horizon increases, for a given cost/rice ratio the units ordered increase. This is exlained by the fact that as the horizon increases, the benefit from learning at the first iteration also increases and hence the olicy orders more to learn more about the demand. The exression for the objective function for the olicy is the same as in the Dist olicy, b n E n [rofit] = a n ) ) b n a n b n + x n cx n+, 6) but where x n is the otimal value under the Knowledge-Gradient olicy, comuted in 6). The limit of x n as a n, b n = λa n is λ log c ), which is the same limiting values as with the other two olicies. Again, this is natural because in the limit the Bayes estimator is accurate, and therefore λ really is equal to ˆλ.

9 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR Otimal Policy We now summarize the result in Lariviere & Porteus 999) that we use to imlement the otimal olicy. The otimal olicy is found by solving a dynamic rogram with two state variables, a and b, and using a dimensionality reduction technique ioneered by Clark & Scarf 96) and extended by Azoury 985), which eliminates the scale arameter b from the analysis. Lariviere & Porteus 999) rove that the otimal level of the inventory after ordering in eriod n, x n a, b) can be scaled as x n a, b) = bx n a). Then, using α as a discount factor, they rove that ) βn a a) x n a) =, c where β n a) + α[ρ n+ a + ) ρ n+ a)], ρ n a) = c acx n a) + αρ n+ a + ). We use these exressions to imlement the otimal olicy and comare it to the other three olicies. 2.3 Simulations and Results We now use numerical simulation to comare the olicy to the other three olicies discussed: Point Estimate, and Otimal. Due to the randomness in the demand and the fact that λ will be different for every run of our exeriments, it is worthwhile to talk about samle behaviors - how the olicies can behave during a secific run - and about average behaviors - how the olicies behave over an average of runs. Below, Section 2.3. discusses average behaviors, while Section discusses samle behaviors Average behavior. By average behavior we mean an average over many samle runs in which we draw many different λ s from the rior. It is this kind of average erformance that the Bayesian objective function calls for us to maximize, and under which the otimal olicy is otimal. We consider horizons increasing from day to days, and for each horizon, comute an average over samles of the difference in cumulative rofits between the otimal olicy and each of the other olicies. In each samle, we draw λ from the Gammaa, b ) sace. We also comute the standard deviation of each average, and lot them form of error bars. The results are resented in Figure 2. As we can see, the mean difference between the Otimal olicy and the Point Estimate and olicies are significantly large, but the mean difference between the Otimal olicy and the Knowledge Gradient is extremely small.

10 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR.6 Cumulative Profits Below Otimal Otimal PointEstimate Otimal Otimal day Figure 2: The means and standard deviations reresented by error bars) for cumulative rofits of the four olicies Samle Behaviors of the Four Policies In analyzing samle runs we are mainly interested in comaring the rofits and cumulative rofits for the four olicies, in observing how our belief on λ changes over time, how the order quantities comare to one another and whether the ratio of our estimates, a b converges to the value of the λ of that run. Then we examine how the order quantities and rofits of comare to those of the otimal olicy if we enforce the same udates for both. Profits and udating the belief about the demand We first visualize the behavior of the four olicies over a horizon of days. Figures 3, 4, and 5 show how the beliefs about the demand change over iterations for the four olicies first two anels), how fast the ratio a b converges to the true value of λ for each olicy, how the order decisions comare to the actual demand left anel on the bottom) and what the daily and cumulative rofits for the olicies are. In the run resented in Figure 3, the otimal olicy is doing the best: the rofits and cumulative rofits are biggest for the otimal olicy, and also the ratio a b converges to the true λ of that run the fastest. However, that is not always the case due to the randomness in the demand and in the λ arameter. Figure 4 shows an instance where the Knowledge Gradient

11 2 EXPONENTIAL DEMAND WITH GAMMA PRIOR a 5 5 a vs. b b vs. a n /b n a n /b n vs. Point Estimate Otimal λ = a true /b true units Point Estimate Otimal Demand rofits Profits vs cumulative rofits Cumulative Profits vs Figure 3: A samle of the behavior of the four olicies when run for iterations. olicy is doing best: the rofits and cumulative rofits are noticeably higher and the ratio a b converges to the true λ the fastest. Figure 5 shows a case where the cumulative rofit for the olicy is the worst. Uon further examination of the lots however, we notice that although the cumulative rofit is worst, the ratio a b converges to the true λ the fastest for the Knowledge Gradient, which would suggest that the Knowledge Gradient is making more accurate udates of the demand. This is because it ordered a lot more than the other olicies at the beginning as is noticeable in the lot of x and d), which leads it to learn the actual λ faster, but also to order more than the demand in a few instances, causing significant losses. Stocking levels and cumulative rofits under the same beliefs To comare the the stocking decisions of the Knowledge Gradient to the Otimal Policy, we run samles where we force the beliefs about the demand of the two olicies, a and b, to be the same at each iteration. The results, deicted in Figure 6, show that indeed the Knowledge Gradient olicy orders more at the beginning than the Otimal olicy. This is exected since the olicy maximizes the value of taking a decision, therefore taking into account that it might learn more at early iterations from over ordering, which can hel make better decision in future iterations.

12 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 2 a a vs. Iteration # b 2 5 b vs Iteration # a n /b n 2.5 a n /b n vs. Iteration # Point Estimate Otimal λ = a true /b true units 5 Actual Demand and Order Quantities vs. Iteration # 2 Point Estimate 5 Otimal Demand 5 rofits 5 Profits vs. Iteration # 2 3 cumulative rofits 5 Cumulative Profits vs. Iteration # Figure 4: Another samle of the behavior of the four olicies when run for iterations. a 2 5 a vs. b b vs. a n /b n 2.5 a n /b n vs Point Estimate Otimal λ = a true /b true units 5 Actual Demand and Ordered Quantities vs. Point Estimate 8 6 Otimal Demand rofits 5 Profits vs. 5 cumulative rofits 5 Cumulative rofits vs Figure 5: Behavior of the four olicies when run for iterations. is ordering more than the other olicies and therefore loses a lot of rofit when the actual demand is small. 3 General Demand with Finite Suort and Dirichlet rior While the revious section demonstrated the erformance of the olicy by comarison to the otimal olicy, we now illustrate its flexibility by imlementing it in a more general setting. The need for a more general distribution of demand comes naturally if we realize that the exonential demand assumed in the revious section is most likely unrealistic. We therefore

13 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 3 Ordered Quantities under Same Belief vs Iteration # 9 Otimal 8 2 Profits under Same Belief vs Iteration # Otimal Units ordered Profits Iteration # Iteration # Figure 6: Comaring the stocking levels and Profits of and Otimal if they are forced to have the same belief about the demand a n, b n ). orders more than the otimal olicy in the beginning. consider a general demand distribution on a finite set of integers. We imlement a myoic olicy Dist), which is the general demand version of the olicy from the exonential case, and the Knowledge Gradient ) algorithm and comare their behavior. Of course, the order quantities will differ from those in the exonential case since the assumed demand distribution is different. Section 3. resents the belief udating equation and derivation of the Dist and olicies, and Section 3.2 resents the results obtained from imlementing the olicies. 3. Model We assume that the demand can take values, 2,..., k, where k is an integer 2. We then define θ i as the robability given the full distribution of demand that the demand on any given day is equal to i. We take a Dirichlet rior on θ with arameter vector α, under which: Pθ du) k j= u αj j for vectors u with k j= u j =, u j. To exemlify this rior, Figure 7 shows a few demand distributions generated from a rior α = 2,,,, 2). In this case the demand can take 5 ossible values, and the high value on the first osition in the rior assigns a high robability of the demand taking the first value, the 2 on the last osition in the rior vector assigns a smaller, but significant robability of

14 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR Figure 7: Samle demand distributions generated from the rior α = 2,,,, 2) the demand taking the fifth value, while the ones in ositions 2, 3 and 4 translate into small robabilities that the demand can take the second, third or fourth values. Such a rior might be used, for examle, when we believe that the demand usually takes low values, but we also think there might be a significant chance of the demand taking high values. Another illustration of this Dirichlet rior is in Figure 8, where we simulated demand distributions from the rior α =,,,, ). As can be seen from the figure, the demand distributions can vary quite a bit because the rior doesn t assign a higher likelihood for any articular value of the demand. Such a rior might be used when we have very limited information about the demand and we want to start with a less informative rior. While the udates after each observation are easy to comute for the exonential demand with gamma rior, they are much more comlicated in the case of a general demand distribution. Although censoring revents a closed form udate of the osterior distribution, we can use a result from Ferguson & Phadia 979) to comute our myoic and measurement olicies. We briefly describe the result here. Let u,..., u N be the distinct values of the observations minx n, ˆD n+ ) ordered so that u < u 2 <... < u N, let δ,..., δ N denote the number of uncensored observations at u,..., u N, and µ,..., µ N denote the number of censored observations at u,..., u N. Define h j = N i=j+ δ i + µ i ) as the number of observations of the demand greater than u j. α t) = i t α i. Also define jt) as the number of u i less than or equal to t, αt) = i>t α i and Letting St) reresent the robability that the demand is greater than t given the demand

15 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR Figure 8: Samle demand distributions generated from the rior α =,,,, ) distribution, Theorem 8 of Ferguson & Phadia 979) then gives the osterior exectation of St) after n observations to be E[St)] = αr) αt) + h jt) αr) + n jt) αr) α u i ) + h i ) αr) α u i ) + h i + δ i ). 7) i= The Ferguson & Phadia 979) result alies to the more general case of Dirichlet rocess riors on the set of measures on R, of which Dirichlet riors on measures of fixed finite suort is a secial case. We now comute the Dist and olicies. We define Z n to be the vector {x < ˆD}, minx, ˆD ),..., {x n < ˆD n }, minxn, ˆD n ) reresenting all the data we have collected in the first n iterations. Let Q Dist,n+ i, Z n ) be the exected rofit in iteration n + from ordering i units given Z n. The olicy at iteration n + is then x Dist,n+ = arg max Q Dist,n+ i, Z n ). 8) i We comute it as Q Dist,n+ i, Z n ) = E[ min ˆD, i) ci Z n ] = E n [ ˆD{ ˆD<i} ] + ie n [ { ˆD i} ] ci.

16 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 6 The first exectation is, using the tower roerty, [ ] [ ]] E n ˆD{ ˆD<i} = E n [E n ˆD{ ˆD<i} S = E n P{ ˆD { ˆD<i} > j S} i i = E n Sj) Si )) = E n + Sj) isi ). j= Similarly, the second equation is: j= [ ] [ E n { ˆD i} = E n [E { ˆD ]] i} S = E n [Si )]. Bringing them together, we get that i Q Dist,n+ i, Z n ) = E n + Sj) isi ) + isi ) ci j= i = E n + Sj) ci. j= Using this and 7), we can comute the exected value of the rofit, Q Dist,n+ i, Z n ), given any value i. Then, by evaluating Q Dist,n+ i, Z n ) for i between and k, we can comute the olicy using 8). j= We now comute the order quantity under the olicy. We define Q,n+ i, Z n ) to be the value of ordering i units in iteration n +, learning from the resulting observation, and then choosing the best stocking decision for all subsequent time eriods. Also, we define V Dist Z n ) = max i Q Dist,n+ i, Z n ). This is the value under the no learning assumtion of the olicy at iteration n +. Then, if N is the horizon and n the number of the current observation, [ Q,n i, Z n ) = Q Dist,n i, Z n ) + N n)e V Dist Z n, min ˆD ] n, i)) The olicy is then x,n Z n ) = arg max Q,n i, Z n ). i However, to comute the values of all Q,n i, Z n ), we use the difference: Q,n+ i +, Z n ) Q,n+ i, Z n ) = Q Dist,n+ i +, Z n ) Q Dist,n+ i, Z n ) + N n)e[v Dist Z n, min ˆD, i + )) V Dist Z n, min ˆD, i))] = E[Si) Z n ] c + N n)p{ ˆD = i Z n }[V Dist Z n, ˆD = i) V Dist Z n, ˆD i)] + N n)p{ ˆD > i Z n }[V Dist Z n, ˆD i) V Dist Z n, ˆD i)] = E[Si) Z n ] c + N n)si ) Si))[V Dist Z n, ˆD = i) V Dist Z n, ˆD i)] + N n)si)[v Dist Z n, ˆD i + ) V Dist Z n, ˆD i)]. 9)

17 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 7 Above, V Dist Z n, ˆD = i) is the value from making uncensored observation i and V Dist Z n, ˆD i) is the value from making a censored observation i. The transition from the first line to the second line above was ossible because for ˆD < i, V Dist Z n, min ˆD, i + )) V Dist Z n, min ˆD, i)) =. Again using the results from Ferguson & Phadia 979), we can reconstruct all the Q,n+ i) and then choose the demand i that maximizes Q,n+. This is the decision of the olicy. The next section discusses the results obtained from imlementing these olicies. 3.2 Simulations and Results Since the results are subject to the randomness in the demand, we again talk about about average behaviors Section 3.2.) samle behaviors in Section 3.2.2) Average behaviors In exloring the average behavior of the olicies, we first considered the following scenario: we retend to be the manager of a store who has to decide how much of a secific roduct to order. The demand for similar roducts is tyically low, but once in a while a secific roduct has an unusually high demand we can imagine this roduct to be for examle a DVD. The demand for a tyical DVD is not very high, but a oular DVD can have a very high demand). We want to satisfy about 8% of the demand and therefore we set our cost c to be 2 and rice to be. We are now interested in seeing how our olicy does in this setting, in which learning the new demand is imortant for keeing u with the cometing stores. We illustrate this in Figure 9, where we lot the mean difference in rofits between and Dist for k = 5, cost = 2, rice = and the rior underestimates the truth the rior vector α is of the form 2,,,,2), while the actual demand is generated from a vector,,,,)). The mean difference is comuted for horizons of u to days and for each day, we average over a hundred runs. As we can see, the olicy erforms significantly better than the olicy, registering a rofit of more than 9 by the th day. The fact that the demand is tyically low encourages the olicy to order a small amount in order to maximize immediate rofits., however, orders more, taking into account that the demand might also have higher values, which enables it to learn the demand better than Dist when the demand actually is high and thus cash in higher rofits. Next, another scenario we imagine is the following: we have very little information about the demand, and therefore we start with the less informative rior consisting of a vector α of ones. We assume the rice to be, and we run the olicies for a horizon of u to observations. For each observation, we average the difference in cumulative rofits between the olicy and

18 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 8 Mean Difference in Cumulative Profits between and day Figure 9: Mean difference in cumulative rofits and standard deviations for k = 5 and cost/rice =.2. the olicy over runs. The true demand is also generated from a vector α of s. These mean differences in rofits enable us to see the tyical behavior of the olicies. In conducting our exeriments we find that the value of learning deends strongly on the cost c and on the number of values k) that the demand can take. We comute the mean difference in cumulative rofits between and for k = 5, and costs c =5, 8 and 9. When k = 5 and c = 8, Figure shows that the mean cumulative rofits of are significantly more than the rofits of Dist. However, when c = 5 and c = 9, Figures and 2 show that the difference in rofits between and Dist becomes statistically insignificant. The first case k = 5, c = 5) can be exlained by the fact that when the cost is relatively low, Dist orders more than one unit because the cost of over-ordering is low, which brings it closer to the olicy. When the cost is very high, over-ordering becomes very costly, which leads to take a more risk averse attitude and order just as much as Dist. It aears that when the cost is not too low and not too high, which is the case when c = 8, the value from learning is highest and s rofits are significantly better than s. Our exeriments show that the value from learning is also deendent uon the number of values that the demand can take. For k =, a cost of 8 no longer brings significant rofits above, as shown in Figure 3. However, increasing the cost to 9 again brings the rofits of above those of Dist as seen in Figure 4. It therefore seems that the value of learning is maximized when the cost is such that the olicy orders more than the myoic olicy, but not too high to cause it imortant enalties for over ordering.

19 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 9 Mean Difference in Cumulative Profits Between and day Figure : Average difference in cumulative rofits and standard deviations for k = 5 and c = Mean Difference in Cumulative Profits day Figure : Average difference in cumulative rofits and standard deviations for k = 5 and c = Samle Behaviors In analyzing the behavior of the olicies in the general case we first run the algorithms for small values of k, where k is the total number of values that the demand can take. Letting k = 5, and having the rior always be given by the vector α =,,..., ), we obtain the following results when running our algorithms simultaneously for various values of the true distribution of the demand. The rice of one unit is set to and the cost is set to 8. If we define the vector olicy as the estimate of the demand distribution at the end of a day using a certain olicy and the vector true as the true demand distribution, we can then define the error of that olicy as i olicy i true i ) 2. We then lot the errors we obtain after each day.

20 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 2 Mean Difference in Cumulative Profits day Figure 2: Average difference in cumulative rofits and standard deviations for k = 5 and c = 9. Mean Difference in Cumulative Profits day Figure 3: Average difference in cumulative rofits and standard deviations for k = and c = 8. Figure 5 shows an instance in which outerforms in both accuracy and cumulative rofits. The run, obtained for k= 5 and a true demand distribution with more mass on the higher values, shows that udates its belief more efficiently than Dist its errors are smaller), and obtains more rofits. The order quantity may be exlained by the fact that the cost/rice ratio is fairly high, which strongly enalizes over-ordering in the myoic objective function. realizes that ordering a little more, while risking higher costs, also will rovide more information. It learns more about the demand, resulting in a more accurate belief. However, does not always outerform in accuracy. Figure 6 shows an instance in which the error in the estimate from is greater than the estimate from Dist. Profits are still higher for than for Dist because orders 2 units as oosed to unit for

21 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 2 35 Mean Difference in Cumulative Profits day Figure 4: Average difference in cumulative rofits and standard deviations for k = and c = 9. Error Profit.35.3 Errors vs Current Day Day Profits on Each Day Day Cumulative Profit Units Ordered Quantities Ordered and Actual Demand Actual Demand Day Cumulative Profits on Each Day Day Figure 5: Behavior of and for k = 5, =, c = 8 and observations when the true distribution of the demand has more mass in the higher value region. Dist, and since the demand is almost always at least 2 - the true robability of demand vector having more mass in the middle - cashes in rofit on the units missed by Dist. Figure 7 shows an instance in which even the cumulative rofits for are slightly worse than those for Dist. The run, obtained for a true distribution with more mass on the lower value region, shows the errors of starting higher than those of Dist, and then droing as observations are made. This time, does not kee ordering 2, but instead, after having incurred losses, decides to order only unit. s weak erformance can be accounted for by the fact that the true demand distribution has more mass in the lower value region, which means that overestimates the otimal order quantity by a lot when trying to learn, and thus incurs

22 3 GENERAL DEMAND WITH FINITE SUPPORT AND DIRICHLET PRIOR 22 error Errors vs Current Day units Quantities ordered and actual demand Actual demand rofit Day Profits on each Day Day cum.rofit Day 5 Cumulative rofits on each Day Day Figure 6: Behavior of and for k = 5 and observations when the true distribution of the demand has more mass in the middle value region. error Errors vs Current Day units Quantities Ordered and Actual Demand Demand rofit Day Profits on Each Day Day cumulative rofit Day Cumulative Profits on Each Day Day Figure 7: Behavior of and for k = 5 and observations when the true distribution of the demand has more mass in the lower value region. losses in rofit due to the unsold items. Having seen these results, we conjecture that, just as in the exonential demand case,

23 4 CONCLUSION 23 does better than Dist when the rior underestimates the truth. 4 Conclusion Both demand distributions illustrate that the Knowledge Gradient algorithm can erform significantly better than standard, no learn olicies. The Knowledge Gradient algorithm tends to order more than no learn algorithms and hence obtains more information about the demand, which hels it to make better decisions in the future. There are, however, instances when the Knowledge Gradient undererforms. These are usually associated with a rior that gives more robability to higher values of the demand, in which case over ordering does not hel as much in learning about the demand, but on the contrary, brings about high enalties from the unsold items which causes to fall behind the myoic olicies in rofits. The exonential demand with gamma rior illustrates the oints made above quite clearly, and also shows that for short horizons, the Knowledge Gradient olicy erforms remarkably close to the otimal olicy described in Lariviere & Porteus 999). The discrete, general demand however roved more comlex to analyze than the continuous, exonential case, although the general observation that does better when the rior underestimates still remains. There is undeniably some value in adoting a learning olicy instead of a ure exloitation olicy. In alications in which one measurement is extremely exensive, learning can therefore have a huge imact on the success of the endeavor.

24 5 APPENDIX: ORDER QUANTITY COMPUTATION 24 5 Aendix: Order Quantity Comutation We have that [ x,n = arg max E n min ˆD ] n+, x n ) cx n + V a n+, b n+ ) x n = x x V a n+, b n+ ) = E n+ [N n) min ˆD n+2, x Dist ) cx Dist)] = N n) max x E n+ [ min ˆD n+2, x n+ ) x n+ = x ] cx n+). Using the same reasoning as in the olicy, we can use the exression derived for E[min D n+ ˆ, x) λ Gammaa, b)] and since x Dist maximizes rofit, we get V a n+, b n+ b n+ ) = N n) max x a n+ + x ) a n+) ) b n+ cx. Just as in equation 5), taking the derivative and setting it to rovides x = b n+ c ) /a n+ ). Substituting x in the exression for V, and ignoring the n + in the suerscrits for simlicity, we get V a, b) = N n)b a c ) ) a c ) )) c a Finally, the exression for x,n is E n [ min ˆD n+, x n ) cx n ] + N n)e n [b n+ x,n = arg max ]) c ) a c n+ )) x n. a n+ The first exectation in the sum was shown to be E n [ min ˆD n+, x n ) cx n b n ) ) b n a n ] = a n b n + x n cx n. ) ) c a n+ We now comute the second exectation, EV E[ N nv a, b)], by conditioning on λ, recalling that a n+ and b n+ are udated according to [2] and [4], and letting y be the observed demand. [ [ EV = E E b n+ a n+ c ) [ x n = E b n c + y) a n c + a n x n b n + x n ) a n+ ) c c ) a n+ ) ) a n + c ) a n ) c c c ] ] )) λ a n, b n, x n ) a n + )) λe λy dy ) )) ] a n λe λy dy.

25 5 APPENDIX: ORDER QUANTITY COMPUTATION 25 Defining A = b n B = c c ) ) a n + a n ) ) a n + a n C = b n c c ) a n ) a n c ) ) c a n +, ) ) c a n +, c ) ) /a n c, the exression inside the revious exectation, which we call I, becomes, I = x n Aλe λy dy + x n Byλe λy dy + x n Cλe λy dy = A e λxn ) Bx n e λxn B λ e λxn ) + Ce λxn ). Integrating the above exression and recognizing some Gamma kernels rovides: ) ) a n ) b n a n ) b n a n b n b B n b n +x n EV = C b n + x n Bx b n + x n a n + A is now b n b n + x n Thus, again droing the suerscrits for simlicity, the function that needs to be maximized b b b+x ) a ) F x) = a + N n) b C b + x cx ) ) a ) a ) b a bb b b+x b Bx + A b + x a b + x ) a n) ) a ).. Taking the derivative with resect to x, setting it to, and solving for x, we get ) a+ c x c + /a a = b ) a+ c + ) a a ) a ) ) a+ c c c + cn N) a + ) + a) This exression can be simlified by dividing both the numerator and denominator by and ) a+ c then by + a. Letting r = c, the exression becomes x = b r [ ] + N n) + ar a a a + )r a a+ a. )

26 REFERENCES 26 References Agrawal, N. & Smith, S. 996), Estimating negative binomial demand for retail inventory management with unobservable lost sales, Naval Research Logistics 436), Azoury, K. 985), Bayes Solution to Dynamic Inventory Models under Unknown Demand, Management Science 39), , 9 Bensoussan, A., Çakanyıldırım, M. & Sethi, S. 27), A Multieriod Newsvendor Problem with Partially Observed Demand, Mathematics of Oerations Research 322), Berk, E., Gürler, Ü. & Levine, R. 27), Bayesian demand udating in the lost sales newsvendor roblem: A two-moment aroximation, Euroean Journal of Oerational Research 82), Clark, A. & Scarf, H. 96), Otimal Policies for a Multi-Echelon Inventory Problem, Management Science 64), , 9 Ding, X., Puterman, M. & Bisi, A. 22), The Censored Newsvendor and the Otimal Acquisition of Information, Oerations Research 53), Ferguson, T. & Phadia, E. 979), Bayesian nonarametric estimation based on censored data, Ann. Statist 7), , 5, 7 Frazier, P., Powell, W. B. & Dayanik, S. 28), A knowledge-gradient olicy for sequential information collection, SIAM Journal on Control and Otimization. 3 Godfrey, G. & Powell, W. 2), An adative, distribution-free aroximation for the newsvendor roblem with censored demands, with alications to inventory and distribution roblems, Management Science 478), 2. 3 Lariviere, M. & Porteus, E. 999), Stalking Information: Bayesian Inventory Management with Unobserved Lost Sales, Management Science 453), , 4, 5, 9, 23 Nahmias, S. & Smith, S. 994), Otimizing Inventory Levels in a Two-Echelon Retailer System with Partial Lost Sales, Management Science 45), Royden, H. 988), Real Analysis, Prentice Hall, Englewood Cliffs, NJ. 6 Ryzhov, I., Powell, W. B. & Frazier, P. 28), The knowledge-gradient algorithm for a general class of online learning roblems, submitted to Oerations Research. 3 Scarf, H. 959), Bayes Solutions of the Statistical Inventory Problem, The Annals of Mathematical Statistics 32), , 3