Using Stochastic Approximation Methods to Compute Optimal Base-Stock Levels in Inventory Control Problems

Transcription

1 Using Stochastic Approximation Mthods to Comput Optimal Bas-Stoc Lvls in Invntory Control Problms Sumit Kunnumal School of Oprations Rsarch and Information Enginring, Cornll Univrsity, Ithaca, Nw Yor 14853, USA Husyin Topaloglu School of Oprations Rsarch and Information Enginring, Cornll Univrsity, Ithaca, Nw Yor 14853, USA May 18, 2007 Abstract In this papr, w considr numrous invntory control problms for which th bas-stoc policis ar nown to b optimal and w propos stochastic approximation mthods to comput th optimal bas-stoc lvls. Th xisting stochastic approximation mthods in th litratur guarant that thir itrats convrg, but not ncssarily to th optimal bas-stoc lvls. In contrast, w prov that th itrats of our mthods convrg to th optimal bas-stoc lvls. Morovr, our mthods continu to njoy th wll-nown advantags of th xisting stochastic approximation mthods. In particular, thy only rquir th ability to obtain sampls of th dmand random variabls, rathr than to comput xpctations xplicitly and thy ar applicabl vn whn th dmand information is cnsord by th amount of availabl invntory.

2 1 Introduction On approach for finding good solutions to stochastic optimization problms is to concntrat on a class of policis that ar charactrizd by a numbr of paramtrs and to find a good st of valus for ths paramtrs by using stochastic approximation mthods. This approach is quit flxibl. W only nd a snsibl guss at th form of a good policy and stochastic approximation mthods allow us to wor with sampls of th undrlying random variabls, rathr than to comput xpctations. Consquntly, paramtrizd policis along with stochastic approximation mthods ar widly usd in practic. In this papr, w analyz stochastic approximation mthods for svral invntory control problms for which th bas-stoc policis ar nown to b optimal. For ths problms, thr xist bas-stoc lvls r 1,..., r τ such that it is optimal to p th invntory position at tim priod t as clos as possibl to r t. In othr words, ltting x t b th invntory position at tim priod t and [x] + = maxx, 0, it is optimal to ordr [r t x t ] + units of invntory at tim priod t. This particular structur of th optimal policy gnrally ariss from th fact that th valu functions in th dynamic programming formulations of ths problms ar convx in th invntory position. In this cas, th computation of th optimal bas-stoc lvls through th Bllman quations rquirs solving a numbr of convx optimization problms. On th othr hand, w los th appaling structur of th Bllman quations whn w try to comput th optimal bas-stoc lvls by using th xisting stochastic approximation mthods in th litratur. As a rsult, th xisting stochastic approximation mthods can only guarant that thir itrats convrg, but not ncssarily to th optimal bas-stoc lvls. Our main goal is to dvlop stochastic approximation mthods that can indd comput th optimal bas-stoc lvls. To illustrat th difficultis, w considr a two-priod nwsvndor problm with bacloggd dmands, zro lad tims for th rplnishmnts, and linar holding and baclogging costs. For this problm, it is nown that th bas-stoc policis ar optimal undr fairly gnral assumptions. Assuming that th purchasing cost is zro and th initial invntory position is x 1, th total xpctd cost incurrd by a bas-stoc policy charactrizd by th bas-stoc lvls r 1, r 2 can b writtn as g(x 1, r 1, r 2 ) = h E [(x 1 r 1 ) d 1 ] + + [max(x 1 r 1 ) d 1, r 2 d 2 ] + + b E [d 1 (x 1 r 1 )] + + [d 2 max(x 1 r 1 ) d 1, r 2 ] +, whr d 1, d 2 ar th dmand random variabls in th two tim priods, h is th pr unit holding cost and b is th pr unit baclogging cost, and w lt x y = maxx, y. Sinc th invntory position aftr th rplnishmnt dcision at th first tim priod is x 1 r 1 and th invntory position aftr th rplnishmnt dcision at th scond tim priod is max(x 1 r 1 ) d 1, r 2, th two xpctations abov rspctivly comput th total xpctd holding and baclogging costs. In this cas, th optimal bas-stoc lvls can b found by solving th problm (r 1, r 2) = argmin (r 1,r 2 ) g(x 1, r 1, r 2 ). (1) On approach to solv this problm is to us stochastic gradints of g(x 1,, ) to itrativly sarch for a good st of bas-stoc lvls. Undr crtain assumptions, it is possibl to show that th itrats of 2

3 Figur 1: Total xpctd cost as a function of th bas-stoc lvls for a two-priod nwsvndor problm. Th problm paramtrs ar x 1 = 0, h = 0.25, b = 0.4, d 1 bta(1, 5), d 2 bta(5, 1). such a stochastic approximation mthod convrg to a stationary point of g(x 1,, ) with probability 1 (w.p.1). Howvr, g(x 1,, ) is not ncssarily a convx function. In particular, a stationary point of g(x 1,, ) may not b an optimal solution to problm (1) and th solution obtaind by a stochastic approximation mthod may not b vry good. For xampl, Figur 1 shows th plot of g(x 1,, ) for a particular problm instanc whr g(x 1,, ) is not convx. This is a rathr surprising obsrvation. If w assum nothing about th structur of th optimal policy and comput it through th Bllman quations, thn th problm is wll-bhaving in th sns that all w nd to do is to solv a numbr of convx optimization problms. On th othr hand, if w xploit th information that th bas-stoc policis ar optimal and us stochastic approximation mthods to solv problm (1), thn w can only obtain a stationary point of g(x 1,, ). In this papr, w mainly considr variants of th multi-priod nwsvndor problm for which th bas-stoc policis ar nown to b optimal. Nvrthlss, our rsults ar fairly gnral and thy can b applid on othr problm classs whos optimal policis ar charactrizd by a finit numbr of bas-stoc lvls. To illustrat this point, w also considr a rlativly nonstandard invntory purchasing problm whr th pric of th product changs randomly ovr tim and w hav to purchas a crtain amount of product to satisfy th random dmand that occurs at th nd of th planning horizon. Although th problms that w wor with ar wll-studid, our papr mas svral substantial contributions. Most importantly, w offr a rmdy for th aformntiond surprising obsrvation by showing that it is indd possibl to comput th optimal bas-stoc lvls through stochastic approximation mthods. Apart from its thortical valu, this rsult allows us to xploit th wll-nown advantags of stochastic approximation mthods whn computing th optimal bas-stoc lvls. A primary advantag of stochastic approximation mthods is that thy allow woring dirctly with th sampls of th dmand random variabls, rathr than th full dmand distributions. In contrast, 3

4 using convntional invntory control modls typically rquirs thr stps. First, th historical dmand data ar collctd. If th historical dmand data includ only th amount of invntory sold but not th amount of dmand, thn w hav a situation whr th dmand information is cnsord by th amount of availabl invntory, and th historical dmand data hav to b uncnsord to hav accss to th sampls of th dmand random variabls. Aftr this, paramtric forms for th dmand distributions ar hypothsizd and th paramtrs ar fittd by using th historical dmand data. Finally, th optimal bas-stoc lvls ar computd undr th assumption that th fittd dmand distributions charactriz th dmand random variabls. In practic, th historical dmand data ar oftn uncnsord by using huristic approachs. Furthrmor, th hypothsizd forms for th dmand distributions usually do not charactriz th dmand random variabls accuratly, causing rrors just bcaus wrong distributions ar hypothsizd to bgin with. On th othr hand, stochastic approximation mthods wor dirctly with th amount of invntory sold, rathr than th amount of dmand. Thrfor, thy do not rquir uncnsoring th historical dmand data. Also, sinc stochastic approximation mthods wor dirctly with th sampls, thy do not rquir hypothsizing paramtric forms for th dmand distributions. Th advantags mntiond in th prvious paragraph unfortunatly com at a cost. On difficulty with stochastic approximation mthods is th choic of th stp siz paramtrs. In gnral, choosing th stp siz paramtrs rquirs som xprimntation, and thr ar no hard and fast ruls for maing th choic. Although this difficulty is always a major concrn, our stochastic approximation mthods appar to b rlativly robust to th choic of th stp siz paramtrs. In particular, w wor with many diffrnt problm classs, dmand distributions and cost paramtrs in our numrical xprimnts, but w us th sam st of stp siz paramtrs throughout. Th sam st of stp siz paramtrs appar to wor wll in all of our numrical xprimnts. Anothr difficulty with stochastic approximation mthods is th choic of th initial bas-stoc lvls. A rough obsrvation from our numrical xprimnts is that if our stochastic approximation mthods start with bas-stoc lvls having 80% optimality gap, thn thy ta 10 to 40 itrations to obtain bas-stoc lvls having 10% optimality gap. This prformanc itslf may b satisfactory in crtain sttings, but w not that this prformanc is obtaind without xploiting prior information about th dmand distributions. In practic, w usually us som prior information to choos bttr initial bas-stoc lvls and th rol of stochastic approximation mthods bcoms that of only fin-tuning th bas-stoc lvls by using th dmand sampls. W also not that vn if w hav accss to th dmand distributions, numrically solving th Bllman quations rquirs discrtization whn th dmand distributions ar continuous. Undr rasonabl assumptions, th bas-stoc lvls obtaind by discrtizing th dmand distributions convrg to th optimal bas-stoc lvls as th discrtization bcoms finr, but our stochastic approximation mthods can b usd as an altrnativ mthod for computing th optimal bas-stoc lvls. Th rmaindr of th papr is organizd as follows. Sction 2 brifly rviws th rlatd litratur. Sctions 3 and 4 considr th multi-priod nwsvndor problm rspctivly with bacloggd dmands and lost sals, and dvlop stochastic approximation mthods to comput th optimal bas-stoc lvls. Sction 5 shows that th proposd stochastic approximation mthods ar applicabl whn th dmand information is cnsord. Sction 6 dvlops a stochastic approximation mthod for an invntory pur- 4

5 chasing problm whr w ma purchasing dcisions for a product whos pric changs randomly ovr tim and w us th product to satisfy th random dmand at th nd of th planning horizon. Sction 7 prsnts numrical xprimnts. 2 Rlvant Litratur In this papr, w mainly considr th multi-priod nwsvndor problm with bacloggd dmands or lost sals. Th problm involvs controlling th invntory of a prishabl (or fashion) product ovr a finit numbr of tim priods. W hav multipl opportunitis to rplnish th invntory of th product bfor th product rachs th nd of its usful lif. A classical xampl is controlling th invntory of a monthly magazin. W ar allowd to rplnish th magazins multipl tims during th cours of a month, but th lft ovr magazins at th nd of a month ar discardd, possibly at a salvag valu. For th multi-priod nwsvndor problm with lost sals, w assum that th lad tims for th rplnishmnts ar zro. All cost functions w dal with ar linar, although gnralizations to convx cost functions ar possibl. Th optimality of th bas-stoc policis for th variants of th multi-priod nwsvndor problm that w considr is wll-nown; s Arrow, Karlin and Scarf (1958), Portus (1990) and Zipin (2000). If th distribution of th dmand is nown, thn th optimal bas-stoc lvls can b computd through th Bllman quations. Significant litratur has volvd around th nwsvndor problm undr th assumption that th distribution of th dmand is unnown. Thr may b diffrnt rasons for mploying such an assumption. For xampl, w may not hav nough data to fit a paramtric dmand distribution or it may b difficult to collct dmand data sinc w ar only abl to obsrv th amount of invntory sold, but not th amount of dmand. Scarf (1960), Iglhart (1964) and Azoury (1985) us Baysian framwor to stimat th dmand paramtrs and to adaptivly updat th rplnishmnt quantitis as th dmand information bcoms availabl. Lvi, Roundy and Shmoys (2005) provid bounds on how many dmand sampls ar ndd to obtain nar-optimal bas-stoc lvls with high probability. Conrad (1976), Bradn and Frimr (1991) and Ding (2002) focus on th cas whr th dmand information is cnsord by th amount of availabl invntory. Godfry and Powll (2001) giv a nic ovrviw of th nwsvndor problm with cnsord dmands. Stochastic approximation mthods can dal with th uncrtainty in th distribution of th dmand and th cnsord dmand information. Thy only rquir th ability to obtain sampls from th dmand distributions. Furthrmor, thy usually do not rquir having accss to th xact valus of th dmand sampls. Instad, only nowing th amount of invntory sold is oftn adquat. Consquntly, stochastic approximation mthods can b usd undr th assumption that a paramtric form for th dmand distribution is not availabl or th dmand information is cnsord by th invntory availability. Th us of stochastic approximation mthods for solving stochastic optimization problms is wllnown. Kushnr and Clar (1978) and Brtsas and Tsitsilis (1996) giv a dtaild covrag of stochastic approximation mthods. As far as th applications ar concrnd, L Ecuyr and Glynn (1994), Fu (1994), Glassrman and Tayur (1995), Bashyam and Fu (1998), Mahajan and van Ryzin (2001), 5

6 Karasmn and van Ryzin (2004) and van Ryzin and Vulcano (2006) focus on quuing, invntory control and rvnu managmnt sttings. Although th objctiv functions that ar considrd in many of ths paprs ar not convx and w can only guarant convrgnc to th stationary points, computational xprinc indicats that stochastic approximation mthods provid good solutions in practic; s Mahajan and van Ryzin (2001) and van Ryzin and Vulcano (2006). Th traditional approach in th stochastic approximation litratur is to concntrat on a class of policis that ar charactrizd by a numbr of paramtrs. Th hop is that this class of policis contain at last on good policy for th problm. In contrast, thr ar numrous rinforcmnt larning mthods that try to avoid this shortcoming by xplicitly approximating th valu functions in th dynamic programming formulation of th problm. Q-larning and tmporal diffrncs larning us sampld stat trajctoris to approximat th valu functions in problms with discrt stat and dcision spacs; s Sutton (1988) and Tsitsilis (1994). Godfry and Powll (2001), Topaloglu and Powll (2003) and Powll, Ruszczynsi and Topaloglu (2004) propos sampling-basd mthods to approximat picwislinar convx valu functions and ths mthods ar convrgnt for crtain stationary problms. Th stochastic approximation mthods that w propos in this papr mbody th charactristics of th two typs of approachs mntiond in th last two paragraphs. Similar to th standard stochastic approximation mthods, w concntrat on th class of policis that ar charactrizd by a finit numbr of bas-stoc lvls, whras similar to th valu function approximation mthods, w wor with th dynamic programming formulation of th problm to sarch for th optimal bas-stoc lvls. 3 Multi-Priod Nwsvndor Problm with Bacloggd Dmands W want to control th invntory of a product ovr th tim priods 1,..., τ. At tim priod t, w obsrv th invntory position x t and plac a rplnishmnt ordr of y t x t units, which costs c pr unit. Th rplnishmnt ordr arrivs instantanously and raiss th invntory position to y t. Following th arrival of th rplnishmnt, w obsrv th random dmand d t and satisfy th dmand as much as possibl. W incur a cost of h pr unit of hld invntory pr tim priod and a cost of b pr unit of unsatisfid dmand pr tim priod. W assum that th rvnu from th sals is zro without loss of gnrality. Th goal is to minimiz th total xpctd cost ovr th planning horizon. Throughout, w assum that th dmand random variabls d t : t = 1,..., τ ar indpndnt and hav finit xpctations, and thir cumulativ distribution functions ar Lipschitz continuous. W assum that th cost paramtrs satisfy b > c 0 and h 0. Th assumption that th cost paramtrs ar stationary and th lad tims for th rplnishmnts ar zro is for notational brvity. It is also possibl to xtnd our analysis to th cas whr th distributions of th dmand random variabls ar discrt. W not that th dmand random variabls do not hav to b idntically distributd. W lt v t (x t ) b th minimum total xpctd cost incurrd ovr th tim priods t,..., τ whn th invntory position at tim priod t is x t and th optimal policy is followd ovr th tim priods t,..., τ. Th functions v t ( ) : t = 1,..., τ satisfy th Bllman quations v t (x t ) = min y t x t c [y t x t ] + E h [y t d t ] + + b [d t y t ] + + v t+1 (y t d t ), (2) 6

7 with v τ+1 ( ) = 0. If w lt f t (r t ) = c r t + E h [r t d t ] + + b [d t r t ] + + v t+1 (r t d t ), (3) thn it can b shown that f t ( ) is a convx function with a finit unconstraind minimizr, say r t. In this cas, it is wll-nown that th optimal policy is a bas-stoc policy charactrizd by th bas-stoc lvls r t : t = 1,..., τ. That is, if th invntory position at tim priod t is x t, thn it is optimal to ordr [rt x t ] + units. Thrfor, w can writ (2) as E h [x t d t ] + + b [d t x t ] + + v t+1 (x t d t ) v t (x t ) = c [rt x t ] + E h [rt d t ] + + b [d t rt ] + + v t+1 (rt d t ) f t (x t ) c x t if x t rt = f t (rt ) c x t if x t < rt. if x t r t if x t < r t It can b shown that f t ( ) and v t ( ) ar positiv, Lipschitz continuous, diffrntiabl and convx functions. W us f t ( ) and v t ( ) to rspctivly dnot th drivativs of f t ( ) and v t ( ). Th following lmma shows that f t ( ) and v t ( ) ar also Lipschitz continuous. (4) Lmma 1 Thr xists a constant L such that w hav f t (ˆx t ) f t ( x t ) L ˆx t x t and v t (ˆx t ) v t ( x t ) L ˆx t x t for all ˆx t, x t R, t = 1,..., τ. Proof W show th rsult by induction ovr th tim priods. Sinc v τ+1 ( ) = 0, this function is Lipschitz continuous. W assum that v t+1 ( ) is Lipschitz continuous. W hav f t (x t ) = c + h P d t < x t b P dt x t + E vt+1 (x t d t ), (5) whr th intrchang of th xpctation and th drivativ abov follows from Lmma in Glassrman (1994). Sinc th composition of Lipschitz continuous functions is also Lipschitz continuous by Lmma in Glassrman (1994), f t ( ) is Lipschitz continuous. To s that v t ( ) is Lipschitz continuous, w us (4) to obtain f t (x t ) c if x t rt v t (x t ) = c if x t < rt. W assum that ˆx t x t without loss of gnrality and considr thr cass. First, w assum that ˆx t r t x t. Sinc r t is th minimizr of f t ( ), w hav f t (r t ) = 0, which implis that v t (ˆx t ) v t ( x t ) = f t (ˆx t ) = f t (ˆx t ) f(r t ) L ˆx t r t L ˆx t x t, whr w us th Lipschitz continuity of f t ( ) in th first inquality. Th othr two cass whr w hav ˆx t x t > r t or r t > ˆx t x t ar asy to show. W now considr computing th optimal bas-stoc lvls r t : t = 1,..., τ through a stochastic approximation mthod. Noting (5) and using 1( ) to dnot th indicator function, w can comput a stochastic gradint of f t ( ) at x t through t (x t, d t ) = c + h 1(d t < x t ) b 1(d t x t ) + v t+1 (x t d t ). (7) (6) 7

8 In this cas, ltting r t : t = 1,..., τ b th stimats of th optimal bas-stoc lvls at itration, d t : t = 1,..., τ b th dmand random variabls at itration and α b a stp siz paramtr, w can itrativly updat th stimats of th optimal bas-stoc lvls through r +1 t = r t α t (r t, d t ). (8) Howvr, this approach is clarly not ralistic bcaus th computation in (7) rquirs th nowldg of v t ( ) : t = 1,..., τ. Th stochastic approximation mthod w propos is basd on constructing tractabl approximations to th stochastic gradints of f t ( ) : t = 1,..., τ. Sinc r t is th minimizr of f t ( ), (5) implis that c = f t (r t ) c = h P d t < r t b P dt rt + E vt+1 (rt d t ). Thrfor, using (5) and (6), w obtain h P d t < x t b P dt x t + E vt+1 (x t d t ) if x t rt v t (x t ) = h P d t < rt b P dt rt + E vt+1 (rt d t ) if x t < rt. (9) From this xprssion, it is clar that h 1(d t < x t ) b 1(d t x t ) + v t+1 (x t d t ) v t (x t, d t ) = h 1(d t < rt ) b 1(d t rt ) + v t+1 (rt d t ) if x t r t if x t < r t (10) givs a stochastic gradint of v t ( ) at x t, satisfying v t (x t ) = E v t (x t, d t ). To construct tractabl approximations to th stochastic gradints of f t ( ) : t = 1,..., τ, w mimic th computation in (10) by using th stimats of th optimal bas-stoc lvls. In particular, ltting r t : t = 1,..., τ b th stimats of th optimal bas-stoc lvls at itration, w rcursivly dfin t h 1(d t < x t ) b 1(d t x t ) + t+1 (x t, d t,..., d τ ) = (x t d t, d t+1,..., d τ ) if x t rt h 1(d t < rt ) b 1(d t rt ) + t+1 (r t d t, d t+1,..., d τ ) if x t < rt, with τ+1 (,,..., ) = 0. At itration, rplacing v t+1(x t d t ) in (7) with t+1 (x t d t, d t+1,..., d τ ), w approximat a stochastic gradint of f t ( ) at x t by using s t (x t, d t,..., d τ ) = c + h 1(d t < x t ) b 1(d t x t ) + t+1(x t d t, d t+1,..., d τ ). (12) Consquntly, w propos th following algorithm to sarch for th optimal bas-stoc lvls. Algorithm 1 Stp 1. Initializ th stimats of th optimal bas-stoc lvls r 1 t : t = 1,..., τ arbitrarily. Initializ th itration countr by stting = 1. Stp 2. Ltting d t : t = 1,..., τ b th dmand random variabls at itration, st for all t = 1,..., τ. Stp 3. Incras by 1 and go to Stp 2. r +1 t = r t α s t (r t, d t,..., d τ ) W lt F b th filtration gnratd by r1 1,..., r1 τ, d 1 1,..., d1 τ,..., d 1 1,..., d 1 τ. Givn F, w assum that th conditional distribution of d t : t = 1,..., τ is th sam as th distribution of 8 (11)

9 d t : t = 1,..., τ. For notational brvity, w us E to dnot xpctations and P to dnot probabilitis conditional on F. W assum that th stp siz paramtr α is F -masurabl, in which cas th stimats of th optimal bas-stoc lvls r t : t = 1,..., τ ar also F -masurabl. Comparing (7) and (12) indicats that if th functions E t+1 (, d t+1,..., d τ ) and v t+1 ( ) ar clos to ach othr, thn th stp dirctions E s t (, d t,..., d τ ) and E t (, d t ) ar clos to ach othr, in which cas using s t (r t, d t,..., d τ ) instad of t (r t, d t ) dos not bring too much rror. In fact, our convrgnc proof is havily basd on analyzing th rror function v t ( ) E t (, d t,..., d τ ). In this sction, w show that lim f t (r t ) = 0 w.p.1 for all t = 1,..., τ for a squnc of bas-stoc lvls r t : t = 1,..., τ gnratd by Algorithm 1 and th total xpctd cost of th policy that uss th bas-stoc lvls rt : t = 1,..., τ convrgs to th total xpctd cost of th optimal policy as. W bgin with svral prliminary lmmas. 3.1 Prliminaris In th following lmma, w driv bounds on t (, d t,..., d τ ) and s t (, d t,..., d τ ). Lmma 2 Thr xists a constant M such that t (x t, d t,..., d τ ) M and s t (x t, d t,..., d τ ) M w.p.1 for all x t R, t = 1,..., τ, = 1, 2,.... Proof W lt N = maxc, h, b. W show by induction ovr th tim priods that t (x t, d t,..., d τ ) 2 [τ t + 1] N w.p.1 for all x t R, t = 1,..., τ, = 1, 2,.... Th rsult holds for tim priod τ by (11). Assuming that th rsult holds for tim priod t + 1, w hav t (x t, d t,..., d τ ) h + b + 2 [τ t] N 2 [τ t + 1] N w.p.1 and this stablishs th rsult. Thrfor, w hav s t (x t, d t,..., d τ ) c + h + b + 2 [τ t] N 2 [τ t + 2] N w.p.1 by (12). Th rsult follows by ltting M = 2 [τ + 1] N. W not that v t ( ), bing th drivativ of th convx function v t ( ), is incrasing. Th following lmma shows that E t (, d t,..., d τ ) also satisfis this proprty. Lmma 3 If ˆx t, x t satisfy ˆx t x t, thn w hav E t (ˆx t, d t,..., d τ ) E t ( x t, d t,..., d τ ) w.p.1 for all t = 1,..., τ, = 1, 2,.... Proof W show th rsult by induction ovr th tim priods. W first show that th rsult holds for tim priod τ. W considr thr cass. First, w assum that rτ ˆx τ x τ. Using (11), w hav E τ (ˆx τ, d τ ) = h P d τ < ˆx τ b P d τ ˆx τ = [h + b] P d τ < ˆx τ b [h + b] P d τ < x τ b = E τ ( x τ, d τ ). Scond, w assum that ˆx τ < rτ x τ. In this cas, (11) and th argumnt in th prvious sntnc imply that E τ (ˆx τ, d τ ) = E τ (rτ, d τ ) E τ ( x τ, d τ ). Third, w assum that ˆx τ x τ < rτ. W hav E τ (ˆx τ, d τ ) = E τ (rτ, d τ ) = E τ ( x τ, d τ ). Thrfor, th rsult holds for tim priod τ. Assuming that th rsult holds for tim priod t + 1, it is asy to chc in a similar fashion that th rsult holds for tim priod t by considring th thr cass rt ˆx t x t or ˆx t < rt x t or ˆx t x t < rt. 9

10 As mntiond abov, our convrgnc proof analyzs th rror function v t ( ) E t (, d t,..., d τ ) xtnsivly. For notational brvity, w lt t (x t ) = v t (x t ) E t (x t, d t,..., d τ ), (13) with τ+1 ( ) = 0. In th following lmma, w stablish a bound on th rror function. This rsult is a dirct implication of th fact that f t ( ) is convx and E t (, d t,..., d τ ) is incrasing. Lmma 4 W hav t (x t ) max f t (r t ) E t+1 (r t d t ), E t+1 (r t d t ), E t+1 (x t d t ) (14) w.p.1 for all x t R, t = 1,..., τ, = 1, 2,.... Proof Using (5) and (11), w obtain h P d t < x t b P d t x t E t (x t, d t,..., d τ ) + E = t+1 (x t d t, d t+1,..., d τ ) if x t rt h P d t < r t b P d t r t + E t+1 (rt d t, d t+1,..., d τ ) if x t < rt f t (x t ) c E vt+1 (x t d t ) + E = t+1 (x t d t, d t+1,..., d τ ) if x t rt f t (r t ) c E vt+1 (rt d t ) + E t+1 (rt d t, d t+1,..., d τ ) if x t < rt. (15) W considr four cass. First, w assum that x t r t and x t r t. Using (6) and (15), w hav t (x t ) = E vt+1 (x t d t ) E t+1 (x t d t, d t+1,..., d τ ) = E t+1 (x t d t ). Thrfor, w obtain t (x t ) E t+1 (x t d t ) by Jnsn s inquality. Scond, w assum that x t r t and x t < r t. W hav E t (x t, d t,..., d τ ) E t (r t, d t,..., d τ ) by Lmma 3. Using this inquality, (6) and (15), w obtain t (x t ) = c E t (x t, d t,..., d τ ) c E t (r t, d t,..., d τ ) = f t (r t ) + E vt+1 (r t d t ) E t+1 (r t d t, d t+1,..., d τ ) = f t (r t ) + E t+1 (r t d t ). Sinc x t < r t and r t is th minimizr of th convx function f t ( ), w hav f t (x t ) 0. Using (15), w also obtain t (x t ) = c E t (x t, d t,..., d τ ) = f t (x t ) + E vt+1 (x t d t ) E t+1 (x t d t, d t+1,..., d τ ) E t+1 (x t d t ). Th last two chains of inqualitis imply that t (x t ) max f t (r t ) E t+1 (r t d t ), E t+1 (x t d t ). 10

11 Third, w assum that x t < r t and x t r t. Sinc f t ( ) is convx, w hav f t (r t ) f t (x t ) f t (rt ) = 0. Using (6) and (15), w obtain t (x t ) = f t (x t ) f t (rt ) + E vt+1 (rt d t ) E t+1 (rt d t, d t+1,..., d τ ) = f t (x t ) f t (r t ) + E t+1 (r t d t ), which implis that Thrfor, w obtain f t (r t ) + E t+1 (r t d t ) t (x t ) E t+1 (r t d t ). t (x t ) max f t (r t ) E t+1 (r t d t ), E t+1 (r t d t ). Fourth, w assum that x t < rt and x t < rt. In this cas, (6) and (15) imply that t (x t ) = f t (rt )+E vt+1 (rt d t ) E t+1 (rt d t, d t+1,..., d τ ) = f t (rt )+E t+1 (rt d t ). Thrfor, w obtain t (x t ) = f t (rt ) E t+1 (rt d t ). Th rsult follows by combining th four cass. 3.2 Convrgnc Proof W hav th following convrgnc rsult for Algorithm 1. Proposition 5 Assum that th squnc of stp siz paramtrs α satisfy α 0 for all = 1, 2,..., =1 α = and =1 [α ] 2 < w.p.1. If th squnc of bas-stoc lvls rt : t = 1,..., τ ar gnratd by Algorithm 1, thn th squnc f t (rt ) convrgs w.p.1 for all t = 1,..., τ and w hav lim f t (rt ) = 0 w.p.1 for all t = 1,..., τ. Proof All statmnts in th proof ar in w.p.1 sns. W us induction ovr th tim priods to show that th following rsults hold for all t = 1,..., τ. (A.1) Th squnc f t (rt ) convrgs. (A.2) W hav =1 α[ f t (rt ) 2 f t (rt ) E t+1 (rt d t ) ] + <. (A.3) W hav lim f t (rt ) = 0. (A.4) W hav t (x t ) τ s=t f s (rs ) for all x t R, = 1, 2,.... (A.5) Thr xists a constant A t such that w hav for all x t R, = 1, 2,.... t (x t ) 2 A t s=t [ f s (r s ) 2 f s (r s ) E s+1 (r s d s) ] + (16) W bgin by showing that (A.1)-(A.5) hold for tim priod τ. Sinc w hav rτ +1 = rτ α s τ (rτ, d τ ), using Lmma 1 and th Taylor sris xpansion of f τ ( ) at rτ +1, a standard argumnt yilds f τ (r +1 τ ) f τ (r τ ) α f τ (r τ ) s τ (r τ, d τ ) [α ] 2 L s τ (r τ, d τ ) 2 ; (17) s (3.39) in Brtsas and Tsitsilis (1996). Sinc w hav E s τ (rτ, d τ ) = c + h P d τ < rτ b P d τ rτ = f τ (rτ ), taing xpctations in (17) and using Lmma 2 yild E fτ (rτ +1 ) f τ (rτ ) α [ f τ (rτ )] [α ] 2 L M 2. (18) 11

12 Sinc f τ ( ) is positiv and =1 [α ] 2 <, w can now us th suprmartingal convrgnc thorm to conclud that th squnc f τ (rτ ) convrgs and =1 α [ f τ (rτ )] 2 < ; s Nvu (1975). Thrfor, sinc τ+1 ( ) = 0 by dfinition, (A.1) and (A.2) hold for tim priod τ. Sinc w hav E s τ (rτ, d τ ) = f τ (rτ ), th itration rτ +1 = rτ α s τ (rτ, d τ ) is a standard stochastic approximation mthod to minimiz f τ ( ) and w hav lim f τ (rτ ) = 0; s Proposition 4.1 in Brtsas and Tsitsilis (1996). Thrfor, (A.3) holds for tim priod τ. Sinc τ+1 ( ) = 0, Lmma 4 shows that (A.4) and (A.5) hold for tim priod τ. Thrfor, (A.1)-(A.5) hold for tim priod τ. Assuming that (A.1)-(A.5) hold for tim priods t + 1,..., τ, Lmmas 6-8 blow show that (A.1)- (A.5) also hold for tim priod t. This complts th induction argumnt, and th rsult follows by (A.1) and (A.3). Lmmas 6-8 complt th induction argumnt givn in th proof of Proposition 5. All statmnts in thir proofs should b undrstood in w.p.1 sns. Lmma 6 If (A.1)-(A.5) hold w.p.1 for tim priods t + 1,..., τ, thn (A.1) and (A.2) hold w.p.1 for tim priod t. Proof Using (5) and (12), w hav E s t (rt, d t,..., d τ ) = c + h P d t < rt b P d t rt + E t+1 (rt d t, d t+1,..., d τ ) = f t (rt ) E v t+1 (rt d t ) + E t+1 (rt d t, d t+1,..., d τ ). Similar to (17) and (18), using th quality abov, Lmma 1 and th Taylor sris xpansion of f t ( ) at rt +1, w hav E ft (r +1 t ) f t (r t ) α f t (r t ) E s t (r t, d t,..., d τ ) [α ] 2 L M 2 = f t (r t ) α f t (r t ) [ f t (r t ) E t+1 (r t d t ) ] [α ] 2 L M 2. Ltting X = α f t (rt ) [ f t (rt ) E t+1 (rt d t ) ], th xprssion abov is of th form E ft (rt +1 ) f t (rt ) [X ] + + [ X ] + + [α ] 2 L M 2 /2. Thrfor, if w can show that =1 [ X ] + <, thn w can us th suprmartingal convrgnc thorm to conclud that th squnc f t (rt ) convrgs and =1 [X ] + <. W now show that =1 [ X ] + <. If [ X ] + > 0, thn w hav 0 [ f t (r t )] 2 < f t (r t ) E t+1 (r t d t ) f t (r t ) E t+1 (r t d t ). Dividing th xprssion abov by f t (r t ), w obtain f t (r t ) < E t+1 (r t d t ). Thrfor, having [ X ] + > 0 implis that [ X ] + = α [ f t (r t ) E t+1 (r t d t ) [ f t (r t )] 2] + α [ f t (r t ) E t+1 (r t d t ) ] + α f t (r t ) E t+1 (r t d t ) α E t+1 (r t d t ) 2 α E t+1 (r t d t ) 2. 12

13 W not that th xprssion on th right sid of (16) dos not dpnd x t and it is F -masurabl. In this cas, using th chain of inqualitis abov and th induction hypothsis (A.5), w obtain [ X ] + = =1 1([ X ] + > 0) [ X ] + =1 =1 s=t+1 α E t+1 (rt d t ) 2 =1 α A t+1 [ f s (r s ) 2 f s (r s ) E s+1 (r s d s) ] + = s=t+1 =1 α A t+1 [ f s (r s ) 2 f s (r s ) E s+1 (r s d s) ] + <, whr xchanging th ordr of th sums in th scond quality is justifid by Fubini s thorm and th last inquality follows from th induction hypothsis (A.2). Thrfor, w can us th suprmartingal convrgnc thorm to conclud that f t (rt ) convrgs and =1 [X ] + <, which is to say that =1 α[ f t (rt ) 2 f t (rt ) E t+1 (rt d t ) ] + <. Lmma 7 If (A.1)-(A.5) hold w.p.1 for tim priods t + 1,..., τ, thn (A.3) holds w.p.1 for tim priod t. Proof W first show that liminf f t (rt ) = 0. By th induction hypothsis (A.4), w hav t+1 (r t d t ) τ s=t+1 f s (rs ). Taing xpctations and limits, and using th induction hypothsis (A.3), w obtain lim E t+1 (rt d t ) = 0. Thrfor, for givn ɛ > 0, thr xists a finit itration numbr K such that E t+1 (r t d t ) 2 ɛ for all = K, K + 1,.... By Lmma 6, (A.2) holds for tim priod t. Sinc w hav =1 α =, (A.2) implis that [ liminf f t (rt ) 2 f t (rt ) E t+1 (rt d t ) ] + [ = 0. In particular, w must hav f t (rt ) 2 f t (rt ) E t+1 (rt d t ) ] + 3 ɛ 2 for infinit numbr of itrations. Thrfor, aftr itration numbr K, w must hav f t (rt ) 2 2 f t (rt ) ɛ f t (rt ) 2 f t (rt ) E t+1 (rt d t ) 3 ɛ 2 for infinit numbr of itrations, which implis that f t (rt ) [ ɛ, 3 ɛ] for infinit numbr of itrations. Sinc ɛ is arbitrary, w obtain liminf f t (rt ) = 0. By xamining th so-calld upcrossings of th intrval [ɛ/2, ɛ] by th squnc f t (r t ) and following an argumnt similar to th on usd to show Proposition 4.1 in Brtsas and Tsitsilis (1996), w can also show that limsup f t (r t ) = 0 and this stablishs th rsult. W dfr th proof of this part to th appndix. Lmma 8 If (A.1)-(A.5) hold w.p.1 for tim priods t + 1,..., τ, thn (A.4) and (A.5) hold w.p.1 for tim priod t. Proof Th induction hypothsis (A.4) implis that t+1 (x t d t ) τ s=t+1 f s (rs ) for all x t R and t+1 (r t d t ) τ s=t+1 f s (rs ). Taing xpctations and using ths xpctations in (14), it is asy 13

14 to s that (A.4) holds for tim priod t. For all x t R, squaring (14) also implis that t (x t ) 2 [ f t (r t )] 2 2 f t (r t ) E t+1 (r t d t ) + [ E t+1 (r t d t ) ] 2 + [ E t+1 (r t d t ) ] 2 + [ E t+1 (x t d t ) ] 2 2 [ [ f t (r t )] 2 f t (r t ) E t+1 (r t d t ) ] E t+1 (r t d t ) 2 + E t+1 (x t d t ) 2 2 [ [ f t (r t )] 2 f t (r t ) E t+1 (r t d t ) ] A t+1 s=t+1 [ f s (r s ) 2 f s (r s ) E s+1 (r s d s) ] +, whr w us th induction hypothsis (A.5) in th last inquality. Ltting A t = max2, 3 A t+1, (A.5) holds for tim priod t. W clos this sction by invstigating th prformancs of th policis charactrizd by th basstoc lvls r t : t = 1,..., τ. Th policy charactrizd by th bas-stoc lvls r t : t = 1,..., τ ps th invntory position at tim priod t as clos as possibl to r t. W lt V t (x t ) b th total xpctd cost incurrd by this policy ovr th tim priods t,..., τ whn th invntory position at tim priod t is x t. Th functions Vt ( ) : t = 1,..., τ satisfy V t (x t ) = E h [xt d t ] + + b [d t x t ] + + V t+1 (x t d t ) if x t r t c [r t x t ] + E h [r t d t ] + + b [d t r t ] + + V t+1 (r t d t ) if x t < r t. In contrast, th function v 1 ( ) givs th minimum total xpctd cost incurrd ovr th tim priods 1,..., τ. Proposition 9 shows that lim V 1 (x 1) v 1 (x 1 ) = 0 w.p.1 for all x 1 R and stablishs that th policis charactrizd by th bas-stoc lvls r t : t = 1,..., τ ar asymptotically optimal. (19) Proposition 9 Assum that th squnc of stp siz paramtrs α satisfy α 0 for all = 1, 2,..., =1 α = and =1 [α ] 2 < w.p.1. If th squnc of bas-stoc lvls rt : t = 1,..., τ ar gnratd by Algorithm 1, thn w hav lim V1 (x 1) v 1 (x 1 ) = 0 w.p.1 for all x 1 R. Proof All statmnts ar in w.p.1 sns. W first show that lim f t (r t ) = f t (r t ) for all t = 1,..., τ. By Proposition 5, th squnc f t (rt ) convrgs, which implis that thr xists a subsqunc r j t j with th limit point ˆr t. Sinc th squnc f t (rt ) convrgs to 0 by Proposition 5, w must hav f t (ˆr t ) = 0. Thrfor, sinc f t ( ) is convx, ˆr t is a minimizr of f t ( ) satisfying f t (ˆr t ) = f t (rt ). In this cas, th subsqunc f t (r j t ) j convrgs to f t (rt ). Sinc th squnc f t (rt ) convrgs, w conclud that this squnc convrgs to f t (rt ). Noting (3), w can writ (19) as V t (x t ) = f t (x t ) c x t + E V t+1 (x t d t ) v t+1 (x t d t ) if x t r t f t (r t ) c x t + E V t+1 (r t d t ) v t+1 (r t d t ) if x t < r t. W now us induction ovr th tim priods to show that 0 Vt (x t ) v t (x t ) τ [ s=t fs (rs ) f s (rs) ] for all x t R, t = 1,..., τ, in which cas th final rsult follows by noting that f t (rt ) convrgs to 14 (20)

15 f t (r t ) for all t = 1,..., τ. It is asy to show th rsult for th last tim priod. Assuming that th rsult holds for tim priod t + 1, w now show that th rsult holds for tim priod t. W considr four cass. Assum that r t x t < r t. Sinc r t is th minimizr of th convx function f t ( ), w hav f t (x t ) f t (r t ). In this cas, using (4), (20) and th induction hypothsis, w obtain 0 V t (x t ) v t (x t ) = f t (x t ) f t (r t ) + E V t+1 (x t d t ) v t+1 (x t d t ) f t (r t ) f t (r t ) + s=t+1 [ fs (r s ) f s (r s) ]. Th othr thr cass whr w hav r t x t < r t, or r t x t and r t x t, or r t x t and r t x t can b shown similarly. 4 Multi-Priod Nwsvndor Problm with Lost Sals This sction shows how to xtnd th idas in Sction 3 to th cas whr th unsatisfid dmand is immdiatly lost. W us th sam assumptions for th cost paramtrs and th dmand random variabls. In particular, w hav b > c 0, h 0 and th dmand random variabls at diffrnt tim priods ar indpndnt, but not ncssarily idntically distributd. Howvr, w nd to strictly impos th assumption that th lad tims for th rplnishmnts ar zro. Othrwis, th bas-stoc policis ar not ncssarily optimal. In addition, our prsntation hr strictly imposs th assumption that th cost paramtrs ar stationary, but th onlin supplmnt xtnds our analysis to th cas whr th cost paramtrs ar nonstationary. Ltting v t (x t ) hav th sam intrprtation as in Sction 3, th functions v t ( ) : t = 1,..., τ satisfy th Bllman quations v t (x t ) = min y t x t c [y t x t ] + E h [y t d t ] + + b [d t y t ] + + v t+1 ([y t d t ] + ), (21) with v τ+1 ( ) = 0. W also lt f t (r t ) = c r t + E h [r t d t ] + + b [d t r t ] + + v t+1 ([r t d t ] + ). It can b shown that v t ( ) and f t ( ) ar positiv, Lipschitz continuous, diffrntiabl and convx functions, and f t ( ) has a finit unconstraind minimizr. r t : t = 1,..., τ ar th minimizrs of th functions f t ( ) : t = 1,..., τ. Sinc w hav In this cas, th optimal bas-stoc lvls f t (r t ) = c + h P d t < r t b P dt r t + E vt+1 (r t d t ) 1(d t < r t ), (22) w can comput a stochastic gradint of f t ( ) at x t through t (x t, d t ) = c + h 1(d t < x t ) b 1(d t x t ) + v t+1 (x t d t ) 1(d t < x t ) (23) and itrativly sarch for th optimal bas-stoc lvls through (8). Howvr, this approach rquirs th nowldg of v t ( ) : t = 1,..., τ. W now us idas similar to thos in Sction 3 to approximat th stochastic gradints of f t ( ) : t = 1,..., τ in a tractabl mannr. 15

16 Using th optimal bas-stoc lvl rt, w writ (21) as E h [x t d t ] + + b [d t x t ] + + v t+1 ([x t d t ] + ) if x t rt v t (x t ) = c [rt x t ] + E h [rt d t ] + + b [d t rt ] + + v t+1 ([rt d t ] + ) if x t < rt. (24) Sinc r t is th minimizr of f t ( ), (22) implis that c = f t (r t ) c = h P d t < r t b P dt r t + E vt+1 (r t d t ) 1(d t < r t ). Thrfor, using this xprssion in (24), w obtain v t (x t ) = h P d t < x t b P dt x t + E vt+1 (x t d t ) 1(d t < x t ) if x t rt h P d t < rt b P dt rt + E vt+1 (rt d t ) 1(d t < rt ) if x t < rt. (25) In this cas, v t (x t, d t ) = clarly givs a stochastic gradint of v t ( ) at x t. h 1(d t < x t ) b 1(d t x t ) + v t+1 (x t d t ) 1(d t < x t ) h 1(d t < r t ) b 1(d t r t ) + v t+1 (r t d t ) 1(d t < r t ) if x t r t if x t < r t (26) To construct tractabl approximations to th stochastic gradints of f t ( ) : t = 1,..., τ, w mimic th computation in (26) by using th stimats of th optimal bas-stoc lvls. In particular, ltting r t : t = 1,..., τ b th stimats of th optimal bas-stoc lvls at itration, w rcursivly dfin h 1(d t < x t ) b 1(d t x t ) t + t+1 (x t, d t,..., d τ ) = (x t d t, d t+1,..., d τ ) 1(d t < x t ) if x t rt h 1(d t < rt ) b 1(d t rt ) + t+1 (r t d t, d t+1,..., d τ ) 1(d t < rt ) if x t < rt, with τ+1 (,,..., ) = 0. At itration, rplacing v t+1(x t d t ) in (23) with t+1 (x t d t, d t+1,..., d τ ), w us s t (x t, d t,..., d τ ) = c + h 1(d t < x t ) b 1(d t x t ) + t+1(x t d t, d t+1,..., d τ ) 1(d t < x t ) (28) to approximat th stochastic gradint of f t ( ) at x t. Thus, w can us Algorithm 1 to sarch for th optimal bas-stoc lvls. Th only diffrnc is that w nd to us th stp dirction abov in Stp 2. Th proof of convrgnc for this algorithm follows from an argumnt similar to th on in Sctions 3.1 and 3.2. In particular, w can follow th proof of Lmma 2 to driv bounds on t (, d t,..., d τ ) and s t (, d t,..., d τ ). It is possibl stablish an analogu of Lmma 3 to show that E t (, d t,..., d τ ) is incrasing. W dfin th rror function as t (x t ) = v t (x t ) 1(x t > 0) E t (x t, d t,..., d τ ) 1(x t > 0), with τ+1 ( ) = 0. In this cas, w can show that th sam bound on th rror function givn in Lmma 4 holds. Onc w hav this bound on th rror function, w can follow th sam induction argumnt in Proposition 5, Lmmas 6-8 and Proposition 9 to show th final rsult. Th dtails of th proof ar givn in th onlin supplmnt. 16 (27)

17 5 Cnsord Dmands This sction considrs th multi-priod nwsvndor problm with lost sals and cnsord dmands. Dmand cnsorship rfrs to th situation whr w only obsrv th amount of invntory sold, but not th amount of dmand. In this cas, our dmand obsrvations ar truncatd whn th amount of dmand xcds th amount of availabl invntory. Our goal is to show that w can still comput th stp dirction in (28), which implis that th algorithm proposd in Sction 4 rmains applicabl whn th dmand information is cnsord. W not if th unsatisfid dmand is bacloggd, thn w can always obsrv th amount of dmand and th cnsord dmand information is not an issu. If th unsatisfid dmand is immdiatly lost and th dmand information is cnsord, thn w do not obsrv th random variabls d t : t = 1,..., τ in Stp 2 of Algorithm 1. Instad, w simulat th bhavior of th policy charactrizd by th bas-stoc lvls r t : t = 1,..., τ and Stp 2 of Algorithm 1 is rplacd by th following stps. Stp 2.a. Initializ th bginning invntory position x 1. St t = 1. Stp 2.b. Plac a rplnishmnt ordr of [rt x t ] + units to rais th invntory position to maxrt, x t. St th invntory position aftr th rplnishmnt ordr as y t = maxr t, x t. Stp 2.c. Comput th invntory position at th nxt tim priod as x t+1 = y t miny t, d t. If t < τ, thn incras t by 1 and go to Stp 2.b. Stp 2.d. St r +1 t = r t α s t (r t, d t,..., d τ ) for all t = 1,..., τ. Thrfor, w only hav accss to miny t, d t : t = 1,..., τ, but not th dmand random variabls thmslvs. Proposition 10 shows that this information is adquat to comput th stp dirction. Proposition 10 Knowldg of r t : t = 1,..., τ, y t : t = 1,..., τ and miny t, d t : t = 1,..., τ is adquat to comput s t (r t, d t,..., d τ ) in (28) for all t = 1,..., τ. Proof It is possibl to show th rsult by induction ovr th tim priods, but w us a constructiv proof, which is mor instructiv and asir to follow. W bgin with a chain of inqualitis that dirctly follow from Stps 2.a-2.c abov. For any tim priod s, w hav ys rs, ys x s and x s+1 y s d s, from which w obtain rs d s ys d s x s+1 y s+1, y s+1 d s+1 y s+2,..., y t 1 d t 1 y t for all t = s + 1,..., τ. Combining ths inqualitis, w hav rs d s d s+1... d t 1 y t for all t = s + 1,..., τ. Consquntly, if w hav minyt, d t = yt for any tim priod t, thn w must hav rs d s + d s d t for all s = 1,..., t 1. Assum that w want to comput s t (rt, d t,..., d τ ), whr w hav s t (rt, d t,..., d τ ) = c + h 1(d t < rt ) b 1(d t rt ) + t+1 (r t d t, d t+1,..., d τ ) 1(d t < rt ). W considr two cass. Cas 1. Assum that minyt, d t = yt. In this cas, w can dduc that d t yt rt. Thrfor, w hav s t (rt, d t,..., d τ ) = c b and w ar don. Cas 2. Assum that minyt, d t < yt. In this cas, sinc w now th valu of minyt, d t, w can dduc th valu of d t as bing qual to minyt, d t. Thus, sinc w now th valus of d t and rt, w can 17

18 comput 1(d t < r t ) and 1(d t r t ). Thrfor, it only rmains to comput t+1 (r t d t, d t+1,..., d τ ) for a nown valu of r t d t. W considr two subcass. Cas 2.a. Assum that miny t+1, d t+1 = y t+1. In this cas, w can dduc that d t+1 y t+1 r t+1. By th inquality w driv at th bginning of th proof, w hav rt d t + d t+1. Using (27), w hav h 1(d t+1 < r t d t ) b 1(d t+1 r t d t ) t+1(r t d t, d t+1,..., d + t+2 τ ) = (r t d t d t+1, d t+2,..., d τ ) 1(d t+1 < r t d t ) if rt d t rt+1 h 1(d t+1 < r t+1 ) b 1(d t+1 r t+1 ) + t+2 (r t+1 d t+1, d t+2,..., d τ ) 1(d t+1 < r t+1 ) if r t d t < rt+1, which is qual to b in ithr on of th cass and w ar don. Cas 2.b. Assum that minyt+1, d t+1 < y t+1. In this cas, w can dduc th valu of d t+1 as bing qual to minyt+1, d t+1. Thus, sinc w now th valus of r t, rt+1, d t and d t+1, w can comput 1(d t+1 < r t d t ), 1(d t+1 r t d t ), 1(d t+1 < r t+1 ) and 1(d t+1 r t+1 ) in th xprssion abov. Thrfor, it only rmains to comput t+2 (r t d t d t+1, d t+2,..., d τ ) and t+2 (r t+1 d t+1, d t+2,..., d τ ) for nown valus of rt d t d t+1 and r t+1 d t+1. Th rsult follows by continuing in th sam fashion for th subsqunt tim priods. 6 Invntory Purchasing Problm undr Pric Uncrtainty W want to ma purchasing dcisions for a product ovr th tim priods 1,..., τ. Th pric of th product changs randomly ovr tim and th goal is to satisfy th dmand for th product at th nd of th planning horizon with minimum total xpctd cost. W borrow this problm from Nascimnto and Powll (2006). A possibl application ara for it is th situation whr w nd to las storag spac on an ocan linr. Th pric of storag spac changs randomly ovr tim and th amount of storag spac that w actually nd bcoms nown just bfor th dpartur tim of th ocan linr. W lt p t b th pric at tim priod t, d b th dmand and b b th pnalty cost associatd with not bing abl to satisfy a unit of dmand. W assum that th random variabls p t : t = 1,..., τ and d ar indpndnt of ach othr, ta positiv valus and hav finit xpctations. W assum that th cumulativ distribution function of d is Lipschitz continuous and p t has a finit support P t. Whn th distinction is crucial, w us ˆp t to dnot a particular ralization of th random variabl p t. Ltting x t b th total amount of product purchasd up to tim priod t, th optimal policy is charactrizd by th Bllman quations with v τ+1 (x τ+1 ) = b E [d x τ+1 ] +. Ltting v t (x t ) = E min p t [y t x t ] + v t+1 (y t ), (29) y t x t f t (r t, p t ) = p t r t + v t+1 (r t ), it can b shown that f t (, p t ) is a convx function with a finit unconstraind minimizr, say r t (p t ). In this cas, it is asy to s that th optimal policy is a pric-dpndnt bas-stoc policy charactrizd 18

19 by th bas-stoc lvls r t (ˆp t ) : ˆp t P t, t = 1,..., τ. That is, if th total amount of product purchasd up to tim priod t is x t and th pric of th product is ˆp t, thn it is optimal to purchas [r t (ˆp t ) x t ] + units at tim priod t. It can b shown that f t (, p t ) and v t ( ) ar positiv, Lipschitz continuous, diffrntiabl and convx functions. Sinc w hav w can comput th drivativ of f t (, p t ) at x t through f t (r t, p t ) = p t + v t+1 (r t ), (30) t (x t, p t ) = p t + v t+1 (x t ), (31) whr f t (, p t ) rfrs to th drivativ with rspct to th first argumnt. Sinc r t (ˆp t ) is th minimizr of f t (, ˆp t ), w can sarch for th optimal bas-stoc lvls through r +1 t (ˆp t ) = r t (ˆp t ) α t (r t (ˆp t ), ˆp t ) for all ˆp t P t, t = 1,..., τ, whr r t (ˆp t ) : ˆp t P t, t = 1,..., τ ar th stimats of th optimal bas-stoc lvls at itration. Similar to Sctions 3 and 4, w now approximat th drivativs of f t (, ˆp t ) : ˆp t P t, t = 1,..., τ in a tractabl mannr. Using th optimal bas-stoc lvl rt (p t ), w writ (29) as v t (x t ) = E v t (x t, p t ), whr v t+1 (x t ) if x t rt (p t ) v t (x t, p t ) = p t [rt (p t ) x t ] + v t+1 (rt (p t )) if x t < rt (p t ). Thrfor, a stochastic gradint of v t ( ) at x t can b obtaind through v t+1 (x t ) if x t rt (p t ) v t (x t, p t ) = p t if x t < rt (p t ). Sinc rt (p t ) is th minimizr of f t (, p t ), (30) implis that p t = v t+1 (rt (p t )) and w obtain v t+1 (x t ) if x t rt (p t ) v t (x t, p t ) = v t+1 (rt (p t )) if x t < rt (p t ). (32) (33) (34) To construct tractabl approximations to th drivativs of f t (, ˆp t ) : ˆp t P t, t = 1,..., τ, w mimic th computation abov by using th stimats of th optimal bas-stoc lvls. In particular, ltting rt (ˆp t ) : ˆp t P t, t = 1,..., τ b th stimats of th optimal bas-stoc lvls at itration, w dfin t t+1 (x t, p t,..., p τ, d) = (x t, p t+1,..., p τ, d) if x t rt (p t ) t+1 (r t (p t ), p t+1,..., p τ, d) if x t < rt (35) (p t ), with τ+1 (x τ+1, d) = b 1(d x τ+1 ). At itration, rplacing v t+1 (x t ) in (31) with t+1 (x t, p t+1,..., p τ, d), w us s t (x t, p t,..., p τ, d) = p t + t+1(x t, p t+1,..., p τ, d) to approximat th drivativ of f t (, p t ) at x t. Consquntly, w propos th following algorithm to sarch for th optimal bas-stoc lvls. 19

20 Algorithm 2 Stp 1. Initializ th stimats of th optimal bas-stoc lvls r 1 t (ˆp t ) : ˆp t P t, t = 1,..., τ arbitrarily. Initializ th itration countr by stting = 1. Stp 2. Ltting p t : t = 1,..., τ b th pric random variabls and d b th dmand random variabl at itration, st r +1 t (p t ) = r t (p t ) α s t (r t, p t,..., p τ, d ) for all t = 1,..., τ. Furthrmor, st r +1 t (ˆp t ) = r t (ˆp t ) for all ˆp t P t \ p t, t = 1,..., τ. Stp 3. Incras by 1 and go to Stp 2. W mphasiz that only th bas-stoc lvls r t (p t ) : t = 1,..., τ ar updatd at itration in Stp 2 of Algorithm 2. Th othr bas-stoc lvls r t (ˆp t ) : ˆp t P t \ p t, t = 1,..., τ rmain th sam. Th proof of convrgnc for Algorithm 2 follows from an argumnt similar to th on in Sctions 3.1 and 3.2. W can follow th proof of Lmma 2 to driv bounds on t (, p t,..., p τ, d ) and s t (, p t,..., p τ, d ), and th proof of Lmma 3 to show that E t (, p t,..., p τ, d ) is incrasing. W dfin th rror function as t (x t, ˆp t ) = v t (x t, ˆp t ) E t (x t, ˆp t, p t+1,..., p τ, d ), with τ+1 (, ) = 0. In this cas, w can show that t (x t, ˆp t ) max f t (r t (ˆp t ), ˆp t ) E t+1 (r t (ˆp t ), p t+1), E t+1 (r t (ˆp t ), p t+1), E t+1 (x t, p t+1) w.p.1 for all x t R, ˆp t P t, t = 1,..., τ, = 1, 2,.... Onc w hav this bound on th rror function, w can follow th sam induction argumnt in Proposition 5, Lmmas 6-8 and Proposition 9 to show th final rsult. In particular, w can show that lim f t (r t (ˆp t ), ˆp t ) = 0 w.p.1 for all ˆp t P t, t = 1,..., τ. Th dtails of th proof ar givn in th onlin supplmnt. 7 Numrical Illustrations This sction focuss on th problms dscribd in Sctions 3, 4 and 6, and numrically compars th prformancs of Algorithms 1 and 2 with standard stochastic approximation mthods. 7.1 Multi-Priod Nwsvndor Problm with Bacloggd Dmands W considr a policy charactrizd by th bas-stoc lvls r t : t = 1,..., τ. That is, if th invntory position at tim priod t is x t, thn this policy ordrs [r t x t ] + units. If w follow this policy starting with th initial invntory position x 1 and th dmands ovr th planning horizon turn out to b d t : t = 1,..., τ, thn th invntory position at tim priod t is givn by x t = max x 1 t 1 s=1 d s, r 1 t 1 s=1 d s,..., r t 1 t 1 s=t 1 d s. 20

21 This is asy to s by noting that th invntory position at tim priod t + 1 is maxx t, r t d t and using induction ovr th tim priods. In this cas, th holding cost that w incur at tim priod t is H t (x 1, D r) = h [maxx t, r t d t ] + = h [ max x 1 D1 t 1, r 1 D1 t 1,..., r t 1 Dt 1 t 1, r ] + t dt = h max x 1 D t 1, r 1 D t 1,..., r t 1 D t t 1, r t D t t, 0, (36) whr w lt D t s = d s d t for notational brvity and us D to dnot th cumulativ dmands D t s : s = 1,..., τ, t = s,..., τ and r to dnot th bas-stoc lvls r t : t = 1,..., τ. Similarly, th baclogging cost that w incur at tim priod t is B t (x 1, D r) = b [d t maxx t, r t ] + = b [ d t max x 1 D1 t 1, r 1 D1 t 1,..., r t 1 Dt 1 t 1, r ] + t whras th rplnishmnt cost that w incur at tim priod t is = b max min D t 1 x 1, D t 1 r 1,..., D t t 1 r t 1, D t t r t, 0, (37) C t (x 1, D r) = c [r t x t ] + = c [ r t max x 1 D1 t 1, r 1 D1 t 1,..., r t 1 Dt 1 t 1 ] + = c max min r t x 1 + D t 1 1, r t r 1 + D t 1 1,..., r t r t 1 + D t 1, 0. (38) Thrfor, w can try to solv th problm min r E τ [ t=1 Ht (x 1, D r)+b t (x 1, D r)+c t (x 1, D r) ] to comput th optimal bas-stoc lvls. Howvr, it is asy to chc that th objctiv function of this problm is not ncssarily diffrntiabl with rspct to r. W ovrcom this tchnical difficulty by using an approach proposd by van Ryzin and Vulcano (2006). In particular, w lt ζ t : t = 1,..., τ b uniformly distributd random variabls ovr th small intrval [0, ɛ] and prturb th bas-stoc lvls by using ths random variabls. As a rsult, w solv th problm [ min E Ht (x 1, D r + ζ) + B t (x 1, D r + ζ) + C t (x 1, D r + ζ) ], (39) r t=1 whr w us r + ζ to dnot th prturbd bas-stoc lvls r t + ζ t : t = 1,..., τ. It is now possibl to show that th objctiv function of problm (39) is diffrntiabl with rspct to r and its gradint is Lipschitz continuous. Thrfor, w can us a standard stochastic approximation mthod to solv this problm. If ɛ is small, thn solving problm (39) instad of th original problm should not caus too much rror. Aftr straightforward algbraic manipulations on (36) and (37), it is asy to s that th r s -th componnt in th gradint of H t (x 1, D r) with rspct to r is givn by h 1(r s Ds t 0) 1(r s Ds t x 1 D1 t) s H t (x 1, D r) = 1(r s Ds t r 1 D1 t)... 1(r s Ds t r t Dt) t if s t 0 if s > t, (40) 21