Sensitivity Analysis of a Dynamic Fleet Management Model Using Approximate Dynamic Programming

Transcription

1 OPERATIONS RESEARCH Vol. 55, No. 2, March April 2007, pp issn X eissn informs doi /opre INFORMS Sensiiviy Analysis of a Dynamic Flee Managemen Model Using Approximae Dynamic Programming Huseyin Topaloglu School of Operaions Research and Indusrial Engineering, Cornell Universiy, Ihaca, New York 14853, opaloglu@orie.cornell.edu Warren B. Powell Deparmen of Operaions Research and Financial Engineering, Princeon Universiy, Princeon, New Jersey 08544, powell@princeon.edu We presen racable algorihms o assess he sensiiviy of a sochasic dynamic flee managemen model o flee size and load availabiliy. In paricular, we show how o compue he change in he objecive funcion value in response o an addiional vehicle or an addiional load inroduced ino he sysem. The novel aspec of our approach is ha i does no require muliple simulaions wih differen values of he model parameers, and in his respec i differs from rial-anderror-based wha-if analyses. Numerical experimens show ha he proposed mehods are accurae and compuaionally aracive. Subjec classificaions: ransporaion: models, nework; dynamic programming/opimal conrol: applicaions. Area of review: Opimizaion. Hisory: Received February 2004; revisions received May 2005, February 2006; acceped February Inroducion Given a vehicle flee and a sochasic process characerizing he load arrivals in a ransporaion nework, he primary objecive of flee managemen models is o make he vehicle reposiioning and vehicle-o-load assignmen decisions so ha some performance measure (profi, cos, deadhead miles, number of served loads, ec.) is opimized. However, besides making hese vehicle reposiioning and assignmen decisions, an imporan quesion ha is commonly overlooked by many flee managemen models is how he performance measures would change in response o a change in cerain model parameers. For example, freigh carriers are ineresed in how much heir profis would increase if hey inroduced an addiional vehicle ino he sysem or if hey served an addiional load on a cerain raffic lane. Railroad companies wan o esimae he minimum number of railcars ha is necessary o cover he random shipper demands. The Airlif Mobiliy Command is ineresed in he impac of limied airbase capaciies on he delayed shipmens. Answering such quesions, in one way or anoher, requires sensiiviy analysis of he underlying flee managemen model responsible for making he vehicle allocaion decisions. In his paper, we develop efficien sensiiviy analysis mehods for a sochasic flee managemen model previously developed in Godfrey and Powell (2002a, b). This model formulaes he problem as a dynamic program, decomposing i ino ime-saged subproblems, and replaces he value funcions wih specially srucured approximaions ha are obained hrough an ieraive improvemen scheme. Two aspecs of his model are crucial o our work: (1) Due o he special srucure of he value funcion approximaions, he subproblem ha needs o be solved for each ime period is a min-cos nework flow problem. This enables us o use he well-known relaionships beween he sensiiviy analyses of min-cos nework flow problems and min-cos flow augmening rees. In paricular, we can use he fac ha he change in he opimal soluion of a mincos nework flow problem in response o a uni change in he supply of a node or a uni change in he upper bound of an arc is characerized by a min-cos flow augmening pah. (2) Leing be he se of ime periods in he planning horizon, jus as he value funcions V describe an opimal vehicle allocaion policy hrough he so-called opimaliy equaion (see Puerman 1994), a se of value funcion approximaions V describes a (possibly subopimal) vehicle allocaion policy. Thus, given a rajecory of load realizaions, one can hink of he rajecory of vehicle allocaion decisions x induced by policy under load realizaion rajecory d =. In his paper, we exploi he aforemenioned sensiiviy relaionships o compue he change in he decision rajecory x in response o a change in a problem parameer and o assess how changes in he curren decisions affec he fuure ime periods. In paricular, we develop mehods o compue how much he profis would 319

2 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 320 Operaions Research 55(2), pp , 2007 INFORMS change if an addiional vehicle or an addiional load were inroduced ino he sysem. Our work is moivaed by he fac ha he freigh ransporaion indusry is ineresed in wha-if scenarios, and his convenionally refers o changing cerain parameers of he model and re-running i. However, here are obvious advanages associaed wih being able o exrac sensiiviy informaion from a single model run. For example, here can be many parameers whose impac on he model performance is of ineres, and making one (or more) run for each parameer migh be impossible. Furhermore, he decision maker simply migh no have an idea abou he criical parameers, and i is imporan o poin ou where he bigges bang for he buck lies. Also, sensiiviy informaion is useful in deermining he opimal flee size and mix, or making pricing decisions. A well-known class of models, from which one can quickly obain sensiiviy informaion, formulaes he problem over a sae-ime nework, where he nodes represen he supply of vehicles in differen saes a differen ime periods and he arcs represen he vehicle movemens. Examples of such models from hree differen indusries are Danzig and Fulkerson (1954), Hane e al. (1995), and Holmberg e al. (1998), and we refer he reader o Dejax and Crainic (1987) and Powell e al. (1995) for deailed surveys. For hese ypes of models, sensiiviy informaion is readily obained by using he dual soluion. However, hese models are inherenly deerminisic and can incorporae he random fuure load arrivals only hrough he expeced values. The model we analyze in his paper falls ino he caegory of sochasic models, which decompose he problem wih respec o ime periods and assess he impac of he curren decisions on he fuure hrough value funcions. However, because pracical flee managemen models involve large numbers of decision variables and possible load realizaions, sandard sochasic opimizaion mehods are no feasible for compuing he value funcions. Therefore, mos of he sochasic flee managemen models revolve around he idea of approximaing he value funcion in a racable manner. For sochasic flee managemen models oher han he one in Godfrey and Powell (2002a, b), we refer he reader o Franzeskakis and Powell (1990), Crainic e al. (1993), Carvalho and Powell (2000), and Topaloglu and Powell (2006). Given a se of value funcion approximaions, hese models behave jus like simulaion models, generaing differen rajecories of vehicle allocaion decisions for differen rajecories of load realizaions. Therefore, a key quesion for heir sensiiviy analysis is o be able o assess how he decision rajecories change when cerain model parameers are perurbed. Our approach has similariies wih infiniesimal perurbaion analysis, which refers o compuing he gradien of a performance measure in a discreeeven dynamic sysem wih respec o an inpu parameer (see Glasserman 1991, Ho and Cao 1991). However, for he sysems we consider, he ransiions beween he saes are more complex han hose ha are convenionally considered by discree-even dynamic sysems because hese ransiions are governed by he soluions of min-cos nework flow problems. In his paper, we make he following research conribuions. We develop efficien sensiiviy analysis mehods for a sochasic flee managemen model previously developed in Godfrey and Powell (2002a, b). In paricular, we show how o compue he change in he objecive value of his model in response o an addiional vehicle or an addiional load inroduced ino he sysem. An accurae, bu edious, sensiiviy analysis mehod is o physically change a parameer of ineres and rerun he model. We compare our mehods wih his brue force approach and show ha hey are quie accurae even when heoreical analysis requires cerain approximaions. The res of his paper is organized as follows. In 1, we briefly describe he flee managemen model used hroughou he paper. Undersanding his model is imporan because we analyze he sensiiviy of he objecive value under he policy prescribed by his paricular model. In 2, we consider problems wih a single vehicle ype and show how o compue he change in he objecive value in response o an addiional vehicle or an addiional load inroduced ino he sysem. Secion 3 exends hese resuls o problems wih muliple vehicle ypes. The compuaional experimens presened in 4 show he accuracy of he proposed sensiiviy analysis mehods. Secion 5 conains concluding remarks. 1. Problem Formulaion We have a flee of vehicles o serve he loads of differen ypes ha arrive over ime. A every ime period, a cerain number of loads ener he sysem, and we have o decide which loads o cover and o which locaions we should reposiion he empy vehicles. We are ineresed in maximizing he oal expeced profi over a finie horizon, bu we formulae our model o minimize cos for compaibiliy wih he min-cos nework flow lieraure. We assume ha advance informaion abou he fuure loads is no available and he loads ha canno be covered in a given ime period are served by an emergency subconracor. These enable us o assume ha he uncovered loads immediaely leave he sysem. For noaional breviy, we assume ha i akes one ime period o move beween any pair of locaions. I is sraighforward o exend our analysis o he case where here are muliperiod ravel imes by using he approach described in Topaloglu and Powell (2006). We define he following: = se of ime periods in he planning horizon, = 1 T. I = se of locaions in he ransporaion nework. = se of available vehicle ypes. L = se of movemen modes, L = 0 L. (Movemen modes represen differen ways in which a vehicle

3 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 321 can move from one locaion o anoher. Movemen mode 0 always corresponds o empy reposiioning, whereas oher modes correspond o carrying differen ypes of loads.) xijl k = number of vehicles of ype k dispached from locaion i o j a ime period using movemen mode l. cijl k = cos of dispaching one vehicle of ype k from locaion i o j a ime period using movemen mode l. D ijl = random variable represening he number of loads ha need o be carried from locaion i o j a ime period and correspond o movemen mode l. As will be clear shorly, he random variable D ijl serves as an upper bound on he decision variables xijl k k. Because he movemen mode 0 corresponds o empy reposiioning and he empy reposiioning movemens are no bounded, we assume ha D ij0 = for all i j I,. In pracice, he movemen modes in L\ 0 can correspond o differen ypes of loads or differen shippers, and we usually have cijl k < 0 when l L\ 0. The vehicle ype migh reflec he size of he vehicle, he skill level of he driver of he vehicle, he abiliy of he vehicle o saisfy cerain safey or saniary requiremens, or a combinaion of hese facors, which ulimaely deermine wheher i is feasible o cover a cerain ype of load wih a cerain ype of vehicle and wha profi is obained by doing so. If i is infeasible o cover a load of ype l wih a vehicle of ype k, hen we capure his by leing cijl k = for all i j I,. Throughou his paper, we use d ijl o denoe a paricular realizaion of D ijl. By suppressing some of he indices in he variables above, we denoe a vecor composed of he componens ranging over he suppressed indices. For example, we have d = ijl i j I l L and d = ijl i, j I, l L,. To capure he sae of he sysem a ime period, we define ri k = number of vehicles of ype k ha are available a locaion i a ime period. The vecor r = ri k i I k compleely defines he sae of he vehicles a ime period. Given his sae vecor and he realizaion of he loads a ime period, he se of feasible decision vecors and he se of sae vecors generaed by hese decisions a he nex ime period are given by { r = x r +1 x k ijl = r k i j I l L for all i I k (1) x k ijl r k j +1 = 0 i I l L for all j I k (2) x k ijl d ijl k for all i j I l L (3) x k ijl + } for all i j I l L k (4) We are ineresed in Markovian deerminisic policies ha minimize he oal expeced cos over he planning horizon. A Markovian deerminisic policy can be characerized by a sequence of decision funcions X such ha X maps he sae vecor r and he realizaion of he loads d a ime period o a decision vecor x. One can also define he sae ransiion funcions R +1 of policy such ha R +1 maps he sae vecor and he realizaion of he loads a ime period o a sae vecor for he nex ime period. We noe ha given X, R +1 can easily be defined by noing he sae ransiion consrains in (2). Then, for a given sae vecor r and realizaion of fuure loads T a ime period, he cumulaive cos funcion for policy can be wrien recursively as F r +1 T =c X r +F +1 R +1 r T (5) wih he boundary condiion FT +1 = 0. By repeaed applicaion of (5), we obain F 1 r 1 1 T = c 1 X 1 r c 2 X 2 R 2 r 1 1 d 2 + +c T X T R T R T 1 T 2 d T 1 d T (6) which is he oal cos incurred over he whole planning horizon when we use policy ; he iniial sae vecor is r 1 and he realizaion of he loads is 1 T. Assuming ha, given r, D is independen of D = 1 1, i can be shown ha he opimal policy is Markovian deerminisic, saisfying = arg min Ɛ F 1 r 1 D 1 D T r 1 where is he se of Markovian deerminisic policies. This opimal policy can be found by compuing he value funcions hrough he so-called opimaliy equaion (see Puerman 1994): { V r = Ɛ min c x + V x r +1 r D +1 r +1 r } (7) In his case, he decision and ransiion funcions for he opimal policy become X r R +1 r = arg min c x + V +1 r +1 (8) x r +1 r Throughou his paper, o keep he presenaion simple, we assume ha he cos vecor c is perurbed by small random amouns so ha problem (8) has a single opimal soluion. Under his assumpion, he decision and sae ransiion funcions are properly defined, and our proofs become easier.

4 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 322 Operaions Research 55(2), pp , 2007 INFORMS Compuing he value funcions V hrough (7) is inracable for almos all problem insances of pracical significance, because i requires enumeraing over all possible values of r and aking an expecaion over he mulidimensional random variable D for all. In his paper, we follow a class of (subopimal) policies proposed in Godfrey and Powell (2002a), which are obained by replacing V in (8) wih separable approxima- of he form V k i r k i (9) ions V V r = i I k where each V i k is a one-dimensional piecewise-linear convex funcion wih poins of nondiffereniabiliy being a subse of posiive inegers. In his case, for a policy characerized by separable piecewise-linear convex value funcion approximaions V, we can define he decision and sae ransiion funcions as X r R +1 r = arg max c x + V +1 r +1 (10) x r +1 r We noe ha alhough he value funcion approximaions V are separable, he cumulaive cos funcions F T for policy are no necessarily separable. Furhermore, we have V r = Ɛ F r D D T r for he opimal policy by he principal of opimaliy (see Puerman 1994), bu we do no necessarily have V r = Ɛ F r D D T r for he policy characerized by he value funcion approximaions V. Godfrey and Powell (2002a) give a sampling-based algorihm ha can be used o obain a good se of value funcion approximaions. The quesion of wheher hese subopimal policies yield high-qualiy soluions is ouside he scope of his paper. We refer he reader o Godfrey and Powell (2002a, b) and Topaloglu and Powell (2006), where he experimenal work indicaes ha his class of policies bea sandard benchmarks by significan margins. Here, we assume ha we already have a good policy, and we are ineresed in compuing he change in F1 r 1 1 T induced by changing an elemen of he sae vecor r 1 or he load availabiliy vecor d 1. We make his quesion precise in he nex wo secions. However, before going ino he specific deails, we can summarize he conens of he nex wo secions as follows: (1) We noe ha if we use a policy characerized by separable piecewise-linear convex value funcion approximaions, hen problem (10) is a min-cos nework flow problem. (2) We use he well-known relaionships beween he sensiiviy analyses of min-cos nework flow problems and min-cos flow augmening rees o find how he soluion of problem (10) a ime period 1 changes in response o an addiional vehicle or an addiional load inroduced ino he sysem. (3) We find how he sae vecor a ime period 2 changes in response o he change in he soluion of problem (10) a ime period 1. (4) Finally, we find how he soluion of problem (10) a ime period 2 changes in response o he change in he sae vecor a ime period 2. We repea he same argumen in a recursive fashion for he subsequen ime periods. In 2, we sar by considering problems wih a single vehicle ype. We generalize he ideas o muliple vehicle ypes in Problems wih a Single Vehicle Type In his secion, we assume ha =1 and drop he vehicle ype superscrip, in which case (9) becomes V r = i I V i r i. Leing R be he oal number of available vehicles, he relevan domain of V i is 0 1 R. Therefore, assuming ha V i 0 = 0 wihou loss of generaliy, we can represen V i by a sequence of numbers ṽi q q = 1 R, where ṽ i q is he slope of V i q over q 1 q. Tha is, we have ṽ i q = V i q V i q 1. In his case, problem (10) can explicily be wrien as min ijl x ijl + x r +1 z +1 i j I l Lc R ṽ j +1 q z j +1 q (11) j I q=1 subjec o x ijl = r i for all i I (12) j I l L x ijl r j +1 = 0 for all j I (13) i I l L r j +1 R z j +1 q = 0 for all j I (14) q=1 x ijl d ijl for all i j I l L (15) z j +1 q 1 for all j I q=1 R (16) x ijl r j +1 z j +1 q + for all i j I l L q= 1 R (17) where we use a sandard echnique o embed he piecewiselinear convex funcions V j +1 j I ino he problem above hrough he decision variables z j +1 q j I q = 1 R. In paricular, due o he convexiy of V j +1, wehaveṽ j +1 1 ṽ j +1 2 ṽ j +1 R. Because he objecive funcion is minimized, noing consrains (14), (16), and (17), we mus have R q=1 r ṽ j +1 q z j +1 j +1 q = ṽ j +1 q = V j +1 r j +1 q=1 in he opimal soluion. Therefore, he second erm in (11) compues j I Vj +1 r j +1 (see Nemhauser and Wolsey 1988). Alhough consrains (13) and (14) can be combined ino i I l L x ijl R q=1 z j +1 q = 0, we leave hem separae o emphasize ha (13) handles he sae ransiion, whereas (14) handles he compuaion of he value funcion approximaion. I is easy o see ha problem (11) is he min-cos nework flow problem in Figure 1. In his

5 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 323 Figure 1. a b c Problem (11) is a min-cos nework flow problem. n n m a b c x ijl r j, +1 z j, +1 (q) Sink node Noe. The pah in bold arcs represens a possible min-cos flow augmening pah from node a on he lef side o he sink node. Such min-cos flow augmening pahs will be useful in 2.1. figure, we assume ha I = a b c and L = n m. Consrains (12), (13), and (14), respecively, correspond o he flow balance consrains for he whie, gray, and black nodes. The ses of decision variables x ijl i j I, l L, r j +1 j I, and z j +1 q j I q = 1 R, respecively, correspond o he arcs ha leave he whie, gray, and black nodes Policy Gradiens wih Respec o Vehicle Availabiliies In his secion, we develop a mehod o compue how much he oal cos under policy would change if an addiional vehicle were inroduced ino he sysem. We le x and r be he sequences of decisions and saes visied by he sysem under policy and load realizaion d =. Tha is, x and r are recursively compued by x = X r r Then, noing (5), F +1 = R +1 r wih r 1 = r 1 (18) r T becomes he oal cos incurred a ime periods T under policy and load realizaion d. Leing e i be he I -dimensional uni vecor wih a one in he elemen corresponding o i I, our objecive in his secion is o compue e i = F r + e i T r T (19) for all i I,. Then, Ɛ 1 e i D ells us how much he oal expeced cos under policy would change by inroducing an addiional vehicle a locaion i a he firs ime period. We noe ha e i can be compued by wo simulaions of policy under load realizaion d, one of which sars wih he sae vecor r and he oher wih + e i. However, doing his for all i I, and r for muliple load realizaions can ge ime consuming. Our objecive is o be able o compue e i for all i I, from a single simulaion. Using (5) and (18), (19) can be wrien as e i = c X r + F +1 R +1 r +1 R +1 r = c X r + e i X r + e i d +1 T d +1 T + e i x + F +1 R +1 r + e i d +1 T +1 r T (20) As will be clear shorly, compuing X r + e i x and R +1 r +e i r+1 is key o compuing e i. Because we have x X r r +1 = arg min x r +1 r c x + V +1 r +1 (21) + e i x and R +1 r + e i r+1 are relaed o how he soluion of problem (11) changes when he righ side of consrains (12) is increased from r r + e i. Consider problem (21) and is nework represenaion in Figure 1. Se he flows on he arcs in his nework such ha hese flows correspond o he opimal soluion x r+1. Le e i be he min-cos flow augmening pah from node i I on he lef side o he sink node in his figure. One possible flow augmening pah when i = a is shown in bold arcs. We define he vecor e i = ıjl e i d ı j I l L as +1 if he arc corresponding o variable x ıjl is a forward arc in e i ıjl e 1 if he arc corresponding o variable i = x ıjl is a backward arc in e i 0 if he arc corresponding o variable x ıjl is no in e i Similarly, we define he vecor +1 e i = j +1 e i j I as +1 if he arc corresponding o variable r j +1 is a forward arc in e i 1 j +1 e i = if he arc corresponding o variable r j +1 is a backward arc in e i 0 if he arc corresponding o variable r j +1 is no in e i For example, for he flow augmening pah in Figure 1, we have abn e a =+1, cbn e a = 1, ccm e a = +1, and c +1 e a =+1. The following resul characerizes how he soluion of problem (21) changes when he number of vehicles available a locaion i is increased by 1. o (22)

6 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 324 Operaions Research 55(2), pp , 2007 INFORMS Lemma 1. The following resuls hold: (1) We have X r + e i = x + e i and R +1 r + e i = r e i. (2) One elemen of he vecor +1 e i is equal o +1 and he oher elemens are equal o 0. Proof. The firs par is a direc resul of Theorem 1 in Powell (1989). The second par holds because any acyclic pah from node i I on he lef side of Figure 1 o he sink node raverses exacly one of he arcs corresponding o one of he variables r j +1 j I. The second par of he lemma implies ha if an addiional vehicle is inroduced a a cerain locaion a ime period, hen exacly one elemen of he sae vecor a ime period + 1 will increase by 1. The following proposiion gives an efficien mehod o compue e i. Proposiion 2. We have e i = c e i e i for all i I, wih he boundary condiion T +1 = 0. Proof. Using Lemma 1, (20) can be wrien as e i = c e i ( + F +1 r e ) i d d +1 T ( +1 r +1 ) +1 T = c e i e i d d The firs erm in c e i e i capures how much he cos incurred a ime period changes in response o an addiional vehicle a locaion i a ime period, whereas he second erm capures how much he cos incurred a ime periods + 1 T changes in response o an addiional vehicle a locaion i a ime period. Thus, he idea is o sar wih he las ime period T and le T e i = c T T e i for all i I. Then, we move o ime period T 1. Because T e i is always a posiive ineger uni vecor, T 1 e i can easily be compued as c T 1 T 1 e i + T T e i. We coninue in a similar fashion unil we reach he firs ime period. We noe ha o evaluae he expeced cos impac of an addiional vehicle, we need o compue Ɛ e i D as opposed o e i for a paricular load realizaion d. In his case, because compuing his expecaion is usually inracable, we can sample N load realizaions, say d 1 N, use he mehod described in his secion o compue e i n for all n = 1 N, and use he sandard confidence inerval mehodology o esimae Ɛ e i D. By carrying ou a pilo run ha uses a small number of load realizaions, we can assess he number of load realizaions ha are needed o esimae Ɛ e i D wih a cerain precision (see Law and Kelon 2000). Finally, we noe ha a similar mehod o compue e i = F r e i T r T (23) can be developed by using min-cos flow decreasing rees. This will be useful in he nex secion Policy Gradiens wih Respec o Load Availabiliies Freigh carriers coninuously face he problem of evaluaing newly arriving loads o decide wheher o accep or rejec hem. In his secion, we develop a mehod o compue how much he oal cos under policy would change if an addiional load were inroduced ino he sysem. This informaion can, in urn, be used for load evaluaion decisions. The class of policies we consider assume ha here is no advance informaion abou fuure load realizaions. For his reason, we assume ha if is he curren ime period, hen he addiional load ha is inroduced ino he sysem is a load ha needs o be served a ime period. Leing e ijl and e ijl be he I 2 L and I 2 Limensional uni vecors wih a one in he elemen corresponding o i j I, l L, and i j I, l L, we wan o compue e ijl = F r +e ijl + e ijl +1 T r +1 T (24) which is he change in he oal cos of policy under load realizaion d in response o an addiional load of ype l on lane i j a ime period. Using an argumen similar o he one in (20) and noing ha r +e ijl is a funcion of he load realizaions up o (bu no including) ime period, (24) can be wrien as e ijl = F r + e ijl +1 T = c X r + F r +1 T +1 R +1 r + e ijl x + e ijl d +1 T +1 r T (25) To compue e ijl, we now need o characerize X r + e ijl x and R +1 r + e ijl r+1. These quaniies are relaed o how he soluion of he mincos nework flow problem (11) changes when he upper bound on he decision variables x is increased from d o d + e ijl. Consider problem (21) and is nework represenaion in Figure 2. Se he flows on he arcs in his nework such ha hese flows correspond o he opimal soluion x r+1. Le e i e j be he min-cos flow augmening pah from node j I in he middle secion o node i I on he lef side in Figure 2. Denoe he cos of his min-cos flow augmening pah by e i e j. One possible flow augmening pah when i = a, j = c is shown in dashed arcs. We define

7 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 325 Figure 2. a b c e a e c is he min-cos flow augmening pah from node c in he middle secion o node a on he lef side. n m n a b c Sink node Proof. See he appendix. Therefore, if e i e j +c ijl 0, hen an addiional load of ype l on lane i j does no change he soluion of problem (21). We noe ha he min-cos flow augmening pah e i e j may no include any arcs corresponding o one of he variables r j +1 j I. In his case, we have +1 e i e j = 0 and we can se + +1 e i e j = +1 e i e j in he second par of Lemma 3. The following proposiion gives an efficien mehod o compue e ijl. Proposiion 4. Leing x ijl r j, +1 z j, +1 (q) he vecor e i e j = ıjl e i e j d ı j I l L as +1 if he arc corresponding o variable x ıjl is a forward arc in e i e j 1 ıjl e i e j = if he arc corresponding o variable x ıjl is a backward arc in e i e j 0 if he arc corresponding o variable x ıjl is no in e i e j We also define he vecor +1 e i e j similar o (22), bu using he flow augmening pah e i e j. F or example, for he flow augmening pah in Figure 2, we have aan e a e c = 1, cbm e a e c =+1, ccn e a e c = 1, a +1 e a e c = 1, and b +1 e a e c =+1. The following resul characerizes how he soluion of problem (21) changes when he number of loads of ype l on lane i j is increased by one. Lemma 3. The following resuls hold: (1) Leing { 1 if a<b 1 a<b = 0 oherwise we have X r + e ijl = x + 1 e i e j +c ijl <0 e i e j + e ijl R +1 r + e ijl = r e i e j +c ijl <0 +1 e i e j (2) The vecor +1 e i e j can be wrien as +1 e i e j = + +1 e i e j +1 e i e j where one elemen of each of he vecors + +1 e i e j and +1 e i e j is equal o +1 and he oher elemens are equal o 0. e ijl = 1 e i e j +c ijl <0 e i e j + e ijl for noaional breviy, we have he following resuls: (1) If e i e j + c ijl 0 or +1 e i e j = 0, hen we have e ijl = c e ijl. (2) If F+1 +1 T is a separable funcion, e i e j + c ijl < 0 and +1 e i e j 0, hen we have e ijl = c e ijl e i e j e i e j (26) for all i j I, l L,, where +1 e i is as defined in (19) and (23). Proof. Under he condiions saed in he firs par, Lemma 3 implies ha R +1 r + e ijl = r+1 and (25) becomes e ijl = c X r + e ijl x. Then, he firs par follows by he definiion of e ijl and Lemma 3. By using Lemma 3, (25) becomes e ijl = c e ijl + F +1 (r e i e j +1 e i e j ) d +1 T +1 r T Because F T is separable, Lemma 8 in he appendix implies ha e ijl = c e ijl ( + F +1 r e ) i e j d d +1 T +1 r T ( + F +1 r e ) i e j d d +1 T +1 r T The firs par of he proposiion corresponds o he case where an addiional load of ype l on lane i j a ime period eiher does no change he decisions a ime period or does no change he sae vecor a ime period +1. The second par corresponds o he case where

8 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 326 Operaions Research 55(2), pp , 2007 INFORMS an addiional load of ype l on lane i j a ime period does change he sae vecor a ime period + 1. Given he fac ha + +1 e i e j and +1 e i e j are posiive ineger uni vecors, (26) can be easily compued once we know +1 e i for all i I. As noed in 1, F+1 +1 T is no necessarily a separable funcion. However, we propose using (26) as an approximaion o e ijl even when F+1 +1 T is no separable. Our compuaional experimens show ha his approximaion yields accurae resuls. We believe ha he accuracy of his approximaion is due o he following. The expression in (26) capures he change in he oal cos of policy under load realizaion d in response o an addiional load of ype l on lane i j a ime period. Among he hree erms on he righ side of (26), he firs erm accuraely capures he change in he cos incurred a ime period, whereas he sum of he second and hird erms approximaely capures he change in he cos incurred a ime periods + 1 T. Therefore, accuraely capuring he change in he cos incurred a he curren ime period and approximaely capuring he change in he cos incurred a he fuure ime periods appear o be adequae o obain a good approximaion. Assuming ha e a e c + c acm < 0 and noing ha a +1 e a e c = 1, b +1 e a e c =+1 for he min-cos flow augmening pah in Figure 2, he approximaion in (26) can be inerpreed as follows. The firs erm gives he change in he immediae cos due o he change in he decisions a ime period. The second erm gives he change in he fuure cos due o having an addiional vehicle a locaion b a ime period + 1. The hird erm gives he change in he fuure cos due o having one less vehicle a locaion a a ime period Problems wih Muliple Vehicle Types In his secion, we exend he ideas in 2 o he case where here are muliple vehicle ypes. Topaloglu and Powell (2006) noe ha if here are muliple vehicles ypes and policy is characerized by a se of separable piecewise-linear convex value funcion approximaions, hen problem (10) becomes a min-cos ineger mulicommodiy nework flow problem, and his inhibis exploiing properies of mincos flow augmening and decreasing rees as we did in he previous secion. Noneheless, hey also show ha if each V is a linear funcion of he form V r = i I k ṽi k ri k, hen problem (10) is a min-cos nework flow problem. Furhermore, over a limied domain, say 0 1 or 0 2, linear funcions approximae piecewiselinear funcions quie well. Because he sum of he elemens of he I -dimensional vecor r is always equal o he number of available vehicles, say R, if I R, hen we expec he elemens of he vecor r o be mosly 0s, 1s, or 2s. In his case, using piecewise-linear approximaions does no bring oo much advanage over linear approximaions. For hese reasons, when working on problems wih Figure 3. Differen locaions Differen vehicle ypes (a, f ) (a,g) (b, f ) (b,g) Problem (27) is a min-cos nework flow problem. k x ijl d aan d aam d abn d abm d ban d bam d bbn d bbm Sink node Upper bounds are d ijl muliple vehicle ypes, we use policies characerized by linear value funcion approximaions. We now exend he resuls of 2.1 o he case of muliple vehicle ypes. Noing (2), he objecive funcion of problem (10) under a policy defined by linear value funcion approximaions is c x + V +1 r +1 = i j I l L k c k ijl xk ijl + j I k ṽ k j +1 ( x k ijl i I l L Then, he decision funcion for policy can be wrien as X r = arg min c k ijl +ṽ k j +1 xk ijl (27) x i j I l L k subjec o (1), (3), (4) which is he min-cos nework flow problem in Figure 3. In his figure, we assume ha I = a b, L = n m, and = f g. Consrains (1) represen he flow balance consrains for he gray nodes. Defining he addiional decision variables w ijl i j I l L and spliing consrains (3) ino wo ses of consrains k xijl k w ijl = 0 and w ijl d ijl for all i j I, l L, he firs se represens he flow balance consrains for he whie nodes in he middle secion. We le be a policy characerized by he linear value funcion approximaions V wih he decision, sae ransiion, and cumulaive cos funcions X, R +1, F. We also define x and r similar o heir counerpars in 2.1. Leing ei k be he I -dimensional uni vecor wih a one in he elemen corresponding o i I, k, we wan o compue ek i = F r + ei k T F r T for all i I, k,. Consider problem (27) and is nework represenaion in Figure 3. Se he flows on he arcs in his nework such ha hese flows correspond o he opimal soluion x. )

9 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 327 Le ek i be he min-cos flow augmening pah from node i k I on he lef side o he sink node in his figure. One possible flow augmening pah when i k = a f is shown in bold arcs. We define he vecor ek i = ıjl ek i d ı j I l L as ıjl ek i = +1 if he arc corresponding o variable xıjl is a forward arc in ek i 1 if he arc corresponding o variable x ıjl is a backward arc in ek i 0 if he arc corresponding o variable xıjl is no in ek i We also define he vecor +1 ek i as j +1 ek i = ı I l L = j +1 ek i d j I ıjl ek i (28) For example, for he flow augmening pah in Figure 3, we have f aan e f a =+1 f a +1 e f a =+1 g aan ea f = 1 g a +1 ea f = 1 g abm ef a =+1 and g b +1 ef a =+1 The following resul characerizes how he soluion of problem (27) changes when he number of vehicles of ype k available a locaion i is increased by one. Lemma 5. The following resuls hold: (1) We have X r + ei k = x + ei k = r R +1 r + ek i and ek i. (2) There exis wo disjoin subses of I, say + +1 ei k and +1 ek i, such ha +1 ek i can be wrien as +1 ek i = j + +1 ek i e j j +1 ek i e j Proof. In he firs par, he firs equaliy is a direc resul of Theorem 1 in Powell (1989) and he second equaliy follows from he definiion of +1 ek i in (28) and he sae ransiion consrains (2). We show he second par in he appendix. The second par of he lemma shows ha an addiional vehicle of ype k a locaion i a ime period could change he sae vecor a ime period + 1 in a complicaed manner, bu each componen of he sae vecor a ime period + 1 changes by a mos one. The following proposiion gives an efficien mehod o compue ek i. Proposiion 6. If F T is a separable funcion, hen we have ek i = c ek i + + j + +1 ek i +1 e j +1 e j (29) j +1 ek i for all i I, k, wih he boundarycondiion T +1 = 0. Proof. Using Lemma 5, we have ek i =c ek i ( ) +F +1 r +1 + e j e j +1 T j + +1 ek i j +1 ek i F +1 r +1 T =c ek i { + F +1 r +1 +e j +1 T j + +1 ek i F +1 r T } { + F +1 r +1 e j +1 T j +1 ek i F +1 r T } where he second equaliy uses he separabiliy assumpion and Lemma 8 in he appendix. On he righ side of (29), he firs erm capures how much he cos incurred a ime period changes in response o an addiional vehicle of ype k a locaion i a ime period, whereas he second and hird erms capure how much he cos incurred a ime periods +1 T changes in response o an addiional vehicle of ype k a locaion i a ime period. F+1 +1 T is no necessarily a separable funcion. However, we propose using (29) as an approximaion o ek i even when F+1 +1 T is no separable. 4. Compuaional Experimens This secion focuses on he resuls of 2.2 and 3 and numerically esablishes he accuracy of he mehods proposed o compue e ijl and ek i. In paricular, we use a variey of es problems o show ha (26) and (29) can approximae e ijl and ek i accuraely even when F+1 +1 T is no a separable funcion. The mehod proposed o compue e i in 2.1 is exac and does no require numerical validaion. Our es problems involve 40 locaions and 41 movemen modes (40 load ypes and one movemen mode

10 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 328 Operaions Research 55(2), pp , 2007 INFORMS Table 1. Accuracy of he policy gradiens wih respec o load availabiliies. Hisogram Coeff. of variaion Corr. Avg. Time Problem coeff. % dev. 2.5% 5% 10% 25% (sec.) Avg. 20% 80% Noe. Percen deviaion is e ijl n 1 e ijl n / 1 e ijl n, and we ignore he daa poins wih 1 e ijl n =0. for empy reposiioning). We label our es problems by T D K R e, where T is he lengh of he planning horizon, D is he expeced number of loads over he planning horizon, K is he number of vehicle ypes, R is he number of available vehicles, and e is he empy reposiioning cos applied on a per-mile basis. Accuracy of he Policy Gradiens wih Respec o Load Availabiliies. We sar by esing he accuracy of he mehod proposed o compue e ijl. Our experimenal seup is as follows. For each es problem, we firs obain a good vehicle allocaion policy by using he sampling-based mehod of Godfrey and Powell (2002a). Having obained a policy, we sample N load realizaions, say d 1 N. For each load realizaion d n, we approximae 1 e ijl n for all i j I, l L by using (26). We le 1 e ijl n i j I l L n= 1 N be hese approximaions. Because he mehod given in 2.2 requires simulaing he behavior of policy under load realizaion d n, a his poin we can also compue F1 r 11 nn 2 n T as c x n. We hen physically increase he number of loads of ype l on lane i j a ime period 1 by 1 and compue F1 r 11 n +e ijl2 n n T by simulaing he behavior of policy under load realizaion d n +e ijl1. In his way, we can accuraely compue 1 e ijl n in a brue force fashion as F1 r 11 n +e ijl d2 n n T F1 r 11 nn 2 n T. Our aim is o compare he approximaion 1 e ijl n ha is compued hrough (26) wih 1 e ijl n ha is compued in a brue force fashion. Table 1 summarizes our findings. The firs se of columns give he average percen deviaion and he coefficien of correlaion beween 1 e ijl n i j I l L n=1 N and 1 e ijl n i j I l L n=1 N. The second se of columns give a hisogram for he percen deviaions ha shows wha fracion of he percen deviaions is less han 2.5%, 5%, 10%, and 25%. The nex column gives he average ime o compue 1 e ijl n i j I l L for a paricular load realizaion d n. This ime includes he ime spen simulaing he behavior of policy under load realizaion d n. The las se of columns give summary saisics for he coefficiens of variaion of 1 e ijl D i j I l L. We esimae he coefficien of variaion of 1 e ijl D as ijl / ijl, where ijl and 2 ijl are he sample mean and sample variance of 1 e ijl n n= 1 N. Using his esimae of coefficien of variaion, one can have an idea of how many load realizaions are needed o esimae Ɛ 1 e ijl D wih a cerain precision (see Law and Kelon 2000). Because ijl / ijl depends on i j I, l L, we give he mean, and he 20h and 80h perceniles of ijl / ijl i j I l L. We noe ha he coefficien of variaion esimaes are highly problem-specific and one should no draw general conclusions from hem. The high coefficien of correlaion and he low average percen deviaion figures in Table 1 show ha 1 e ijl n i j I l L n=1 N and 1 e ijl n i j I l L n=1 N are in close agreemen. The hisograms show ha, abou 90% of he ime, our approximaions are wihin 25% of he rue value. If we were working on problems wih deerminisic load arrivals, hen 1 e ijl could also be approximaed by using he opimal value of he dual variable associaed wih he load availabiliy consrain x ijl1 ijl1 in he sae-ime nework formulaion of he problem. Powell (1989) repors ha, 10% of he ime, approximaing 1 e ijl by using he dual soluion brings an error of 50% or more. Therefore, our mehod can approximae 1 e ijl noiceably beer han he dual soluion of he sae-ime nework formulaion. Accuracy of he Policy Gradiens wih Respec o Vehicle Availabiliies. We now compare he approximaions obained hrough (29), say 1 ek i n i I k n=1 N, wih he values of 1 ek i n i I k n=1 N obained in a brue force fashion by increasing he number of vehicles of ype k a locaion i by one and simulaing he behavior of policy under load realizaion d n. The resuls in Table 2 indicae ha (29) yields accurae resuls.

11 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 329 Table 2. Accuracy of he policy gradiens wih respec o vehicle availabiliies. Hisogram Coeff. of variaion Corr. Avg. Time Problem coeff. % dev. 2.5% 5% 10% 25% (sec.) Avg. 20% 80% Conclusions We presened efficien mehods o assess he sensiiviy of a sochasic dynamic flee managemen model o flee size and load availabiliy. Numerical experimens indicaed ha hese mehods are accurae and compuaionally racable. Informaion abou he cos impac of an addiional vehicle or an addiional load can, in urn, be used when making flee sizing, load evaluaion, and pricing decisions. Using he mehod described in 2.2 for load pricing is he opic of anoher paper (see Topaloglu and Powell 2005). Appendix This secion presens he omied proofs. The following resul is useful when proving Lemma 3. Lemma 7. If e i e j +c ijl 0, hen we have +e ijl =x X r. Proof of Lemma 7. Consider problem (21) and is nework represenaion in Figure 2. In his problem, d ijl acs as an upper bound on he decision variable x ijl, and a min-cos nework flow problem wih upper bounds can be convered o an equivalen problem wihou upper bounds by he ransformaion shown in Figure A.1 (see Vanderbei 1997). Therefore, if d ijl is increased by one, hen he change in he opimal soluion of problem (21) is given by a min-cos flow augmening pah from node j o node i j l in Figure A.1(b). We denoe his min-cos flow augmening pah by. Because node i j l has exacly wo inbound arcs, here are wo possible cases o consider for : (1) Eiher includes only he bold arc ha connecs node j o node i j l in his case, he cos of is 0; or (2) connecs node j o node i, and hen node i o node i j l. We le e i e j be he cos of he min-cos flow augmening pah from node j o node i in Figure A.1(a). Then, he cos of he min-cos flow augmening pah from node j o node i in Figure A.1(b) is also e i e j. Hence, for he second case, he cos of is e i e j +c ijl. Because is he min-cos flow augmening pah, if e i e j +c ijl >0, hen he firs case mus hold for. Thus, we have X r +e ijl =X r. We conclude by noing ha he possibiliy of having e i e j +c ijl =0 is ruled ou by he random perurbaion of he coss so ha problem (21) does no have alernaive opima. Proof of Lemma 3. To show he firs par, we consider wo cases depending on he sign of e i e j +c ijl : (1) If e i e j +c ijl 0, hen by Lemma 7, an addiional load of ype l on lane i j does no change he soluion of problem (21). Therefore, X r +e ijl = x and R +1 r +e ijl =r+1 hold. (2) Following an argumen similar o he proof of Lemma 7, if we inroduce an addiional load of ype l on lane i j, hen he change in he soluion of problem (21) is given by he min-cos flow augmening pah from node j o i j l in Figure A.1(b). Le his flow augmening pah be.if e i e j + c ijl <0, hen he second case in he proof of Lemma 7 holds. Therefore, firs connecs node j o node i, and

12 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model 330 Operaions Research 55(2), pp , 2007 INFORMS Figure A.1. (a) i (b) i [c ijl, ] (a) The cos and he upper bound for he bold arc ha connecs node i o node j are c ijl and d ijl. (b) The min-cos nework flow problem in Figure A.1(a) can be convered o one wihou he aforemenioned upper bound by a simple ransformaion. We inroduce an exra node i j l wih supply d ijl and se he supply of node j o +d ijl. (i, j,l) d ijl [c ijl, d ijl ] [0, ] j r j, +1 j +d ijl r j, +1 hen, node i o node i j l. This means ha is equal o he min-cos flow augmening pah e i e j appended by he arc ha connecs node i o node i j l. Then, he resul follows. The second par holds because any acyclic pah from node j I in he middle secion of Figure A.1(a) (or Figure A.1(b)) o node i I on he lef side raverses eiher zero or wo of he arcs corresponding o he variables r j +1 j I. If a pah raverses wo of hese arcs, hen one of hese arcs is a forward arc and he oher is a backward arc in he pah. Proof of Lemma 5. We now show he second par. We firs noe ha any acyclic pah from node i k I on he lef side of Figure 3 o he sink node can only visi he nodes i k k. Assume ha he resul does no hold. This means ha +1 ek i canno be wrien as a vecor whose elemens are +1, 1, or 0, and hence, here exis j I, k such ha k j +1 ek i 2. Assume ha k j +1 ek i 2. Because we have k j +1 ek i = k ıj l ek i ı I l L here exis i i I and l l L such ha i k j l ek i = +1 and i k j l ek i =+1. Because of our iniial observaion, we mus have i =i =i. Bu, having ij k l ek i =+1 and ij k l ek i =+1 implies ha, on he min-cos flow augmening pah from node i k o he sink node, here are wo forward arcs ha leave node i k. This conradics he fac ha he min-cos flow augmening pah is acyclic. One can also reach a conradicion by assuming ha k j +1 ek i 2. Proofs of Proposiions 4 and 6 use he following resul. Lemma 8. For a separable funcion G n, we have ( n n G x+ i e i ) G x = G x+ i e i G x i=1 i=1 where i for all i =1 n, and e i is he n-dimensional uni vecor wih a one in he ih elemen. Proof of Lemma 8. Leing G x = n i=1 g i x i, where g i for all i =1 n, wehave ( n G x+ i e i ) G x = = i=1 n g i x i + i g i x i i=1 {[ n i 1 i=1 j=1 g j x j +g i x i + i + Acknowledgmens n j=i+1 n j=1 ] g j x j } g j x j The auhors graefully acknowledge he commens of wo anonymous referees ha ighened he presenaion significanly. The work of he firs auhor was suppored in par by Naional Science Foundaion gran DMI The work of he second auhor was suppored in par by Air Force Office of Scienific Research gran AFOSR- FA and Naional Science Foundaion gran CMS References Carvalho, T. A., W. B. Powell A muliplier adjusmen mehod for dynamic resource allocaion problems. Transporaion Sci Crainic, T., M. Gendreau, P. Dejax Dynamic and sochasic models for he allocaion of empy conainers. Oper. Res Danzig, G., D. Fulkerson Minimizing he number of ankers o mee a fixed schedule. Naval Res. Logis. Quar Dejax, P., T. Crainic A review of empy flows and flee managemen models in freigh ransporaion. Transporaion Sci Franzeskakis, L., W. B. Powell A successive linear approximaion procedure for sochasic dynamic vehicle allocaion problems. Transporaion Sci. 24(1) Glasserman, P Gradien Esimaion via Perurbaion Analysis. Kluwer Academic Publishers, Norwell, MA.

13 Topaloglu and Powell: Sensiiviy Analysis of a Dynamic Flee Managemen Model Operaions Research 55(2), pp , 2007 INFORMS 331 Godfrey, G. A., W. B. Powell. 2002a. An adapive, dynamic programming algorihm for sochasic resource allocaion problems I: Single period ravel imes. Transporaion Sci. 36(1) Godfrey, G. A., W. B. Powell. 2002b. An adapive, dynamic programming algorihm for sochasic resource allocaion problems II: Muli-period ravel imes. Transporaion Sci. 36(1) Hane, C., C. Barnhar, E. Johnson, R. Marsen, G. Nemhauser, G. Sigismondi The flee assignmen problem: Solving a large-scale ineger program. Mah. Programming Ho, Y.-C., X.-R. Cao Perurbaion Analysis of Discree Even Dynamic Sysems. Kluwer Academic Publishers, Norwell, MA. Holmberg, K., M. Joborn, J. T. Lundgren Improved empy freigh car disribuion. Transporaion Sci Law, A. L., W. D. Kelon Simulaion Modeling and Analysis. McGraw-Hill, Boson, MA. Nemhauser, G., L. Wolsey Ineger and Combinaorial Opimizaion. John Wiley & Sons, Chicheser, UK. Powell, W. B A review of sensiiviy resuls for linear neworks and a new approximaion o reduce he effecs of degeneracy. Transporaion Sci. 23(4) Powell, W. B., P. Jaille, A. Odoni Sochasic and dynamic neworks and rouing. C. Monma, T. Magnani, M. Ball, eds. Neworks. Handbook in Operaions Research and Managemen Science. Norh- Holland, Amserdam, The Neherlands, Puerman, M. L Markov Decision Processes. John Wiley & Sons, New York. Topaloglu, H., W. B. Powell Incorporaing pricing decisions ino he sochasic dynamic flee managemen problem. Technical repor, School of Operaions Research and Indusrial Engineering, Cornell Universiy, Ihaca, NY. Topaloglu, H., W. B. Powell Dynamic programming approximaions for sochasic, ime-saged ineger mulicommodiy flow problems. INFORMS J. Compu. 18(1) Vanderbei, R Linear Programming: Foundaions and Exensions. Kluwer s Inernaional Series, Norwell, MA.