Approximate Dynamic Programming by Linear Programming for Stochastic Scheduling

Size: px

Start display at page:

Download "Approximate Dynamic Programming by Linear Programming for Stochastic Scheduling"

Moris Ferguson
5 years ago
Views:

1 Approximate Dyamic Programmig by Liear Programmig for Stochastic Schedulig Mohamed Mostagir Nelso Uha 1 Itroductio I stochastic schedulig, we wat to allocate a limited amout of resources to a set of jobs that eed to be serviced. Ulike i determiistic schedulig, however, the parameters of the system may be stochastic. For example, the time it takes to process a job may be subject to radom fluctuatios. Stochastic schedulig problems occur i a variety of practical situatios, such as maufacturig, costructio, ad compiler optimizatio. As i determiistic schedulig, the set of stochastic schedulig problems is icredibly large ad diverse. Oe importat class of models ivolves schedulig a fixed umber of jobs o a fixed umber of idetical parallel machies while miimizig a give performace measure. The processig times of the jobs are assumed to follow some joit probability distributio. I additio, there may be precedece costraits, or iterdepedecies betwee the jobs that require certai jobs ot be scheduled util others are completed. The determiistic couterpart to this class of problems has bee studied extesively [KSW98]. A aïve approach to these problems would be to take the expected processig times ad use the algorithms for the determiistic problems. Ufortuately, it is easy to costruct examples that shows that this approach ca produce solutios that are arbitrarily bad. Fortuately, however, this class of stochastic schedulig problems ca be easily cast as a Markov decisio process (MDP) ad therefore ca be attacked by dyamic programmig methods. Results for this class of stochastic schedulig problems are somewhat scattered, ad have bee obtaied usig a variety of methods. Rothkopf [R66] showed that for oe machie without precedece costraits, a idex rule miimizes the expected sum of weighted completio times for arbitrary processig time probability distributios. Möhrig, Radermacher ad Weiss [MRW84, MRW85] study the aalytic properties of various classes of schedulig policies, ad determie optimal policies for special cases. Möhrig, Schulz, ad Uetz [MSU99] developed approximatio algorithms for a variety of stochastic 1

2 schedulig problems usig techiques from combiatorial optimizatio. Bertsekas ad Castaño [BC99] focus o a differet class of stochastic schedulig problems, the quiz problem ad its variatios. They show how rollout algorithms ca be implemeted efficietly, ad preset experimetal evidece that the performace of rollout policies is ear-optimal. For this project, we cosider a problem with oe machie ad a arbitrary ormalized regular ad additive objective fuctio. We recast our fiite-horizo decisio problem ito a stochastic shortest path problem. We show that for a relaxed formulatio of our problem, the error of the solutio obtaied by the approximate liear programmig approach to dyamic programmig as preseted i de Farias ad Va Roy [dv03] is uiformly bouded over the umber of jobs that eed to be scheduled, provided that the expected job processig times are fiite. Fially, we argue usig results from dyamic programmig that the approximate solutio for the relaxed formulatio of our problem is also ot that far away from the optimal solutio of the origial problem. 2 The problem Cosider the followig problem. We have a set of jobs N = {1,...,} to be processed o oe machie. The machie ca oly process oe job at a time. The processig time of job i follows a discrete probability distributio p i, ad the distributios p 1,..., p are assumed to be pairwise stochastically idepedet. The jobs have to be scheduled opreemptively, that is, oce a job has started, it must be processed cotiuously util it is fiished. The schedule must also respect ay precedece costraits, or iterdepedecies betwee jobs that require certai jobs ot be scheduled before others are completed. We would like to miimize the expected value of the objective fuctio ( ) 1 ( ) γ C (1),...,C () = 1 h (R i ) C (i+1) C (i) i=0 where C (i) deotes the time of the ith job completio (C (0) = 0), R i is the set of jobs remaiig to be processed at the time of the ith job completio, ad h is a set fuctio such that h ( ) = 0. Such a objective fuctio is said to be additive. The fuctio h ca be iterpreted as the holdig cost per uit time o the set of ucompleted jobs. We also assume that γ is odecreasig i the job completio times. May commo objective fuctios i schedulig are odecreasig ad additive. For example, h (S) = j S w j for all S N geerates the ormalized 1 sum of weighted completio times, j N w j C j. 2.1 Formulatio as a fiite horizo MDP We ca formulate this problem as a fiite horizo MDP with a fiite state space S. For each state x S, there is a set of available actios A x. Takig actio 2

3 a A x i state x, we trasitio to state y with probability P a (x, y) ad icur cost g a (x, y). Sice we oly have oe machie, the state of the system x is sufficietly characterized by the set of jobs remaiig to be processed R x ad the maximum completio time of the jobs completed so far, C max (x): x = (C max (x),r x ) S. The size of the state space is expoetial i the umber of jobs. Sice the objective fuctio is odecreasig i completio times, ad sice the jobs processig times are idepedet of their start times, ay optimal policy eed ot leave the machie deliberately idle at ay time. Therefore, a actio i our problem is simply the ext job to be processed: at every state x, the actio set is A x R x. If there are o precedece costraits, A x = R x. The time stage costs are 1 g a (x, y) = h (R x )(C max (y) C max (x)). The trasitio probabilities are { p a (t) if R y = R x \ {a} ad C max (y) = C max (x) + t P a (x, y) = 0 otherwise. The problem is the to solve for the followig fiite-horizo cost-to-go fuctio J (x, ) = 0 J (x, t) = mi P a (x, y)(g a (x, y) + J (y, t + 1)), t = 0, 1,..., 1. a A x 2.2 Reformulatio ito a stochastic shortest path problem Sice our state space is expoetial i size, we caot hope to solve our problem exactly usig dyamic programmig methods. All of the major methods for approximate dyamic programmig cosider ifiite-horizo discouted cost problems. I order to use these methods for our fiite-horizo stochastic schedulig problem, we recast our problem ito a stochastic shortest path (SSP) problem. We refer to the followig formulatio of our problem as the origial SSP formulatio. We itroduce a termiatig state x with the followig property: oly the states that have o more remaiig jobs (R x = ) ca reach x i oe step. Observe that for all states such that R x = ad at the termiatig state x, the set of actios available A x is empty, ad therefore the trasitio probabilities ad time stage costs ivolvig x are ot affected by what actio is take. The trasitio probabilities ivolvig x are 1 x such that R x = P a (x, x) = 1 if x = x 0 otherwise 3

4 ad the time stage costs ivolvig x are { 0 x such that R x = g a (x, x) = 0 if x = x. The cost-to-go fuctio is therefore T(x) J (x) = mi E g u (x t, x t+1 ) x 0 = x. u where T (x) is the time stage whe the system reaches the termiatig state. Recall that a statioary policy u is called a proper policy if, whe usig this policy, there is positive probability that the termiatig state will be reached after a fiite umber of stages. Also recall that if all statioary policies are proper, the the cost-to-go fuctio for the SSP problem is the uique solutio to Bellma s equatio [B01]. Sice ay policy i our schedulig problem requires oe additioal job to be scheduled at each time stage, we must always reach the termiatig state with probability 1, regardless of the policy used. Therefore, to solve our problem exactly, we ca solve Bellma s equatio. 3 Solutio method ad bouds We cosider the liear programmig approach to dyamic programmig by de Farias ad Va Roy [dv03] as a solutio method to our problem. We briefly review the method ad related results. 3.1 The liear programmig approach to dyamic programmig Defie the operator TJ = mi {g u + αp u J}. u where the miimizatio is carried out compoet-wise. We wat to determie,α J, the uique solutio to Bellma s equatio, TJ = J. The exact liear programmig (ELP) approach to solvig Bellma s equatio solves the followig optimizatio problem for ay c > 0: maximize T c J subject to TJ J. Note that this optimizatio problem ca be recast as the followig liear program: T maximize c J subject to g a (x) + α P a (x, y)j (y) J (y), x S, a A x. 4

5 The approximate liear programmig (ALP) approach to dyamic programmig reduces the umber of variables i the exact liear programmig by the use of basis fuctios. Give a set of basis fuctios φ 1,...,φ K, we defie the matrix Φ = φ 1 φ K, ad i the hopes of computig a vector r such that Φ r is a close approximatio,α to J, we solve the followig optimizatio problem: maximize subject to T c Φr (1) TΦr Φr. This problem ca be recast as a liear program just as i the ELP approach. Give a solutio r, we hopefully ca obtai a good policy by usig the greedy policy with respect to Φ r, u (x) = arg mi g a (x) + α P a (x, y)(φ r)(y). a A x de Farias ad Va Roy [dv03] proved the followig boud o the error of the ALP solutio. Theorem 3.1. (de Farias ad Va Roy 2003) Let r be a solutio of the approximate LP (1), ad α (0, 1). The, for ay v R K such that (Φv) (x) > 0 for all x S ad αhφv < Φv,,α J 2c T Φv Φ r 1,c mi J 1 β Φv r where (HΦv)(x) = max P a (x, y)(φv) (y) a A x ad,α α (HΦv)(x) β Φv = max. x (Φv) (x) Φr 3.2 A uiform boud over umber of jobs,1/φv We relax our stochastic shortest path problem by itroducig discout factors α (0, 1) at each time stage. We refer to this formulatio as the α-relaxed SSP formulatio. The cost-to-go fuctio for this problem is T(x),α J (x) = mi E α t g u (x t, x t+1 ) x 0 = x. u For this relaxatio, we show that the error of the approximate liear programmig solutio is uiformly bouded over the umber of jobs to be scheduled. 5

6 Theorem 3.2. Assume that the holdig cost h (S) is bouded above by M for all subsets S of N. Let r be the ALP solutio to the α-relaxed SSP formulatio of the stochastic schedulig problem. For α (0, 1),,α 2M max i N E [p i ] J Φ r 1,c. Proof. Due to the ature of our problem, we kow that for ay state x, T (x) = R x +1. However, sice g a (x, x) = 0 for all x such that R x =, ad g a ( x, x) = 0, whe computig the cost-to-go fuctio at state x, we oly eed to cosider time stage costs for time stages 0 through R x 1. Sice the holdig cost h is bouded by above by M for all subsets of N, for some policy u we have R x 1,α J (x) = mi E α t g u(xt) (x t, x t+1 ) x 0 = x u R x 1 E α t g u(xt) (x t, x t+1 ) x 0 = x R x 1 E g u(xt) (x t, x t+1 ) x 0 = x R x 1 1 = E h (R xt )(C max (x t+1 ) C max (x t )) x0 = x, u R x 1 1 = E h (R xt )p u(xt) x 0 = x R x 1 1 E Mp u(xt) x 0 = x M = E [p i ] where p u(xt) is the processig time of the job chose at state x t. Cosider the fuctio k V (x) = E [p i ] for some costat k. The for all x S α (HV )(x) = α max P a (x, y)v (y) a A x 1 = αk max P a (x, y) E [p i ] a A x i R y 6

7 ( ) k k = α E [p i ] E [p a ] ( ) αe [p a ] = V (x) α E [p i ] < V (x) where a arg max a Ax P a (x, y)v (y). Therefore, V is a Lyapuov fuctio, ad there is a β < 1 idepedet of the umber of jobs such that αhv βv. Also ote that,α mi J r Φr J,α,1/V,1/V = max x S max,α (x) J V (x) [ ] M E x S k M = k [ i Rx p ] p i This is uiformly bouded over the umber of states. Fially, we cosider c T V. Let c be some probability distributio such that c (x) > 0 for all x S. ( ) k c (x) V (x) = c (x) E [p i ] x S x S ( ) 1 = k c (x) E [p i ] x S ( ) k c (x) max E [p i ] i N x S = k max E [p i ]. i N This is uiformly bouded over the umber of jobs, provided that the expected processig times of the jobs are bouded. Therefore, by Theorem 3.1, provided that E [p i ] is oe of our basis fuctios, we have that,α J 2c T Φv Φ r 1,c mi J α Φr 1,1/Φv βφv r 2M max i N E [p i ] which is uiformly bouded over the umber of jobs,. i 7

8 3.3 What happes whe α = 1? Ufortuately, the boud i Theorem 3.2 explodes for α = 1, ad therefore does ot directly apply to the origial SSP formulatio. The problem lies i the choice of the Lyapuov fuctio. If we use the same Lyapuov fuctio as i the proof of Theorem 3.2, we are left with the followig equality ( ) E [p a ] (HV )(x) = V (x) 1. E [p i ] As the umber of jobs, the ratio of the expected processig time of ay oe job to the sum of the expected processig times of all jobs goes to zero. Therefore, by usig the methods from the proof of Theorem 3.2, we caot fid a β < 1 such that HV βv regardless of the umber of jobs. However, sice the ALP solutio for the α-relaxed formulatio is ot that far away from the optimal solutio to the α-relaxed formulatio, we ca show that the ALP solutio obtaied for the α-relaxed SSP formulatio is also ot that far away from the optimal solutio of the origial SSP formulatio, depedig o the value of α. First, we briefly preset a well-kow result from dyamic programmig that relates the costs of the ifiite-horizo discouted cost ad average-cost problems. Lemma 3.1. For ay statioary policy u ad α (0, 1), we have J u J α,u = + h u + O ( ) where ( ) N 1 J α,u = α k k 1 P u g u, J u = lim Pu k g u, N N k=0 k=0 h u is a vector satisfyig J u + h u = g u + P u h u, ad O ( ) is a α-depedet vector such that lim O ( ) = 0. α 1 For the stochastic schedulig problem formulated as a stochastic shortest path problem, the J u i Lemma 3.1 is equal to zero. Recall that J is the cost-to-go fuctio for the origial SSP formulatio. Therefore, Lemma 3.1 implies,α,α J J J J = O ( ) J J,α O ( ) where J is the cost of usig the optimal greedy policy for the α-relaxed SSP formulatio i the origial SSP formulatio (h u i Lemma 3.1). Sice the costto-go fuctios for the origial ad α-relaxed SSP formulatios are ot that far 8

9 apart whe α is large, the ALP solutio for the α-relaxed SSP formulatio is ot that far away from the optimal solutio to the origial SSP formulatio whe α is large. Corollary 3.1. Let r be the optimal solutio to the ALP for the α-relaxed SSP formulatio. The for α (0, 1), J Φ r 1,c 2M max i N E [p i ] T c O ( ). Proof. The result follows immediately from Theorem 3.2 ad the argumets above: J Φ r J J,α,α 1,c 1,c + J Φ r 1,c T c J J,α + 2H max i N E [p i ] 2H max i N E [p i ] T c O ( ) Although we have had difficulty i obtaiig a error boud uiform over the umber of jobs for the ALP solutio whe α = 1, recet work [d04] idicates that relaxig the problem to iclude discout factors is the right approach to obtaiig good bouds for the origial SSP formulatio. 4 Coclusio We have show that i theory, the approximate liear programmig approach to dyamic programmig should be a good solutio method for the stochastic schedulig problem we studied. By relaxig the problem to iclude a discoutig factor at each time stage, we obtaied a boud o the error of the ALP solutio that is uiform over the size of the problem. Usig well-kow results i dyamic programmig, we also showed that by solvig the relaxed problem, we obtai a solutio that is ot that far away from the optimal solutio to the origial problem. The aalysis above also provides a error boud uiform over the umber of jobs for the multiple machie case. Although the state ad actio spaces of the problem with multiple machies are very complex, a upper boud o the cost-to-go fuctio at ay state ca always be obtaied by cosiderig the cost-to-go fuctio o just oe machie. Oe immediate aveue of further research would be to determie a error boud that is uiform over the umber of jobs for the ALP solutio to the origial SSP formulatio. Although our error boud does ot grow as the size of the problem grows, the error boud is still potetially very large. It would be ice to see if a tighter boud o the errors ca be obtaied. Aother directio of ivestigatio would be to ru computatioal experimets o usig ALP 9

10 for our stochastic schedulig problem. It would be ice to see how the ALP approach performs i practice, ad whether or ot i reality ALP solves the origial SSP formulatio (with α = 1) poorly as the size of the problem grows. Fially, the ALP method relies o solvig a liear program with a costrait for every state-actio pair, which i our problem would result i a extremely large optimizatio problem. Oe might ivestigate how the costrait samplig LP approach to dyamic programmig [dv01] performs with the stochastic schedulig problem we studied, both theoretically ad practically. Refereces [B01] [BC99] Bertsekas, D. (2001). Dyamic Programmig ad Optimal Cotrol. Athea Scietific, Belmot MA. Bertsekas, D., D. Castaño (1999). Rollout algorithms for stochastic schedulig problems. Joural of Heuristics 5, pp [d04] de Farias, D. P. (2004). Private commuicatio, May 10, [dv03] [dv01] de Farias, D. P., B. Va Roy (2003). The liear programmig approach to approximate dyamic programmig. Operatios Research 51, pp de Farias, D. P., B. Va Roy (2001). O costrait samplig for the liear programmig approach to dyamic programmig. Mathematics of Operatios Research, forthcomig. [KSW98] Karger, D., C. Stei, J. Wei (1998). Schedulig algorithms. CRC Algorithms ad Theory of Computatio Hadbook, Chapter 35. [MRW84] Möhrig, R. H., F. J. Radermacher, G. Weiss (1984). Stochastic schedulig problems I: geeral strategies. Zeitschrift für Operatios Research 28, pp [MRW85] Möhrig, R. H., F. J. Radermacher, G. Weiss (1985). Stochastic schedulig problems II: set strategies. Zeitschrift für Operatios Research 29, pp [MSU99] Möhrig, R. H., A. S. Schulz, M. Uetz (1999). Approximatio i stochastic schedulig: the power of LP-based priority policies. Joural of the ACM 46, pp [R66] [U96] Rothkopf, M. H. (1966). Schedulig with radom service times. Maagemet Sciece 12, pp Uetz, M. (1996). Algorithms for determiistic ad stochastic schedulig. Ph.D. dissertatio, Istitut für Mathematik, Techische Uiversität Berli, Germay. 10

Markov Decision Processes

Markov Decision Processes Markov Decisio Processes Defiitios; Statioary policies; Value improvemet algorithm, Policy improvemet algorithm, ad liear programmig for discouted cost ad average cost criteria. Markov Decisio Processes