A Queueing Model for Call Blending in Call Centers

A Queueing Model for Call Blending in Call Centers Sandjai Bhulai and Ger Koole Vrije Universiteit Amsterdam Faulty of Sienes De Boelelaan 1081a 1081 HV Amsterdam The Netherlands E-mail: {sbhulai, koole}@s.vu.nl Abstrat In this paper we study the sheduling of jobs with a onstraint on the average waiting time in the presene of bakground jobs. The objetive is to shedule to s servers suh that the throughput of the bakground traffi is maximized while satisfying the response time onstraint on the foreground traffi. A typial appliation of this model is all blending in all enters. Here the servers are alled agents, the foreground traffi are the inoming alls and the bakground traffi are the outgoing alls. The arrivals are determined by a Poisson proess and the servie times of the jobs are independent exponentially distributed. We onsider both the situation where servie requirements by both types of jobs are equal and unequal. The first situation is solved to optimality, for the seond situation we find the best poliy within a ertain lass of poliies. Optimal shedules always keep part of the servie apaity free for arriving foreground jobs. 1 Introdution In this paper we onsider a queueing system with two types of jobs. The first type of jobs has a onstraint on the performane, i.e., the average waiting time has to be below a ertain level. Next to this time-onstrained type there is a seond type of jobs, available in an infinite quantity, for whih the objetive is to serve as many as possible. The arrivals of the first job type are determined by a Poisson proess and the servie times of both job types are independent exponentially distributed. Both job types are served by a ommon pool of s servers under a non-preemptive regime. The question that we will answer in this paper is how to shedule these s servers as to maximize the throughput of type 2 jobs while satisfying the waiting time onstraint on the type 1 jobs. Sheduling a type 1 job and delaying a type 2 job does not hange the throughput, sine we fous on the long-term throughput. Therefore the question is, when a server beomes free and there are no type 1 jobs in the queue, whether this server should start serving a type 2 job or wait for a type 1 job to arrive. The optimal deision will appear to be a funtion of the state of the other servers. 1

A typial appliation of our model is all blending in all enters. Modern all enters deal with a mixture of inoming and outgoing alls. Of ourse there are onstraints on the waiting time of inoming alls. The traditional solution is to designate all enter employees often alled agents ) to either inoming or outgoing alls. However, the all rate flutuates over the day and in order to handle the alls during peak periods usually a substantial amount of agents needs to be assigned to inoming alls. This leads to a low produtivity of the agents during other periods. On the other hand, designating less agents to inoming alls inreases the produtivity, but leads to longer waiting times. Hene there is a need to balane produtivity and waiting times. The solution is all blending, dynamially assigning agents either to inoming or outgoing traffi. The model that we study in this paper represents exatly this situation: there are s agents, the time onstrained inoming alls are modeled as type 1 ustomers, and the type 2 alls represent the baklog of outgoing alls. The objetive is to obtain a simple model that provides insight into the system behaviour. Moreover the aim is to derive simple sheduling poliies that an be implemented in all enter software. The organization of this paper is as follows. In the next setion we give the exat model formulation. In Setion 3 we analyze the ase of equal servie requirements. In Setion 4 we analyze the ase of different servie requirements. 2 Model and first results The exat model formulation is as follows. There are two types of traffi, type 1 and type 2, having independent exponentially distributed servie requirements with rates µ 1 and µ 2. Type 1 jobs arrive aording to a Poisson proess with rate λ and there is an infinite waiting queue for jobs that annot be served yet. There is an infinite supply of type 2 jobs. There are in total s servers. The long-term average waiting time of the type 1 jobs should be below a onstant α. Waiting exludes the servie time; if the response time is to be onsidered, then the average servie time, 1/µ 1, should be added to the average waiting time. The objetive for type 2 jobs is to maximize its throughput, i.e., to serve on average per unit of time as many type 2 jobs as possible, of ourse at the same time obeying the onstraint on the type 1 waiting time. The following ontrol ations are possible. The moment a server finishes servie, or more general at any moment that a server is free, it an take one of the following three ations: start serving a type 1 job, if one or more are waiting in the queue for servie; start serving a type 2 job; remain idle. This finishes the desription of our model. In the next two setions we will deal with the ase µ 1 = µ 2 and µ 1 µ 2, respetively. A first question that has to be answered is whether it is at all possible to find a poliy that satisfies the waiting time onstraint of the type 1 traffi. 2

Lemma 2.1: Let ρ = λ/µ 1. The onstraint EW q α is satisfiable when λ, µ 1, and s are suh that ] 1 EW q ρ s s 1 ρ = µ 1 s 1)!s ρ) x ρ s 2 x=0 x! + α. s 1)!s ρ) Proof : It is lear that the waiting time for type 1 jobs is minimized if we shedule no type 2 jobs at all. Without the type 2 jobs the system beomes a standard M/M/s queue. If we denote with W q the stationary waiting time of this queue, then the question is whether EW q α. If the system is unstable, i.e., λ/µ 1 s, then EW q =. But the ondition λ/µ 1 < s is not suffiient. Thus EW q should first be alulated. Define Cs,ρ) as the stationary delay probability, for s servers and a load of ρ = λ/µ 1 Erlang. Then EW q is given by the wellknown formula with Cs,ρ) given by Cs,ρ) = EW q = Cs,ρ) µ 1 s ρ), ] 1 ρ s s 1 ρ s 1)!s ρ) x ρ s x=0 x! +. 1) s 1)!s ρ) By substitution of this term into EW q the result diretly follows. From now on we assume that λ, µ 1, and s are suh that EW q α. Cheking whether this is the ase an be easily done using the formulas above. In our model preemption of jobs in servie is not allowed. When preemption is allowed there is atually no problem to solve. The optimal poliy will assign all servers to type 2 jobs when no type 1 jobs are present in the system. When a type 1 job arrives then it is learly optimal to interrupt the servie of a type 2 job and to serve the type 1 job. Hene the waiting time onstraint is satisfied and the type 2 throughput is equal to µ 2 s λ 1 /µ 1 ). Note that any work-onserving poliy that satisfies the waiting time onstraint is optimal and ahieves the same throughput. In pratie the jobs that an be preempted in all enters are email messages. Therefore it is benefiial for all enters to stimulate email traffi. While formulating the model we stated that a free server an shedule a type 2 or a type 1 job when available) at any moment. Due to the fat that we are onsidering long-term average performane it is only optimal to shedule jobs at ompletion or arrival instants. Indeed, if it is optimal to keep a server idle at a ertain instant, then this remains optimal until the next event in the system. This follows diretly from the ontinuous-time Bellman equation. Therefore it suffies to onsider the system only at ompletion or arrival instants. Beause of this, and beause of the fat that the maximum total rate is uniformly bounded by λ +smax{µ 1,µ 2 }, we an use the well-known uniformization tehnique see Setion 11.5 of Puterman 6]). This allows us to use disrete-time dynami programming to ompute performane measures and to find the optimal poliy. 3

However, our system is not a standard Markov deision proess MDP), beause of the different objetives for queue 1 and queue 2. The form of the problem makes it a onstrained MDP; maximize the type 2 throughput with a onstraint on the type 1 waiting time. Constrained MDP s an be solved using various tehniques. Here we use one that introdues the onstraint in the objetive using a Lagrange multiplier. Under weak onditions it an be seen that the optimal poliy for a ertain Lagrange multiplier is optimal for the onstrained problem if the value of the onstraint under this poliy attains exatly α. From the theory on onstrained MDP s it follows that this poliy is stationary and randomizes in at most 1 state. For this and other results on onstrained MDP s, see the reent book of Altman 1]. 3 Equal servie requirements Let µ := µ 1 = µ 2. Consider the event that a server beomes free, and that there are one or more type 1 jobs waiting. Then the ontroller has to hoose between sheduling a type 1 or a type 2 job or idling, but this is evidently suboptimal). Giving priority to a type 2 job and delaying type 1 jobs obviously leads to higher waiting times. Delaying the proessing of a type 2 job does not hange the performane for this lass, as we are interested in the long-term throughput. This intuitive argument implies that, when a server beomes free and a type 1 job is waiting, it is optimal to assign this type 1 job to the server. The following oupling argument shows that this is indeed true. Theorem 3.1: Suppose that a server beomes free while there are type 1 jobs waiting in the queue. Then the optimal ation is to shedule a type 1 job. Proof : Let π be an arbitrary poliy whih respets the waiting time onstraint on the type 1 jobs. Suppose that there is a time instant, say t 1, where a type 2 job is sheduled, given that there is a type 1 job waiting in the queue. Sine π respets the waiting time onstraint on type 1 jobs there will be a later time instant, say t 2 where this type 1 job will be sheduled. Now onsider the poliy π whih follows all ations of π exept that it shedules a type 1 job at t 1 and a type 2 job at t 2. Note that this interhange does not hange the deision epohs for π, sine the servie requirement is µ for both job types. The total number of type 2 ustomers served after t 2 is equal under both polies, thus also the throughput. However, the average waiting time under poliy π is lower than under π, sine the type 1 job is served earlier. Hene the result follows. Sine both types of jobs have equal servie requirements, we need not make a differene between type 1 or type 2 jobs in servie. Therefore the state of the system is ompletely desribed by the number of jobs in servie plus the number of type 1 jobs in the queue. Thus the state spae is X = N 0. By Theorem 3.1 there annot be less than s type 1 jobs in servie while there are type 1 jobs waiting in the queue. Hene there is only a hoie in state x X if x < s; otherwise a type 1 job is automatially sheduled. The possible ations are denoted with a = 0,...,s x, orresponding to sheduling a type 2 jobs. We denote the transition rate of going from x to y before taking any ation) with px,y). Then we have px,x 1) = min{x,s}µ and px,x + 1) = λ. After suh an event an ation an be hosen, if the new state is below s. If ation a is hosen in x with a s x), then the system moves to x + a. 4

The objetives are modeled as follows. If ation a is hosen, then a reward of a is reeived, 1 for eah type 2 job that enters servie. By Little s law, the average waiting time is related to the average number of jobs in queue L q, by the following relation: EL q = λew q. Thus to obtain the average waiting osts EW q we an take rate osts equal to x s) + /λ. Next we uniformize the system see Setion 11.5 of Puterman 6]). For simpliity we assume that sµ+ λ 1. We an always get this by saling.) Uniformizing is equivalent to adding dummy transitions from a state to itself) suh that the rate out of eah state is equal to 1; then we an onsider the rates to be transition probabilities. Note also that rate osts in this ase are equivalent to lump osts at eah epoh. The two objetives are merged into a single reward funtion using a Lagrange parameter γ. Define the dynami programming operator T as follows: T f x) = γx s)+ λ + y {x 1,x,x+1} px, y) max a {0} {1,...,s y} {a + f y + a)}, with the onvention that {1,..., s y} = /0 if s y 0. Note the plae of the maximization: this makes that the ation depends on the state hanges that ourred. The dp operator an easily be rewritten in the standard formulation with a single overall maximization see Chapter 5 of Koole 3]), but this would onsiderably ompliate the notation.) The long-term average optimal ations are a solution of the optimality equation in vetor notation) h + g = T h. Another way of obtaining them is through value iteration, by reursively defining V n+1 = TV n, for arbitrary V 0. For n the maximizing ation onverges to the optimal ones. For existene and onvergene of solutions and optimal poliies we refer to Puterman 6].) We will next show that the optimal poliy is of threshold type. Theorem 3.2: There is a level, alled the threshold, suh that if x <, then the optimal ation is x. If x, then 0 is the optimal ation. Proof : We first derive ertain monotoniity properties of the value funtion V n with V 0 = 0. This is easier if we rewrite T as follows: T f x) = γx s)+ λ + px,y)u s) f y), y {x 1,x,x+1} with U s) the s-fold onvolution of the operator U, and U defined as { max{ f x),a + f x + 1)} if x < s, U f x) = f x) otherwise. It is obvious that both forms are equivalent: sheduling a group of type 2 jobs at the same time is equivalent to sheduling them one by one. We ontinue with the fat that we are only interested in γ 0. For γ = 0 the situation is as if the waiting time onstraint poses no real onstraint; for γ < 0 it pays to have a poliy that tries to derease the type 1 waiting time. For γ 0 the diret rewards γx s) + /λ are dereasing onave dv) as a funtion of x. The funtion V 0 = 0 is also dv. It is 5

straightforward to show that V n is dv for all n this is equivalent to the results derived in Setion 3 of Koole 4]). When applying U to f, we see that it is optimal to shedule a type 2 job if and only if f x) f x + 1) 1. If f is dv, then f x) f x + 1) inreases in x. Thus for eah f there is a threshold level at and above whih it is not optimal to shedule type 2 jobs, and below whih it is. Beause the U operator is repeated s times after eah event the threshold level will be reahed. In fat, there will never be sheduled more than one type 2 job, as the jump size is maximal 1.) We need to stress that it an be the ase that it is both optimal to shedule a type 2 job and not to shedule one. This ours if f x) = 1 + f x + 1). In this ase two threshold poliies, with threshold level and +1, are both optimal, and all the poliies that randomize between these two poliies. We will see that in general to find a threshold poliy that satisfies EW q = α, we need to randomize. Next we alulate for a fixed threshold poliy that randomizes between and + 1 its stationary type 1 waiting time EW q and its type 2 throughput. We assume that the poliy is suh that if a transition from + 1 to ours then with probability 1 δ a type 2 job is immediately sheduled. This results in a transition rate p + 1,) from + 1 to of p + 1,) = δ + 1)µ. The lowest possible state is, as the state moves immediately up to as soon as 1 is reahed. The other positive transition rates are px,x + 1) = λ for all x and px,x 1) = min{s,x}µ for all x > + 1. This results in a birth-death proess from whih the average waiting time and the throughput an be omputed as follows. Theorem 3.3: Let ρ = λ/µ. Then the average waiting time as a funtion of the threshold and the randomization parameter δ is given by EW q,δ) = ρ s! µδs 1)!s ρ) 2 q with q = The throughput ξ of type 2 jobs is given by ξ,δ) = µq s 1 x=max{,1} s 1 1 + x=+1 ρ x! δx! ] ρ x! sρ s! δx 1)! + λ. δs 1)!s ρ) + ] 1 ρ s!. δs 1)!s ρ) Proof : Let us alulate the stationary probabilities, whih are denoted with q x, for the birth-death proess. It is readily seen that with ρ = λ/µ) and with q = q x = ρx! q for all < x s δx! q x = ρx! δs!s x s q for all x > s, s 1 1 + x=+1 ρ x! δx! + ] 1 ρ s!. δs 1)!s ρ) 6

We define the probability of delay for this,δ) poliy by C,δ) s,ρ) = x=s q x = ρ s! δs 1)!s ρ) q. If = 0 and δ = 1 then no type 2 jobs are admitted, and indeed C 0,1) s,ρ) = Cs,ρ) is given by 1). The waiting time under the threshold poliy is also ompletely equivalent to the one without type 2 jobs: EW q,δ) = C,δ)s,ρ) µs ρ). This is the formula for the type 1 waiting times. Next we derive an expression for the type 2 throughput, whih we will denote with ξ. The throughput of type 2 is the total throughput minus the type 1 throughput. Therefore ] s 1 ρ ξ,δ) = µmin{x,s}q x λ = µq x! sρ s! x= δx 1)! + λ. δs 1)!s ρ) x=max{,1} The term max{,1} is there to prevent 1)! from appearing in ase = 0. Figure 1 displays the behaviour of the waiting time of type 1 jobs line) and the throughput of type 2 jobs dotted line) when varies. The used values of the parameters are: λ = 1/2, µ = 1/3 and s = 5. The figure should be interpreted as follows: to a given waiting time guarantee α to type 1 jobs, one an read the optimal threshold value in the figure. Next, one an read the throughput of type 2 jobs assoiated with α under the poliy using the threshold. 1.2 waiting time throughput 1 0.8 0.6 0.4 0.2 0 0 1 2 3 4 5 Figure 1: λ = 1/2, µ = 1/3 and s = 5. It is interesting to note that the average waiting time inreases slowly, while the throughput inreases nearly linearly. In the all enter framework one ould linearly inrease the produtivity of the agents, while very little additional waiting ours. 7

4 Unequal servie requirements When µ 1 µ 2 the analysis is more ompliated. In this ase we have to differentiate between type 1 ustomers and type 2 ustomers. The optimal poliy will depend on these different lasses and an be very ompliated. From a pratial viewpoint these poliies an also be diffiult to implement in all enter software. Therefore we prefer to study simpler poliies and we restrit ourselves to the lass of threshold poliies. This hoie is partially supported by the intuitive reasoning behind Theorem 3.1. Numerial experiments indiate that this theorem also holds in ase of unequal servie requirements. The restrition to the lass of threshold poliies fores the poliies to be simple and appealing. Moreover we will show by numerial omputation that threshold poliies are a good approximation to the optimal poliy. Let x denote the number of type 1 jobs in servie and in the queue, and let y denote the number of type 2 jobs in servie. Let the threshold be fixed. Then the stationary probabilities q x,y are determined by the following set of equilibrium equations. For y = we have s )µ1 + λ + µ 2 ) qx, = λq x 1, + s )µ 1 q x+1, x s, 2) xµ 1 + λ + µ 2 )q x, = λq x 1, + x + 1)µ 1 q x+1, 0 < x < s, 3) λq 0, = µ 1 q 1, + µ 1 q 1, 1. 4) For 0 < y < the set of equations beomes ) s y)µ1 + λ + yµ 2 qx,y = λq x 1,y + s y)µ 1 q x+1,y + y + 1)µ 2 q x,y+1 x s y, 5) xµ 1 + λ + yµ 2 )q x,y = λq x 1,y + x + 1)µ 1 q x+1,y + y + 1)µ 2 q x,y+1 y < x < s y, 6) y)µ1 + λ ) q y,y = y + 1)µ 2 q y,y+1 + y + 1)µ 1 q y+1,y + y + 1)µ 1 q y+1,y 1. 7) Finally for y = 0 we have sµ 1 + λ)q x,0 = λq x 1,0 + sµ 1 q x+1,0 + µ 2 q x,1 x s, 8) xµ 1 + λ)q x,0 = λq x 1,0 + x + 1)µ 1 q x+1,0 + µ 2 q x,1 < x < s, 9) µ 1 + λ)q,0 = µ 2 q,1 + + 1)µ 1 q +1,0. 10) These equations determine the dynamis of the system. Figure 2 summarizes the system dynamis graphially using state transition diagrams. λ xµ 1 λ s )µ 1 λ µ 2 µ 2 y)µ 1 λ x + 2)µ 1 λ s y)µ 1 λ yµ 2 yµ 2 µ 1 λ x + 4)µ 1 λ sµ 1 λ s Figure 2: System dynamis. 8

This infinite set of equilibrium equations an be numerially solved by using the matrixgeometri approah developed by Neuts 5] or by the spetral expansion method as desribed by Chakka and Mitrani 2]. However, note that in the equations there is no flow from q x,y towards level y + 1 or higher when x > y, sine jobs of type 1 are given priority over type 2 jobs. Due to this speial struture of the equations, we an solve this part of the system analytially using standard results from the theory of linear differene equations. The equilibrium equations are solved by solving the equations for y = and afterwards working the way down from y = 1 to y = 0. After solving these equations we obtain a finite set of equations still to be solved. However, these equations an easily be omputed numerially by applying the reurrene relations on the obtained solutions. Theorem 4.1: The solution to equations 2), 5) and 8) is given by q x,y = i=y K y,i y z x s i) i 0 y, x s y. 11) The onstants z i for i = 0,..., are the solution to a homogeneous differene equation and are given by z i = s i)µ1 + λ + iµ 2 ) s i)µ1 + λ + iµ 2 ) 2 4λs i)µ1 2s i)µ 1. The onstants K y,i are given by the reurrene relation K y,i = y + 1)µ 2 iµ 1 + µ 2 ) iµ 1 z y+i K y+1,i 1 0 y <,1 i y. 12) Proof : Fous the attention on a partiular row y with 0 y. Consider the orresponding homogeneous equations assoiated with y and x > s y; thus we are onsidering the homogeneous parts of 2), 5) or 8). Its solution is given by q x,y = q s y,y z x s y) y, where z y is the root of the polynomial λ s y)µ1 + λ + yµ 2 ) z + s y)µ1 z 2. By straightforward omputation one shows that one of the two roots is larger than 1, and therefore not useful. The other root, whih is positive and less than one, is given by z y = s y)µ1 + λ + yµ 2 ) s y)µ1 + λ + yµ 2 ) 2 4λs y)µ1 2s y)µ 1. Note that the + 1 roots z 0,...,z are different from eah other. Now we will show that the expressions 11) to 12) satisfy the differene equations. The ase y = has already been onsidered during the derivation of z, sine equation 2) is a homogeneous differene equation. Now let 0 y < and onsider equations 5) and 8) given by λq x 1,y s y)µ1 + λ + yµ 2 ) qx,y + s y)µ 1 q x+1,y + y + 1)µ 2 q x,y+1 = 0. 9

Substitute expression 11) into the left-hand side of this equation. Fix 1 i y and look at all fators z y+i. This yields the following expression K y,i z x s y i) 1 ) ] y+i λ s y)µ1 + λ + yµ 2 zy+i + s y)µ 1 z 2 y+i + y + 1)µ 2 K y+1,i 1 z x s y i) y+i. Now subtrat the following quantity from this. K y,i z x s y i) 1 ) ] y+i λ s y i)µ1 + λ + y + i)µ 2 zy+i + s y i)µ 1 z 2 y+i. This quantity is equal to zero, sine the terms between the brakets is the solution to the homogeneous differene equation for level y + i. The result of the subtration is given by ] K y,i z x s y i) 1 y+i iµ 1 + µ 2 )z y+i + iµ 1 z 2 y+i + y + 1)µ 2 K y+1,i 1 z x s y i) y+i. whih is equal to z x s y i) ) ] y+i iµ1 + µ 2 ) iµ 1 z y+i Ky,i + y + 1)µ 2 K y+1,i 1. Substituting expression 12) into this equation yields zero, showing that expressions 11) to 12) indeed are the solution to the differene equations 2), 5) and 8). At this point equations 2), 5) and 8) have been solved up to K 0,0,...,K,0. We still have a finite number of equations to solve, whih an be done numerially as follows. Using relations 3), 6) and 9) one an ompute the values of q x,y for 0 y with y < x < s y expressed in the unknown onstants. Then the boundary onditions 4), 7) and 10) relate the onstants to eah other leaving only one onstant to be determined. This onstant is finally determined by the ondition x,y q x,y = 1. Now the omplete system is determined. One the probabilities q x,y are known, the performane harateristis of Setion 3 for the two types of jobs an be easily omputed. The probability of delay is given by C ) s) = y=0 x=s y The waiting time of type 1 jobs is given by q x,y = y=0 i=y K y,i y 1 z i z i y i. EW q ) = 1 ] + µ 1 x + y) s q x,y, with x] + = max{x,0}. And finally the throughput of type 2 jobs is given by x,y ξ ) = µ 2 yq x,y. x,y Figure 3 illustrates the behaviour of the waiting time of type 1 jobs line) and the throughput of type 2 jobs dotted line) when µ 1 µ 2. The hosen parameters in this ase are λ = 1/2, µ 1 = 4/10, µ 2 = 3/10 and s = 5. Again we see that the average waiting time inreases slowly, while the throughput inreases nearly linearly. Sine the threshold poliy does not need to be the optimal poliy, it is interesting to numerially ompute the performane of the optimal poliy. The graph for the optimal poliy is generated as follows. 10

1.2 waiting time throughput optimal 1 0.8 0.6 0.4 0.2 0 0 1 2 3 4 5 Figure 3: λ = 1/2, µ 1 = 4/10, µ 2 = 3/10 and s = 5. For various values of the Lagrange parameter γ we derived the waiting times with the orresponding throughput. Mathing the throughput with the throughput omputed by the threshold poliy yields a value of. Then the minimal waiting time that an be ahieved for that level is the waiting time omputed by the optimal poliy, whih is denoted with a dot in the graph. It is interesting to note that the optimal poliy yields a performane very lose to the approximative threshold poliy. Extensive experiments show that for other parameter values the same result is obtained. The experiments indiate that the optimal poliy behaves nearly as a threshold poliy, but minor differenes our when x + y is lose to the threshold value. Referenes 1] E. Altman. Constrained Markov Deision Proesses. Chapman and Hall, 1999. 2] R. Chakka and I. Mitrani. Heterogeneous multiproessor systems with breakdowns: performane and optimal repair strategies. Theoretial Computer Siene, 125:91 109, 1994. 3] G.M. Koole. Stohasti sheduling and dynami programming. CWI Trat 113. CWI, Amsterdam, 1995. 4] G.M. Koole. Strutural results for the ontrol of queueing systems using event-based dynami programming. Queueing Systems, 30:323 339, 1998. 5] M.F. Neuts. Matrix-geometri solutions in stohasti models. Johns Hopkins University Press, 1981. 6] M.L. Puterman. Markov Deision Proesses. John Wiley & Sons, 1994. 11