An Optimization Framework for Practical Multipath Routing in Wireless Mesh Networks

Transcription

1 An Optimization Framework for Pratial Multipath Routing in Wireless Mesh Networks Božidar Radunović 1, Christos Gkantsidis 1, Peter Key 1, Pablo Rodriguez 2, and Wenjun Hu 3 1 Mirosoft Researh 2 Telefonia Researh 3 Cambridge University Cambridge, UK Barelona, Spain Cambridge, UK {bozidar,hrisgk,peter.key}@mirosoft.om pablorr@tid.es wenjun.hu@l.am.a.uk June 2007 Tehnial Report MSR-TR Mirosoft Researh Mirosoft Corporation One Mirosoft Way Redmond, WA

2 1 Abstrat We onsider wireless mesh networks, and exploit the inherent broadast nature of wireless by making use of multipath routing. We present an optimization framework that enables us to derive optimal flow ontrol, routing, sheduling, and rate adaptation shemes, where we use network oding to ease the routing problem. We prove optimality and derive a primal-dual algorithm that lays the basis for a pratial protool. Optimal MAC sheduling is diffiult to implement, and we use random sheduling rather than optimal for omparisons. Under random sheduling, our protool beomes fully deentralised. We use simulation to show on realisti topologies that we an ahieve % throughput improvement ompared to single path routing, and several times ompared to a reent related opportunisti protool (MORE). 1 Introdution One of the main hallenges in building wireless mesh networks ([1, 2, 3]) is to guarantee high performane. The diffiulty is mainly aused by the unpreditable and highly-variable nature of the wireless hannel. However, the use of wireless hannels presents some unique opportunities that an be used to improve the performane. For example, the broadast nature of the medium an be used to provide opportunisti transmissions as suggested in [4]. Also, in wireless mesh networks, there are typially multiple paths onneting eah soure destination pair; using some of these paths in parallel an improve performane [5, 6]. The optimal use of multiple paths and of opportunisti transmissions is the main fous of this work. We use network oding [7] to simplify the problem of sheduling paket transmissions aross multiple paths, similarly to [5, 6, 8]. We propose a network optimization framework that optimizes the rate of paket transmissions between soure and destination pairs. In order to use the resoures of the wireless mesh network effiiently, the system needs to take into aount: (a) the existene of multiple paths, (b) the unreliable nature of the links, () the existene of multiple transmission powers and rates (whih in turn affets the probability of orret paket reeption), (d) the broadast nature of the hannel, (e) the ompetition among many flows, (f) fairness and effiieny. Observe that optimizing aross all these parameters results in optimizing aross multiple layers of the networking stak; for example, the hoie of transmission power and rate is typially done at the physial layer, whereas oordination among different flows is typially done at the network layer. As we shall see, it is important to optimize all these parameters simultaneously to ahieve optimal performane. We use an optimization framework to design a distributed maximization algorithm. We aount for transport layer ontrols and address questions of fairness by maximizing the aggregate utility of the end-to-end flows, where we assoiate a utility funtion U( ) with a flow. Beause we use network oding, our optimization framework borrows heavily from [9, 8]. Our algorithm is a primal-dual algorithm [10]. The primal formulation expresses the optimization problem as a funtion of the rates of the various flows in the network; the dual formulation uses as variables the queue lengths (per flow and per node). The main advantage of using the dual formulation of the optimization problem is that the dual variables (also referred as shadow pries) relate to queue lengths and an be diretly used by bak-pressure algorithms for flow ontrol [11, 12]. As a simple example, a large number of queued pakets for a partiular flow at an internal node an be interpreted that the path going through that node is ongested and should be avoided. The main advantage of using the primal-dual formulation is that it adapts the primal variables (i.e. flow rates) more slowly, hene, allows TCP-like window-based rate ontrol modeling (as originally mentioned by Erylimaz et al. [11]). We propose a novel algorithm for ross-layer optimization and prove, using Lyapunov funtions, that it onverges to the optimal rate alloation. The proposed optimization framework is diffiult to implement; indeed, the joint sheduling, rate and power ontrol problem is NP-hard [13]. Additionally, urrent wireless MAC protools use unontrolled randomized hannel sheduling. We propose a distributed heuristi based on the optimal algorithm. We show that, even in the absene of optimal hannel sheduling, the other aspets of the optimization problem (i.e. flow seletion and transmission rate seletion) still give performane advantages. Hene, our heuristi an be implemented in pratial systems. The fundamental idea of our algorithm (and, of the distributed implementation) is to assign redits to nodes,

3 2 transfer redits between nodes, and shedule on the basis of redits (see Se. 3 for more details). The main ontributions of our paper are as follows: We propose a network wide optimization algorithm that maximizes rate-based global network performane, and extends previous work by inorporating broadast (opportunisti routing), multi-path routing, and fairness/rate ontrol (Setions 2 and 3). We introdue a notion of virtual pakets, alled redits, that enable us to deouple routing and flow ontrol from atual paket transmissions. We prove the optimality of the algorithm. Based on the optimization algorithm, we give a distributed implementation of routing, rate adaptation, and flow ontrol for networks with random sheduling (Setion 3.3) that outperforms existing algorithms. We prove that our algorithms extends and outperforms MORE [5]. The distributed algorithm an be used with the urrent MAC, and indeed is MAC independent. We also show how it an be used with pratial network oding shemes with finite generation sizes (Setion 4.1). We demonstrate that rate seletion is important for optimizing performane in a networks (Setion 4.2). We onfirm the findings from [5] that suh optimizations are not neessary for b networks. Using simulation on realisti topologies, we show we an ahieve % throughput improvement with our distributed implementation ompared to single path routing, and % ompared to MORE [5] (Setion 5). 1 2 Model In this setion we introdue the notation used in the paper. We extend the model of wireless erasure network developed in [14] to inlude multiple flows. Vetors are denoted in bold. 2.1 PHY and MAC Charateristis We onsider a network omprising of a set of nodes N, N = N. Whenever a node transmits a paket, several 1 Observe that MORE optimizes the number of transmitted pakets for flows in isolation, and may perform worse than single path, w.r.t. rates, when multiple flows are ative. 1 p12 p p24 p34 4 y 12 C 12 C 1{2,3} Figure 1: A network with 4 nodes is given on the left. For example, an ativation profile {(1, 2), (3, 4)} depits a profile where nodes 1 and 3 are transmitting, nodes 2 is reeiving a paket from node 1, while node 4 is reeiving from node 3. Profile {(1, {2, 3})} depits node 1 transmitting and nodes 2 and 3 reeiving (the same) paket from node 1. Feasible rate region for (y 12, y 13 ) is given on the right, desribe by inequalities: j J y 1j C 1J for all J {2, 3}. nodes may reeive it. We model paket transmission from node i to a set of nodes J N with a hyperar (i, J). We define an ativation profile S = {S l } to be a set of hyperars ative at the same time. There may be several onstraints on feasible ativation profiles. For example, a node may be limited to reeive from but one node, or transmit to only one node at a time. The only ondition we shall impose is that a node an be the soure of only one hyperar in one ativation profile. We denote by S the set of feasible ativation profiles and let SRC(S) = {i N J N, (i, J) S} be the set of transmitters in ativation profile S. Eah transmission has two assoiated parameters, power P P and rate R R, where P is the set of allowed transmission powers (e.g. P [0, P M ], where P M is given by regulations) and R is sets of available PHY transmission rates, defined by supported spreading, oding, and modulations. Consider an ativation profile S in whih node i transmits to set of nodes J, and suppose node i is transmitting with power P i and rate R i. We an assoiate power vetor P = (P i ) i N rate vetor P = (P i ) i N to these transmissions. Let T ij = 1 if a paket is suessfully transmitted from i to j J. We define p ij (P, R i, S) = Prob(T ij (P, R i, S) = 1) to be the probability that node j J will suessfully reeive the paket from i, given the above onditions. We also assume that T ij and T lk are independent for i l or j k, whih is justified C 13 y 13

4 3 by measurements (.f. [15]). By onvention, we assume p ij (P, R i, S) = 0 if j J, for (i, J) S. We an now alulate C ik (P, R i, S) the average number of pakets per unit time onveyed from node i to any of the nodes in K N. We have C ik (P, R i, S) = R i 1 (1 p ij (P, R i, S)). j K (1) Note that C ik (P, R i, S) = 0 if K J =, for (i, J) S. 2.2 Traffi, Routing and Flow Sheduling There is a set of uniast end-to-end flows C in the network, and eah flow C has a soure and a destination node Sr(), Dst() N respetively. Eah flows may take an arbitrary route from a soure to a destination, subjet to onstraints imposed by the set of ativation profiles. We denote by f the rate of flow C. One flow may take multiple routes to the destination. We assume routing is done at eah hop. Eah relay will deide how muh traffi from a flow it will forward to other nodes. This deision is made through redit assignment, as desribed next in Setion 2.4. Whenever a node is ative, it needs to deide whih flow it will transmit. This is defined through a flowsheduling profile matrix A. If node i transmits a paket from flow we set A i = 1, otherwise A i = 0. We say that a flow sheduling profile is valid if for eah i N there exists only one C suh that A i = 1. Let A be the set of all valid flow sheduling profiles. To illustrate the use of flow sheduling profile, onsider the example in Figure 1 having two flows C = {1, 2}, both from 1 to 4. The number of pakets for flow sent by node 1 and reeived by node 2 equals A 1 (C 12 (P, R i, S) + C 1{2,3} (P, R i, S)) and it depends not only on how often (1, 2) is sheduled, but also on how often (1, {2, 3}) is sheduled. This is why the flow sheduling deision is assigned to a node, instead to a link as in [16, 17, 11]. 2.3 Network Coding We assume network oding per flow is used [14, 5]. The main benefit of network oding is that it failitates sheduling. If the same paket is reeived by several nodes, a mehanism is needed to prevent two or more nodes forward the same pakets [4]. To eliminate this problem, eah relay forwards a random linear ombination of all previously reeived pakets from the same flow. Ideally, network oding should be performed at the sale of the whole flow. However, this is not pratial. Instead, pakets are divided in generations and only pakets from the same generation are ombined. For more details see [14, 5]. In Setion 3 we analyze networks with very large generation sizes (as in [14]) and we address the finite generation sizes in Setion Credits Whenever a paket is transmitted, it may be reeived by several nodes, and it is important to deide whih should forward pakets, to avoid redundant transmissions (as explained in [18, 5]). We will use a onept of redits, whih is similar to the ontrol deision variable of Neely [18]. Credits are reated for eah paket at the soure node, and identified with a generation, not a speifi paket. They are interpreted as the number of pakets of a speifi flow to be transferred by the node. Credits are onserved until they arrive at the flow s destination. The main advantage of the redit sheme is that it simplifies sheduling. Credits are delarations of intent. The atual paket transmissions may our at arbitrary time instants. Due to the use of network oding, we only need to ensure that the total number of pakets per generation transmitted between eah two nodes orresponds to the number of redits. Thus, sheduling is done at a generation level and not at the paket level, inuring signifiantly smaller overhead (espeially when the generation size is large). In pratie, redits an be piggybaked with paket transmissions. The reeiving node only updates its redits when a suessful paket transmission atually ours. In this work we assume there is an ideal (no loss and no delay) signaling plane that transmits redits and feedbaks. Our results an be extended to onsider imperfet signaling (f. [12]). As eah redit delegates one paket to a node, we may express all the rates in the system in terms of redits. For example, yij is the rate of redits of flow passed from node i to node j. Theorem 1 shows that the rate of in-

5 4 dependent pakets reeived at a destination of eah flow will orrespond to the number of redits delivered, when the generation size is large. 2.5 Dynamis and Stability We further assume the system is slotted in time. In eah slot t = 0, 1,... a medium aess protool assigns an ativation profile S(t) and a flow-sheduling profile A(t), and to eah transmitter i SRC(S(t)) we assign transmit power P i (t) and rate R i (t). We further denote by yij (t) the number of redits for flow transmitted from node i to node j during slot t, and with x ij (t) the number of pakets of flow atually transmitted from i to any of the nodes in J during slot t. Let f (t) be the number of fresh pakets/redits generated at the soure of flow. Note that, beause eah paket transmission is always assoiated to a redit transmissions, we look at redit queues. Let qi (t) be the amount of redits of flow queued at node i. The system is stable if every queue size is bounded. We will define stability more formally in Setion Optimal Flow Control For Fairness In this Setion we introdue the optimization problem (Se. 3.2), propose an algorithm for solving it (Se. 3.3), and prove that the algorithm onverges (Se. 3.4). Se.3.1 introdues some further notation that is needed for the desription of the optimization problem. Finally, in Se. 3.5 we ompare it theoretially with the MORE algorithm proposed in [5]. 3.1 Feasible Rate Set Assume an assignment of end-to-end rates f, for eah flow, and denote the rate vetor by f = (f ) C. The vetor of rates is valid under the following three onditions. First, by onservation of the flow of redits at eah node i Dst(): yji + f 1 i=sr() yij (2) j i j i Also, y ij 0. Seond, due to the onstraints of the broadast regions (see also Fig. 1): yij x ij (3) j J Assume now that variables α S,R,P,A define a shedule and denote a fration of time network uses sheduling profile S, routing profile A and power and rate alloations R, P. By definition, α S,R,P,A 0 and S,R,P,A α S,R,P,A 1. The third ondition omes from sheduling onstraints: x ij S,A,R,P α S,R,P,A A i C ij (P, R i, S) (4) Note that (as explained in Setion 2.2), although (4) implies { x ij } i,j belongs to Hull({C ij (P, R i, S)} i,j ), the onverse is not true. We will use the following haraterization of feasible rates from [8]: Definition 1. Vetor f is said to be feasible if eah flow an transport information from Sr() to Dst() at rate f. Theorem 1. Let F be the set of end-to-end rate vetor f = (f ) C suh that there exists vetors y = (y ij ) i,j N, C, x = (x ij ) i N,J N, C, and α = (α S,R,P,A ) S S,R RN,P P N,A A that satisfy (2), (3), and (4) subjet to α S,R,P,A 0 and S,R,P,A α S,R,P,A 1. The vetor f is feasible when oding generation size goes to infinity if and only if it belongs to F. Moreover, the set of feasible end-to-end rates F is onvex. Proof. Follows diretly from [8]. 3.2 Utility Maximization For eah flow C we define a utility funtion U ( ) to be a stritly onave, inreasing funtion of end-to-end flow rate f. The utility of flow is then U (f ). For example, U (f ) = log(f ) represents proportional fairness [19] and U (f ) 1/f approximates TCP s utility [10]. The goal of utility maximization is to ahieve trade-off between effiieny and fairness. A typial example of suh approah is proportional fairness [19].

6 5 We an write the network-wide optimization problem as max C U (f ) (5) s.t. f F. Sine set F is onvex and the objetive is stritly onave, there exists a unique solution f to the maximization problem. Corresponding y, x also exist but are not neessarily unique. Let us denote with µ i and ξ ij the Lagrangian multipliers assoiated with inequalities (2) and (3), respetively. To simplify the notation we will also define µ Dst() = 0. We an write the KKT onditions at the optimal point µ i yij yji f 1 i=sr() = 0, (6) j i j i f ξ ij x ij j J yij ( ) U (f ) µ Sr() = 0, (7) = 0, (8) We see that µ i an be positive only if more traffi of flow omes into node i than leaves it. Hene intuitively we an relate µ i to qi (t), the number of redits for flow queued at i. Similarly we an relate ξij to the number of pakets queued for broadasting at i. In Setion 3.3 we will express this relationship more formally. We will also use (8) to develop a flow ontrol algorithm. As a onsequene of KKT, using some elementary algebra one an derive 0 µ i µ j J N j J CiJ = argmax C ij {C ij (P,R i,s)} i ξ ij, (9) max J ξ ij C ij (10) Notably we will use (10) in Setion 3.3 to derive the optimal sheduling. 3.3 Maximization Algorithm We next present an algorithm that onverges to the optimal value of (5). In the following we assume that the feedbak is ideal, hene that the aknowledgments and redits are transmitted instantaneously and without errors. We leave the analysis of signaling with losses and delays for future work. Node and Transport Credits : Reall that qi (t) is the amount of redits of ommodity queued at node i. We all these redits node redits. In addition, let wij (t) be the number of redits of ommodity queued at i and orresponding to the pakets that have to be delivered to any of the nodes in J (as previously deided by the redit transmission sheme). We all these redit transport redits. When a redit for flow is passed from node i to node j, we derease qi, we inrease q j, and we inrease w ij for all J j (all of them by one unit). We derease wij when a paket from flow is atually transmitted from i to any of the nodes in J. Routing protool: Node redits represent intentions of paket transmissions and a routing protool desribes when and how are node redits transferred. Let yij (t) be the number of node redits for flow transferred from node i to node j at time t and let us define wij (t) = X N j X w ix (t). A bak-pressure between nodes i and j is defined as z ij(t) = q i (t) w ij(t) q j(t), the differene between the exess redits queued of flow at node i not destined for node j (qi w ij ) and the node redits at node j (qj ). A redit is routed from i to j only if the bak-pressure is positive: y ij(t) = M 1 {z ij (t)>0}, (11) where 1 {x>0} is 1 if x > 0 or 0 otherwise. In Setion 3.4 we will derive onditions on M to guarantee onvergene of the algorithm. Sheduling, rate and power ontrol: The optimal sheduling, rate and power ontrol algorithm is the tuple (S(t), P (t), R(t), A(t)) that solves the following op-

7 6 timization problem w i (t, P, R i, S) = max (S(t), P(t), R(t)) = argmax S,P,R J wij(t)c ij (P, R i,(12) S), i N w i (t, P, R i, S), (13) C ij (t) = C ij (P(t), R i (t), S(t)), (14) i (t) = argmax wik(t)c ik (t),(15) A i (t) = 1 {= i (t)}, (16) x ij(t) = A i C ij (t) (17) Equations (12)-(17) represent a joint sheduling, rate, and power ontrol problem. We find the optimal sheduling, power and rate ontrol (S(t), P (t), R(t)) by solving (13). Then, equation (15) is used to selet whih flow will be transmitted by eah node in slot t. Note that, as explained in Setion 2.2, we annot deouple the flow seletion proess A(t) and routing/sheduling/rate/power ontrol as it was done in similar approahes that do not use opportunisti routing (e.g. [16, 17, 11]). Also, unlike in [16, 17, 11, 18], we do not expliitly use bak-pressure information for sheduling in (12)-(17); instead we use transmission redits wij (t). Observe that all equations exept (13) use loal information only. Hene, with the exeption of (13) (assuming that we ould somehow ompute the values C ij (t)), the problem ould have been solved with a distributed algorithm. Reall from Se. 2.1 that C ij (t) relate to hannel sheduling. If hannel sheduling was determined by the MAC protool, as is typial in most urrent wireless tehnologies (where nodes randomly ompete for the wireless hannel), then the quantities C ij (t) are expliitly known and the rest of the optimization problem an be omputing using loal omputations. Of ourse, with random sheduling it is not possible to ahieve optimal performane, but, on the other hand, we an implement the optimization framework with a distributed algorithm and still observe performane benefits (as we shall see in Se. 5). Flow ontrol: The optimal flow rate at the soure, f (t) an be alulated using a primal-dual approah, as in [11] f (t + 1) = K [ ( )] + f (t) + γ U (f (t)) qsr() (t), (18) where [x] + = max{x, 0}. Eah flow adapts its rate based on the previous rate and urrent number of redits queing for transmission at the soure node for that flow (q Sr() ). The primal-dual approah well desribes additive-inrease multipliative-derease transport protools, like TCP [10]. 3.4 Convergene Of The Algorithm We now onsider a fluid model of the system, and show that it onverges to the optimal point. Analysis of a disrete-time model an be derived from our fluid-model analysis, using a similar approah to [11]. We assume that time is ontinuous and that queue evolutions are governed by following differential equations q i (t) = ẇ ij(t) = f (t)1 i=sr() + yji(t) j j j J y ij(t) x ij(t) yij(t) (19) q i (t) 0 wij (t) 0 (20) where (x) y>z equals x is y > z or 0 otherwise. Similarly, flow rate evolution in the fluid-model is given by f (t) = γ ( U (f (t)) q Sr() )f (t) 0. (21) We next prove that the algorithm presented in Setion 3.3 stabilizes the system with flow rates that maximize the optimization problem (5). Let us first define an ative link Definition 2. We say that link (i, j) is ative for flow if there exist a finite number T suh that for eah t that satisfies y ij (t) > 0, there exists t, t < t < t + T suh that y ij (t ) > 0. Intuitively, if a node is ompletely disonneted from the rest of the network, or in any way not used by a flow, redits will neither arrive to nor will leave the node. Thus tehnially, we annot guarantee that an arbitrary initial number of redits at this node will onverge to any partiular value. Instead, we onsider only ative links (and the orresponding nodes) whose average traffi is at least M/T. We then have:

8 7 Theorem 2. If M > max f then, starting from any vetors f(0), q(0), w(0), the rate vetor f(t) onverges to f as t goes to infinity. Furthermore, queue sizes qi (t), q j (t) and wij (t) on all ative links (i, j) for flow are bounded, and ξ ij re- and onverge to the shadow pries µ i spetively., µ j Proof. The proof uses a Lyapunov funtion with stability defined on the set of ative link, and we show that on all ative links that arry a positive amount of traffi, the delays are bounded hene the system is stable. The details are given in the appendix. 3.5 Comparison with MORE In this setion we ompare the performane of our algorithm with the MORE algorithm presented in [5, 20]. Our algorithm onverges to the optimal solution of the optimization problem (5) (as shown in Theorem 2), hene MORE at best is as good as our algorithm. We first show under what onditions MORE is guaranteed to give the optimal solution. We then also illustrate by two examples that MORE an yield stritly suboptimal rate alloations. Theorem 3. If there is only one flow in the system, if transmission rates and powers of all nodes are fixed and if only one node an transmit at a time (that is SRC(S) = 1 for all S S),MORE and our algorithm give the same performane. Proof. Sine only one node an transmit at a time, we have S = N. Furthermore, transmission powers and rates are fixed, hene (3) and (4) reads as j J y ij α i C ij. We also omit as there is only one flow in the system. We start with the MORE optimization problem, as given in [20], and we introdue f = 1/ ( i N z i), y ij = f x ij and α i = f z i. The optimization from [20, Eq.(1)-(4)] is then equivalent to min 1/f (22) s.t. y ij y ji = f1 {i=sr} (23) j N j N α i C ij y ij, (24) j J α i = 1, (25) i N 1 p 12 p p 24 p 35 y 1 y 3 Wall y 2 Figure 2: A network with 6 nodes. Due to a wall in between, nodes 2 and 4 do not interfere with nodes 3 and 5. The set of ativation profiles is S = {{(1, {2, 3})}, {(2, 3)}, {(3, 5)}, {(4, 6)}, {(5, 6)}, {(2, 4), (3, 5)}, {(2, 4), (5, 6)}, {(3, 5), (4, 6)}}. There are two flows in the system, flow f 1 = y 1 + y 2 whih is assigned 2 routes (y 1 = y and y 2 = y ) and flow f 2 = y 3 whih is assigned a single route (y 3 = y 2 4 ). 4 5 p 46 p 56 whih is exatly the optimization problem (5). We next give two examples where the performane of MORE is stritly suboptimal. Consider a hexagonal network depited in Figure 2. Let us first onsider a ase with a single flow f 1, (f 2 = 0), and in whih p 12 = p 24 = p 46 = 0.8 and p 13 = p 35 = p 56 = 0.2. Sine not all links interfere, the onditions of Theorem 3 are learly not satisfied. The optimal rate alloation that maximizes (5) is f = with y = and y = However, MORE will transmit all pakets over the path , hene the total rate will be f MORE = 0.267, some 15% less than the optimal. Intuitively, the reason why MORE is suboptimal is that it does not onsider possibility that links 3 5 and 5 6 transmit in parallel with 4 6 and 2 4. (Reall that MORE s goal is to minimize the number of transmissions and not to maximize the flow rate.) It will then onlude that forwarding any paket to 3 is largely suboptimal, sine links 3 5 and 5 6 are of a bad quality. Thus, if p 35 and p 56 are suffiiently smaller than p 12, p 24 and p 46, as in this example, MORE will not use route at all. In the seond example we again onsider the same hexagonal network but with both flows ative. Sine MORE algorithm does not take into onsideration ontention among flows, it will again assign all traffi to the path This traffi will ontend with 6

9 8 the traffi from flow 2, and feasible end-to-end rate alloations have to satisfy 3f 1 + f 2 = p. Note that the routing sheme is fixed by MORE and does not depend on flow ontrol applied by transport layer. If for example transport layer on the top of MORE is designed to maximize log utility, the optimal rates will be f1 MORE = p 12 /6 = 0.133, f2 MORE = p 12 /2 = 0.4. On the ontrary, our distributed algorithm will adapt routing to ontentions among flows, and it will assign y 1 = 0.12, y 2 = 0.03, f 1 = 0.15 and f 2 = y 3 = 0.4. As we an see, our algorithm balaned flow 1 by dereasing y 1 and inreasing y 2. As a result, the rate of flow f 2 stayed the same while the rate of f 1 inreased. 4 Pratial Issues In this setion we onsider two pratial issues that onern implementation of the protool proposed in Setion 3 in a mesh network: finite oding generation size and rate adaptation for randomized sheduling. We leave other pratial issues, suh as the effet of delayed feedbak, for future work. 4.1 Finite Generation Size Previous results assume that generation size used for network oding tends to infinity (see Theorem 1). Pratial reasons, suh as omplexity and performane of deoding, and header overhead for storing the oeffiient vetor, require us to limit the size of the header; some pratial systems limit the size to 32 pakets [5, 6]. We next modify our optimization framework to inlude finite generation size. Let G be the set of generations. Let us define qi (t, g) and wij (t, g) be the number of redits and transmission redits for generation g G of flow queued at i. Similarly, we define f (t, g), yij (t, g), x ij (t, g) as before, only onstrained on generation g (thus we have for example yij (t) = g G y ij (t, g), and by analogy for the other variables). The enoding proesses f (t, g) are defined at the soure. For example, if the size of eah generation is G, we have t f (t, g)dt = G for all g G. Extending the distributed maximization algorithm from Setion 3.3 to this setting is not straightforward. For example, a naive way to modify sheduling rule (17) would be to shedule the oldest generation available g i (, t) = min{g ( J)w ij(t, g) > 0}, (26) that is x ij (t, g) = C ij(t), if = i (t) and g = gi ( i (t), t). However, this rule may yield poor rates. To see why, onsider the example of a network with nodes {1, 2, 3} and a single flow, going from 1 to 3 (diretly or via node 2). Furthermore, suppose p 12 = p 23 = 1 and p 13 is lose to 0. Due to the nature of the bakpressure algorithm, queue w 13 will be filled with redits, to prevent 1 assigning more redits to y 13. Suppose that the oldest generation that has a redit in w 13 is generation g 13 = min{g w13(t, 1 g) > 0}. Sine p 13 is lose to 0, queue w 13 will take long time to get rid of redits from generation g 13. At the same time, sine p 12 = 1 queues w 12 and w 1{2,3} will quikly get rid of all generations older or equal to g 13. Thus, whenever node 1 is seleted to transmit, it will transmit a paket from generation g1(t) = g 13, aording to (26). Sine w 12 (t, g 13 ) = w 1{2,3} (t, g 13 ) = 0, node 2 doesn t need more pakets from generation g 13 and the paket it reeives will be useless. Indeed, it will almost never be reeived by node 3, yielding end-to-end rate f 0. Nevertheless, we still want to keep link 1-3 ative beause a few pakets transmitted over that link still improve overall performane. Finding a jointly optimal oding and sheduling strategy that maximizes system utility for finite generation sizes is a diffiult problem. In addition to the previously mentioned sheduling issue, when the generation size beomes finite, the results from [14] do not hold either. This implies that pakets reeived at the destination will not neessarily be linearly independent and Theorem 1 does not hold. Instead, we propose a heuristi inspired by the proof of Theorem 2, whih minimizes the drift Ẇ (f(t), q(t), w(t)). It onsists of modifying rules (11) and (17) to y ij(t, g) = g i (, t) = argmin g x ij(t, g) = { M, q i (t, g) qj (t, g) + w ij (t, g), 0, qi (t, g) > q j (t, g) + w ij (t, g) (27), wij(t)x ij(t) 1 {w ij (t,g)<0} (28) J { CiJ (t), = i (t), g = g i (, t), 0, otherwise (29) An explanation how this poliy is derived is given in the

10 9 Appendix. We demonstrate its performane, for finite generation sizes, by simulations in Setion Rate Adaptation For ompatible Sheduling Rates [Mbps] As shown in [13], finding the optimal sheduling rule (13) is an NP-hard entralized optimization problem. Some reent works [21, 13, 22] explore deentralized implementations of similar problems. Applying these ideas to our setting is out of sope of this paper, and left for future work. Instead, we will onsider a more realisti, suboptimal sheduling proess and we will show how our algorithm an be applied as a distributed heuristi. Like in most existing wireless mesh networks deployment, we will assume that nodes always transmit paket at the full power P i (t) = {0, P MAX }. We all a set of feasible ativation profiles S ompatible if for all S S and for all (i 1, J 1 ) S there is no (i 2, J 2 ) S suh that p i1,i 2 > 0, ( j J 2 )p i1,j > 0 or ( j J 1 )p i2,j > 0. Intuitively, this orresponds to like protool with RTS/CTS mehanism. When node i 1 establishes ommuniation with nodes J 1, all nodes involved in ommuniation send an RTS/CTS. All nodes that hear the RTS/CTS (p > 0) will be prevented from transmission or reeption during the same slot. Furthermore, we will assume that the underlying sheduling proess is not under our ontrol. At every time t, sheduling proess will selet a set of noninterfering nodes I(t) N to transmit (i.e. for eah i, j I(t), p ij = 0). Eah node i I(t) has a set of possible destinations J i (t) = {j N p ij > 0, ( k I(t), k i), p kj = 0}, whih in turns define a shedule S(t) = {i, J i } i I(t). A set of ativation profiles S = {{i, J i } i I I P(N )} is learly ompatible. Assuming previously defined shedule S(t), the (12) - (17) simplifies to ( i (t), R i (t)) = argmax,r i K J i w ik(t)c ik (R i ), (30) whih an be easily solved in a distributed manner, separately at eah node. We note that, for an arbitrary sheduling proess {S(t)} t, the distributed routing (11) and rate adaptation 0 Single path Multi path MORE Figure 3: End-to-end rate alloation example: eight flows among randomly seleted soure-destination pairs. On the y axis we give rates. Small bars denote rates per flow. Large bars denote total rate. Blak line gives utilities with an arbitrary saling (to fit the figure). Absolute improvement in utility (b) Run Relative improvement in total rate (a) Run Figure 5: Cumulative performane improvement of our algorithm over MORE: (a) Improvement in utility ( log(f m ) log(f s )); (b) Improvement in total rate ( f m / f s ). In all ases we run 100 experiments and we sort them by performane improvement. (30) algorithm does not neessarily minimize the optimization problem (5). The optimal algorithm will depend on the harateristis of {S(t)} t and it is diffiult to haraterize. We present (11) and (30) as a heuristi that an be used as a pratial implementations of opportunisti multi-path routing in networks with ompatible sheduling. We illustrate by simulations in Setion 5 that in the ase of random, ompatible sheduling, the heuristi (30) outperforms a onventional, single-path routing approah.

11 10 (b) (a) () Absolute improvement in utility Run Relative improvement in total rate Run Relative effiieny for total rate Run Figure 4: Cumulative performane improvement of multi-path over single-path: (a) Absolute improvement in utility ( log(f m ) log(f s )); (b) Relative improvement in total rate ( f m / f s ); () Relative effiieny for total rate ( f finite / f infinite ) for multi-path routing due to finite generation size of G = 32. In all ases we run 100 experiments and we sort them by performane improvement. In many ases, for single-path routing, some flow had zero rates for the duration of the simulation, due to slow onvergene. Sine this would yield to inifinite utility differene, we don t plot these runs. Fration of time Nodes 54.00Mbps 18.00Mbps 12.00Mbps 9.00Mbps 6.00Mbps Figure 6: Optimal distribution of PHY rates for roofnet network with a ards, for one random seletion of 8 soure-destination pairs. In ase of b ards, almost all nodes are assigned the highest, 11 Mbps rate. 5 Simulation Results We now present simulation results whih quantify the performane advantages of the opportunisti routing, sheduling and flow ontrol algorithms defined in the previous setions. We are primarily interested in pratial algorithms that an be applied in like mesh networks, where sheduling algorithm is not under our ontrol. Hene in our simulations we used an ompatible shedule {S(t)} t, as defined in Setion 4.2, assuming I(t) is randomly seleted among baklogged nodes. We ompared our algorithm with a onventional, single path routing algorithm, and with the MORE algorithm [5]. To make the omparison fair, we assumed that the single-path routing algorithm used the same kind of jointly-optimal routing and flow-ontrol approah as our sheme, whih boils down to [11]. In ontrast, MORE does not integrate flow ontrol or flow sheduling with the routing algorithm. When simulating the MORE algorithm, defined in [5, 20], we assumed that eah soure had a large baklog of pakets to transmit, and that eah relay performed FIFO sheduling among pakets from different flows. We used the roofnet network topology based on b ards, given in [4], for our simulations. Transmission probabilities between eah pair of nodes for different transmission rates are given in [1]. We used U ( ) = log( ), hene the rate alloation that maximizes (5) is the proportionally fair rate alloation [19]. We looked at two performane metris. The first one is the improvement in total utility U(f ) U(f ). Alloation f is better than f if the sum is positive. The proportional fair rate maximizes the optimization problem (5) hene has the highest utility. 2 The seond metri 2 Sine in the simulations we use random and not the optimal shedul-

12 11 is the total rate improvement f / f. Alloation f is better than f if the quotient is larger than 1. The proportionally fair alloation does not always have highest total rate. We developed a disrete-event simulator that implements the three routing, flow and rate ontrol algorithms. We ran simulations to obtain end-to-end rate alloations. Figure 3 illustrates the optimal rate alloations, obtained by our algorithm, for 8 randomly seleted souredestination pairs on one example. In this ase, we an see both utility and total rate inrease if we use multiple paths instead of a single one, where we benefit from broadast and multi-path diversity. We then ran the previous experiment with 100 random traffi matries and ompared the performanes of the different algorithms with respet to the two performane metris. Single vs. Multiple Paths We start by illustrating the benefits of the multi-path routing over the single-path routing in Figure 4. We first look at the network utility. The rate alloation obtained by the optimal algorithm (Setion 3.3) always maximizes the utility. However, this is not the ase for the distributed heuristi (Setion 4.2). In our simulations we saw that in about 90% of the runs, the distributed heuristi for multi-path routing ahieves higher utility than does single-path routing. In only about 10% of ases is the utility for single-path routing higher. Also, in more than 80% of runs, our deentralized heuristi ahieved higher total rate than the onventional, single-path algorithm. In more than half of the runs, the total rate has inreased by 20%, and in some ases by over 100%. From these results we see that there is a signifiant advantage in using our multi-path routing algorithm over the single-path one. We see that the advantage is signifiant even in number of flows is large (see next paragraph for explanation). Deentralized Heuristi vs. MORE We next ompare our deentralized heuristis with MORE. The results are depited in Figure 5. Network utility is inreased in about 90% of the runs. Total rate is inreased in almost all of the runs, sometimes up to a fator of 4. ing, the resulting rate alloation does not neessarily have the highest utility. From these results we an see that in many ases MORE behaves worse then the single-path routing. This resonates with the findings of [5] where the benefits of multi-path derease as the number of flows inrease (for an explanation, see Setion 3.5). In addition, MORE is only a routing protool whereas our single-path routing algorithm also inludes more intelligent flow ontrol and flow sheduling. Effets of Finite Generation Size Figure 4, () illustrates the impat of a finite generation size. The performane drop is due to imperfet sheduling (Setion 4.1) and oasional linear dependeny of reeived pakets. We an see that in most of the ases the effiieny loss is less than 5%. 3 The Optimal PHY Rate Seletion Finally, we onsider the optimal PHY rate seletion at different nodes. We analyze how frequently eah node uses eah PHY rate. In the ase of roofnet topology with b ards we find that in almost all ases it is optimal to use the highest rate of 11 Mbps, whih onfirms the findings from [5]. Furthermore, we use the SNR data from roofnet topology and measurements from [23] to analyze the approximatively optimal performane in the same network topology with a ards. The results are depited in Figure 6. As expeted, the optimal PHY rate seletion is no longer uniform, due to the large number of available rates. This demonstrates a need for an intelligent PHY rate seletion algorithm in this framework. 6 Related Work One of the first uses of opportunisti routing for uniast sessions in wireless mesh networks is presented in [4]. It has been extended in [5] to inlude network oding to failitate sheduling. However, in [5] the authors do not expliitly onsider multiple flows, fairness, nor sheduling, and in fat show that the performane benefit drops as number of flows inreases. An optimization framework for opportunisti routing that minimizes power onsumption is presented in [18], and shows signifiant benefits 3 Note that an additional performane drop for finite generation size may our due to imperfet signaling, but we do not onsider it in our model.

13 12 over [4]. Nevertheless, [18] does not onsider network oding, rate maximization, nor the TCP-like primal-dual rate adaptation. Cross-layer design for network oding with uniast or multi-ast sessions is onsidered in [8]; however, [8] onsiders only stability and not any form of rate maximization. Several theoretial analysis of linear network oding algorithms for uniast sessions have been performed [14, 9]. Network oding for uniast sessions is used also in COPE [24]. Compared to COPE, we perform enoding operations only between pakets of the same flow; in that respet our approah is orthogonal to COPE. Our work is an example of ross-layer optimization. Cross-layer design in wireless is a widely researh topis (see [12, 25] and referenes therein). Optimizing network performane in terms of network utility is originally proposed in [19]; see [10] for an overview. Our primal-dual approah is similar to [10], where it is shown that it an apture different versions of TCP. 7 Conlusions This paper proposes an optimization framework for addressing questions of multi-path routing in wireless mesh networks. We have extended previous work by inorporating the broadast nature of wireless and simultaneously addressing fairness issues. Impliit in our approah is the use of network oding, whih enables us to define notions of redits that are assoiated with number of pakets in a generation, rather than speifi pakets. Using our framework we show that our algorithm signifiantly outperforms single-path routing and MORE [5]. In the ase when sheduling is determined by a MAC, suh as by random sheduling or like sheduling, we have shown how our approah leads to a distributed heuristi, whih still outperforms existing approahes. Using a simulation results on a realisti topology, we found for our examples that for b, using maximal rate is optimal, but for a this was not the ase. We have addressed some of the pratial issues assoiated with having a finite generation size for network odes. Our primal-dual rate adaptation an be used to model window-based flow ontrol shemes, suh as TCP. The performane of appliations that run on top of our system and use TCP is an interesting open problem. Another interesting diretion is to analyze the performane of our protool with more realisti signaling shemes. Referenes [1] MIT roofnet - publiations and trae data, [2] J. Eriksson, S. Agarwal, P. Bahl, and J. Padhye, Feasibility study of mesh networks for all-wireless offies, in ACM/Usenix MobiSys, [3] M. Caesar, M. Castro, E. Nightingale, G. O Shea, and A. Rowstron, Virtual ring routing: Network routing inspired by DHTs, in ACM SigComm, [4] S. Biswas and R. Morris, ExOR: opportunisti multi-hop routing for wireless networks, in ACM SIGCOMM, [5] S. Chahulski, M. Jennings, S. Katti, and D. Katabi, Trading struture for randomness in wireless opportunisti routing, in ACM SigComm, [6] B. Radunovi, C. Gkantsidis, P. Key, S. Gheorgiu, W. Hu, and P. Rodriguez, Multipath ode asting for wireless mesh networks, in MSR-TR , Marh [7] R. Ahlswede, N. Cai, S. R. Li, and R. W. Yeung, Network information flow, IEEE Transations on Information Theory, [8] T. Ho and H. Viswanathan, Dynami algorithms for multiast with intra-session network oding, in 43rd Allerton Annual Conferene on Communiation, Control, and Computing, [9] D. Lun, M. Medard, and R. Koetter, Network oding for effiient wireless uniast, in IEEE International Zurih Seminar on Communiations, February [10] R. Srikant, The Mathematis of Internet Congestion Control. Birkhauser, 2003.

14 13 [11] A. Eryilmaz and R. Srikant, Joint ongestion ontrol, routing and ma for stability and fairness in wireless networks, IEEE Journal on Seleted Areas in Communiations, vol. 24, no. 8, pp , August [12] L. Georgiadis, M. J. Neely, and L. Tassiulas, Resoure alloation and ross-layer ontrol in wireless networks, Foundations and Trends in Networking, vol. 1, no. 1, pp , [13] G. Sharma, R. Mazumdar, and N. Shroff, On the omplexity of sheduling in wireless networks, in Pro. MOBICOM, [14] D. Lun, M. Medard, and M. Effros, On oding for reliable ommuniation over paket networks, in Pro. 42nd Allerton Conferene, [15] A. K. Miu, H. Balakrishnan, and C. E. Koksal, Improving Loss Resiliene with Multi-Radio Diversity in Wireless Networks, in 11th ACM MOBICOM Conferene, Cologne, Germany, September [16] L. Tassiulas and A. Ephremides, Stability properties of onstrained queueing systems and sheduling poliies for maximum throughput in multihop radio networks, IEEE Trans. on Automati Control, vol. 37, no. 12, [17] M. Neely, E. Modiano, and C. Rohrs, Dynami power alloation and routing for time-varying wireless networks, IEEE Journal on Seleted Areas in Communiations, vol. 23, no. 1, pp , January [18] M. Neely, Optimal bakpressure routing for wireless networks with multi-reeiver diversity, in Conferene on Information Siene and Systems, Marh [19] F. P. Kelly, A. Maulloo, and D. Tan, Rate ontrol in ommuniation networks: shadow pries, proportional fairness and stability, Journal of the Operational Researh Soiety, vol. 49, pp , [20] S. Chahulski, M. Jennings, S. Katti, and D. Katabi, MORE: A network oding approah to opportunisti routing, in MIT-CSAIL-TR , [21] P. Chaporkar, K. Kar, and S. Sarkar, Throughput guarantees through maximal sheduling in wireless networks, in Pro. Allerton, [22] E. Modiano, D. Shah, and G. Zussman, Maximizing throughput in wireless networks via gossiping, in Pro. ACM SIGMETRICS / IFIP Performane 06, June [23] L. Huang, B. Johnson, D. Tadas, and M. Stoler, a performane over various hannels, in IEEE / k, [24] S. Katti, H. Rahul, W. Hu, D. Katabi, M. Medard, and J. Crowroft, XORs in the air: Pratial wireless network oding, in ACM SigComm, [25] X. Lin, N. B. Shroff, and R. Srikant, A tutorial on ross-layer optimization in wireless networks, IEEE J. on Seleted Areas in Comm., vol. 24, no. 8, Aug Appendix Before proving Theorem 2 we first introdue the following lemma Lemma 1. The following equalities and inequalities hold i,j N C yij (µ i µ j ) = f µ i, (31) C i N,J P(N ) C i N,J P(N ) C i,j N, C ξij x ij = ξij ξ ij x ij ξ ij x ij j J y ij, (32) i N,J P(N ) C i,j N, C ξij x(33) ij, ξ ij y ij, (34) y ij (q i (t) q j(t)) = C f q Sr(), (35) wij i N,J P(N ) j J i N,J P(N ) C wij(t)x ij(t) yij = wijy ij, (36) i,j N i,j N, C w ij(t)y ij (37)

15 14 Proof. Equalities (31) and (32) follow diretly from (6) and (7). From (4) we further have C x ij = C ij. Together with (10) we an derive i,j, ξ ij x ij i,j, i,j, = i,j, ( J ( J ξij x ij ξ ij C ij ) ξ ij C ij whih proves (33). From (32) and (33) we derive (34). Sine by definition qdst() (t) = 0 we have i,j N, C y ij (t)(q i (t) q j(t)) = C,j N ) y Sr()j q Sr() from whih we derive (35). Equality (36) follows from the definition of wij. Also, by definition of sheduling (12)-(17), we have i,j, w ij(t)x ij(t) i i,j, max J wij(t)c ij ( C(38) ij ) w ij(t)x ij (39) is Ẇ (f(t), q(t), w(t)) (41) = + i, (f (t) f )(U (f (t)) q Sr() (t)) f (t) 0 (q i (t) µ i ) f i (t) + j yji(t) k + (wij(t) ξij ) yij(t) x ij(t) i,j, j J yik(t) wij (t) 0. As in (10)-(13) from [11] we have that, when qi (t) < 0 the derivative q i (t) is by definition positive; also µ i 0. The same holds for wij (t) and f (t) and we an upperbound q i (t) 0 Ẇ (f(t), q(t), w(t)) (f (t) f )(U (f (t)) qsr() (t)) + i, (q i (t) µ i ) f i (t) + j yji(t) k yik(t) + (wij(t) ξij ) yij(t) x ij(t) (42) i,j, j J Let us further add and substrat U (f ) = µ [11], to obtain Sr(), as in whih yields (37). Proof of Theorem 2: We will follow the idea of the proof of Theorem 2 from [11]. First, let us define Lyapunov funtion W (f, q, w) = 1 2γ (f f ) C i N,J P(N ) C i N, C (q i µ i ) 2 (wij ξij ) 2. (40) Ẇ (f(t), q(t), w(t)) + i, + i, µ i k qi (t) j y ik(t) j (f (t) f )(U (f (t)) U (f )) yji(t) k y ji(t) f i y ik(t) + f i + (wij(t) ξij ) yij(t) x ij(t). i,j, j J We want to show that the derivative Ẇ 0. For brevity, we define fi (t) = f (t)1 {i=sr()}. The derivative of W Due to onavity of U we have (f (t) f )(U (f (t)) U (f )) 0. Next, let us pik any set of link rates {yij } i,j, that orrespond to the optimal flow alloation

16 15 {f }. We expand f using (31) and (35) to obtain Ẇ (f(t), q(t), w(t)) i,j, (µ i + (qi qj )(yij yij(t)) i,j, µ j )(y ij(t) y ij ) + (wij(t) ξij ) yij(t) x ij(t) i,j, j J Let us denote with zij (34), (36) and (37) we have = µ i µ j ξij. Then, from Ẇ (f(t), q(t), w(t)) (yij(t) yij ) (43) i,j, [ (µ i µ j ξij ) (qi (t) qj(t) wij(t)) (44) ] = (yij(t) yij )(zij zij(t)) (45) i,j, (a) = yij(t)z ij (yij(t) yij )zij(t) (46) i,j, (b) (yij(t) yij )zij(t) () 0. (47) i,j, where (a) follows from KKT and the fat that yij z ij = 0, (b) from (9) and () from the fat that when zij (t) > 0 then yij (t) = M > max f max i,j, yij. Hene we have that Ẇ (f(t), q(t), w(t)) 0 for all f(t) > 0, q(t) 0, w(t) 0. Let us define Q(t) = (q, w) (yij(t) yij )(zij zij(t)) i,j, (48) Let us define E = {(f, q, w) Ẇ (f, q, w) = 0}. It is easy to see from (45) that E Q. We an further apply LaSalle s invariane priniple as in [11] to show that f(t) onverges to f and (q(t), w(t)) onverges to Q(t). However, set Q(t) is not bounded in general. If link (i, j) is ative for flow then for every t, yij (t) > 0 we have qj (t) + w ij (t) = q i (t) z ij. If the maximum node degree in a network is D, we have that qj (t) + w ij (t) qi (t) z ij + 2DT. Sine q Sr() onverges to U (f ) we see that queues qi (t), q j (t) and w ij (t) are bounded for all ative links (i, j) of eah flow. Derivation of (27)-(29): Let us write modified queue evolution equations q i (t, g) = (f i (t, g) + j y ji (t, g) ) j y ij (t, g) q i (t,g) 0 and ẇ ij (t, g) = ( j J y ij (t, g) x ij (t, g) )w ij (t,g) 0. Note that (19) and (20) do not hold anymore. Consequently, we annot laim that (42) follows from (41) and the proof of Theorem 2 annot be applied. We first onsider q i (t, g). We see from (27) that yik (t, g) > 0 only if q i (t, g) > 0. Thus, the exat queue evolution desribed with q i (t, g) = f i (t, g)+ j y ji (t) k y ik (t). Next, let us look at the evolution of w ij (t, g). We have that ẇij (t, g) = 0 only if w ij (t, g) = 0 and x ij (t, g) > 0, thus if g = g (, t). Therefore, we an write ẇ ij(t, g) j J ẇ ij(t) j J and from (41) we an write y ij(t) x ij(t, g) 1 {w ij (t,g) 0}, y ij(t) x ij(t) 1 {w ij (t,g (,t)) 0}. Ẇ (f(t), q(t), w(t)) (49) + i, (f (t) f )(U (f (t)) qsr() (t)) (50) (q i (t) µ i ) f i (t) + j yji(t) k yik(t) (51) + (wij(t) ξij ) yij(t) x ij(t) (52) i,j, j J + i,j, w ij(t)x ij(t) 1 {w ij (t,g (,t))<0}. (53) Intuitively (53) means that if we deide to transmit generation g (, t), we will not remove any redit from queues for whih there is no suh generation queued (that is wij (t, g (, t)) < 0). Note that this annot happen with infinite generation sizes as wij (t, g (, t)) < 0 implies wij (t) < 0. Sine we have already proven in Theorem 2 that (50) + (51) + (52) 0, we want to minimize (53), whih is indeed done in (28) and (29). It is also easy to see from (53) that the naive poliy (26) may yield an unbounded drift.