An Adaptive Opportunistic Routing Scheme for Wireless Ad-hoc Networks

Transcription

1 An Adaptive Opportunistic Routing Scheme for Wireess Ad-hoc Networks A.A. Bhorkar, M. Naghshvar, T. Javidi, and B.D. Rao Department of Eectrica Engineering, University of Caifornia San Diego, CA, 9093 {abhorkar, naghshvar, tjavidi, Abstract In this paper, an adaptive opportunistic routing scheme for muti-hop wireess ad-hoc networks is proposed. The proposed scheme utiizes a reinforcement earning framework to achieve the optima performance even in the absence of reiabe knowedge about channe statistics and network mode. This scheme is shown to be optima with respect to an expected average per packet cost criterion. The proposed routing scheme jointy addresses the issues of earning and routing in an opportunistic context, where the network structure is characterized by the transmission success probabiities. In particuar, this earning framework eads to a stochastic routing scheme which optimay expores and expoits the opportunities in the network. I. INTRODUCTION Opportunistic routing for muti-hop wireess ad-hoc networks has seen recent research interest to overcome deficiencies of conventiona routing [1] [6] as appied in a wireess setting. Opportunistic routing decisions are made in an on-ine manner, choosing the next reay based on the actua transmission outcomes as we as a rank ordering of reays. This on-ine and sampe-path dependent structure of opportunistic schemes improves the performance of routing by expoiting the broadcast nature of wireess transmissions as we as the inherent path and muti-user diversity present in a network. The authors in [1], [6] provided a Markov decision theoretic formuation for opportunistic routing. In particuar, it is shown that the optima routing decision at any epoch is to seect the next reay node based on an index summarizing the expectedcost-to-forward from that node to the destination. This index is shown to be computabe in a distributed manner and with ow compexity using the probabiistic description of wireess inks. The study in [1], [6] provides a unifying framework for amost a versions of opportunistic routing such as SDF [], GeRaF [3] and EXOR [4]. 1 The opportunistic agorithms proposed in [1] [6] impicity depend on a precise probabiistic mode of wireess connections and oca topoogy of the network. In a practica setting, however, these probabiistic modes have to be This work was partiay supported by the UC Discovery Grant #com , Ericsson, Inte Corp., QUALCOMM Inc., Texas Instruments Inc., CWC at UCSD, and NSF CAREER Award CNS The variations in [] [4] are due to the authors choices of cost measures to optimize. For instance, an optima route in the context of EXOR is computed so as to minimize the expected number of transmissions (ETX), whie GeRaF uses the smaest expected geographica distance from the destination as a criterion for seecting the next-hop. earned and maintained. With the exception of [7], which provides a sensitivity anaysis of opportunistic routing when channe modes are erroneous, by and arge, the question of earning and estimating channe statistics has not been expored in the opportunistic routing context. In this paper, using a reinforcement earning framework, we propose an adaptive opportunistic routing (AdaptOR) agorithm which minimizes the expected average per packet cost when zero or erroneous knowedge of transmission success probabiities and network topoogy is avaiabe. We woud ike to point out that our proposed scheme aso provides a generaization and an anaytica framework for the ticket-based probing heuristics in [8]. The rest of the paper is organized as foows: In Section II, we discuss the system mode and formuate the probem. Section III-A formay introduces our proposed routing agorithm, AdaptOR. We then state and prove the optimaity for AdaptOR agorithm in Section III-B. Finay, we concude the paper and discuss future work in Section IV. We end this section with a note on the notations used. For a vector x R D, D 1, we use x() to denote the th eement of the vector. We use n + to denote the time just after the start of sot [n, n + 1) and (n + 1) to denote the time just before the end of the sot [n, n + 1). II. SYSTEM MODEL We consider the probem of routing packets from the source node o to a destination node d in a wireess ad-hoc network of d + 1 nodes denoted by the set Θ = {o, 1,,..., d}. The time is sotted and indexed by n 0. Packets indexed by m 1 are generated at the source node o at a (possiby random) time m s according to an arbitrary distribution with rate λ > 0. We assume that the successfu reception of the packet transmitted by a node occurs according to a fixed conditiona probabiity distribution over the set of nodes in the network. Furthermore, we assume that successfu transmissions over different time sots are independent and identicay distributed. In particuar we characterize the behavior of the wireess channe using a probabiistic oca broadcast mode [6]. The oca broadcast mode is defined using the transition probabiity P (S i), S Θ, i Θ, where P (S i) denotes the probabiity of successfu reception of the packet transmitted by node i by a the nodes in S. Note that for a S S, successfu reception at S and S are mutuay excusive and

2 S Θ P (S i) = 1. Moreover, to mode i s abiity to reca the packet just transmitted, we assume that node i is aways a recipient of its own transmission, i.e. P (S i) = 0 if i / S. Loca broadcast mode generaizes the notion of ink and aows for correation of successfu receptions. When successfu transmission to various nodes are independent, P (S i) can be written as Π j S P ij where 0 P ij 1 represents the ink quaity. The successfu reception of the packet by the neighbors is assumed to be known at the centraized controer with zero error and deay. Given a successfu transmission from node i to the set of nodes S, the next (possiby randomized) routing decision incudes 1) retransmission by node i, ) reaying packet by a node j S, or 3) dropping the packet a together. If the controer decides to use node j for reay, then node j is assumed to transmit the packet at the next sot, whie other nodes k j, k S drop that packet. We assume a fixed transmission cost c i > 0 is incurred upon a transmission from node i. Transmission cost c i can be considered to mode the amount of energy used for transmission, the expected time to transmit a given packet, or the hop count when the cost is equa to unity. We define the termination event for packet m to be the event that packet m is either received by the destination or is dropped by a reay before reaching the destination. We define termination time e m to be a random variabe at which packet m is terminated. We discriminate amongst the termination events as foows: We assume that upon the termination of a packet at the destination (successfu deivery of a packet to the destination), a fixed and given positive reward R is obtained, whie if the packet is terminated (dropped) before it reaches the destination, no reward is obtained. Let r m denote the random reward obtained at the termination time e m, i.e. it is either zero if the packet is dropped prior to reaching the destination node or R if the packet is received at the destination. Given the assumptions and mode, the routing scheme can be viewed as seecting a (possiby random) sequence of nodes {i n,m } for reaying packets m = 1,,.... As such, the expected average per packet reward associated with routing packets aong sequence of {i n,m } is: im E N r m, (1) where denotes the number of packets terminated upto time N, i n,m denotes the index of the node which transmits packet m at time n, and the expectation is taken over the events of transmission decisions, successfu packet receptions, and packet generation times. 3 Probem (P) : We are interested in maximizing (1) by choosing the sequence of reay nodes {i n,m } in the absence The packets are indexed according to the termination order. 3 Our main resut estabishes the existence of an optima poicy which maximizes the im in (1). This is a strong notion of optimaity and impies that the proposed agorithm s expected average reward is greater than the best case performance (im sup) of a poicies [9, Page 344 ]. of knowedge about the oca broadcast mode. Remark The probem of opportunistic routing for mutipe source-destination pairs can be effectivey decomposed to the probem above where routing from one node to a specific destination is addressed. In proposing a soution to the Probem (P), we wi need the foowing definitions of action space, state space, and reward function. The set of a actions or the action space, is given by, A = Θ {f}, i.e. the set of reay nodes aong with the termination action f. The state space is given by a set S, S = i Θ {S : P (S i) > 0} {F }, denoting the sets of potentia reception outcomes from every node i Θ together with a termination state F. The termination state F is the state visited by the system when termination action f is chosen, i.e. P (F f) = 1. Given a set S of nodes that have received a packet, the set of aowabe actions is denoted by A(S) = S {f}. The aowabe action in the termination state F is f, i.e. A(F ) = {f}. It remains to define the reward function g : S A R to represent the reward obtained from taking an action at a given state. In summary, g(s, a) is given as: c a if a S g(s, a) = R if a = f and d S. 0 if a = f and d / S Let S n,m and a n,m be respectivey the state of the system and the routing decision at time n for packet m. Let H n be the σ-fied generated by s m, S m s,m, a m s,m,..., S n 1,m, a n 1,m, S n,m for a m such that s m n. An admissibe routing poicy φ is a random sequence of actions {a m s,m, a m s +1,m,...} for a packets m taking vaues on the aowabe action space A such that the event {a n,m = a} beongs to the σ-fied H n. The set of admissibe poicies for Probem (P) is denoted by Φ. III. THE ALGORITHM AND MAIN RESULTS A. Agorithm AdaptOR In this section, we present our Adaptive Opportunistic Routing (AdaptOR) agorithm to sove Probem (P). At each time sot n, AdaptOR uses a score vector Λ n in R v, where v = S S A(S) is the cardinaity of the domain S A. Remark Λ n (S, a) evauated at state S S and action a A(S), can be considered to be an estimate of the expected reward obtained by taking action a at state S at time sot n. AdaptOR is parametrized by a scaer constant 0 < γ 1 and a sequence of positive scaers {α n } n=1. During any time sot [n,n+1), the agorithm uses two counting random variabes ν n (S, a), N n (S), and two random sets W n and Y n to make routing decisions as we as to update the n th iterate Λ n.

3 3 Counting random variabes ν n (S, a) and N n (S) are equa to the number of times state-action pair (S, a) and state S have been reached respectivey upto time n. Random set W n Θ denotes the set of transmitting nodes during time sot [n 1, n), whie random set Y n consists of the set of potentia reays associated with transmissions from nodes in W n 1. Random counters ν n, N n, random sets Y n, W n, and Λ n are initiaized as foows: ν 0 (S, a) = 0, N 0 (S) = 0, Y 0 = {o}, W 0 = {o}, { R if (S, a) = (F, f) Λ 0 (S, a) = 0 otherwise To better conceptuaize the working of AdaptOR, we divide the execution of the agorithm into three stages of reception, adaptive computation, and reay/transmission. 1) Reception and Acknowedgment Stage: This stage is assumed to occur at time n. W n Θ denotes the (random) set of nodes each of which has transmitted one packet at time n. For any transmitter node a W n, et Sn a denote the (random) set of nodes that have successfuy received the packet from node a. In the reception and acknowedgment stage, the successfu reception of the transmitted packet is acknowedged by a the nodes in the set Sn a for a a W n. These nodes form the set of potentia reays for node a; coectivey they form random set Y n+1, i.e. Y n+1 := {Sn a : a W n }. Upon reception and acknowedgment, the counting random variabes are incremented as foows: { Nn 1(S) + 1 if S Y N n(s) = n+1, N n 1(S) if S / Y n+1 and ν n(s, a) = { νn 1(S, a) + 1 if (S, a) Y n W n ν n 1(S, a) if (S, a) / Y n W n. ) Adaptive Computation Stage: This stage is assumed to occur at n +. In this stage, for a (S, a) Y n W n, Λ n is updated as foows: Λ n(s, a) = Λ n 1(S, a)+ ( α νn(s,a) Λ n 1(S, a) + g(s, a) ) + max Λ n 1(Sn, a j) j A(Sn a). () For the state-action pair (S, a) / Y n W n, Λ n remains unchanged, i.e. Λ n (S, a) = Λ n 1 (S, a). 3) Reay/Transmission Stage:. This stage is assumed to occur at (n + 1). In this stage, the next set of reay nodes (actions) are seected. In particuar, for a S Y n+1, random action a S n+1 A(S) is seected according to the foowing (randomized) rue parameterized by γ ɛ n (S) = N n (S) + 1 with probabiity (1 ɛ n (S)), is seected, 4 with probabiity ɛn(s) A(S), a S n+1 arg max Λ n (S, j) a S n+1 A(S) is seected at random. At time (n + 1), the set of transmitters W n+1 = {a : S Y n+1, a Θ and a = a S n+1} is updated. A nodes in W n+1 transmit a packet at time (n+1). B. Optimaity of AdaptOR We wi now state our main resut on the optimaity of AdaptOR, φ Φ. Theorem 1. For a φ Φ, im im sup N E φ We prove the optimaity of AdaptOR in two steps. In the first step, we show that Λ n converges amost surey. In the second step we use this convergence resut to show that AdaptOR is optima for Probem (P). Let U : R v R v be an operator on vector Λ such that, (UΛ)(S, a) = g(s, a) + S P (S a) max j A(S ) Λ(S, j). Let Λ R v denote the fixed point of operator U, 5 i.e. Λ (S, a) = g(s, a) + S P (S a) max j A(S ) Λ (S, j), (3) Λ (F, f) = R. (4) The foowing theorem estabishes the convergence of recursion () to the fixed point of U, Λ. Theorem. Let (J1) Λ 0 (.,.) = 0 and Λ 0 (F, f) = R, (J) n=0 α n =, n=0 α n <. Then iterate Λ n obtained by the stochastic recursion () converges to Λ amost surey. 4 In case ambiguity, node with smaest index is chosen. 5 Existence and uniqueness of Λ is provided in [10].

4 4 Proof: The proof foows using known resuts on the convergence of a certain stochastic process presented in Theorems 1, in [11]. The detaied proof is provided in [10]. Using the convergence resut of Λ n, next we show that the expected average per packet reward under AdaptOR is equa to the optima expected average per packet reward obtained for a genie-aided system where the oca broadcast mode is known perfecty. In proving the optimaity of AdaptOR agorithm for Probem (P), we take cue from known resuts of a cosey reated Auxiiary Probem (AP) wherein the controer has perfect knowedge of oca broadcast mode as presented in [1], [6]. Let P be the sampe space of the random probabiity measures for the oca broadcast mode. Specificay, P := {p R d R d : p is a eft stochastic matrix}. Moreover, et P P be the trivia σ-fied generated by the oca broadcast mode P P (sampe point in P), i.e. P P = {P, P\P,, P}. 6 Let F n be the product σ-fied P P H n [1]. For Auxiiary Probem (AP), et admissibe routing poicy π be a sequence of actions {a m s,m, a m s +1,m,...} for packet m taking vaues on the aowabe action space A such that the event {a n,m = a} beongs to the σ-fied F n. Furthermore, et Π denote the set of admissibe poicies for Auxiiary Probem (AP). The reward associated with poicy π Π for routing a singe packet m from the source to the destination is given by J π ({o}) := E π r m F 0, (5) n=0 where F 0 = P P. Now, in this setting, we are ready to formuate the foowing Auxiiary Probem (AP) as a cassica shortest path Markov decision probem (MDP). Auxiiary Probem (AP) Find an optima poicy π such that, J π ({o}) = sup J π ({o}). (6) π Π Auxiiary Probem (AP) has been extensivey studied in [1], [6], [13] and the foowing theorem is estabished. Fact 1 (Theorem.1 [6]). There exists a function π : S A such that the poicy {a n,m = π (S n,m )} is an optima soution for the Auxiiary Probem (AP). 7 Furthermore, π is such that π (S) arg max V (j), (7) where (vaue) function V : A R is the unique soution to the foowing fixed point equation: V (d) = R (8) V (i) = max({ c i + S P (S i)(max j S V (j))}, 0)(9) V (f) = 0. (10) 6 σ-fied captures the knowedge of the reaization of oca broadcast mode and assumes a we-defined prior on these modes. 7 In other words there exists a stationary, deterministic, and Markov optima poicy for Auxiiary Probem (AP). Lasty, V (j) is the maximum expected reward for routing a packet from node j to destination d: V (j) = J π ({j}) = sup J π ({j}). π Π Lemma 1 beow states the reationship between the soution of Probem (P) and that of the Auxiiary Probem (AP) by estabishing V (o) as an upper bound for the soution to Probem (P). Lemma 1. Consider any admissibe poicy φ Φ for Probem (P). Then for a N = 1,,... E φ V (o). Proof: The proof is given in [10]. Intuitivey the resut hods because the set of admissibe poicies Φ in (P) is a subset of admissibe poicies Π in (AP). Lemma gives the achievabiity proof for Probem (P) by showing that the expected average per packet reward of AdaptOR is no ess than V (o). Lemma. For any δ > 0, im inf Proof: The proof is given in Appendix A. Lemmas 1 and impy that im V (o) δ. exists and is equa to V (o). This together with Lemma 1 estabishes the proof of Theorem 1. IV. CONCLUSIONS In this paper, we proposed an adaptive opportunistic routing scheme which maximizes the expected average per packet reward from the source to the destination in the absence of knowedge regarding network topoogy and ink quaities. We beieve that a decentraized and asynchronous extension of AdaptOR is straight-forward. The detais are subject of ongoing research. The broadcast mode used in this paper assumes a decouped operation at the MAC and network ayer. Whie this assumption seems reasonabe for many popuar MAC schemes based on random access phiosophy, it ignores the potentiay rich interpays between scheduing and routing which arises in many TDM based schemes such as [14]. The joint design of MAC and routing remains an important area of future research.

5 5 REFERENCES [1] C. Lott and D. Teneketzis, Stochastic Routing in Ad hoc Wireess Networks, Decision and Contro, 000. Proceedings of the 39th IEEE Conference on, vo. 3, pp vo.3, 000. [] P. Larsson, Seection Diversity Forwarding in a Mutihop Packet Radio Network with Fading channe and Capture, ACM SIGMOBILE Mobie Computing and Communications Review, vo., no. 4, pp. 4754, October 001. [3] M. Zorzi and R. R. Rao, Geographic Random Forwarding (GeRaF) for Ad Hoc and Sensor Networks: Mutihop Performance, IEEE Transactions on Mobie Computing, vo., no. 4, 003. [4] S. Biswas and R. Morris, ExOR: Opportunistic Muti-hop Routing for Wireess Networks, ACM SIGCOMM Computer Communication Review, vo. 35, pp. 3344, October 005. [5] S.R. Das S. Jain, Expoiting Path Diversity in the Link Layer in Wireess Ad hoc Networks, Word of Wireess Mobie and Mutimedia Networks, 005. WoWMoM 005. Sixth IEEE Internationa Symposium on a, pp. 30, June 005. [6] C. Lott and D. Teneketzis, Stochastic Routing in Ad-hoc Networks, IEEE Transactions on Automatic Contro, vo. 51, pp. 5 7, January 006. [7] T. Javidi and D. Teneketzis, Sensitivity Anaysis for Optima Routing in Wireess Ad Hoc Networks in Presence of Error in Channe Quaity Estimation, IEEE Transactions on Automatic Contro, pp , August 004. [8] W. Usahaa and J. Barria, A Reinforcement Learning Ticket-Based Probing Path Discovery Scheme for MANETs, Esevier Ad Hoc Networks, vo., Apri 004. [9] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, New York: John Wiey & Sons, [10] A.A. Bhorkar and M. Naghshvar and T. Javidi and B.D. Rao, An Adaptive Routing Scheme for Wireess Ad-hoc Networks, preprint. [11] J.N. Tsitsikis, Asynchronous Stochastic Approximation and Q- earning, Proceedings of the 3nd IEEE Conference on Decision and Contro, vo. 1, pp , Dec [1] Sidney Resnick, A Probabiity Path, Birkhuser, Boston, [13] Dimitri P. Bertsekas and John N. Tsitsikis, Parae and Distributed Computation: Numerica Methods, Athena Scientific, [14] M.J. Neey, E. Modiano, and C.E. Rohrs, Dynamic power aocation and routing for time varying wireess networks, INFOCOM 003, vo. 1, pp vo.1, March 003. A. Proof of Lemma APPENDIX Proof: From (3)-(4) and (8)-(9) we obtain the foowing equaity Let b = min arg max min S S i, Λ (S,i) Λ (S,j) V (j) = arg max Λ (S, j). (11) Λ (S, i) Λ (S, j). (1) Theorem impies that, in an amost sure sense, there exists packet index m 1 < such that for a n > m1 s, Λ n (S, a) Λ (S, a) b S S, a A(S). (13) Therefore, for a n > s m1, given any set of S, probabiity that AdaptOR chooses an action a A(S) such that Λ (S, a) max Λ (S, j) is upper bounded by ɛ n (S). Furthermore, each state is visited infinitey often (N n (S) ). As a resut, amost surey for any given η > 0, there exists packet index m < such that for a n > s m and for a S, ɛ n (S) < η. Let m 0 = max{m 1, m }. For a packets with index m m 0 the overa expected reward is upper-bounded by m 0 R max < and ower-bounded by m0 λ d max i c i >, hence their presence does not impact the expected average reward. Consequenty, we ony need to consider the errors due to random decisions of poicy φ (exporation) for packets m > m 0. Consider the m th packet generated at the source. Let Bk m be an event for which there exist k instances at which routing agorithm routes packet m differenty from the possibe set of optima actions. Mathematicay speaking, event Bk m occurs iff there exists instances s m n m 1 n m... n m k e m and actions {a n m 1, a n m,..., a n m k } such that for a = 1,,..., k Λ (S n m, a n m ) max j A(S n m ) Λ (S n m, j), where S n m is the set of nodes that have successfuy received packet m at time n m. We ca event Bk m a mis-routing of order k. For m > m 0, P rob(b m k ) (max S ɛ n(s)) k η k. For any packet m > m 0, et us consider the expected differentia reward under poicies π and φ : e m 1 E π rm e m 1 F 0 E φ rm = V (o) E φ = k=0 E φ V (o) Bm k P rob(bk m ) k R P rob(bk m ) (14) k=0 R kη k (15) k=1 = δ, where δ = ηr (1 η). Inequaity (14) is obtained by noticing that maximum oss in the reward occurs if agorithm AdaptOR decides to drop packet m (no reward) whie there exists a node j in the set of potentia forwarders such that V (j) R. Thus the expected average per packet reward under poicy φ is bounded as im inf im inf = V (o) δ. [ MN (V (o) δ) ]