A Survey on Delay-Aware Resource Control. for Wireless Systems Large Deviation Theory, Stochastic Lyapunov Drift and Distributed Stochastic Learning

Transcription

1 A Survey on Deay-Aware Resource Contro 1 for Wireess Systems Large Deviation Theory, Stochastic Lyapunov Drift and Distributed Stochastic Learning arxiv: v1 [cs.pf] 20 Oct 2011 Ying Cui Vincent K. N. Lau Rui Wang Huang Huang Shunqing Zhang Abstract In this tutoria paper, a comprehensive survey is given on severa major systematic approaches in deaing with deay-aware contro probems, namey the equivaent rate constraint approach, the Lyapunov stabiity drift approach and the approximate Markov Decision Process (MDP) approach using stochastic earning. These approaches essentiay embrace most of the existing iterature regarding deay-aware resource contro in wireess systems. They have their reative pros and cons in terms of performance, compexity and impementation issues. For each of the approaches, the probem setup, the genera soution and the design methodoogy are discussed. Appications of these approaches to deay-aware resource aocation are iustrated with exampes in singe-hop wireess networks. Furthermore, recent resuts regarding deay-aware muti-hop routing designs in genera muti-hop networks are eaborated. Finay, the deay performance of the various approaches are compared through simuations using an exampe of the upink OFDMA systems. Index Terms Deay-aware resource contro, arge deviation theory, Lyapunov stabiity, Markov decision process, stochastic earning. Ying Cui and Vincent K. N. Lau are with Department of Eectronic and Computer Engineering, Hong Kong University of Science and Technoogy, Hong Kong. Rui Wang, Huang Huang and Shunqing Zhang are with Huawei Technoogies Co., Ltd., China. This work was supported by GREAT Project, Huawei Technoogies Co., Ltd.

2 2 Fig. 1. Iustration of cross-ayer resource aocation with respect to both the MAC ayer state (QSI) and PHY ayer state (CSI). I. INTRODUCTION There is penty of iterature on cross-ayer resource optimization in wireess systems. For exampe, there are papers on joint power and subcarrier aocations to maximize the sum throughput for OFDMA systems [1], [2]. There are aso papers on joint power and precoder optimization to boost the sum rate, weighted sum MMSE or SINR for MIMO wireess systems [3], [4]. A these papers iustrate that significant throughput gain can be obtained by joint optimization of radio resource across the Physica (PHY) and the Media Access Contro (MAC) ayers. However, a typica assumption in these papers is that the transmitter has an infinite backog and the information fow is deay insensitive. As a resut, these papers focus ony on optimizing the PHY ayer performance metrics such as sum throughput, MMSE, SINR or proportiona fairness, and the resuting contro poicy is adaptive to the channe state information (CSI) ony. In practice, it is very important to consider random bursty arrivas and deay performance metrics in addition to the conventiona PHY ayer performance metrics in cross-ayer optimization, which may embrace the PHY, MAC and network ayers. A combined framework taking into account both queueing deay and PHY ayer performance is not trivia as it invoves both queueing theory (to mode the queue dynamics) and information theory (to mode the PHY ayer dynamics). The system state invoves both the CSI and the queue state information (QSI) and the deay-optima contro poicy shoud be adaptive to both the CSI and the QSI of wireess systems as iustrated in Fig. 1. This design approach is fundamentay chaenging for the foowing

3 3 reasons. First, there may not be cosed-form expressions reating the optimization objective (such as the average deay) and the optimization variabes (power, precoder, etc). Second, it is not cear if the optimization probems are convex (in most cases, they are not convex). Third, there is the curse of dimensionaity due to the exponentia growth of the cardinaity of the system state space as we as the arge dimension of the contro action space invoved (i.e., set of actions). For exampe, consider a queueing network with N queues, each with finite buffer size N Q. The size of the system state space is O(NQ N ), which is unmanageabe even for sma number of users N and buffer ength N Q. There are various approaches to dea with deay-aware resource contro in wireess networks [5], [6]. One approach converts average deay constraints into equivaent average rate constraints using the arge deviation theory and soves the optimization probem using a purey information theoretica formuation based on the rate constraints [7] [12]. Whie this approach aows potentiay simpe soutions, the resuting contro poicies are ony functions of the CSI and such contros are good ony for the arge deay regime where the probabiity of empty buffers is sma. In genera, optima contro poicies shoud be functions of both the CSI and QSI. In addition, due to the compex couping among queues in muti-hop wireess networks, it is difficut to express the average deay in terms of a the contro actions. Therefore, it is not easy to generaize this approach to joint resource aocation and routing in muti-hop wireess networks. A second approach to dea with deay-aware resource contro utiizes the notion of Lyapunov stabiity and estabishes throughput-optima contro poicies (in the stabiity sense). The throughput-optima poicies ensure the stabiity of the queueing network if stabiity can be indeed achieved under any poicy. Three casses of poicies that are known to be throughput-optima incude the Max Weight rue [13], the Exponentia (EXP) rue [14] and the Log rue [15]. Among the three casses, the throughput-optima property of the Max Weight type agorithms [16] and the Log rue [15] are both proved by the theory of Lyapunov drift, whereas the EXP rue is proved to be throughput-optima by the fuid imit technique aong with a separation of time scaes argument in [14]. Specificay, the genera Max Weight type agorithms are proved to minimize the Lyapunov drift, and hence, are throughput-optima. Many dynamic contro agorithms beong to this type, which incude optimizing the aocation of computer resources [17], and stabiizing packet switch systems [18] [21] and sateite and wireess systems [22] [24]. The Lyapunov drift theory (which ony focuses on controing a queueing network to achieve stabiity) is extended

4 4 to the Lyapunov optimization theory (which enabes stabiity and performance optimization to be treated simutaneousy) [13], [25] [27]. For exampe, utiizing the Lyapunov optimization theory, the Energy-Efficient Contro Agorithm (EECA) proposed in [27] stabiizes the system and consumes an average power that is arbitrariy cose to the minimum power soution with a corresponding tradeoff in network deay. In transport ayer fow contro and network fairness optimization, the Cross Layer Contro (CLC) agorithm was designed in [25] to achieve a fair throughput point which is arbitrariy cose to optima with a corresponding tradeoff in network deay, when the exogenous arriva rates are outside of the network stabiity region. In [28] and [29], the authors consider the asymptotic singe-user and muti-user power-deay tradeoff in the arge deay regime and obtain insights into the structure of the optima contro poicy in the arge deay regime. Athough the derived poicy (e.g., dynamic backpresssure agorithm) by the Lyapunov drift theory and the Lyapunov optimization theory may not have good deay performance in moderate and ight traffic oading regimes, it aows potentiay simpe soutions with throughput optimaity in muti-hop wireess networks. However, throughput optimaity is a weak form of deay performance and it is aso of great interest to study scheduing poicies that minimize average deay of queueing networks. A more systematic approach in deaing with deay-optima resource contro in genera deay regime is the Markov Decision Process (MDP) approach. In some specia cases, it may be possibe to obtain simpe deay-optima soutions. For exampe, in [30], [31], the authors utiize Stochastic Majorization to show that the ongest queue highest possibe rate (LQHPR) poicy is deay-optima for mutiaccess systems with homogeneous users. However, in genera, the deayoptima contro beongs to the infinite horizon average cost MDP, and it is we known that there is no simpe soution associated with such MDP. Brute force vaue iterations or poicy iterations [32], [33] coud not ead to any viabe soutions due to the curse of dimensionaity. In addition to the above chaenges, the probem is further compicated under distributed impementation requirements. For instance, the deay-optima contro actions shoud be adaptive to both the goba system CSI and QSI. However, these CSI and QSI observations are usuay measured ocay at some nodes of the network and hence, centraized soutions require huge signaing overhead to deiver a these oca CSI and QSI to the centraized controer. It is very desirabe to have distributed soutions where the contro actions are computed ocay based on the oca CSI and QSI measurements.

5 5 A systematic understanding of deay-aware contro in wireess communications is the key to truy embracing both the PHY ayer and the MAC ayer in cross-ayer designs. In this paper, we give a comprehensive survey on the major systematic approaches in deaing with deayaware contro probems, namey the equivaent rate constraint approach, the Lynapnov stabiity drift approach and the approximate MDP approach using stochastic earning. These approaches essentiay embrace most of the existing iterature regarding deay-aware resource contro in wireess systems. They have their reative pros and cons in terms of performance, compexity and impementation issues. For each of the approaches, we discuss the probem setup, the genera soution, the design methodoogy and the imitations of deay-aware resource aocations with simpe exampes in singe-hop wireess networks. We aso discuss recent advances in deay-aware routing designs in muti-hop wireess networks. The paper is organized as foows. In Section II, we eaborate on the basic concepts of crossayer resource aocation, which consists of the system mode, the source mode, the contro poicies, the queue dynamics and the genera resource contro probem formuation. In Section III, we eaborate on the theory and the framework of the first approach (equivaent rate constraint). In Section IV, we eaborate on the theory and the framework of the second approach (Lynapnov stabiity drift). In Section V, we eaborate on the theory and the framework of the third approach (MDP) and iustrate how the approximate MDP and stochastic earning coud hep to obtain ow compexity and distributed deay-aware contro soutions. In Section VI, we discuss the deay-aware routing designs in muti-hop wireess networks. In Section VII, we compare the performance of the aforementioned approaches in a common appication topoogy, namey the upink OFDMA systems with mutipe users. Finay, we concude with a brief summary of the resuts in Section VIII. II. SYSTEM MODEL AND GENERAL CROSS-LAYER OPTIMIZATION FRAMEWORK In this section, we eaborate on the system mode, the queue mode, the framework of resource contro for genera wireess networks. We aso use the upink OFDMA systems as an exampe in the eaboration to make the description easy to understand.

6 Fig. 2. Iustrative diagram of a muti-hop wireess network with N = {1, 2,..., 9}, L = {1, 2,..., 1} and C = {1, 2, 3}. A. System Mode In this paper, we study deay-aware resource contro in a genera muti-hop wireess network with a set of N nodes N = {1, 2,..., N} and a set of L transmission inks L = {1, 2,..., L} as iustrated in Fig. 2. Each ink in set L denotes a communication channe for direct transmission from node s N to node d N, and is abeed by the ordered pair 1 (s, d). We denote s() and d() as the transmit node and the receive node of the -th ink, respectivey. The network is assumed to work in sotted time with sot boundaries that occur at time instances t {1, 2,...}. We use sot t to denote the time interva [t, t + 1). Denote H(t) = [H 1 (t), H 2 (t),..., H L (t)] H as the CSI of a L inks in set L in sot t, where H denotes the system CSI state space. We have the foowing assumption on the channe fading. Assumption 1 (Assumption on the Channe Fading): Each eement in H takes vaue from the discrete state space H and the system CSI H(t) is a Markov process, i.e., Pr [ H(t) H(t 1), H(t 2),..., H(0) ] = Pr [ H(t) H(t 1) ]. 1 Note that (s, d) and (d, s) denote two different transmission inks: the former is the ink from the s-th node to the d-th node, whereas the atter is the ink from the d-th node to the s-th node.

7 7 The genera network mode described above encompasses a wide range of practica network topoogies. B. Source Mode A data that enters the network is associated with a particuar commodity 2 c C, which minimay defines the destination of the data, but might aso specify other information, such as the source node of the data or its priority service cass [13]. C = {1, 2,, C} represents the set of C commodities in the network. Let λ (c) n (t) denote the amount of new commodity c data (in number of bits) that exogenousy arrives to node n at the end of sot t. We make the foowing assumption on the arriva process. Assumption 2 (Assumption on Arriva Process): The packet arriva process λ (c) n (t) [0, λ (c) n,max] is i.i.d. over scheduing sots foowing genera distribution with average arriva rate E[λ (c) n (t)] = λ (c) n. Each node n maintains a set of queues for storing data according to its commodity. Let Q (c) n (t) denote the queue ength (in number of bits) of commodity c stored at node n. Note that we et Q (c) n (t) = 0 for a t if node n is the destination of commodity c. Let µ (c) (t) denote the rate offered to commodity c over ink during sot t. Therefore, the system queue dynamics is given by [13] Q (c) n (t+1) max Q(c) n (t) {:s()=n} (t), 0 µ (c) +λ(c) n (t)+ {:d()=n} µ (c) (t), n N. (1) The above expression is an inequaity rather than an equaity because the actua amount of commodity c data arriving to node n during sot t may be ess than {:d()=n} µ(c) (t) if the neighboring nodes have itte or no commodity c data to transmit. For notationa convenience, we define the QSI as Q(t) = [ Q (c) n (t) ] Q, where Q denotes the system QSI state space. C. Contro Poicy and Resource Contro Framework Let χ(t) = {H(t), Q(t)} X be the system state which can be estimated by the resource controer at the t-th sot, where X = H Q is the fu system state space. In practice, different contro poicies may be adaptive to partia or fu system states. For exampe, a CSI-ony contro 2 The commodity index c can be interpreted as the data fow index in the network.

8 8 poicy has contro actions that are adaptive to the partia system state CSI ony. A QSI-ony contro poicy has contro actions that are adaptive to the partia system state QSI ony. A crossayer contro poicy has contro actions that are adaptive to the fu system state, i.e., the CSI and the QSI. We define Ω : X A to be the contro poicy, which is a mapping from the fu system state space X to the action space A. The contro poicy may incude the resource aocation poicy (e.g., power aocation poicy, subcarrier aocation poicy, precoder design poicy, etc) and the routing poicy. by Under contro poicy Ω, the average queue ength of commodity c stored at node n is given Q (c) n = im sup T + 1 T T t=1 E Ω [Q (c) n (t)], n N, c C, where E Ω [ ] means the expectation operation taken w.r.t. the measure induced by the given poicy Ω. We aso introduce the average drop rate as a performance metric in our genera system mode to incorporate deay-aware resource contro in queueing networks with finite buffer size (c.f., [32]), where data dropping is necessary when a buffer overfows. For queueing networks with finite buffer size N Q, the average drop rate of commodity c stored at node n is defined as d (c) n = im sup T + 1 T T t=1 E Ω[ I[Q (c) n (t) = N Q ] ], n N, c C. Taking the effect of data dropping into consideration, we refer to the average deay as the average time that a piece of data stays in the network before reaching the destination (averaged over the data that are not dropped 3 ). This is because the penaty of data dropping is accounted for separatey in the average drop rate. The foowing emma extends Litte s Law to the case with data dropping. Lemma 1 (Litte s Law with Data Dropping): The average deay of a the commodities and 3 For exampe, suppose 100 packets enter a singe-hop network, among which, 10 packets are dropped and the other 90 packets are successfuy deivered to their destinations. Furthermore, the tota time taken by the 90 packets to reach their destinations is 90. The average deay is given by 90/90 = 1 and the average drop rate is given by 10/100 = 0.1.

9 9 commodity c in the network are given by D D (c) n N n N c C Q(c) n c C (1 d(c) n )λ (c) n n N Q(c) n n N (1 d(c) n )λ (c) n, (2), (3) where the above two inequaities are asymptoticay tight for genera muti-hop networks as d (c) n 0 for a n N and c C. In addition, in singe-hop queueing networks, the inequaities in (2) and (3) are tight for any d (c) n node n is given by and λ (c) n,max = 1, and the average deay of commodity c at D (c) n = Q (c) n (1 d (c) n )λ (c) n. Proof: The proof can be easiy extended from the standard Litte s aw [34] by considering the data that are not dropped 4. We omit the detais due to page imit. Remark 1 (Interpretation of Lemma 1): Lemma 1 estabishes the reationship among the average deay, the average queue ength and the average drop rate in genera networks. Given the average drop rate, the average deay bound for genera muti-hop networks or the average deay for singe-hop networks is proportiona to the average queue ength. Thus, the average queue ength (and the average drop rate if data dropping happens) is commony used in the existing iterature [13] as the deay performance measure. Moreover, under contro poicy Ω, the average throughput of ink L and the average power consumption of node n N are given by [ ] 1 T T = im sup E Ω µ (c) (t), L T + T t=1 c C 1 T P n = im sup E Ω p (c) (t), T + T t=1 {:s()=n} c C n N respectivey, where p (c) (t) denotes the power aocated to commodity c over ink at sot t. 4 Since a the data (incuding the data that are utimatey dropped) contributes to the queue ength process (before being dropped), we have inequaities in (2) and (3).

10 10 Therefore, deay-aware resource contro probems for wireess networks can be divided into the foowing three categories: Category I: Maximize the average weighted sum system throughput (or average arriva rate) subject to average deay constraints, average power constraints and average drop rate constraints. Thus, the deay-aware resource contro probem can be expressed as max w T (4) L s.t. Q (c) n Q (c) n, n N, c C P n P n, n N d (c) n d (c) n, n N, c C, where w is the weight for the -th ink, Q n (c), P n and d (c) n are the average deay constraint, the average power constraint and the average drop rate constraint for commodity c at node n, respectivey. Category II: Minimize the average weighted sum deay subject to average power constraints and average drop rate constraints for given arriva rates at a sources. Thus, the deay-aware resource contro probem can be expressed as min w n (c) Q (c) n (5) n N c C s.t. P n P n, n N d (c) n d (c) n, n N, c C, where w (c) n is the weight for commodity c at node n. Category III: Minimize the average weighed sum power consumption subject to average deay constraints and average drop rate constraints for given arriva rates at a sources. Thus, the deay-aware resource contro probem can be expressed as min n N w n P n (6) s.t. Q (c) n Q (c) n, n N, c C where w n is the weight for node n. d (c) n d (c) n, n N, c C,

11 11 Fig. 3. Bock diagram of upink OFDMA systems. Remark 2 (Unified Optimization Framework): Note that the Lagrangian function of a the above optimization probems can be written in a unified form: L = ξ T + ( ) ν n (c) Q n (c) + η n (c) d n (c) + γ n P n, L n N c C n N where ξ, ν n (c), η n (c) and γ n can be Lagrange Mutipiers associated with the constraints or weights in the objective function. Hence, these probems can be soved by a common optimization framework. D. Upink OFDMA Systems In this part, we iustrate the genera network mode in Section II-A with a simpe exampe of one-hop upink OFDMA systems. This exampe topoogy wi aso be used as iustration to the deay-aware resource contro in ater sections. In the upink OFDMA system exampe iustrated in Fig. 3, we assume the set of N nodes N are mobie stations (MSs) that communicate with one base station (BS). Each MS and the BS is equipped with a singe antenna. Therefore, the set of inks L corresponds to the set of a upink channes from the N MSs to the BS (with L = N). Furthermore, there are N data fows (with C = N), and for notation simpicity, we use {1, 2,, N(= L)} to denote the ink index, the node index as we as the commodity index. We consider communications over a wideband frequency seective fading channe, and the whoe spectrum is divided into N F orthogona fat

12 12 fading frequency bands (subcarriers). Let H,m (t) denote the CSI of the -th upink on the m-th (m {1, 2,, N F }) subcarrier and et H (t) = [H,1 (t), H,2 (t),..., H,NF (t)] denote the aggregate CSI over a subcarriers of the -th ink. The system CSI H(t) = [H 1 (t), H 2 (t),..., H N (t)] is a Markov process satisfying Assumption 1. In addition, we assume {H,m (t)} are i.i.d. w.r.t. {1, 2,, L} and m {1, 2,, N F }. Let Q (t) and s,m (t) {0, 1} denote the queue ength and the subcarrier aocation for the -th ink on the m-th subcarrier at sot t, respectivey. The received signa from the -th user on the m-th subcarrier of the BS at sot t is given by Y,m (t) = s,m (t) (H,m (t)x,m (t) + Z,m (t)), L, m {1, 2,, N F }, where X,m (t) is the transmit symbo and Z,m (t) CN (0, 1) is the channe noise of the -th ink on the m-th subcarrier at sot t. Hence, the data rate of the -th ink on the m-th subcarrier at sot t is given by R,m (t) = s,m (t) og 2 ( 1 + p,m (t) H,m (t) 2), L, m {1, 2,, N F }, where p,m (t) is the transmit power over the -th ink on the m-th subcarrier at sot t. The sum rate of the -th ink at sot t is given by µ (t) = N F m=1 R,m(t). In this upink OFDMA system exampe, the contro poicy for the -th ink is given by Ω = (Ω,p, Ω,s ), where the power aocation poicy Ω,p and the subcarrier aocation poicy 5 Ω,s are defined as foows. Definition 1 (Power Aocation Poicy): The power aocation poicy of the -th ink is a mapping X P from the system state to the power aocation action, which is given by { } Ω,p (χ) = p,m 0 : m {1, 2,, N F } P, L, (7) where p,m is the transmit power on the m-th subcarrier of the -th ink. Definition 2 (Subcarrier Aocation Poicy): The subcarrier aocation poicy of the -th ink is a mapping X S from the system state to the subcarrier aocation action, which is given by Ω,s (χ) = { } s,m {0, 1} : m {1, 2,, N F } S, L, (8) 5 Pease note that when N F = 1, i.e., there is ony one carrier, the subcarrier aocation is reduced to ink seection. Therefore, the subcarrier aocation poicy considered in the foowing probem formuation covers most of the cases in resource aocations for singe-hop wireess networks.

13 13 where s,m = 1 means that the m-th subcarrier is used by the -th ink for data transmission, and s,m = 0 otherwise. III. EQUIVALENT RATE CONSTRAINT APPROACH The first attempt in the iterature to dea with the compicated deay contro probem is to consider an equivaent probem in the PHY ayer domain ony, i.e., converting average deay constraints into average rate constraints using the arge deviation theory [7], [8], [10] [12]. This approach can be traced back to the eary 90 s, when the statistica quaity of service (QoS) requirements have been extensivey studied in the context of effective bandwidth theory [35] [38], which asymptoticay modes the statistica behavior of a source traffic process in the wired networks (e.g., asynchronous transfer mode (ATM) and Internet protoco (IP) networks). For notationa simpicity, we consider a singe queue in the foowing introduction of the known resuts on the arge deviation theory. Let A(t) represent the amount of source data (in number of bits) over the time interva [0, t). Assume that the Gartner-Eis imit of A(t), expressed as Λ B (θ) = im t og E[e θa(t) ] t is defined as exists for a θ 0. Then, the effective bandwidth function of A(t) E B (θ) = Λ B(θ) θ 1 = im t θt og E [ e θa(t)]. (9) Consider a queue with infinite buffer size served by a channe with constant service rate R. By the arge deviation theory [39], it is shown in [35] that the probabiity of the deay D(t) at time t exceeding a deay bound D max satisfies: sup Pr[D(t) D max ] γ(r)e θ(r)dmax, (10) t where γ(r) = Pr [D(t) 0] is the probabiity that the buffer is nonempty and θ(r) = RE 1 B (R) is the QoS exponent (i.e., the soution of E B (θ) = R mutipied by R). Both γ(r) and θ(r) are functions of the constant channe capacity R. Thus, a source, which has a common deay bound D max and can toerate a deay bound vioation probabiity of at most ɛ, can be modeed by the pair {γ(r), θ(r)}, where the constant channe capacity shoud be at east R with R being the soution of γ(r)e θ(r)dmax = ɛ. The intuitive expanation is that the tai probabiity that the deay D(t) exceeds D max is proportiona to the probabiity that the buffer is nonempty and decays exponentiay fast as the threshod D max increases. The QoS exponent [7] θ(r) can be

14 14 interpreted as the indicator of the QoS requirement, i.e., a smaer θ(r) corresponds to a ooser QoS requirement and vice versa. As a resut, the effective bandwidth is defined as the minimum service rate required by a given arriva process for which the QoS exponent requirement is satisfied. Inspired by the effective bandwidth theory, where the constant service rate R is used in the source traffic modeing in wired networks, the authors in [7] use the constant source traffic rate λ to mode a wireess communication channe. They propose the effective capacity, which is the dua of the effective bandwidth. Let S(t) t τ=1 R(τ) represent the amount of service (in number of bits) over the time interva [0, t). Assume that the Gartner-Eis imit of S(t), expressed as Λ C (θ) = im t og E[e θs(t) ] t is defined as exists for a θ 0. Then, the effective capacity function of S(t) E C (θ) = Λ C(θ) θ 1 = im t θt og E [ e θs(t)]. (11) If we further assume the process {R(t)} is uncorreated, then the effective capacity reduces to E C ( θ ) = 1 θ og ( E{e θr(t) } ). (12) Consider a queue of infinite buffer size served by a data source of constant data rate λ (in number of bits). Simiar to the effective bandwidth case, it is shown in [7] that the probabiity of the deay D(t) at time t exceeding a deay bound D max satisfies: sup Pr[D(t) D max ] γ(λ)e θ(λ)dmax, (13) t where γ(λ) = Pr[D(t) 0] is the probabiity that the buffer is nonempty and the QoS exponent is θ(λ) = λe 1 (λ). Both γ(λ) and θ(λ) are functions of the constant source rate λ. Thus, C a source, which has a common deay bound D max and can toerate a deay bound vioation probabiity of at most ɛ, can be modeed by the pair {γ(λ), θ(λ)}, where the constant data rate shoud be at most λ with λ being the soution of γ(λ)e θ(λ)dmax = ɛ. Therefore, as the dua of the effective bandwidth, the effective capacity is defined as the maximum constant arriva rate that a given service process can support in order to guarantee a QoS requirement specified by θ. With the above observation, we can incorporate the QoS requirement into a pure PHY ayer requirement. By interpreting θ as the QoS constraint, the throughput maximization probem subject to the deay QoS constraint (in terms of the exponentia tai probabiity of the queue

15 15 ength distribution) can be directy transformed into an effective capacity maximization probem for a given QoS exponent θ. This approach is widey used when the QoS constraint is specified in terms of the QoS exponent θ. Interested readers can refer to [9] [11] and references therein for more detaied descriptions. A more carefu treatment of the average deay constraint is deveoped in [8], where packet fow mode is considered. The principe idea behind this approach is to estabish the reationship among the average deay requirement D, the average arriva rate λ and the average service rate µ using the queueing theory framework [34]. The foowing procedures are performed to obtain this reationship. 1) Express the average system deay in terms of the average residue service time and the average queueing deay. 2) Express the average residue service time in terms of the moments of the service process. 3) Estabish the reationship among the average deay requirement, the average arriva rate and the average service rate. Exampe 1 (Equivaent Rate Constraint Approach for Upink OFDMA Systems): In the upink OFDMA system exampe, we assume the buffer size for each ink is infinite (as in [8]). Hence, the optimization probem (6) can be simpified as foows [8]: [ L ] N F min E Ω p,m Ω={Ω,s,Ω,p } =1 m=1 s.t. s,m {0, 1}, L, m {1, 2,, N F }, [ NF E Ω m=1 (14) L s,m = 1, L (15) p,m ] P, L (16) ] λ E [Q Ω (t) D, L. (17) Foowing the standard procedures shown above, the average deay constraint (17) can be repaced with an equivaent average rate constraint given by [8, Lemma 1] [ NF ] E Ω s,m og 2 (1 + p,m H,m 2 ) (2D λ + 2) + (2D λ + 2) 2 8D λ N, (18) 4D m=1 where N and λ are the average packet size and the average arriva rate of the -th ink, respectivey. Appying the above resuts, the origina optimization probem (14) can be reformuated =1

16 16 as foows: min Ω={Ω,s,Ω,p } E Ω [ L N F =1 m=1 p,m ] (19) s.t. (15), (16), (18). (20) Optimization probem (19)-(20) is a mixed combinatoria (w.r.t. integer variabes {s,m }) and convex optimization probem (w.r.t. {p,m }). If we reax the integer constraint s,m {0, 1} into rea vaues, i.e., s,m [0, 1], the resutant probem (19) woud be a convex maximization probem. Using standard Lagrange Mutipier techniques, we can derive the optima subcarrier and power aocation as foows: { } 1, if X,m = max j Xj,m > 0 s,m = (21) 0, otherwise ( 1 + ν p,m = s,m 1 ) +, (22) γ H,m 2 where X,m = (1 + ν ) og 2 ( ( H,m 2 ν 1 ) ) + γ H,m 2 ( 1 + ν γ 1 ) +, γ H,m 2 γ is the Lagrange mutipier corresponding to the average power constraint in (16) and ν is the Lagrange mutipier corresponding to the transformed average rate constraint (18) for the -th MS. This approach provides potentiay simpe soutions for singe-hop wireess networks in the sense that the cross-ayer optimization probem is transformed into a purey information theoretica optimization probem. Then, the traditiona PHY ayer optimization approach, such as power aocation and subcarrier aocation, can be readiy appied to sove the transformed probem. The optima contro poicy is a function of the CSI with some weighting shifts by the deay requirements and hence it is simpe to impement in practica communication systems. However, the impicit assumption behind this approach is that the user traffic oading is quite high, or equivaenty, the probabiity that a buffer is empty is quite ow (i.e., the arge deay regime). For genera deay regime, the deay-optima contro poicy shoud be adaptive to both the CSI and the QSI and the performance of the equivaent rate constraint approach is not promising, as we sha iustrate in Section VII. In addition, due to the compex couping among the queues in muti-hop wireess networks, it is difficut to express deay constraints in terms of a the

17 17 contro actions, incuding routing. Therefore, this approach cannot be easiy extended to mutihop wireess networks. IV. STOCHASTIC LYNAPNOV STABILITY DRIFT APPROACH Another important method to dea with deay-aware resource contro in wireess networks is to directy anayze the characteristics of the contro poicies in the stochastic stabiity sense using the Lyapunov drift technique. The Lyapunov drift theory has a ong history in the fied of discrete stochastic processes and Markov chains [40], [41]. The authors of [42] first appied the Lyapunov drift theory to deveop a genera agorithm which stabiizes a muti-hop packet radio network with configurabe ink activation sets. The concepts of maximum weight matching (MWM) and differentia backog scheduing, deveoped in [42], pay important roes in the dynamic contro strategies in queueing networks. The Lyapunov drift theory is then extended to the Lyapunov optimization theory. In this section, we first introduce the preiminaries and the main resuts on the Lynapnov stabiity anaysis; after that, we present two exampes, one for the Lyapunov drift theory and the other for the Lyapunov optimization thoery. A. What is Queue Stabiity? First, we introduce the definition of queue stabiity as foows. Definition 3 (Queue Stabiity): T 1 1) A singe queue is strongy stabe if im sup E [Q(t)] <. T T + t=1 2) A network of queues is strongy stabe if a the individua queues of the network are strongy stabe. A queue is strongy stabe if it has a bounded time average backog. Throughout this paper, we use the term stabiity to refer to strong stabiity. Based on the definition of stabiity, we define the stabiity region as foows. Definition 4 (Stabiity Region): The stabiity region Λ Ω of a poicy Ω is the set of average { } arriva rate vectors λ (c) n n N, c C for which the system is stabe under Ω. The stabiity { } region of the system Λ is the cosure of the set of a average arriva rates λ n (c) n N, c C for which a stabiizing contro poicy exists. Mathematicay, we have Λ = Ω G Λ Ω, (23)

18 18 where G denotes the set of a stabiizing feasibe contro poicies. Definition 5 (Throughput-Optima Poicy): A throughput-optima poicy dominates 6 any other poicy in G, i.e. has a stabiity region that is a superset of the stabiity region of any other poicy in G. Therefore, it shoud have a stabiity region equa to Λ. In other words, throughput-optima poicies ensure that the queueing system is stabe as ong as the average arriva rate vector is within the system stabiity region. Three casses of poicies that are known to be throughput-optima are the Max Weight rue (aso known as M-LWDF/M- LWWF [16] in singe-hop wireess queueing systems), the Exponentia (EXP) rue [14] and the Log rue [15]. The throughput-optima property of Max Weight type agorithms is proved by the theory of Lyapunov drift [16], which is introduced in the next part. The Log rue is aso proved to be throughput-optima by the theorem reated to the Lyapunov drift in [15]. On the other hand, the EXP rue is proved to be throughput-optima by the fuid imit technique aong with a separation of time scaes argument in [14]. B. Main Resuts on Lyapunov Drift In order to show the stabiity property of the queueing systems, we rey on the we-deveoped stabiity theory in Markov Chains using negative Lyapunov drift [20], [21], [43] [45]. We use the quadratic Lyapunov function L(Q) = ( (c)) 2 n N,c C Q n for the system queue state Q through the rest of the paper. Based on the Lyapunov function, we define the (one-sot) Lyapunov drift as the expected change in the Lyapunov function from one sot to the next, which is given by (Q(t)) E [L(Q(t + 1)) L(Q(t)) Q(t)]. (24) Therefore, the theory of Lyaponov stabiity is summarized as foows [13], [46]: Theorem 1 (Lyapunov Drift): If there are positive vaues B, ɛ such that for a time sot t we have: (Q(t)) B ɛ n N,c C then the network is stabe, and the average queue ength satisfies: 1 T im sup E [ Q n (c) (t) ] B T T ɛ. t=1 n N,c C Q (c) n (t), (25) 6 A poicy Ω 1 dominates another poicy Ω 2 if Λ Ω2 Λ Ω1

19 19 Note that if the condition in (25) hods, then the Lyapunov drift (Q(t)) δ ( δ > 0) whenever n N,c C Q(c) n (t) B+δ. Intuitivey, this property ensures network stabiity because ɛ whenever the queue ength vector eaves the bounded region for the sum queue ength, i.e., { Q 0 : } n N,c C Q(c) n (t) B+δ, the negative drift (Q(t)) δ eventuay drives it back ɛ to this region. Next, we iustrate how to use the Lyapunov drift to prove the stabiity of queueing networks and deveop stabiizing contro agorithms. Define the maximum input rate and output rate of node n as µ in max,n = sup t c C {:d()=n} µ(c) (t) and µ out max,n = sup t c C {:s()=n} µ(c) (t), respectivey. They are finite due to the resource aocation constraints. Assume the tota exogenous arriva to node n is bounded by a constant λ max n = sup t c C λ(c) n (t). From the queue dynamics in (1), we have the foowing bound for Lyapunov drift [13]: (Q(t)) B + 2 Q (c) n (t)λ (c) n where 2E n N,c C n N,c C n (t) Q (c) B n N {:s()=n} µ (c) (t) {:d()=n} µ (c) (t) Q(t), (26) ( (µ out max,n) 2 + (λ max n + µ in max,n) 2). (27) The dynamic backpressure agorithm (DBP) is designed to minimize the upper bound of the Lyapunov drift (the R.H.S. of (26)) over a poicies at each time sot. For singe-hop wireess networks, we use the ink index to specify each queue instead of the node index n and the commodity index c for notationa simpicity. From (26), the singe-hop dynamic backpressure agorithm (M-LWDF/M-LWWF) maximizes L Q (t)µ (t) under resource aocation constraints. Based on Theorem 1, it is shown that the DBP agorithm is throughput-optima [25], [26]. After the introduction of dynamic contro agorithm designs in [42], the Lyapunov drift approach is successfuy used to optimize the aocation of computer resources [17], stabiize packet switch systems [18] [21], sateite and wireess systems [22] [24]. For exampe, the concepts of MWM and differentia backog scheduing are first deveoped in [42] based on the Lyapunov drift theory. Using the inear programming argument and the Lyapunov drift theory, it is proved that a MWM agorithm can achieve a throughput of 100% for both uniform and

20 20 nonuniform arrivas in [21]. Based on the anaytica techniques of the Lyapunov drift, the bounds on the average deay and queue size averages as we as variances in input-queued ce-based switches under MWM are derived in [20]. By the Lyapunov drift theory, the Longest Connected Queue (LCQ) agorithm is proved in [22] to stabiize the system under certain conditions and minimize the deay for the specia case of symmetric queues (i.e., queues with equa arriva, service and connectivity statistics). Due to page imitation, we refer the readers to the above references for the detais. C. Main Resuts on Lynapnov Optimization The Lyapunov drift theory is extended to the Lyapunov optimization theory, through which we can stabiize queueing networks whie additionay optimize some performance metrics and satisfy additiona constraints [13]. Let x(t) = ( x 1 (t), x 2 (t),, x K (t) ) represent any associated vector contro process that infuences the dynamics of the vector queue ength process Q(t). Let g : R K R be any scaar vaued concave function. Define x(t ) 1 T T t=1 E[x(t)] and g im sup T 1 T E[g(x(t))]. Suppose the goa is to stabiize the Q(t) process whie maximizing g( ) of the time average of the x(t) process, i.e., maximizing g(x), where x = im sup T x(t ). Let g represent a desired target utiity vaue. The theory of Lyapunov optimization is summarized as foows: Theorem 2 (Lyapunov Optimization): If there are positive constants V, ɛ, B such that for a t and a Q(t), the Lyapunov drift satisfies: then, we have: (Q(t)) V E[g(x(t)) Q(t)] B ɛ im sup T 1 T T t=1 n N,c C im inf T 1 T n N,c C Q (c) n (t) V g, (28) E [ Q (c) n (t) ] B + V (g g ), (29) ɛ T g (x(t)) g B V. (30) t=1 Note that a simiar resut can be shown for minimizing a convex function h : R K defining g( ) = h( ) and reversing inequaities where appropriate. R by

21 21 Theorem 2 is most usefu when the quantity g g can be bounded by a constant. Specificay, if g g G max, then the ower bound on the achieved utiity g can be pushed arbitrariy cose to the target utiity g with a corresponding increase (inear in V ) in the upper bound 1 on im sup T T T t=1 n N,c C E[Q(c) n (t)]. The Lyapunov optimization theorem in Theorem 2 suggests that a good contro strategy is to greediy minimize the foowing drift metric at every time sot i.e., the L.H.S. of (28). (Q(t)) V E [g(x(t)) Q(t)], Next, we introduce the Energy-Efficient Contro Agorithm (EECA) [27], which utiizes the Lyapunov optimization theory to deveop an agorithm that stabiizes the system and consumes an average power that is arbitrariy cose to the minimum power soution. From (26), we have [ ] (Q(t)) + V E P n (t) Q(t) B + 2 Q (c) n (t)λ (c) n n N nc E 2 Q (c) n (t) µ (c) (t) µ (c) (t) V P n (t) Q(t). (31) nc n N {:s()=n} {:d()=n} EECA is designed to minimize the R.H.S. of the inequaity in (31) over a possibe power aocation strategies. For singe-hop wireess networks, we use ink index to denote the node index as we as the commodity index. From (31), we have that the singe-hop EECA maximizes L ( 2Q (t)µ (t) V P (t) ) over a possibe power aocation strategies at each sot t. Based on Theorem 2, it is shown that the EECA is throughput-optima and can achieve [O(1/V ), O(V )] power-deay tradeoff by adjusting the parameter V [13]. The Lyapunov optimization theory aso has appications in the transport ayer fow contro and network fairness optimization when the exogenous arriva rates are outside of the network stabiity region. The Cross Layer Contro (CLC) agorithm is designed to greediy minimize the R.H.S. of (28) to achieve a utiity of exogenous arriva rates, which is arbitrariy cose to optima whie maintaining network stabiity [13], [25], [26]. Remark 3: Note that the average deay bounds deveoped in Theorem 1 and Theorem 2 are tight ony when the traffic oading is sufficienty high, whie the tightness of such deay bounds for moderate and ight traffic oading is not known.

22 22 D. Methodoogy and Exampe To stabiize queueing networks using the Lyapunov drift theorem in Theorem 1 or stabiize queueing networks whie additionay optimizing some performance metrics (e.g., maximizing the average weighted sum system throughput in (4), minimizing the average weighted sum power in (6), etc) using the Lyapunov optimization theorem in Theorem 2, the procedure can be summarized as foows: 1) Choose a Lyapunov function and cacuate the Lyapunov drift (Q(t)) or (Q(t)) V E[g(x(t)) Q(t)], where g(x) is the utiity to be maximized. 2) Based on the system state observations, minimize the upper bound of (Q(t)) or (Q(t)) V E[g(x(t)) Q(t)] over a poices at each time sot. 3) Transform other average performance constraints into queue stabiity probems using the technique of virtua cost queues [13], [26] if needed 7. In the foowing, we iustrate how to appy the Lynapnov drift approach and the Lynapnov optimization approach in resource aocation for upink OFDMA systems, respectivey. Exampe 2 (Lynapnov Drift Approach for Upink OFDMA Systems): In the upink OFDMA system exampe, the dynamic backpressure agorithm under the subcarrier aocation constraints in (15) and the average power constraints in (16), can be obtained by soving the foowing optimization probem max Ω={Ω,s,Ω,p } L Q (t) =1 N F m=1 R,m (t), t (32) s.t. (15), (16) are satisfied. (33) Simiar to Exampe 1, by appying continuous reaxation (i.e., s,m [0, 1]) and standard convex optimization techniques, we can derive the optima subcarrier and power aocation as foows: { } 1, if X,m = max j Xj,m > 0 s,m = (34) 0, otherwise ( Q (t) p,m = s,m 1 ) +, (35) γ H,m 2 7 Pease refer to [13], [26] for the detais of the virtua cost queue technique, which we omit here due to paper imitation. In a the exampes, we use the Lagrangian techniques to dea with these average constraints to faciitate the derivation of the cosed-form soution to obtain certain insights and comparisons of the soutions obtained by different techniques.

23 23 where X,m = Q (t) og 2 ( ( 1 + H,m 2 Q (t) 1 ) ) + γ H,m 2 ( Q (t) γ 1 ) +, γ H,m 2 and γ is the Lagrange mutipier corresponding to the average power constraint in (16) for the -th MS. Exampe 3 (Lynapnov Optimization Approach for Upink OFDMA Systems): In the upink OFDMA system exampe, the average sum power minimization in (6) (with a weights 1 for iustration simpicity) under network stabiity constraint and the subcarrier aocation constraints in (15) based on Theorem 2 is given by ( L max 2Q (t) Ω={Ω,s,Ω,p } =1 N F m=1 R,m (t) V N F m=1 p,m ), t (36) s.t. (15) is satisfied. (37) Simiar to the previous exampe, we can derive the optima subcarrier and power aocation as foows: where X,m = 2Q (t) og 2 ( { } 1, if X,m = max j Xj,m > 0 s,m = 0, otherwise (38) ( 2Q (t) p,m = s,m 1 ) +, V H,m 2 (39) 1 + H,m 2 ( 2Q (t) V 1 ) ) + V H,m 2 ( 2Q (t) Note that the parameter V is used to adjust the average power-deay tradeoff. V 1 ) +. H,m 2 Remark 4: The Lyapunov stabiity drift approach provides a simpe aternative to dea with deay-aware contro probems. The derived cross-ayer contro poicies are adaptive to both the CSI and the QSI. The derived poicies are aso throughput-optima (in stabiity sense). However, as we sha iustrate, throughput optimaity (stabiity) is ony a weak form of deay performance and the derived poicies may not have good deay performance especiay in the sma deay regime. There are many recent studies focusing on deay reduction in the traditiona DBP agorithm in muti-hop networks and we sha further eaborate this in Section VI.

24 24 V. MARKOV DECISION PROCESS AND STOCHASTIC LEARNING APPROACH In wireess networks, the system state can be characterized by the aggregation of the CSI and the QSI. In fact, under Assumptions 1 and 2, the system state dynamics evoves as a controed Markov chain and the deay-optima resource contro can be modeed as an infinite horizon average cost MDP [32]. MDP is a systematic approach for deay-optima contro probems, which in genera coud give optima soutions for any operating regime. However, the main issue associated with the MDP approach is the curse of dimensionaity. For instance, the cardinaity of the system state space is exponentia w.r.t. the number of queues in the wireess network and hence soving the MDP is quite compicated in genera. In addition, the optima contro actions are adaptive to the goba system QSI and CSI but in some cases, these CSI and QSI observations are obtained ocay at each node. Hence, a brute-force centraized soution wi ead to enormous compexity as we as signaing oading to deiver the goba CSI and QSI to the controer. In this section, we briefy summarize the key theories of MDP and stochastic approximation (SA) and iustrate how we coud utiize the techniques of approximate MDP and stochastic earning to overcome the compexity as we as the distributed impementation requirement in deay-aware resource contro. A. Why Deay-Optima Contro is an MDP? In genera, an MDP can be characterized by four eements, namey the state space, the action space, the state transition probabiity and the system cost, which are defined as foows: X = {χ 1, χ 2, }: the finite state space with X states; A = {a 1, a 2, }: the action space; Pr[χ χ, a]: the transition probabiity from state χ to state χ under action a; and g(χ, a): the system cost in state χ under action a. Therefore, an MDP is a 4-tupe ( X, A, Pr[, ], g(, ) ). A stationary and deterministic contro poicy Ω : X A is a mapping from the state space X to the action space A, which determines the specific action taken when the system is in state χ. Given poicy Ω, the corresponding random process of the system state and the per-stage cost ( χ(t), g(t) ) evoves as a Markov chain with the probabiity measure induced by the transition kerne Pr[χ χ, Ω(χ)]. The goa of the infinite horizon average cost probem is to find an optima poicy Ω such that the ong term average

25 25 cost is minimized among a feasibe poicies, i.e., 1 T min im sup E Ω [g (χ(t), Ω (χ(t)))], Ω T T t=1 where E Ω denotes the expectation operator taken w.r.t. the probabiity measure induced by the contro poicy Ω. If the set of feasibe poicies are unichain poicies, then the optimization probem can be written as 1 min im sup Ω T T T t=1 E Ω [g (χ(t), Ω(χ(t)))] = min Ω E π(ω) [g (χ, Ω(χ))], where π(ω) is the unique steady state distribution given poicy Ω. Assume the buffer size is finite and denoted as N Q. The system queue dynamics Q(t) evoves according to (1) with projection onto [0, N Q ] and the arriva, departure and the CSI processes are Markovian under Assumptions 1 and 2. Hence, the system state χ(t) is a finite state controed Markov chain with the foowing correspondence: The system state space in the deay-optima contro probem is defined as the aggregation of the system CSI and the system QSI, thus X = H Q. Q and X are both finite. The action space is the space of a contro actions, incuding the resource aocation actions (e.g., power aocation actions and subcarrier aocation actions) and the routing actions. The transition kerne is given by Pr [ χ(t + 1) χ(t), Ω(χ(t)) ] = Pr [ Q(t + 1) χ(t), Ω(χ(t)) ] Pr [ H(t + 1) H(t) ]. By the standard Lagrangian approach, the optimization probem in (5) can be transformed as foows min Ω L Ω = im sup T where γ n and η (c) n 1 T [ T ( E Ω ν (c) n Q (c) n (t) + η n (c) I [ ] ]) Q n (c) (t) = N Q + γ n P n (t), t=1 n N c C n N are Lagrange mutipiers corresponding to the average power and average drop rate constraints. Therefore, the per-stage system cost function given a system state χ is defined as g ( χ, Ω(χ) ) = ( ν (c) n Q (c) n + η n (c) I[Q (c) n = N Q ] ) + γ n P n, (40) n N c C n N As a resut, there is a one-one correspondence between the deay-optima contro probem and the MDP. The average deay minimization probem under average power constraint in (5) can be modeed as an infinite horizon MDP to minimize the average cost (deay) per stage as foows

26 26 Probem 1 (Deay-Optima MDP Formuation): min L Ω 1 T = min im sup E Ω [g (χ (t), Ω (χ (t)))]. (41) Ω Ω T T t=1 Note that we restrict our poicy space to the unichain poicies where the induced Markov chains under a feasibe unichain poicies are ergodic and share the same state space X. In addition, we assume the induced Markov chains are irreducibe and hence, the chains are ergodic with steady state distribution π(ω). In this case, the imit of infinite horizon average cost under poicy Ω (i.e., L Ω ) exists with probabiity 1 (w.p.1) and is independent of the initia state. B. Optima Soution of the Deay-Optima MDP Under the unichain poicy space assumption, the deay-optima contro poicy of the above MDP is given by the soution of the Beman equation [32]. This is summarized in the foowing Lemma. Lemma 2 (Beman Equation): If a scaar θ and a vector V = [V (χ 1 ), V (χ 2 ), ] satisfy the Beman equation for the deay-optima MDP in Probem 1: θ + V (χ i ) = min Ω g ( χ i, Ω(χ i ) ) + Pr [ χ j χ i, Ω(χ i ) ] V (χ j ), χi X, (42) χ j then θ is the optima average cost per stage θ = min Ω L Ω = L. (43) Furthermore, if Ω attains the minimum in (42) for any χ i X, it is the optima contro poicy. The Beman equation (42) can be soved numericay by Offine Reative Vaue Iteration [32] under certain conditions. Whie the genera soution of the MDP in (41) can be expressed as a Beman equation in (42), this is sti quite far from getting a desired soution. There are two major issues, namey the compexity issue and the signaing overhead issue. Athough the reative vaue iteration approach [32] can give optima soution to the MDP in (41), the soution is usuay too compicated to compute due to the curse of dimensionaity. For exampe, consider a wireess network with N queues; the tota number of the system QSI states is (N Q + 1) N (N Q is the buffer size of each queue), which grows exponentiay with the number of queues. Thus, it is essentiay impossibe to compute the potentia function at every possibe state even for