Optimal Cross Layer Scheduling of Transmissions over a Fading Multiaccess Channel

Transcription

1 Optimal Cross Layer Sceduling of Transmissions over a Fading Multiaccess Cannel Munis Goyal, Anurag Kumar, Vinod Sarma Department of Electrical Communication Engg Indian Institute of Science, Bangalore, India goel.munis@gmail.com,anurag, vinod@ece.iisc.ernet.in Abstract We consider te problem of several users transmitting packets to a base station, and study an optimal sceduling formulation involving tree communication layers, namely, te medium access control, link and pysical layers. We assume Markov models for te packet arrival processes and te cannel gain processes. Perfect cannel state information is assumed to be available at te transmitter and te receiver. Te transmissions are subject to a long-run average transmitter power constraint. Te control problem is to assign power and rate dynamically as a function of te fading and te queue lengts so as to imize a weigted sum of long run average packet transmission delays. First, we study te problem for a single user system and obtain structural properties of te optimal policy. We obtain numerical results for te delay-power tradeoff. Ten, we consider te multiuser system and obtain a value iteration algoritm for computing te optimal policy. We identify tat te problem is computationally intractable and consider an approximation for te cost to go function. Te approximating function provides a tigt upped bound for te optimal cost function and tus a onestep value iteration could result in a close to optimal policy. A one-step value iteration is ten carried out to improve upon te policy. We obtain structural properties of te one-step iterated policy and sow tat te resulting policy can be obtained via solving a fixed point iteration on a family of suitably modified single user optimal control policies. Keywords: power and rate control in wireless networks, constrained Markov decision processes I. INTRODUCTION Te dream of anyone, anywere, anytime communication can only be realized by te widespread deployment of ig quality wireless access networks. Te traditional approac to network arcitecture is based on a stack of protocol layers wit well defined layer functionality and interlayer interfaces. Tis flexible and transparent approac is mainly responsible for te success of today s wired networks. In te context of wireless networks, owever, it as been observed tat truly efficient use of te wireless communication resources spectrum and power) requires adaptability to canging cannel and network caracteristics in all layers. Tis leads to a concept called cross-layer design, te idea of joint optimization at two or more of te layers of communication. Suc an approac can elp address te unique callenges of te wireless environment suc as te time-varying or fading nature of wireless cannels, and te limited battery energy available in wireless andsets. One of te main ideas of cross layer design is to permit te excange of information across layers, someting tat would be considered a layer violation in traditional design. Tis additional information is used by te layers to better adapt to varying transmission conditions. In tis work, we will concentrate on cross layer design problems involving tree of te wireless communication system layers: te data link layer, te medium access control MAC) layer and te pysical layer. Te approac combines fundamental communication limits of pysical layer capacity captured via te information teoretic cannel capacity), wit a iger layer quality of service measure, namely, long run average packet delay. We consider random packet arrivals and queueing at te transmitters and incorporate te effect of temporal variations in te cannel. Te pysical layer constraint is te average transmission energy available wic relates to battery life). Te control variables are te amount of data released from te link layer and te transmission power used at te pysical layer. An information teory based analysis elps in obtaining te limits of wat could possibly be acieved using an efficient cannel coding-decoding sceme, and also provides insigt into good rate and power control policies. Background Literature: In [8] Tse and Hanly considered a resource allocation problem for a multiaccess fading cannel wit te objective of maximizing te trougput capacity, and in te sequel [0], tey went on to discuss te capacity region wen users need delay guarantees. Tey considered a framework in wic delay guarantees are acieved if eac user transmits at a fixed guaranteed rate. Tis is a restrictive assumption and can lead to te wastage of resources. Collins and Cruz [7] considered a rate and power sceduling problem for a point-to-point wireless system wit te objective of imizing average transmitter power subject to an average transmission delay constraint. Tey assumed tat te received power is always constant. Our researc as been in te spirit of Berry and Gallager [3] wo considered a problem similar to te one considered in [7] but witout te constant received power assumption. Berry and Gallager [3] obtain structural results exibiting a tradeoff between te optimal transmitter power and te mean queueing delay. Tey sow tat te optimal power versus te optimal delay curve is convex, and as te average power available for transmission increases, te acievable mean delay decreases. Tey also provide some structural results for te optimal policy tat acieves any point on te power-delay curve. In [], Berry furter considered te wireless multiaccess fading system wit te objective of imizing a weigted sum of average packet transmission delays and transmission power. Te autor obtained a class of

2 simple rate and power sceduling policies tat were sown to be nearly optimal wen te average delay constraint is large. A recent survey paper [4] discusses fundamental limits of crosslayer design algoritms for multiaccess wireless networks. Te survey paper contains a good list of references on cross-layer design problems in wireless networks. Te work presented in tis paper is an extension of our earlier work reported in [] on power and delay optimal transmission policies for a wireless link. In tis paper we extend te work in [] to te Markov arrival and fading settings and also to te multiuser case. Contributions: In tis paper, we provide several new results on te single user problem introduced in [3] and ten we develop te multiuser problem. We cast te single user problem as a constrained Markov decision process and draw upon results from average cost Markov decision teory to establis detailed structural results for te optimal policy in te single user case. Our approac permits us to go beyond te results in [3] in te following ways. ) We obtain a complete structural caracterization of te policy in te i.i.d. arrival and fading case. Furter new structural results are obtained in te Markov arrivals and fading case in Teorem 3.. ) In Section III-E, we utilise a relative value iteration algoritm to obtain numerical results for te average cost problem. 3) We ten develop te constrained Markov decision problem approac for te multiuser case. In Section IV, we approximate te cost to go function by replacing it wit an additive separable function tat tigtly upper bounds te original cost to go function. A one-step value iteration is ten carried out to obtain an improved policy wic could be close to optimal since te cost to go function tigtly upper bounds te original cost to go function. 4) We obtain structural properties of te one-step iterated policy and sow tat te policy is obtained essentially via a fixed point iteration on a family of single user control policies. 5) In a special case of on-off control, te control policy of a tagged user, given te control decisions of all te oter users, is sown to possess a simple tresold form. Paper Outline: Tis paper is organized as follows. In Section II, we discuss te model of te system under consideration and formulate te controller objectives as a constrained optimization problem. In Section III, we analyse te single user system wit a mean delay objective. We use a result from [5] to convert te single user problem into a family of unconstrained optimization problems. Tis unconstrained problem is a Markov decision problem MDP) wit te average cost criterion. We sow te existence of stationary average cost optimal policies and teir structural properties in Section III-B. A corresponding discounted cost problem is studied in Appendix C. In Section III-C, we obtain conditions for te existence of a Lagrange multiplier suc tat te optimal policy corresponding to tat value for te multiplier is also optimal for te original constrained MDP. We provide numerical results for te optimal policy and obtain te powerdelay tradeoff curve in Section III-E. We analyse te M user mean delay imization problem in Section IV. We observe tat te general problem is too complicated to work wit. We approximate te cost to go function wit an additive separable function and carry out a one-step value iteration to derive te control policy. Te proofs of teorems are given in Appendix D. II. THE SYSTEM MODEL We consider a discrete time model of a multiaccess fading cannel wit M users communicating to a receiver as sown in Figure. Time is divided into slots of lengt τ units eac. Let N be te number of cannel uses per slot. Packets generated at te iger layer arrive at te link layer at te end of every slot as sown in te top part of Figure te process A[n]). Te link layer is modeled as a queue of infinite capacity were te arriving packets are kept before forwarding to te medium access control MAC) layer. A controller co-located at te receiver decides about te number of packets to be forwarded to te MAC for transmission in any given slot. Te decision is based on te number of packets buffered at eac user s queue and te cannel gain attenuation) for eac transmit-receive pair. Te queue lengt information is communicated troug te eader of te last packet transmitted in a slot wereas te cannel gain for eac transmit-receive pair is measured at te receiver. Te control decisions are communicated to te transmitters during a control period at te beginning of eac slot see Figure ). In practice, WIMAX based on an ortogonal frequency division multiple access OFDMA) sceme provides suc a mecanism for excange of information between a centralized controller and spatially distributed transmitters. Te pysical layer as te responsibility of transmitting bits on te multipat fading wireless cannel. Te transmission power is constrained by battery life. We use bold symbols to represent vectors of lengt M te number of users); i.e. x represents x, x,, x M. Eac source generates fixed size packets eac of lengt b bits) according to a finite state ergodic Markov cain A[n]; let P a be te transition probability matrix. At time nτ, let Q[n] be te queue lengt and R[n] be te number of packets to be released in te current slot as per te control decision. Te evolution equation of te buffer lengt process is given by Q[n + ] = Q[n] R[n]) + + A[n], ) were x) + is a notation for maxx, 0. We assume a long run average transmitter power constraint of P at eac transmitter. Let te transmitted signal from te i t transmitter be X i [n]. Let H i [n] be te fading process seen by te i t user s transmission and Z[n] be an additive wite Gaussian noise receiver noise) process wit zero mean and variance σ. Te signal received at te receiver is ten Y [n] = Hi [n]x i [n] + Z[n].

3 3 Data from iger layer A[n ] HIGHER LAYER USER M Excange of Contol information Data transmission Data transmission nτ n+)τ n+)τ Time TRANSMIT BUFFER Q [k] USER Q [k] M R [k] R [k] M A[n] FADING CHANNEL H [K] H [K] M Σ AWGN RECEIVER & CONTROLLER Fig.. System model for an M user multiaccess fading cannel. Te controller, based on te cannel gain vector [k] and te queue lengt vector q[k], decides upon r i [k], te number of packets to be transmitted during slot k and p i [k], te transmitter power during slot k from te i t user. We assume tat te cannel state information is available perfectly at bot te transmitter as well as te receiver end. We model te fading process as block-fading were te cannel gain fade) stays constant over te duration of a slot, i.e, for τ time units. Te cannel gain process H i [n], embedded at te slot boundaries, is assumed to be a finite state ergodic Markov cain; let P be te transition probability matrix. We assume, for eac n, A [n], H [n], A [n], H [n],..., A M [n], H M [n]) are mutually independent random variables. We will make an additional assumption tat te fading is bounded away from zero and takes values in a finite subset of [ 0, ] were 0 > 0. We address te optimal power and rate allocation problem for te M user multiaccess system wit te objective of imizing mean packet transmission delay subject to an average power constraint. Te receiver acts as a central controller, wic, depending upon te transmitter buffer lengts and te cannel gains of eac user, allocates te packet transmission rates R[n] and powers P[n] to individual users. Based on te control decisions, te controller informs te pysical layer as to wat transmission rate and power to use. Te pysical layer ten encodes te data cog from te MAC layer at tat rate and transmits te encoded data over te cannel at te sceduled power level. Te random processes R[k], Q[k], P[k] correspond to packet transmission rate vector, queue lengt vector and power allocation vector respectively. We ave te following sceduling constraints. First, tere is te natural constraint tat R[k] Q[k], were te representation R[k] Q[k] means R i [k] 0,,, Q i [k] for i,,, M. Furter, we will ave te multiple users capacity constraint in eac slot, i.e., te allowed values of R[k] given a power allocation vector P[k]. Te capacity of a multiaccess system depends upon te decoding sceme employed at te receiver. Given a power vector p and a cannel gain vector, we assume tat te maximum number of packets r tat can be transmitted reliably for a system tat employs successive decoding sould satisfy te capacity constraint r C g, p) were C g, p) is te set of rate vectors satisfying, r j θ ln j S + ) jp j σ j S for every S,,, M and θ := ln)b N. Remark.: Note tat, by our model, a codeword can at te most stretc to a slot to ensure decoding at te end of every slot. Tis assumption of finite lengt code would result in a strictly positive decoding error probability [8]. Since te capacity function is averaged only over Gaussian noise, a relatively sort code block lengts are required to approximate te asymptotic capacity results. Under an assumption tat te coding/sceduling frame time is sorter tan te cannel coerence time and te number of cannel uses in tis time are large enoug for reliable communication, te capacity formula could be a reasonable approximation for te purpose of exploring tis problem [3]. Furter, We need to pay a marginal rate or a power penalty to acieve a target decoding error probability wile using codewords of lengt N cannel uses per slot). Te penalty factors could also be incorporated easily by using te error exponent bounds on te probability of error [8]. Te Multiuser Problem: At time nτ, te state of te system is represented by X[n] := Q[n], H[n], A[n]). Recall tat te process Q[n] is te queue lengt process at time instant nτ. At te n t decision instant time instant nτ, n 0), te controller decides upon te number of packets R[n] to be transmitted in te current slot and P[n], te transmitter power required for reliable transmission depending on te entire istory of state evolution, i.e., X[k] for k = 0,,,, n tat imizes a weigted sum of a long run average delay subject to te average power constraint P. Now since delay is related to te amount of data in te buffer by Little s formula [0], te control objective is equivalent to imizing a weigted sum of te average queue lengts. Te controller s objective is to obtain an optimal sequence of pairs R[n], P[n]) tat solves te following optimization problem. subject to lim sup n n n E k=0 ω i Q i [k] R[k] Q[k] and R[k] C g H[k], P[k]); for k 0 [ n ] lim sup n n E P[k] P, 3) k=0 were w i are nonnegative weigts associated wit user i and define te relative importance of user i over oter users. Te Single User Problem: We first analyse te single user system wit one transmitter and one receiver controller) and ten specialize te results obtained to te multiuser scenario employing successive decoding at te receiver. Te single user model is sown in Figure. In te single user setting, te power, p, required for reliable transmission of r packets in a slot gets fixed as p = σ eθr ). Tus te control objective is to obtain te sequence R[n] as a function of X[0], X[],, X[n], tat solves te following )

4 4 Higer Layer Fig.. Data Arrives Excange of Contol from iger layer information a[n ] a[n] Excange of Contol information Data transmission Data transmission nτ n+) τ n+) τ Time TRANSMITTER QUEUE q[n] r[n] Cannel [n] AWGN Model of a single user point to point wireless link. optimization problem. and lim sup n lim sup n n E n n E k=0 Q[k] subject to R[k] Q[k] for k 0 [ n k=0 + σ H[k] eθr[k] ) III. THE SINGLE USER SYSTEM ] Controller as Receiver) P. 4) Te single user control problem, as stated in Equation 4, is a constrained dynamic optimization problem. We first convert it into a family of unconstrained optimization problems and analyse tem. Te unconstrained optimization problems belong to a category of average cost Markov decision problems MDP). We caracterize te optimal policies for tese MDPs. We ten sow ow tese policies result in a solution to te original constrained problem. A. Formulation as a Markov Decision Process MDP) Let X[n], n 0,,, denote a controlled Markov cain, wit state space X = Z + 0, ] Z +, and action space Z +, were Z + denotes te set of nonnegative integers. Te set of feasible actions r in state x = q,, a) is te set of all integers belonging to Rx) = 0,,, q. Let K be te set of all feasible state-action pairs. Te transition kernel on X given an element x, r) K is denoted by Γ. Define te mapping p : K R + by px, r) = σ eθr ), te power required to transmit r in a slot wit θ = ln)b N. Define a policy π = π 0, π, π, ) tat at time instant n generates an action r[n] depending upon te entire istory of te process, i.e., at decision instant n 0,,,, π n is a mapping from K n X to RX[n]). Let Π be te space of all suc policies. A stationary policy is of te form π = f, f, f, ) were f is a measurable mapping from X to RX[n]). For a policy π Π, and initial state x X, we define two cost functions Bx π, te buffer cost, and Kx π, te power cost by, B π x = lim sup n K π x = lim sup n n n Eπ x k=0 n n Eπ x k=0 Q[k]; px[k], R[k]). Given te power constraint P > 0, denote by Π P te set of all admissible control policies π Π wic satisfy te long run transmitter power constraint K π x P. Ten te controller objective can be restated as a constrained optimization problem CP) defined as, CP ) : Minimize B π x subject to π Π P 5) Te problem CP) can be converted into a family of unconstrained optimization problems troug a Lagrangian relaxation [5]. For every β > 0, define a mapping c β : K R + as c β x, r) = q + βpx, r). Define a corresponding functional for any policy π Π by, J π β x) = lim sup n n n Eπ x k=0 c β X[k], R[k]). Given β > 0, define te unconstrained problem UP β ) UP β ) : Minimize J π β x) subject to π Π 6) Te following teorem gives sufficient conditions under wic an optimal policy for an unconstrained problem is also optimal for te original constrained control problem CP). Teorem 3.: [5] Let, for some β > 0, π Π be te policy tat solves te unconstrained problem UP β suc tat π yields te expressions B π and K π as limits for all x X, and in addition, for all x, K π = P. Ten te policy π is optimal for te constrained problem CP). Proof: See [5] We analyse te problem UP β ) in Section III-B. We verify tat te conditions stated in te ypotesis of te Teorem 3. are valid and tus obtain te constrained solution in Section III-C. B. Structure of te Optimal Policy for UP β Te problem UP β ) is a standard Markov decision problem wit an average cost criterion. We now study te unconstrained problem and obtain te structural properties of te optimal policy. As β > 0 is fixed for te analysis in tis section, for notational simplicity we suppress te subscript β. Define a discounted cost MDP wit discount factor α 0, ) corresponding to te problem U P ), for eac initial state x = q,, a), wit value function, V α q,, a) = π Π Eπ x [ k=0 ] α k Q[k] + βpx[k], R[k])). 7) We call te optimal solution for te discounted problem a discount optimal policy. Te following lemma states tat te average cost problem can be studied as a limit of discounted cost problems as te discount factor α increases to one and also proves its existence. Lemma 3.: Tere exists a stationary deteristic policy rq,, a) tat solves te unconstrained problem UP for eac β > 0. Te stationary optimal policy tat solves te unconstrained problem U P is limit discount optimal in te sense tat te policy can be obtained as a limit of discount optimal policies as te discount factor increases to one. Proof: See Appendix B)

5 5 First, we study te discounted cost problem and obtain structural properties of te discount optimal policy see Appendix C). In ligt of te result of Lemma 3., using te structural properties of te discount optimal policies, we obtain structural properties of rq,, a) for eac q,, a). For a state-action pair x = q,, a), r), define u := q r, u 0,,, q, as te number of packets not served wen te system is in state x. Tus uq,, a) = q rq,, a) also defines te optimal stationary policy for te single user problem. We note tat te value function V α q,, a) is convex in q refer Lemma C- in Appendix C). Define a differential of te value function as G α q,, a) = V α q,, a) V α q,, a) and Zq,, a) = e θq lim α αe,a [G α q + A, H, A)], were E,a [ ] denotes expectation wit respect to te transition probability of H and A wit initial state and a respectively. Note tat Zq,, a) is monotone increasing in q as te value function V α ) is convex in q. Define a function u q,, a) as te value of u tat solves te following inequalities, for given q,, a), c) Te optimal solution is see Figure 4): e 0 if θq < Z0) βσ e θ ) uq, ) = e q if θq > Zq) βσ e θ ) u q, ) o.w. Proof: Let x = q,, a). Denote by u α x), te discount optimal policy. Teorem A- states tat te average cost optimal policy ux) is te limit of discount optimal policies wic migt be optimal for some close neigbourood of x rater tan x itself. Lemma D- provides a stronger result tat ux) is limit of discount optimal policies u α x) as discount α increases to one. Te results of te teorem now follow from te analysis of te discounted cost problem see Appendix C). i) Follows from Teorem C- and Teorem C-. ii) Follows from Teorem C-3 iii) Follows from Lemma C- iv) Follows from Teorem C-5 v) Follows from Teorem C-6 Figure 3 depicts te structure of te optimal policy depicted in terms of te unsent data uq,, a). Zu,, a) βσ eθq e θ ) Zu +,, a). 8) Te optimal policy uq,, a) equals u q,, a), q. In section III-D, we state an algoritm to compute u q,, a) and ence te optimal policy using te above relation. Te following teorem gives te structural properties of te optimal policy. Teorem 3.: Structural properties of te stationary optimal policy for UP): i) Te optimal policy uq,, a) is monotonic nondecreasing in q and rq,, a) = q uq,, a) is monotonic nondecreasing in q as well. ii) Te function u q,, a) is bounded below by, θ ln + 4 β σ 4 η, a)eθq e θ ) βσ η, a) uq,,a) s 0 s s s s 3 4 s 5 s s 6 7 s s 8 9 s 0 q Fig. 3. Te structure of te optimal amount of buffer not served uq,, a) versus q, for a fixed and a. Te dark curve plots a typical policy. Define s i, a) := maxq : u q,, a) q i. For q s 0, no packets are transmitted. Te number of packets transmitted for q s i +, s i+ ) is i +. Tus, for q s +, s ), it is optimal to serve two packets. and bounded above by, ) βσ θ ln eθq e θ ), were η, a) = E,a [ e θa H ] and θ = ln)b N. iii) Te optimal number of packets transmitted rq,, a) increases to infinity as q increases to infinity. iv) Te optimal solution uq,, a) is monotone nondecreasing wit β te power price). v) If te fading and te arrival processes are i.i.d: a) Te function Zq,, a) is only a function of q and W y) defined as te value of u tat solves Zu) e θy Zu + ) is a monotone nondecreasing in y. b) Given any q, ), te optimal policy uq, ), is uq, ) = q, W q θ ln /βσ e θ ) )). SERVE NOTHING 0 SERVE EVERYTHING q Fig. 4. Depiction of te optimal policy for te scenario wen te cannel gain process and te arrival processes are i.i.d. Discussion: Te structural properties of te average optimal policy stated in Teorem 3. goes beyond te results stated in [] and [3].

6 6 i) Given, a), it follows from Teorem 3.i) tat for any q and q satisfying q < q, we ave q uq ) q uq ), i.e., uq ) uq ) q q and uq ) uq ). Tis implies tat te number of packets transmitted rq,, a) grows at a rate slower tan q. Tese caracteristics of te optimal policy are sown in Figure 3. ii) Te lower bound for te optimal policy is especially useful in te sense tat it provides information about te rate of growt of transmission rate wit te queue lengt. iii) Te transmission rate does not saturate to a level as te queue lengt increases to infinity. Te larger te queue te larger te number of packets transmitted. iv) Given q,, a), te optimal number of packets transmitted rq,, a) is nonincreasing in β. Tis is natural to expect since te larger te power price β, te iger is te transmission cost. v) Te following observations can be made for te i.i.d. case, a) it is optimal not to serve anyting wen te cannel is bad i.e., is large); for eac q tere is a small enoug suc tat for cannel pairs worse tan tis, it is optimal not to serve. b) wen te cannel is good, it is optimal to serve everyting until a value of te buffer size q tat increases wit increasing ; c) even under poor cannel conditions, as q increases it becomes optimal to serve data as te delay becomes costlier tan power. Note tat Figure 3 would represent te policy for a small value of, wit noting being served until q s 0. C. Te Power Constrained Delay Optimal Policy We ave given structural results for te optimal policy for te unconstrained problem UP β ). Now invoking Teorem 3.) we sow tat tere exists a β > 0 for wic te optimal policy obtained above is also optimal for te constrained problem CP ). We reintroduce te dependence on te multiplier β. Recollect tat te solution to te problem UP β ) is r β q,, a) = q u β q,, a). We sow tat te conditions under wic Teorem 3. olds are satisfied. First, we need to sow tat for eac β > 0, te lim sup and lim inf are equal. Tis is true if te controlled cain is ergodic as it would imply u β q,, a) yields te expressions B u β and K u β as limits. Teorem 3.ii) states tat te optimal number of packets transmitted increases to infinity as q tends to infinity. By a standard drift argument, it is easy to sow [] tat te system is ergodic for all finite arrival rates. Tis is natural to expect as te relaxed problem is an unconstrained system and one can carry any arrival rate by spending more and more power. Next, we need to sow te existence of a β suc tat te average power cost is equal to te power constraint, i.e., K u β = P. We know tat te policy r st ) = θ ln )) + λσ is stabilising [9], were λ solves for te power constraint P. Define P st R) = E σ H R st P ) = max E[r st H)] : E e θrst H) ) ) : E[r st H)] R, σ H e θrst H) ) ) P, If te mean arrival rate E[A] < R st P ), te mean queue lengt is finite under te stabilizing policy. Te mean delay increases as β increases since te power gets costlier wereas te power cost decreases to P st E[A]) as β increases to infinity. Note tat P st E[A]) is te imum power required to keep te queue stable for an arrival rate E[A] and we need P st E[A]) < P. It as been sown [3] tat te average power is monotone nonincreasing convex function of mean delay. Furter, it is easy to see tat te average transmitter power required is monotone nonincreasing in β and converges to P st E[A]) as β. If tis power cost function is continuous in β, tere always exists a β > 0 suc tat te average power cost for te optimal policy corresponding to tat β equals P. But a monotone function may ave jump discontinuities and tus tere may not be a value of β for wic te average power constraint is satisfied wit equality. Tis is a very standard situation tat even arises in knapsack packing problems. In case tere is no β for wic te average power constraint is satisfied wit equality, we ave two possible solutions. Te first one is to cange te power constraint itself by coosing one of te nearest usually a lower one) number for wic tere is a β satisfying te average power cost wit equality and say tat tere is no advantage in aving a constraint value larger tan tat number. Tus te ypotesis of Teorem 3. would be satisfied and we ave an optimal solution for te constrained problem. Te second approac is to define a randomized policy. Since tere always exist a β for wic te power cost K u β < P because oterwise it contradicts te existence of stabilizing policy. Tus define β 0 as te smallest value of β for wic K u β P. If te equality olds ten we are done. But due to possibility of a discontinuity at β 0, K u β + 0 < P and K u β 0 > P. Define a new randomized policy tat randomizes between u β + and u 0 β and te probabilities 0 are cosen so tat te power constraint is met wit equality. Tis randomized policy defines a constrained solution. Remark 3.: Te monotonic nature of optimal delay and optimal power usage wit respect to beta yields a simple iterative algoritm to compute an appropriate coice for beta tat satisfies te average power constraint or delay constraint). Start wit an arbitrary coice of β suc tat β > 0 and compute te optimal policy and te long run average power required. If te average power required is moreless) tan te constraint, decreaseincrease) te value of beta and recompute. Repeat till we converge to a value of β were monotonicity property guarantees te convergence of tis iteration. If tere is a discontinuity, a randomized policy needs to be considered as discussed and explained above. D. An Algoritm for Computing u ) In order to compute u q,, a), as per te definition in Equation 9 we need to compute Zq,, a). To compute

7 7 Zq,, a) we need an algoritm to compute V α q,, a) as defined by Equation 7 for eac α 0, ). Consider te following iterative algoritm to compute V α q,, a). We suppress te subscript α. For n 0, V n q,, a) = u 0,,,q q + βσ ) e θq u) +αe,a [V n u + A, H, A)], 9) wit V 0 q,, a) = 0. Let G n q,, a) = V n q,, a) V n q,, a) and Z n q,, a) = e θq E,a [G n q + A, H, A)]. Define u nq,, a) be te value of u tat solves te following inequalities, αz n u,, a) βσ eθq e θ ) αz n u +,, a). Note tat Zq,, a) = lim α,n Z n q,, a). Define s i,n), a) = maxq : u nq,, a) q i. Tus based on te constrained solution as defined earlier, given a value of, a), we ave, For q s 0,n) no packet transmission r ) = 0)), G n+ q,, a) = + αe,a [G n q + A, H, A)] For q = s i,n) + and i 0,,, implying tat te number of packets transmitted at q is one larger tan tose transmitted for a queue lengt of q See Figure 3), G n+ q,, a) = + βσ e θi+) e θi)) For q s i,n) +,, s i+,n) and i 0,,, implying tat given i, te number of packets transmitted for tis range of queue lengts is te same, G n+ q,, a) = + αe,a [G n u nq,, a) + A, H, A)] Furter by definition, Z n+ q,, a) = e θq E,a [G n+ q + A, H, A)] Te sequence u nq,, a) converges to te optimal solution u q,, a) in te limit as n tends to followed by te limit as α increases to. E. Numerical Evaluation We numerically evaluate te optimal policy and te powerdelay trade-off as te multiplier β is varied. We will use an average cost value iteration algoritm for numerical computation. Te value iteration algoritm is similar to te iteration in Equation 9 wit α set to. One expects tat if V n x) V n x) converges and is independent of x ten te limiting policy is te average cost optimal and te limiting difference would be te average cost. Te convergence of suc an algoritm is known to be considerably difficult to analyse. Cen and Meyn [6] ave given sufficient condition for te convergence of te value iteration algoritm for problems arising in queueing networks. Te essential idea is to initialize te algoritm wit a value function corresponding to a stable policy, i.e., V 0 x) needs to be appropriately cosen. Tey give counterexamples to sow tat te value iteration if initialized wit V 0 x) = 0 may never converge. Teorem 3.3: Suppose, a) assumes only finitely many values and tere exists a pair 0, a 0 ) among possible pairs of, a) for wic te transition probability matrix for and a as a positive entry at te diagonal. Let tere be a positive probability of arrivals a being equal to 0. Furter, suppose tere exists a t suc tat for any given η > 0 and starting te dynamic system in state q 0,, η and in any, a), we can reac z = 0, 0, a 0 ) at time t wit a positive probability. Ten te value iteration algoritm if initialized wit V 0 x) te cost corresponding to a policy rq,, a) = q) results in convergence and te limiting policy is optimal. Furter, V n z) V n z) converges to te optimal average cost. Proof: See Appendix D We consider te following numerical example. Te ambient noise power σ =. Te number of cannel uses per slot is N = 0. Te cannel gain process is assumed to be i.i.d. and take values in te set.4,.7, wit probabilities.3,.4,.3 respectively. Te packet arrival process is also assumed to be i.i.d. and takes values 0, 00 wit probabilities.5,.5 respectively. In order to verfiy te convergence of te numerical algoritm, we note tat te ypotesis of Teorem 3.3 is satisfied. Since te arrival process is i.i.d., te optimal control related functions are independent of variable a. Te value function V 0 q, ) is q + βσ eθq ) + EV 0 [A, H] wit G 0 q, ) = V 0 q, ) V 0 q, ) = + βσ eθq e θ ). Set state z = 0, ) and V n+ z) is computed from G n q, ) as follows. A V n+ 0, ) = E[V n A, H)] = E[ G n k, H) + V n 0, H)]. k= Te optimal average cost is tus te limit of V n+ z) V n z) as n tends to infinity. Te plot for optimal number of packets transmitted versus te queue lengt for various values of β and cannel gain equal to are sown in Figure 5. Te optimal policy rq, ) for oter values of is r q θ ln ) + ). Te power and delay versus β is sown in Figure 6 and te power-delay tradeoff curve is sown in Figure 7. Recall tat te mean queue lengt is proportional to te mean transmission delay. IV. THE MULTIACCESS SYSTEM We now consider te M user delay imization problem wit an average power constraint Equation 3). Based on te structural properties for te single user system, we wis to obtain similar structural results for te multiaccess system. We convert te problem into a family of unconstrained problems see [5]) by associating multipliers λ i, i,,, M wit te average power constraints. Te controller objective is to allocate optimal rate vector R[n] and a power vector P[n] given te state X[n] = Q[n], H[n], A[n]) wile imizing a weigted sum of te average delay and te average power.

8 β =.00 Packets Transmitted rq) β =0 β POWER Degree Line) Queue lengt q) Queue Lengt Fig. 5. Te optimal number of packets transmitted rq) versus q for various values of te Lagrange multiplier β power price) for te numerical example in Section III-E. Note tat rq) = q for lower values of q and tis 45 degree line is sown. Te values of β cosen are n mod 9) + ) 0 3+ n 9 for n = 0,,,, 8. As β increases, te number of packets transmitted decreases Average Queue Lengt Average Power Fig. 7. Power-Delay trade-off curve for te numerical example in Section III- E R i [k] 0,,, Q i [k] for i,,, M, were subscript x denotes te initial state of te system. Tis is a standard average cost constrained Markov decision problem. As before we consider te corresponding discounted cost problem. R[k],P[k] subject to E x k=0 α k ω i Q i [k] + λ i P i [k]) R[k] C g [k], P[k]); R i [k] 0,,, Q i [k] for i,,, M., ) Te corresponding discounted cost optimality equation is, Fig. 6. Te average transmitter power required and te mean queue lengt for various coices of β for te numerical example in Section III-E. Given a power constraint, one can find a smallest value of β for wic te average power constraint is satisfied and te policy corresponding to tat β would yields te imum mean queue lengt as sown in te plot against tat coice of β. Similarly, if te mean queue lengt constraint is given, one can find a largest value of β for wic te queue lengt constraint is met and te policy corresponding to tat β would yield a imum average power required as per te curve sown in te plot against tat coice of β. We ave te following formulation. n lim sup R[k],P[k] n n E x ω i Q i [k] + λ i P i [k]), 0) subject to β k=0 R[k] C g H[k], P[k]); V α q,, a) = r,p):r q;r C g,p) M ω i q i + λ i p i ) +αe,a [V α q r + A, H, A))], ) were E,a [fa, H)] denote te expectation of f, ) conditioned upon, a), and V α x) is te discounted cost value function wit discount factor α 0, ) wen starting in state x. A. Analysis of te Multiuser Problem Observe tat if we fix r, te objective function in Equation is imized by tat coice of p wic solves M λ i p i ; subject to p : r C g, p). p Tis is true since te cost to go only depends upon te coice of r. But we know see [8]) tat, given,r, and λ, and reindexing te users so tat λ λ λ M M,

9 9 te optimal value for te above problem is, σ λ i e θ i k= rk) e θ i k= r k). 3) i Recall tat te decoding is sequential were a user wit λ te lowest value of is decoded first and te decoded signal is ten subtracted from te received signal see [8]). Wile decoding a signal, te interference only comes from λ transmissions of users aving iger value of tan te user wose signal is being decoded. For eac ordering of λi i, we can similarly obtain te optimal cost. Tere are at most M! distinct orderings possible. We enumerate te possible orderings and define νk, i), te index of te user wit te i t order in te k t ordering. As an example, for M =, tere are two possible orderings, ) and, ) indexed by k = and k = respectively. Ten ν, ) =, ν, ) =, ν, ) = and ν, ) =. Given r, and satisfying order k, te optimal value of λ p for te k t ordering is M σ λ νk,i) νk,i) e θ i j= rνk,j)) e θ i j= r νk,j)). We can rewrite te optimal value for te k t ordering in te following convenient form, σ λνk,i) λ ) νk,i+) e θ i j= rνk,j)) σ λ νk,), νk,i) νk,i+) νk,) 4) were, by convention, λ νk,m+) νk,m+) is zero. Te set of all possible cannel gain vectors can be partitioned into M! subsets suc tat eac subset corresponds to one of te ordering, i.e., k t ordering corresponds to k t subset say H k wit M! k= H k = H, te wole set. Te cannel gain transition probability matrix needs to be redefined. Let P,k H) define te probability tat te next state of te cannel gain vector is H H k given tat te current cannel gain vector is. Let E,k [fh)] defined te conditional expectation of f ) wit respect to tis transition probability matrix. We define M! value functions indexed by k say V k q,, a) were k signifies te fact tat te cannel gain vector H k. We now ave a family of M! coupled discounted cost optimality equations corresponding to te optimality equation of Equation. We drop te subscript α for convenience. Given q, a and H k for k,,, M!, we ave V k q,, a) = r q -e θ ) i j= r νk,j) l= were we note tat te rate-power constraint as been eliated. Te corresponding value iteration algoritm, for k,,, M!, is given by, V k,n q,, a) = r q -e θ ) i j= r νk,j) M ω i q i + σ λ νk,i) e θ i j= r νk,i) νk,i) M! + α E,l,a [V l,n q r + A, H, A)], l= were V l,0 x) = 0 for all x. Recall from MDP teory tat te first expression witin te parenteses on te rigt and side of te above value iteration is called te single stage cost wile te second expression is called te cost to go. Te convergence of te algoritm can be easily sown as in single user case. Te above problem appears intractable for structural analysis mainly due to a ig degree of nonlinearity and coupling of te single stage cost function resulting in a igly complex cost to go expression M! l= E,l,a[V l q r + A, H, A)]. Furter, te uge state space associated wit te above said control problem renders it impractical and computationally inefficient. Hence, tere is a need for some near-optimal approximating sceme to it tat addresses tese analytical and computational difficulties. We proceed as follows: i) Obtain an additive separable approximation of te cost to go expression also known as te value function and study its structural properties. By an additive separable function we mean tat it can be divided into additive terms wit eac being a function of only one user s variables. Suc approximations exist in te literature []. ii) Carry out one-step value iteration wit te approximated cost to go function and obtain te structural properties of te resulting policy. Tis idea of one-step of value iteration was introduced by Krisnan and Ott [3], and as been used in many papers since ten. Te remarkable fact about applying one-step value iteration is tat te resulting policy could be very close to optimal if te cost to go function is cosen appropriately. We assume tat available rate set is bounded above. Tis is not an unrealistic assumption owing to te fact tat only finite rate codewords are practical. Witout loss of generality we assume tat te bound is te same for all te users. B. Cost to Go Approximation Let ˆr be te upper bound on te available rates set. Using te result obtained in Equation 3, te discounted cost problem Equation ) can be restated as, R[n] E x n=0 α n ω i Q i [n] + σ λ νk[n],i) H νk[n],i) e θ i j= Rνk[n],j)[n] e θ ) i j= R νk[n],j)[n] 6) M ω i q i + σ λ νk,i) e θ i j= r νk,j) subject to R i [n] ˆr, 0,,, Q i [n] for i νk,i),,, M, were k[n] is te ordering during n t slot. M! In order to obtain te additive separable cost to go function, we first replace, for eac i,,, M, te + α E,l,a [V l q r + A, H, A)], 5) rates R νk[n],j) [n] wit ˆr for j i, and obtain a reasonable upper bound for te above objective function. Tis replacement is equivalent to an assumption tat every transmission sees maximum possible interference. For example, a user wit te igest value of λ observes no interference wereas a λ user wit smallest value of assumes tat all te oter users transmit at te igest possible rate ˆr. Tis will be te scenario wen users are unaware of te queue lengt of oter users. Next, we optimize te resulting function and obtain a tigt uniform bound for te cost to go function. Define

10 0 µk, i) = j : νk, j) = i, te order of user i in te k t ordering. We obtain te following optimization problem and refer to it as a cost bounding problem. R[n] E x n=0 α n ω i Q i [n] + σ λ i H i [n] eθµk[n],i) )ˆr ) e Ri[n] ). Note tat e θµk[n],i) )ˆr is te total power received from all te users to be decoded after decoding user i wit te assumption tat te users transmit at te rate ˆr, te maximum allowed transmission rate. Te term σ λ i eθµk[n],i) )ˆr H i[n] upper bounds te oter user interference to user i since µk[n], i) is te number of users decoded after decoding te signal of user i. Observe tat te objective function of te cost bounding problem can be separated user wise and yields a cost tat upper bounds te imal cost acievable by te original discounted cost problem Equation 6). We now study te cost bounding problem and obtain a value function wic will serve as te approximate cost to go function. Since te objective function is separable, we tag user and analyse te corresponding discounted cost optimality equation. Te state vector includes te tagged user s queue lengt q, te arrival state a and te cannel gain vector since te ordering k is detered by te complete cannel gain vector. For notational simplicity, we represent µk, ) by µk) and set te weigt of user, ω =. Define β), for H k representing k t ordering, as, β) = σ λ e θµk) )ˆr. We drop te subscript for notational simplicity. Te discounted cost optimality equation is given by, V q,, a) = q + β)e θr ) r q,ˆr +αe,a [V q r) + + A, H, A)],, 7) Note tat te Equation 7 as a close resemblance to te single user problem Equation B- in te Appendix) were te only difference is tat te rate is now constrained to ˆr and β) as replaced β. Along te lines of results for te single user problem, it is easy to prove te following result. Teorem 4.: Te value function V q,, a) is convex and monotone nondecreasing in q. We now carry out one-step value iteration wit V ) serving as te approximating cost to go function. C. One-Step Value Iteration Recall te actual discounted cost problem Equation 6) restated below for convenience. Given x = q,, a), V x) = E x α n ω i Q i [n] + σ λ νk[n],i) R[n] H νk[n],i) n=0 e θ i j= R νk[n],j)[n] e θ i j= R νk[n],j)[n] ). We write te above objective function as a sum of two components, namely, n = 0 representing te first stage cost and n > 0 representing te aggregate expected cost to go. By definition of te initial state of te system x = q,, a), we ave Q i [0] = q i, H i [0] = i and A i [0] = a i for i =,,, M. Let te ordering k[0], corresponding to, be k. Tus, given a rate vector r q, ˆr, te first stage cost equals ω i q i + σ λ νk,i) e θ i j= r νk,j) e θ i j= νk,j) ) r. νk,i) Now tat te vector r is te transmission rate vector for te first stage, te queue lengt vector at te end of te first stage equals q r + a. Te cannel gain vector and te arrival rate vector for te second stage will be random variables wit transition density conditional on and a respectively. Te aggregate expected cost to go tus equals αe,a V q r + a, H, A). We substitute te above cost to go function wit te separable approximation carried out in Section IV-B. Te approximated cost to go function is, αe,ai V i q i r i + A i, H, A i ), were V i ) is as obtained in Equation 7 and satisfies te properties stated in Teorem 4.. Te approximation as stated earlier provides a tigt upper bound for te actual cost to go function and a one step optimization would furter result in a close to optimal solution. Given any q, a and H k, te one-step value iteration is to obtain te rate vector r q, ˆr tat imizes, ω i q i + σ λ νk,i) e θ i j= r νk,j) e θ i j= r νk,j) νk,i) +αe,ai V i q i r i + A i, H, A i )). 8) Tis is an approximate multiuser problem. Te solution tat we obtain by solving tis problem will provide a close upper bound to te optimal system performance. We refer to te solution to tis problem as an one-step iterated policy. Teorem 4. implies tat te objective function Equation 8) is strictly convex in r wic, along wit te fact te decision space in compact, implies te existence of a unique imizer r. We now derive structural properties of te onestep iterated policy. Since is given and fixed, witout loss of generality we assume tat H k and te ordering k is suc tat νk, i) = i. Define r i := r j, j i. Given r i, we study te structural properties of r i and sow tat te optimal one-step iterated policy is te solution to a fixed point equation. We factor out terms involving r i te decision variable) in te objective function. r i q i,ˆr [ e θri e θ i l= r l e θ j l=i+ rl e θ j l=i+ r l σ λ i + M σ λ j i j=i+ j ))] +αe,ai V iq i r i+a i,h,a i),

11 were V i is te solution of Equation 7. If we denote te expression witin te square brackets by gr i ), te optimization problem becomes, e θr i gr i ) + αe,ai V i q i r i + A i, H, A i ). r i q i,ˆr Te analysis of te problem is similar to te analysis of te single user problem. As in Section III-B, we define for i =,,, M a differential of te value function as G i q i,, a i ) = V i q i,, a i ) V i q i,, a i ) and Z i q i,, a i ) = e θqi αe,ai [Gq i + A i, H, A i )], were E,ai [ ] denotes expectation wit respect to te transition probability of H and A i wit initial state and a i respectively. Note tat Zq i,, a i ) is monotone increasing in q i as te value function V q i,, a i ) is convex in q i. Define a function u i q i, i, a i ) as te value of u tat solves te following inequalities, for given q i,, a i ), Z i u,, a i ) e θqi e θ ) Z i u +,, a i ). 9) Te imizing policy r i q i,, a i ) equals ˆr, max q i u i q i + θ ) loggr i)),, a i, 0. We state te following results. Teorem 4.: Given te transmission rates of all te oter users r i and te cannel gain vector, te structural properties of te optimal one-step iterated policy for user i is as follows. i) Te optimal policy r i q i,, a i ) is monotonic nondecreasing in q i and u i q i,, a i ) = q i r i q i,, a i ) is monotonic nondecreasing in q i as well. ii) Te optimal number of packets transmitted r i q i,, a i ) increases to ˆr as q i increases to infinity. iii) Te optimal solution r i q i,, a i ) is monotone nonincreasing in gr i ). Proof: Te proofs of i) and ii) are along te lines of single user discounted cost analysis Refer Appendix C). Result iii) follows from te fact tat as gr i ) increases, te first stage cost increases wile te cost to go remains te same. Remark 4.: According to result iii), if any of te r j, j i increases, te function gr i ) increases and ence r i decreases. We obtain similar results by tagging oter users one-byone. Given te system state q,, a), te unique imizer r = r, r,, r M is obtained by solving te following family of nonlinear equations. For eac i =,,, M, we solve, r i = ˆr, max q i u i q i + ) θ loggr i)),, 0. We now turn our attention to a special case of on-off control. D. On-Off Control In practice, one cannot cange te coding rate every slot and in most systems, only one code book is available at eac of te transmitters implying only one transmission rate. In tis section, we assume tat only one transmission rate is possible. Te transmitter is allowed to decide weter to transmit at tat rate, or not to transmit at all in a given slot. We call a user as ON over a slot if it transmits at te allowed rate oterwise te user is OFF, i.e., it does not transmit anyting in tat slot. At te start of eac slot, depending upon te state of te system, te controller make te on-off decisions. ) Analysis of Cost Bounding Problem: We solve te cost bounding problem to get an approximate cost to go function. Tag user. Based on te system state q,, a), te tagged user may decide to be ON and transmit at a fixed rate r = r or it may just decide to be OFF meaning it does not transmit r = 0. Given te system state, let te ordering be k and define µk) := i : νk, i) =, te index of te tagged user under k t ordering. Recall earlier definition of β). If ON: Te queue lengt at te end of current slot would be q r) + + A were A is a random variable wit probability distribution corresponding to te a t row of te arrival state transition probability matrix. Te costs incurred are te olding cost q and te power price β)e θr ). If OFF: Te queue lengt at te end of current slot would be q + A were A is a random variable wit probability distribution corresponding to te a t row of te arrival state transition probability matrix. Te costs incurred would be te olding cost q. Te above problem can now be formulated as te following Markov decision problem. Let V q,, a) be te discounted cost value function. Te discounted cost optimality equation is, V q,, a) = q + β)e θr ) + αe,a [V q r) + +A, H, A)], αe,a [V q + A, H, A)], 0) were by definition te optimal action is ON if β)e θr ) + αe,a [V q r) + + A, H, A)] is less tan αe,a [V q + A, H, A)], and is OFF oterwise. Note tat β) > 0. Consider te corresponding discounted cost value iteration algoritm. Let V 0 q,, a) = 0. Ten for n, V n+ q,, a) = q + β)e θr ) + αe,a [ V n q r) + + A, H, A)], αe,a [V n q + A, H, A)]. ) Define x =, a) and W n q, x) = E,a [V n q + A, H, A)]. Let W q, x) be te limiting function. Tus V n+ q, x) = q + β)e θr ) + αw n q r) +, x), αw n q, x). ) We now state structural results. Te proofs are given in Appendix D. Teorem 4.3: Given x, te function W q, x) is monotone increasing in q. Teorem 4.4: Given x, te difference W q, x) W q r) +, x) is nondecreasing in q.