Capacity Theorems for Relay Channels

Capacity Theorems for Relay Channels Abbas El Gamal Department of Electrical Engineering Stanford University April, 2006 MSRI-06

Relay Channel Discrete-memoryless relay channel [vm 7] Relay Encoder Y n X n W p(y, y x, x ) Encoder X n Y n Decoder Ŵ Goal: Reliably communicate message W [, 2 nr ] from X to Y with the help of relay node (X,Y ) Relay transmission: Classical: x i = f i (y, y 2,...,y (i ) ) Without-delay: x i = f i (y, y 2,...,y i ) MSRI-06 2

Capacity Capacity C is the supremum over achievable rates R An infinite-letter characterization of the capacity is given by C = lim k sup p(x k ),{f i } k i= k I(Xk ;Y k ) Not considered a satisfactory answer computationally intractable? A computable description of capacity is not known in general What we known: Single-letter charcterizations of capacity for some special classes Single-letter upper and lower bounds on C MSRI-06 3

My Encounter with The Relay Channel Visited University of Hawaii in Spring 976 ARPA Aloha packet radio project David Slepian introduced me to the problem Send data directly and via a satellite Worked on it as part of my PhD thesis and with my first student M. Aref Little interest from information theory and communication community for years Renewed interest motivated by wireless networks work with S. Zahedi, M. Mohseni, N. Hassanpour, and J. Mammen MSRI-06 4

Early Work 997 980 [CE 79 ] Cover, El Gamal, Capacity theorems for the relay channel, IT, 979 Cutset upper bound on C Block Markov coding (decode-and-forward) Capacity of degraded relay channels Capacity of relay channel with feedback Side-information coding (compress-and-forward) General lower bound on C combining partial decode-and-forward and compress and forward [EA 82 ] El Gamal, Aref, The Capacity of the semi-determininstic relay channel, IT Trans., 982 (partial decode-and-forward) El Gamal, On information flow in relay networks, IEEE NTC, 980 (cutset bound for relay networks) Aref, Information flow in relay networks, PhD Thesis, Stanford 980 (capacity of degraded relay networks) MSRI-06 5

Recent Work 2002 Present [EZ 05 ] El Gamal, Zahedi, Capacity of a class of of relay channels with orthogonal components, IT Trans., 2005 (partial decode-and-forward) [EMZ 06 ] El Gamal, Mohseni, Zahedi, Bounds on Capacity and Minimum Energy-Per-Bit for AWGN Relay Channels, to appear in IT Trans. Compress-and-forward with time sharing Capacity of FD-AWGN relay channels with linear relaying functions Bounds on minimum energy-per-bit that differ by factor <.7 [EH 05 ] El Gamal, Hassanpour, Relay-without-delay, ISIT, 2005 El Gamal, Hassanpour, Capacity Theorems for the relay-without-delay channel, Allerton, 2005 [EM 06 ] El Gamal, Mammen, Relay networks with delays, UCSD, 2006 MSRI-06 6

Outline Classical relay: Cutset upper bound Partial decode-and-forward Compress-and-forward Upper and lower bounds on capacity of AWGN relay channel Relay-without-delay: New cutset bound Instantaneous relaying Lower bound and capacity results (see poster by N. Hassanpour) MSRI-06 7

Cutset Upper Bound Upper bound on capacity of classical relay [CE 79]: C max p(x,x ) min {I(X, X ;Y ),I(X;Y, Y X )} Max-flow Min-cut interpretation: X Y : X X Y X Y Multiple access bound Broadcast bound Bound is tight for all classes where capacity is known Not known whether it is tight in general no known tighter bound MSRI-06 8

Partial Decode-and-Forward Lower bound [CE 79]: C max min {I(X, X ; Y ), I(X;Y X,U) + I(U;Y X )} p(u,x,x ) Transmission in B blocks. In block b:... W W 2 W 3 W B W B 2 3... B B The relay and sender coherently send information (X ) to Y The sender also superposes a new message W b (thr X) The relay decodes part of W b (represented by U) The receiver decodes X, which helps it decode W b (U then X) Bound is tight for all cases where capacity is known: Degraded: (X (Y, X ) Y ). Set U = X [CE 79] Semideterministic: (Y = g(x, X )). Set U = Y [EA 82] Relay channel with orthogonal components (X = (X D,X R ), p(y x D, x )p(y x R,x )). Set U = X R [EZ 05] MSRI-06 9

Compress-and-Forward The relay can help even without decoding any part of the message Compress-and-forward scheme [CE 79]: Again transmission in B blocks Relay quantizes the received sequence in previous block and sends it to the receiver (a la Wyner-Ziv) Ŷ Y : X C W X Relay and the sender do not cooperate (interfere with each other) max min{i(x;y,ŷ X ), I(X,X ; Y ) I(Y ;Ŷ X,X )} p(x)p(x )p(ŷ y,x ) Except for some asymptotic results, not optimal for any known class Partial decode-and-forward and compress-and-forward can be combined (Theorem 7 of [CE 79]) Y Ŵ MSRI-06 0

Compress-and-Forward with Time-Sharing [EMZ 06] Compress-and-forward achievable rate is not convex Can be improved by time-sharing, which gives the bound C max min{i(x;y, Ŷ X,Q),I(X, X ; Y Q) I(Y ; Ŷ X,X,Q)}, where the maximization is over p(q)p(x q)p(x q)p(ŷ y,x,q) Example: FD-AWGN relay channel [EMZ 06] MSRI-06

Summary We don t know the capacity of the DM relay channel in general We have upper and lower bounds that coincide in some cases: Cutset upper bound. Tight for all cases where we know capacity. Is it tight in general? Partial decode-and-forward lower bound. Tight for all classes where we know capacity Compress-and-forward with time-sharing? Is it optimal for any class? Partial decode-and-forward and compress-and-forward with time-sharing can be combined. Is it optimal for any class? Neeed new coding strategies and tighter upper bound MSRI-06 2

General AWGN Relay Channel Consider the following AWGN relay channel Z a Y : X b Z W X Y Ŵ a, b > 0 are relative channel gains Z N(0, ) is independent of Z N(0, ) Average power constraints: E(X 2 ) P and E(X) 2 P Capacity with average power constraints not known for any 0 < a, b < MSRI-06 3

Cutset and Decode-and-Forward [EMZ 06] Z a Y : X b Z W X Decode-and-forward Y Cutset Upper Bound Ŵ C ( ( b a 2 + ( a 2 b 2 ) 2 P ( ), if a 2 + b 2 ab+ ) +a C 2 b 2 ) 2 P, if a 2 b 2 a 2 (+a 2 ) C ( max{, a 2 }P ), otherwise C ( ( + a 2 )P ), otherwise C(x) = 2 log 2( + x) Partial decode-and-forward reduces to decode-and-forward Bounds never coincide MSRI-06 4

Weak Channel to Relay (a ) Z a Y : X b Z W X Y Ŵ When a, decode-and-forward lower bound reduces to C (P), i.e., direct transmission lower bound Decoding at the relay is not beneficial since everything the relay can decode is already decoded by the receiver But, relay can still help by forwarding some information about it s received sequence We consider two strategies: Compress-and-forward Linear relaying functions MSRI-06 5

Compress-and-Forward [EMZ 06] Achievable rate using compress-and-forward R = max min{i(x,x ;Y ) I(Y ;Ŷ X,X ),I(X;Y, Ŷ X )}, p(x)p(x )p(ŷ y,x ) The optimal choice of p(x)p(x )p(ŷ y, x ) is not known Assume X, X and Ŷ Gaussian: Ŷ Z α Z a Y : X b Z X C C ( P ( + a 2 b 2 P P( + a 2 + b 2 ) + As b, this bound becomes tight (coincides with broadcast bound) )) Y MSRI-06 6

Comparison of Bounds a = and b = 2 2.5 2 Compress-forward Rate (Bits/Trans.).5 Upper Bound Decode-forward 0.5 0 0 2 3 4 5 6 7 8 9 0 SNR MSRI-06 7

Frequency-Division AWGN Relay Channel [EMZ 06] Model motivated by wireless communication: Z Z R a Y : X b Y R X Y S Bounds on capacity of FD-AWGN relay Decode-and-forward Z S Cutset Upper Bound C ( P ( + b 2 + b 2 P )), if a 2 + b 2 + b 2 P C ( P ( + b 2 + b 2 P )), if a 2 b 2 + b 2 P C ( max{, a 2 }P ), otherwise C ( ( + a 2 )P ), otherwise If a 2 + b 2 + b 2 P capacity is C ( P ( + b 2 + b 2 P )) (decode-forward) Model same as without delay MSRI-06 8

FD-AWGN Weak Channel to Relay (a ) [EMZ 06] Again if a, decode-and-forward reduces to direct transmission Z Z R a Y : X b Y R X Y S Can improve rate using compress-and-forward. Using Gaussian signals, we obtain: ( ( a 2 b 2 )) P( + P) C C P + a 2 P + ( + P)( + b 2 P) As b or P, compress-and-forward becomes optimal Z S MSRI-06 9

FD-AWGN Compress-and-Forward With Time-Sharing For small P, compress-and-forward is ineffective due to low SNR of Y Rate Rate with time-sharing 0 P Rate can be improved by time-sharing at the broadcast side (X sends at power P/α for fraction 0 < α and 0, otherwise) to: C max 0<α αc P α + a 2 ( )/( ) + + a2 P + ( + b P+α 2 P) α MSRI-06 20

Comparison of Bounds for FD AWGN Example: a = and b = 2 2.5 2 Upper Bound Rate (Bits/Trans.).5 Compress-forward Decode-forward 0.5 0 0 2 3 4 5 6 7 8 9 0 SNR MSRI-06 2

Linear Relaying Assume relay can only transmit a linear combination of the past received symbols, i.e., x (i+) = i j= d ijy j Using vector notation X = DY, where D = [d ij ] k k is a strictly lower triangular matrix and X = (X,X 2,...,X k ) T and Y = (Y,Y 2,...,Y k ) T Z n W X n a Y n X n D b Z n Y n Ŵ Let C (l) be the capacity with linear relaying subject to the power constraints MSRI-06 22

Capacity with Linear Relaying Capacity with linear relaying can be expressed as where subject to C (l) = sup k C (l) k k C(l) k = lim k k C(l) k = sup I(X k ; Y k ) P X k,d Sender power constraint: k i= E(X2 i ) kp Relay power constraint: k i= E(x2 i) kp Causality constraint: D strictly lower triangular MSRI-06 23

Example [EMZ 04] Y φ Z φ X 2 X X 2 a Y b Z Y Y 2 W X α Let X N(0, 2αP), and X 2 = α X and X 2 = dy where d is chosen to satisfy relay power constraints The achievable rate is R = 2 I(X, X 2 ; Y,Y 2 ) = max 0 α 2 C 2αP + ( ( ) 2 α)/α + abd + b 2 d 2 Y Ŵ 2 C(l) 2 C (l) MSRI-06 24

Comparison of Bounds Example: a = and b = 2 0.5 0.45 0.4 Upper Bound Rate (Bits/Trans.) 0.35 0.3 0.25 0.2 0.5 0. Linear Relaying Decode-forward 0.05 Compress-forward 0 0 0.05 0. 0.5 0.2 0.25 0.3 0.35 0.4 0.45 0.5 SNR MSRI-06 25

Capacity with Linear Relaying for General AWGN Gaussian distribution P X k maximizes C (l) k = sup PX k,d I(X k ; Y k ) Problem reduces to: Maximize lim k 2k log (I + abd)σ x(i + abd) T + (I + b 2 DD T ) (I + b 2 DD T ) Subject to Σ x 0 tr(σ x ) kp tr(a 2 Σ x D T D + D T D) kp d ij = 0 for j i Σ x = E(XX T ) and D are variables of the problem Sequence of non-convex problems (open problem) Single-letter characterization can be found for FD-AWGN relay channel MSRI-06 26

FD-AWGN Relay Channel with Linear Relaying Example: Consider the following amplify-and-forward scheme X Y Z Z R X a Y b Y R Y R X Y S Y S X N(0, P) and X = dy where d is chosen to satisfy the relay power constraint The achievable rate for this scheme is given by I(X;Y S, Y R ) ( ( C (l) a 2 b 2 )) P = C P + + (a 2 + b 2 )P Z S MSRI-06 27

Example (continued) C (l) is convex for small P, concave for large P Rate can be improved with time-sharing on broadcast side ( ) ( ( C FD L (P) max αc θp θp a 2 b 2 )) P + αc + 0<α,θ α α a 2 θp + b 2 P + α Transmission power θp/α θp/α Direct Transmission Amplify-and-forward ᾱn Transmission index n MSRI-06 28

Capacity of FD-AWGN Relay Channel with Linear Relaying Capacity with linear relaying for the FD-AWGN model can be expressed as C (l) = sup k k C(l) k = lim k k C(l) k, where Subject to: C (l) k = sup I(X k ; YS k, YR) k P X k,d Sender power constraint: k i= E(X2 i ) kp Relay power constraint: k i= E(X2 i ) kp Causality constraint: D lower triangular MSRI-06 29

Optimization Problem Gaussian input distribution P X k maximizes C (l) k Finding C (l) k Maximize = sup PX k,d I(X k ;Y k S,Y k R ) reduces to [ ] I + Σ x abσ x D T 2k log abdσ x a 2 b 2 DΣ x D T + (I + b 2 DD T ) 2 [ ] I 0 0 I + b 2 DD T Subject to Σ x 0 tr(σ x ) kp tr(a 2 Σ x D T D + D T D) kp, and d ij = 0 for j > i Σ x = E(XX T ) and lower triangular D are the optimization variables This is a non-convex problem in (Σ x, D) with k 2 + k variables For a fixed D, problem is convex in Σ X (waterfilling optimal); however, finding D for a fixed Σ X is a non-convex problem MSRI-06 30

Key Steps of the Proof Can show that diagonal Σ x and D suffice Objective function reduces to k Maximize 2k log 2 i= ( ( + σ i + a2 b 2 d 2 i + b 2 d 2 i Subject to σ i 0, for i =, 2,...,k, k d 2 i( + a 2 σ i ) kp i= )) k σ i kp, and This still is a non-convex optimization problem time-share between k amplify-forward schemes At the optimum point: If σ i = 0 it is easy to show that d i = 0 If d i = d j = 0 (direct-transmission) then σ i = σ j (concavity of log function) MSRI-06 3 i=

Proof (continued) Problem reduces to: Maximize 2k log 2 ( + kθ 0 P/k 0 ) k 0 k i=k 0 + Subject to d i > 0, for i = k 0 +,...,k, k i=k 0 + d 2 i( + a 2 σ i ) kp ( + σi ( + a 2 b 2 d 2 i/( + b 2 d 2 i) )) k i=k 0 σ i k( θ 0 )P, and By KKT conditions, at the optimum, there are 4 non-zero (σ j,d j ) 4 Maximize 2k log 2 ( + kθ 0 P/k 0 ) k ( ( 0 + σj + a 2 b 2 d 2 j/( + b 2 d 2 j) )) k j Subject to d j > 0, for j =,...,4, 4 k j d 2 j( + a 2 σ j ) kp j= j= 4 k j σ j k( θ 0 )P, and MSRI-06 32 j=

Capacity of FD-AWGN Relay with Linear Relaying [EMZ 06] Taking the limit as k, we obtain ( ) C (l) θ0 P 4 ( θj P = maxα 0 C + α j C α 0 j= α j ( + a2 b 2 )) η j, + b 2 η j subject to α j,θ j 0, η j > 0, 4 j=0 α j = 4 j=0 θ j =, and 4 j= η ( ) j a 2 θ j P + α j = P Time-share between direct transmission and 4 amplify-forward regimes A non-convex optimization problem, but with only 4 variables and 3 constraints We haven t been able to find any example where optimal requires > 2 amplify-forward regimes (in addition to direct transmission) MSRI-06 33

Comparison with Other Schemes Example: a = and b = 5 0.8 0.7 Compress-forward Rate (Bits/Trans.) 0.6 0.5 0.4 0.3 0.2 Upper Bound Linear Relaying Decode-forward 0. 0 0 0. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 SNR As b, C (l) approaches the cutset upper bound MSRI-06 34

Summary For general AWGN relay channel: Bounds are never tight for any 0 < a,b < Compress-and-forward becomes optimal as b Linear relaying can beat compress-and-forward we don t have a computable expression for capacity with linear relaying For FD-AWGN relay channel: Decode-and-forward is optimal (bound coincides with cutset) when a 2 + b 2 + b 2 P Compress-and-forward becomes optimal as b or P. Time-sharing improves rate for small P We have a computable form of capacity with linear relaying time-sharing between direct transmission and at most 4 amplify-and-forward regimes MSRI-06 35

Relay-Without-Delay (RWD) Suppose the delay from the sender X to the receiver Y is longer than to the relay (X,Y ), so that x i can depend on its current, in addition to past received symbols, i.e., Relay-without-delay: x i = f i (y, y 2,...,y (i ), y i ) Relay Encoder Y n X n W X n p(y x)p(y x, x, y ) Y n Ŵ Again, wish to reliably communicate W [, 2 nr ] from X to Y MSRI-06 36

Generalized Cutset Bound Upper bound on capacity of RWD channel: C max min{i(v, X;Y ), I(X;Y, Y V )}, p(v,x), f(v,y ) where x = f(v, y ), and V min{ Y, X X } + Proof follows similar lines to cutset bound: Define V i = Y i, so X i = f i (V i, Y i ) Key observation: (W, Y i ) (X i,v i ) (Y i,y i ) This bound can be strictly larger than the cutset bound Bound applies to classical case, but x is a function only of v and the bound reduces to the classical cutset bound MSRI-06 37

Proof Use standard Fano s inequality argument. nr I(W; Y n ) + nǫ n, where ǫ n 0 Consider I(W; Y n ) = = n I(W; Y i Y i ) i= n (H(Y i ) H(Y i W, Y i )) i= n i= n i= (H(Y i ) H(Y i W, Y i,y i, X i )) (H(Y i ) H(Y i Y i,x i )) n I(X i,v i ; Y i ) = ni(x Q,V Q ;Y Q Q) ni(x,v ; Y ) i= MSRI-06 38

Next consider I(W; Y n ) I(W; Y n, Y n ) n = I(W; Y i, Y i Y i,y i ) = = i= n i= n i= H(Y i,y i Y i,y i ) H(Y i, Y i Y i,y i, W) H(Y i,y i V i ) H(Y i, Y i Y i, Y i, W, X i ) n H(Y i,y i V i ) H(Y i, Y i V i, X i ) i= n I(X i ; Y i, Y i V i ) = ni(x;y, Y V ) i= MSRI-06 39

Instantaneous Relaying Any lower bound on the capacity of the classical relay channel, e.g., decode-and-forward, is a lower bound on the RWD channel Instantaneous relaying: Ignore the past and set X = f(y ); f(y ) Y X W X p(y x)p(y x, x, y ) Y Ŵ This scheme can achieve R = max I(X;Y ) p(x), f(y ) This strategy can be optimal and can achieve higher rate than classical cutset bound! MSRI-06 40

Sato Example Consider the following DW relay channel [Sato 76]. Y = X and X 0 2 0.5 Y 0 X 0 0.5 0.5 0.5 2 2 X = 0 X = The capacity of the classical case is.6878 bits/transmission [CE 79] Coincides with cutset bound The upper bound on the capacity without delay gives: C 2(log 3 ) =.69925 bits/transmission Instantaneous relaying with input distribution ( 3 9, 2 9, 4 9) and the mapping from X to X of 0 0,, 2 achieves this bound!! Capacity of this RWD channel > classical, which is = cutset bound Y 0 2 MSRI-06 4

AWGN RWD Channel Same model as before (but without relay coding delay) Z a Y : X b Z W X Y Ŵ a, b > 0 are relative channel gains Z N(0, ) is independent of Z N(0, ) x i = f i (y,y 2,...,y i ) Average power constraints: E(X 2 ) P and E(X 2 ) P Recall that capacity of the classical case is not known for any 0 < a,b < MSRI-06 42

Amplify-and-Forward Consider the following special case of instantaneous relaying: Z W X a Y X h b Z Y Ŵ If a b, the upper bound gives C C ( ( + a 2 )P ) Consider the case where a 2 b 2 min{,p/(a 2 P + )} Amplify-and-forward also gives C max p(x), E(X 2 ) P I(X;Y ) = C ( ( + a 2 )P ), Capacity in this case coincides with classical cutset bound Can capacity of AWGN RWD channel be > cutset bound? MSRI-06 43

New Lower Bound on Capacity of RWD Channel Idea: Use a superposition of partial decode-and-forward and instantaneous relaying Achieves the lower bound: C max min{i(v, X;Y ), I(U;Y V ) + I(X;Y V, U)} p(u,v,x), f(v,y ) U represents the information decoded by the relay in partial decode-and-forward and V (replacing X ) represents the information sent coherently by the sender and the relay to help the receiver decode the previous U Each x n codeword in partial decode-and-forward is replaced by a v n codeword, and at time i the relay sends x i = f(v i,y i ) This bound coincides with the generalized cutset bound for degraded and semi-deterministic RWD channels Same superposition idea can be used to obain new compress-and-forward lower bound MSRI-06 44

Achievable Rate for AWGN RWD Channel Restrict previous scheme to superposition of decode-and-forward and amplify-and-forward: Let U = X = V + X, where V N(0, αp) and X N(0, αp) are independent, for 0 α and α = α Let X be a normalized convex combination of Y and V X = h(βy + βv ), 0 β Using the power constraint on the relay sender, we obtain h 2 = P (β + aβ) 2 αp + β 2 (a 2 αp + N) Substituting the above choice of (U,V, X,X ), we obtain C RWD AWGN max α, β min { C ( a 2 αp ), C ( αp(bh(aβ + β) + ) 2 + αp(bhaβ + ) 2 )} ( + (βbh) 2 ) MSRI-06 45

Achievable Rate vs. Cutset vs. Upper Bound Assume a = 2 and b = 3 2.9 Rate (bits/transmission) 2.8 2.7 2.6 2.5 2.4 2.3 2.2 2. Achievable rate for AWGN RWD channel Cutset bound Upper Bound on capacity of AWGN RWD channel 2 5 6 7 8 9 0 2 3 4 5 MSRI-06 46 P

Classical vs. RWD Relay Channels Result Classical Relay RWD Capacity Not known in general Not known in general Can be > classical Upper bound max p(x,x ) min{i(x, X ; Y ), I(X; Y, Y X )} max p(v,x), f(v,y ) min{i(v, X; Y ), I(X; Y, Y V )} Cutset bound Can be > cutset bound Degraded capacity max p(x,x ) min{i(x, X ; Y ), I(X; Y X )} max p(v,x), f(v,y ) min{i(v, X; Y ), I(X; Y V )} X (X, Y ) Y Decode-forward Decode-forward + instantaneous coding Semi-det capacity max p(x,x ) min{i(x, X ; Y ), I(X; Y, Y X )} max p(v,x), f(v,y ) min{i(v, X; Y ), I(X; Y, Y V )} Y = g(x) Partial decode-forward + instantaneous coding AWGN Capacity not known for any a, b 0 Capacity known for a 2 b 2 min{, Amplify-forward Can be in general > classical P a 2 P+N } MSRI-06 47

Conclusion Overview of known upper and lower bounds on relay and RWD channels Bounds are tight only is special cases We know as much about the RWD as classical relay, main difference is instantaneous relaying, but it is not sufficient Generalized cutset bound needed for RWD channel Generalization to relay networks with delays (see poster with J. Mammen) MSRI-06 48