ELEC546 Review of Information Theory

ELEC546 Review of Information Theory Vincent Lau 1/1/004 1

Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random variable with alphabet X and probability mass function p(x), If X is a continuous random variable with p.d.f. f(x), hx ( ) = f( x)log f( xdx ) Joint Entropy ( ) = ( )log ( ) = [ log ( )] H X p x p x E p x x Χ H( X, X ) = p x, x log p x, x ( ) ( ) 1 1 1 x, x 1 Conditional Entropy H( X X1) = p( x1, x) log p( x x1 ) = EX [ ] 1, X log p( X X1) 1/1/004 x1, x

Entropy Properties of Entropy: H( X) 0 Lower Bound: Upper Bound: Discrete X: H( X) log X equality iff p( X) = 1/ X Continuous X: H( X) (1/ ) log πσ e X equality holds iff X ~ Ν µ, σ H( X, X ) = H( X ) + H( X X ) ( ) ( X ) Chain Rule: 1 1 1 Conditioning reduces Entropy: H( X Y) H( X) equality holds iff X &Y are independent Concavity of entropy: H ( X) is a concave function of p( x) If X & Y are independent, then H( X + Y) H( X) Fano s Inequality: Given two random variables X & Y, let Xˆ = gy ( ) be an estimate of X given Y. Define the probability of error P ˆ e = Pr X X, we have: { } ( ) + log ( 1 ) ( ) H P P X H X Y X Y Xˆ e e 1/1/004 3

Mutual Information Consider two random variables X & Y with joint pdf p(x,y). The mutual information between X & Y is given by: p( x, y) p( X, Y) I( X; Y) = p( x, y) log = EXY, log xy, p( x) p( y) p( X) p( Y) I( X; Y) = H( X) H( X Y) Hence, mutual information is the reduction in uncertainty of X due to knowledge of Y mutual information represents the amount of information communicated to the receiver. X Y 1/1/004 4

Mutual Information Conditional Mutual Information: I( XY ; Z) = HX ( Z) HX ( YZ) Properties of Mutual Information Symmetry: I X Y = Self Information: I( X; X) = H( X) Lower Bound: ( ; ) I( Y; X) I( X; Y) = H( X) + H( Y) H( X; Y) I( X; Y) 0 equality holds iff X & Y are independent 1/1/004 5

Mutual Information Properties of Mutual Information Chain Rule of Information: I( X, X,..., X ; Y) = I( X ; Y X,..., X ) 1 N n n 1 1 n= 1 Let ( XY, )~ pxy (, ) = pxpy ( ) ( x). N The mutual information I(X;Y) is a concave function of p(x) for a given p(y x). The mutual information I(X;Y) is a convex function of p(y x) for a given p(x). 1/1/004 6

Mutual Information Properties of Mutual Information Data Processing Inequality: X, Y, Z are said to form a Markov chain { X Y Z} if p(x,y,z)=p(x)p(y x)p(z y). If { X Y Z}, then I( X; Y) I( X; Z). Equality holds iff X Z Y Corollary: Function of the data Y could not increase the information about X. If X Y Z then I( X; Y Z) I( X; Y) The dependency of X & Y is decreased by the observation of a downstream random variable Z. If Z = gy ( ), then X Y gy ( ) and I( XY ; ) I( X; gy ( )) 1/1/004 7

Asymptotic Equipartition Law of Large Number: If X,..., X ~ iid... random variables, then AEP Theorem: If X1 XN p x p X1 XN H X. Typical Set 1 n= 1 N N 1 lim Xn = E[ X] in probability N N 1,..., are i.i.d. ~ ( ), then log (,..., ) ( ) in probability N { N N( H ) ( ) ( ) ( X ) + ε N H ( X ) ε } 1,.., N : ( 1,.., N) A = x x X p x x ε 1/1/004 8

Asymptotic Equipartition Properties of AEP: 1 If ( x1,..., x N) Aε, then H( X) ε log p( x1,..., x N) H( X) + ε N {( 1 N ) ε } Pr x,..., x A > 1 ε for sufficiently large N A ( N) N( H( X) + ε ) ε for sufficiently large N A ( N) ε ( ε ) ( ( ) ε ) N H X 1 for sufficiently large N 1/1/004 9

Jointly Typical Sequences Definition: A ( N ) ε N N N N 1 N ( x, y ) X Y : log p( x ) H( X) < ε, N = 1 N 1 N N log p( y ) H( Y) < ε, log p( x, y ) H( X, Y) < ε N N Theorem (Joint AEP): Let (X N,Y N ) be sequences of length N drawn N N N i.i.d. according to p ( x, y ) = p( xn, yn) ( ) {( x ) } N y N A N ε Pr, 1 N N ( N) N( I( X; Y) + 3ε ) 1/1/004 and Pr X, Y Aε 1 ε for sufficiently large N 10 A n= 1 ( N ) ε ( (, ) + ε ) N H X Y ( ) ( ) ( ) ( ) ( ) { } ( ) ( ) N( I( X; Y) 3ε ) { ε } N N N N N N N If X, Y ~ p x p y, then Pr X, Y A

Properties: Jointly Typical Sequence ( ) There are about NH X typical X sequences ( ) There are about NH Y typical Y sequences However, since there are (, ) only NH X Y jointly typical sequences, not all pairs of typical X and typical Y are also jointly typical. The probability that any randomly chosen pair is jointly typical is about ( ; ) NI X Y 1/1/004 11

Channel Coding Theorem Channel Encoder A mapping from message set to transmitted sequence. Set of codewords X is called a codebook. Code rate ( ) R= log M / N Generic Channel { () 1,..., X ( M) } Probability mapping from X N N to Y p( y x ) Channel Decoder A deterministic decoding function from Y N message index. Error Probability N { g( Y ) m m } Pr is transmitted 1/1/004 1

Channel Coding Theorem Discrete Memoryless Channel: Definition of Channel Capacity: N N ( ) p Y X = p( Yn Xn) p( X) N n= 1 C = max I( X; Y) 1/1/004 13

Examples of Channel Capacity Binary Symmetric Channel X ~ Binary Input {0,1}; Y ~ Binary Output {0,1} 1 p p py ( x) = p 1 p Error probability is independent of the transmit input bit. Example Channel: Channel Capacity: ( ) C = max I( X; Y) = 1+ plog p+ (1 p)log 1 p 1/1/004 14 p( x) p(0) = p(1) = 0.5

Examples of Channel Capacity Discrete Input Continuous Output Channels Input X ~ discrete alphabets, Channel output Y ~ continuous (unquantized). N N Channel is characterized by the transition probability f( y x ) Example case (binary input, continuous output, AWGN) y = x + z z ~ zero mean white Gaussian noise σ n n n n z N N f ( y x ) f( y x ) = n n f( yn 0)~ N( A, σ z ) f( yn 1)~ N( A, σ z ) n Mutual information is maximized when p(0)=p(1)=0.5. Channel Capacity is given by: 1/1/004 py ( A) py ( A) C = 0.5 f( y A)log dy+ 0.5 f( y A)log dy 1 py ( ) py ( ) y y 15

Examples of Channel Capacity Discrete Input Continuous Output Channels 1/1/004 16

Examples of Channel Capacity Discrete Input Continuous Output Channels 1/1/004 17

Examples of Channel Capacity Continuous Input Continuous Output Channels Both input symbol and output symbol are continuous random variable. (Infinitely dense constellation). Example (AWGN Channel): y = x+ z z ~ N(0, σ z ) Channel Capacity: f( y x)~ N( x, σ z ) I( X; Y) = H( Y) H( Y Z) = H( Y) H( Z) Since H(Z) is independent of p(x), the mutual information is maximized when H(Y) is maximized Y is Gaussian X is Gaussian. p( x) ( x z ) C = max I( X; Y) = 0.5log 1 + σ / σ E X = σ x 1/1/004 18

Channel Capacity for continuous time AWGN channel 1/1/004 19

Bandwidth Efficiency 1/1/004 0

Bandwidth Efficiency 1/1/004 1

Bandwidth Efficiency 1/1/004

Bandwidth Efficiency 1/1/004 3

System Performance Spectral efficiency vs power efficiency Various code performance assume to operate at 3 6 BER ~10 10 High Bandwidth Efficiency ~ M-ary modulation High Bandwidth Expansion ~ Orthogonal Modulation, CDMA 1/1/004 4

Channel Capacity of Fading Flat fading channel: Channels Memoryless Fading Channel: ( ) ( ) y = h x + z h ~ CN 0,0.5, z ~ CN 0, σ n n n n n n z p( y x, h ) = p( y x, h ) N N N n n n n= 1 Encoding frame spans over multiple fading coefficients. Fading coefficients are i.i.d. between symbols. Average Power Constraint: Average transmitted power across a coding frame is constrained (short term average). Ergodic Channel Capacity When the transmission time (over a coding frame) is long enough >> Coherence Time, the long term ergodic property of the fading process is revealed. Finite average channel capacity is achievable. N 1/1/004 5

Channel Capacity of Fading Channels Channel Capacity of Fading Channels Fast Fading Channels Ergodic Capacity Capacity is a deterministic no. Zero packet error If R < capacity Capacity with CSIR Capacity with CSIT Capacity with CSIR & CSIT Slow Fading Channels Outage Capacity Capacity is itself a random variable Cannot guarantee zero packet error. Capacity with CSIR Capacity with CSIT Capacity with CSIR & CSIT Capacity with no CSIR, no CSIT Capacity with no CSIR, no CSIT 1/1/004 6

Channel Capacity of Fading Ergodic Channel Capacity (fast fading): Channels Case 1: Perfect CSIR only Channel Capacity: p( X) p( X) p( X) p( X) { } C = max IXYH ( ;, ) = max IXY ( ; H) + IXH ( ; ) CSIR ( ) = max I( X; Y H) = max E I X; Y H = h Example: - Rayleigh fading σγ x C = max EH [ I( X; Y H = h) ] = log 1 f ( γ ) dγ p( X) + σ 0 z 1/1/004 7

Channel Capacity of Fading Channels Case 3: Perfect CSIR & CSIT (fast fading) The channel capacity is given by: CCSIT, CSIR = E h max I( X; Y H = h) p( X h) Example (Temporal Power Water-filling): For Rayleigh fading channel with perfect CSIT & CSIR, the capacity achieving distribution p(x h) is complex Gaussian σ x ( h) with power. The optimal strategy is to adaptive power over a temporal domain: ( ) ( ) 1/1/004 C = C γ f γ dγ 8 γ γ = h

Example Temporal Power Water-filling The optimization problem: Choose optimal transmit power allocation σ X to maximize the channel capacity at an average transmit power constraint (averaged over one encoding frame). ( ) σ x γ γ max L( λσ, ) max log ( ) x = 1 λσ x γ σx σ + x σ z + ( ) ( ) ( ) 1 σ z log γ / λσ z γ λσ z L / σ x = 0 σ x ( γ ) = C = C γ f γ dγ C ( γ ) = λ γ γ 0 otherwise The solution temporal power water-filling - when the channel fading is poor, reduce or shut down the tx power - when the channel fading is good, increase the tx power γ = h 1/1/004 9

Channel Capacity of Slow Fading Channels Outage Capacity: Ergodic assumption is not always valid {e.g. over the entire encoding frame, the fading process is non-ergodic}. Example: Quasi-static fading channel: Channel fading coefficient is constant within an encoding frame. For the case with perfect CSIR only, the instantaneous channel capacity is a function of channel fading the instantaneous channel capacity is itself a random variable. There is no guarantee on error-free transmission of the coded frame. zero ergodic capacity. There may not be classical Shannon s meaning attached to the capacity. {in other words, there is a finite probability that the actual transmission rate, no matter how small it is, exceeds the instantaneous mutual information. The effective capacity is quoted as outage capacity. {i.e. a capacity together with it s c.d.f.} 1/1/004 30

Outage Capacity Example: Rayleigh fading with perfect CSIR: The instantaneous capacity is a random variable given by: γσ X C ( γ ) = log 1+ σ z The outage probability is given by: ( ) ( ) { } ( ) z r Pout r = Pr C γ r = 1 exp 1 σ x P(r)=0 r=0; only the zero rate is compatible with zerooutage, thus eliminating any reliable communication in Shannon s sense. Taking into L retransmission, the average goodput (average number of packets successfully delivered to the receiver) is given by: ρ = rpr{ r C( γ )} 1/1/004 31 σ

The role of CSIT In fast fading channels, CSIT allows power adaptation increases the ergodic capacity: C CSIT, CSIR C CSIR In slow fading channels, CSIT allows power + rate adaptation achieve zero outage probability 100% reliability of packet transmission is possible even in slow fading with CSIT. P (, ) 0 outage CSIT CSIR 1/1/004 3

Summary of Main Points Entropy Measure the degree of randomness of a random variable Physical meaning (by AEP theorem): Consider a sequence of i.i.d. source symbols (X1,,Xn), the size of the typical set ~ NH(X) # of bits required to encode the source symbol ~ H(X) bits per symbol for large N. Mutual Information Measure the reduction of entropy on X by observation of Y. Physical Meaning (by channel coding theorem) Maximum number of information bits per channel use that can be transmitted over a channel with arbitrarily low error probability. C=max_{p(x)} I(X;Y) 1/1/004 33

Summary of Main Points Channel Capacity: AWGN channels discrete time models Continuous time models Fast Flat Fading Channels CSIR only CSIT, CSIR Slow Flat Fading CSIR only packet outage C CSIR and CSIR power adaptation + rate C adaptation at the transmitter no packet outage and achieving the ergodic capacity. C C AWGN AWGN σ x = log 1 + bits per ch use σ z P av = W log 1 + bits/sec Wη0 σ h log 1 bits/ch use x fastfading, CSIR = E + σ z + h = E log bits/ch use fastfading, CSIT, CSIR λσ z 1/1/004 34

Appendix - Advanced Topics 1/1/004 35

Appendix A: Proof of Shannon s Coding Theorem 1/1/004 36

Random Coding Achievability Proof NR Fix p(x) and we generate a independent N codewords at random N according to the distribution p( x ) = p( xn) we have the random n= 1 codebook given by the matrix: x1 (1) x (1) N Ω= R R R NR NR x1 ( ) xn ( ) The code is then revealed to both the transmitter and the receiver. A message W is chosen according to a uniform distribution Pr{ } NR NR W = w =, w { 1,,..., } The w-th codeword is sent out of the transmitter. The receiver receives a sequence Y N according to the distribution N N N ( ( )) n n( ) n= 1 ( ) P y x w = p y x w 1/1/004 37

Random Coding Achievability Proof The receiver guesses which message was transmitted based on typical set decoding. the receiver declares Wˆ was transmitted if N ( ˆ ), N X W Y is jointly typical ( ) There is no other index k, such that ( ) ( X N ( k), Y N ) A N ε If no such Wˆ exists or if there is more than one such, then an error is declared. There is a decoding error if Wˆ W 1/1/004 38

Random Coding Achievability Proof We shall calculate error probability averaged over all possible codebooks. ( N) NR ( ) ( ) ( )( 1/ ) λ ( ) P = P Ω P Ω = P Ω Ω = e e w Ω Ω w NR ( 1/ ) P( Ω) λw ( Ω ) = P( Ω) λ1 ( Ω ) = Pr ( E W = 1) w Ω ( ) { ˆ N Pr ( ) is transmitted} λw Ω = W w X w By symmetry of code construction, the average probability of error averaged over all codes does not depend on the particular index sent. Without loss of generality, assume message W=1 is transmitted. Define the events: i-th codeword and Y N are jointly typical ( ) {( ) ε } N N N NR E = X ( i), Y A i [1,.., ] i 1/1/004 39 Ω

Random Coding Achievability Proof Error occurs when E E E c 1 NR The transmitted codeword and the received sequence are not jointly typical or A wrong codeword is jointly typical with the received sequence Based on Union Bound, we have: NR c c Pe = Pr{ E1 E E NR } P( E ) ( ) 1 + P En n= By Jointly AEP Theorem, P( E c 1 ) ε for sufficiently large N. Since by random code generation, X N (1) and X N (i) are independent and so are Y N and X N (i), we have by Jointly AEP Theorem that: ( ) P E i ( ( ; ) 3ε ) N I X Y Hence, the average error probability is bounded by: P e NR i= ( ( ; ) 3 ε) NR ( ( ; ) 3 ε) 3Nε ( ( ; ) ) ( ) ε + = ε + 1 ε + N I X Y N I X Y N I X Y R 1/1/004 40

Random Coding Achievability Proof Hence, if R< I( X; Y) 3ε, we could choose ε and N such that P e ε Since the average error probability is averaged over all codebooks, * there exists at least one codebook Ω with a small average probability of error P * e ( Ω ) ε. This proved that there exists at least an achievable code with rate R < I(X;Y) for arbitrarily large N. Although the theorem shows that there exists good codes, it does not provide a way of constructing the best codes. We could generate a good code by randomly generating the codewords in a codebook. However, without any structure in the codewords, it is very difficult to decode at large N. 1/1/004 41

Converse Proof ( ) We have to prove that any sequence of NR ( N ), N codes with Pe 0 must have R C. N ( N) ( N) N Fano s Inequality: H( W Y ) 1 + Pe NR, where Pe = Pr g( Y ) W Let Y N be the result of passing XN through a discrete memoryless channel. Then I X N ; Y N NC for all p( x N ) ( ) Let W be the message index drawn according to a uniform distribution over { 1,,..., NR }. Hence, we have: N N N N N NR= HW ( ) = HW ( Y ) + IWY ( ; ) HW ( Y ) + I( X ( W); Y ) 1 + P NR + I( X ( W ); Y ) 1+ P NR + NC ( N) N N ( N) e e R P R+ 1/ N + C P 1 C/ R 1/ NR ( N) ( N) e e Hence, if R>C, the error probability is bounded away from 0. 1/1/004 4

Appendix B: Proof of Continuous time AWGN Capacity. 1/1/004 43

Examples of Channel Capacity Continuous Time Waveform Channels Channel input & output are continuous time random signals (instead of discrete time symbols). yt () = xt () + zt () Vector Representation of continuous time signals T s For any random signal x(t) with finite energy xt () dt<, we have 0 N where { φn () t } is a set of orthogonal basis n= lim E x() t xnφn() t = 0 1,..., N N T n= 1 s * x xt (), φ () t x t φ t dt function = = () () n n n 0 For bandlimited signal x(t) with bandwidth W and duration T where WT >>1, the signal space dimension approaches WT. Converting the continuous time AWGN channel into vector representation (over a dimension of WT), we have: xt ( ) x= x,..., x z'( t) z ' = z,..., z y = x+ z' ( ) ( ) 1 WT 1 WT Note that z(t) requires a higher dimension signal space. Yet, y represents a sufficient statistics on x the noise components in the noise vector z are Gaussian i.i.d. Total noise power = 1 E z ' = ηw ( W) E z = η W T η0 E z n = 0 n 0 1/1/004 44

Examples of Channel Capacity Continuous Time Waveform Channel The channel transition probability: p( y x) = p( y x ) p( y x ) = 1 exp ( y x ) WT n n n n n n n= 1 πη η 0 0 Treating the signal vector x as one super-symbol, the asymptotic channel capacity (per unit time) is given by: ~ bits per second 1 C = lim max I( XY ; ) T p( x) T WT max I( XY ; ) = max I( X ; Y ) = WTlog 1+ n n p( x) p( x ) n= 1 n η0 Since the average transmitted power is given by: T WT 1 1 1 av () x n σ x σ x 0 n= 1 the channel capacity (bit per second) is given by: 1/1/004 45 σ x / Pav P = x t dt = = W x W = T T WT W P av C = W log 1+ Wη0