Lecture 4 Noisy Channel Coding
|
|
- Ariel Robbins
- 5 years ago
- Views:
Transcription
1 Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, / 56 I-Hsiang Wang IT Lecture 4
2 The Channel Coding Problem w Channel Encoder x N Noisy Channel y N Channel Decoder bw Meta Description 1 Message: Random message W Unif [1 : 2 K ]. 2 Channel: Consist of an input alphabet X, an output alphabet Y, and a family of conditional distributions { p ( y k x k, y k 1) k N } determining the stochastic relationship between the output symbol y k and the input symbol x k along with all past signals ( x k 1, y k 1). 3 Encoder: Encode the message w by a length N codeword x N X N. 4 Decoder: Reconstruct message ŵ from the channel output y N. 5 Efficiency: Maximize the code rate R K N bits/channel use, given certain decoding criterion. 2 / 56 I-Hsiang Wang IT Lecture 4
3 Decoding Criterion: Vanishing Error Probability w Channel Encoder x N Noisy Channel A key performance measure: Error Probability P (N) e Question: Is it possible to get zero error probability? y N Channel Decoder bw { } P W Ŵ. Ans: Probably not, unless the channel noise has some special structure. Following the development of lossless source coding, Shannon turned the attention to answering the following question: Is it possible to have a sequence of encoder/decoder pairs such that P e (N) 0 as N? If so, what is the largest possible code rate R where vanishing error probability is possible? Recall: In lossless source coding, we see that the infimum of compression rates where vanishing error probability is possible is H ({S i } ). 3 / 56 I-Hsiang Wang IT Lecture 4
4 Rate R Block Length N Probability of Error P (N) e Capacity: Take N, Require P (N) e 0 = sup R = C. Error Exponent: Take N, Fix rate R = min P (N) e 2 NE(R). Finite Block Length: Fix N, Require ( P ) (N) e ε V = sup R = C N Q 1 (ε) + O log N N. Remark: For source coding, one can establish a similar framework. 4 / 56 I-Hsiang Wang IT Lecture 4
5 In this lecture we only focus on capacity. In other words, we ignore the issue of finite block length (FBL). FBL performance can be obtained via techniques extending from CLT. We do not pursue finer analysis on the error probability via large deviation techniques either. 5 / 56 I-Hsiang Wang IT Lecture 4
6 Discrete Memoryless Channel (DMC) In order to demonstrate the key ideas in channel coding, in this lecture we shall focus on discrete memoryless channels (DMC) defined below. Definition 1 (Discrete Memoryless Channel) A discrete channel ( X, { p ( y k x k, y k 1) k N }, Y ) is memoryless if k N, p ( y k x k, y k 1) = p Y X (y k x k ). In other words, Y k X k ( X k 1, Y k 1). Here the conditional p.m.f. p Y X is called the channel law or channel transition function. Question: is our definition of a channel sufficient to specify p ( y N x N ), the stochastic relationship between the channel input (codeword) x N and the channel output y N? 6 / 56 I-Hsiang Wang IT Lecture 4
7 p ( y N x N) = p ( x N, y N) p (x N ) p ( x N, y N) N = p ( x k, y k x k 1, y k 1) = k=1 N p ( y k x k, y k 1) p ( x k x k 1, y k 1) k=1 Hence, we need to further specify { p ( x k x k 1, y k 1) k N }, which cannot be obtained from p ( x N). Interpretation: { p ( x k x k 1, y k 1) k N } is induced by the encoding function, which implies that the encoder can potentially make use of the past channel output, i.e., feedback. 7 / 56 I-Hsiang Wang IT Lecture 4
8 DMC without Feedback w Channel Encoder x k Noisy Channel y k w Channel Encoder x k y k 1 D Noisy Channel y k No Feedback With Feedback Suppose in the system, the encoder has no knowledge about the realization of the channel output, then, p ( x k x k 1, y k 1) = p ( x k x k 1) for all k N, and it is said the the channel has no feedback. In this case, specifying { p ( y k x k, y k 1) k N } suffices to specify p ( y N x N). Proposition 1 (DMC without Feedback) For a DMC ( X, p Y X, Y ) without feedback, p ( y N x N) = N p Y X (y i x i ). k=1 8 / 56 I-Hsiang Wang IT Lecture 4
9 Overview In this lecture, we would like to establish the following (informally described) noisy channel coding theorem due to Shannon: For a DMC ( X, p Y X, Y ), the maximum code rate with vanishing error probability is the channel capacity C max I (X ; Y ). p X ( ) The above holds regardless of the availability of feedback. To demonstrate this result, we organize the lecture as follows: 1 Give the problem formulation, state the main theorem, and visit a couple of examples to show how to compute channel capacity. 2 Prove the converse part: an achievable rate cannot exceed C. 3 Prove the achievability part with a random coding argument. 9 / 56 I-Hsiang Wang IT Lecture 4
10 Channel Capacity Proof of the Weak Converse Feedback Capacity 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 10 / 56 I-Hsiang Wang IT Lecture 4
11 Channel Capacity Proof of the Weak Converse Feedback Capacity 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 11 / 56 I-Hsiang Wang IT Lecture 4
12 Channel Capacity Proof of the Weak Converse Feedback Capacity Channel Coding without Feedback: Problem Setup w Channel Encoder x N Noisy Channel y N Channel Decoder bw 1 A ( 2 NR, N ) channel code consists of an encoding function (encoder) enc N : [1 : 2 K ] X N that maps each message w to a length N codeword x N, where K NR. a decoding function (decoder) dec N : Y N [1 : 2 K ] { } that maps a channel output sequence y N to a reconstructed message ŵ or an error message. 2 The error probability is defined as P (N) e { } P W Ŵ. 3 A rate R is said to be achievable if there exist a sequence of ( 2 NR, N ) codes such that P (N) e 0 as N. The channel capacity is defined as C sup {R R : achievable}. 12 / 56 I-Hsiang Wang IT Lecture 4
13 Channel Capacity Proof of the Weak Converse Feedback Capacity Channel Coding Theorem for Discrete Memoryless Channel Theorem 1 (Channel Coding Theorem for DMC without Feedback) The capacity C of the DMC p (y x) without feedback is given by C = max I (X ; Y ). (1) p(x) The capacity formula (1) is intuitive, since I (X ; Y ) represents the amount of information about the channel input X that one can infer from the channel output Y. The maximization over p (x) stands for choosing the best possible input distribution so that the amount of information transfer is maximized. 13 / 56 I-Hsiang Wang IT Lecture 4
14 Channel Capacity Proof of the Weak Converse Feedback Capacity Rest of the lecture: 1 First we give some examples of noisy channels to show how to compute capacity. 2 Then, we prove that for any rate R > C, it is impossible to have vanishing error probability (converse). 3 Finally, we prove that for any R < C, there exist a sequence of encoding/decoding schemes such that the error probability vanishes as blocklength tends to (achievability), based on a probabilistic argument called random coding. 14 / 56 I-Hsiang Wang IT Lecture 4
15 Channel Capacity Proof of the Weak Converse Feedback Capacity Binary Symmetric Channel A binary symmetric channel (BSC) consists of Binary input/output X = Y = {0, 1}. [ ] 1 p p Channel law p (y x) =. p 1 p The capacity of BSC is C BSC = 1 H b (p). 1 X 0 1 p p p 1 p Y 0 1 To compute BSC capacity, observe I (X ; Y ) = H (Y ) H (Y X ), and H (Y X = 0 ) = H (Y X = 1 ) = H b (p) = H (Y X ) = H b (p). H (Y ) log 2 = 1, with equality iff Y is uniform. Question: Is it possible to choose a p (x) such that Y is uniform? Ans: Yes, choose X to be uniform = C = max I (X ; Y ) = 1 H b (p). p(x) 15 / 56 I-Hsiang Wang IT Lecture 4
16 Channel Capacity Proof of the Weak Converse Feedback Capacity Binary Erasure Channel A binary erasure channel (BEC) consists of Binary input X = {0, 1} and output with erasure Y = {0, 1, }. [ ] 1 p p 0 Channel law p (y x) =. 0 p 1 p The capacity of BEC is C BEC = 1 p. X p p p 1 p Y 0 1 Suppose we begin with I (X ; Y ) = H (Y ) H (Y X ). Then, H (Y X = 0 ) = H (Y X = 1 ) = H b (p) = H (Y X ) = H b (p). H (Y ) log 3, with equality iff Y is uniform. Question: Is it possible to choose a p (x) such that Y is uniform? Ans: No. So, we cannot say that max p(x) H (Y ) = log / 56 I-Hsiang Wang IT Lecture 4
17 Channel Capacity Proof of the Weak Converse Feedback Capacity X 0 1 p Y 0 Y 0 1 X 0 p 1 p 1 p Instead, we can start with I (X ; Y ) = H (X ) H (X Y ). Then, we have the reverse channel law p (x y) = 1 0 α 1 α, where α P {X = 0}. 0 1 H (X Y = 0 ) = H (X Y = 1 ) = 0, H (X Y = ) = H b (α) = H (X ) = H (X Y ) = P {Y = } = ph (X ). H (X ) 1, with equality iff X is uniform. Hence, C BEC = max p(x) (1 p) H (X ) = 1 p. 17 / 56 I-Hsiang Wang IT Lecture 4
18 Channel Capacity Proof of the Weak Converse Feedback Capacity Erasure Channel We can generalize BEC to the following erasure channel: Input X, output Y = X { }. 1 p, y = x Channel law p (y x) = p, y = 0, otherwise A motivation for this model is from networking, where the erasure models the packet drop. Exercise 1 Show that the capacity of the erasure channel is C EC = (1 p) log X. 18 / 56 I-Hsiang Wang IT Lecture 4
19 Channel Capacity Proof of the Weak Converse Feedback Capacity Symmetric Channel In computing the capacity of BSC, we observe that 1 H (Y X ) = H b (p) regardless of p (x). Why? Because all rows of p (y x) are permutations of a same probability vector [ p 1 p ]. 2 H (Y ) = log Y can be attained, that is, Y can be made uniform by choosing X to be uniform. Why? Because all columns of p (y x), have the same sum x p (y x). Definition 2 (Symmetric Channel) A symmetric channel is a channel with channel law p (y x) satisfying (1) all rows of p (y x) are permutations of a same probability vector p, and (2) all columns of p (y x), have the same sum x p (y x). Exercise 2 Show that the capacity of a symmetric channel is log Y H (p). 19 / 56 I-Hsiang Wang IT Lecture 4
20 Channel Capacity Proof of the Weak Converse Feedback Capacity Computing Capacity of DMC via Convex Optimization For a DMC, we are able to find its capacity efficiently by revoking efficient algorithms in solving convex programs, since I (X ; Y ) is a concave function of p (x) for fixed p (y x). Proposition 2 I (X ; Y ) is a concave function of p (x) for fixed p (y x). pf: By definition, I (X ; Y ) = H (Y ) H (Y X ). H (Y X ) = x p (x) H (Y X = x) is a linear function of p (x), because H (Y X = x) = p (y x) log p (y x) is constant for fixed p (y x). H (Y ) is a concave function of p (y). p (y) is a linear function of p (x) for fixed p (y x). Hence, H (Y ) is a concave function of p (x) for fixed p (y x). Putting the above together, we complete the proof. 20 / 56 I-Hsiang Wang IT Lecture 4
21 Channel Capacity Proof of the Weak Converse Feedback Capacity 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 21 / 56 I-Hsiang Wang IT Lecture 4
22 Channel Capacity Proof of the Weak Converse Feedback Capacity Proof of the (Weak) Converse (1) We would like to show that for every sequence of ( 2 NR, N ) codes such that P (N) e 0 as N, the rate R max I (X ; Y ). p(x) pf: Note that W Unif [1 : 2 K ] and hence K = H (W ). ( ) ( ) NR H (W ) = I W ; Ŵ + H W Ŵ (2) I ( W ; Y N ) ( P (N) e log ( 2 K + 1 )) (3) N k=1 I ( W ; Y k Y k 1 ) ( ) P (N) e (NR + 2) (2) is due to K = NR NR and chain rule. (3) is due to W Y N Ŵ and Fano s inequality. (4) is due to chain rule and 2 K NR NR+1. (4) 22 / 56 I-Hsiang Wang IT Lecture 4
23 Channel Capacity Proof of the Weak Converse Feedback Capacity Proof of the (Weak) Converse (2) ( ) Set ε N 1 N 1 + P (N) e (NR + 2), we see that ε N 0 as N because lim N P (N) e = 0. The next step is to relate N k=1 I ( W ; Y k Y k 1 ) to I (X ; Y ), by the following manipulation: I ( W ; Y k Y k 1 ) I ( W, Y k 1 ; Y k ) I ( W, Y k 1, X k ; Y k ) (5) = I (X k ; Y k ) max I (X ; Y ) (6) p(x) (5) is due to the fact that conditioning reduces entropy. (6) is due to DMC: p ( y k x k, y k 1, w ) = p ( y k x k, y k 1) = p (y k x k ) = Y k X k ( W, X k 1, Y k 1) = Y k X k ( W, Y k 1). 23 / 56 I-Hsiang Wang IT Lecture 4
24 Channel Capacity Proof of the Weak Converse Feedback Capacity Proof of the (Weak) Converse (3) Hence, we have NR N k=1 I ( W ; Y k Y k 1 ) + Nε N N max I (X ; Y ) + Nε N p(x) = R max I (X ; Y ) + ε N, N. p(x) Taking N, we have: R max I (X ; Y ) if it is achievable. p(x) Remark: Similar to the source coding problem, a stronger version of the converse holds in the channel coding problem as well: if R > C, then 1 as N for any encoding/decoding functions. P (N) e 24 / 56 I-Hsiang Wang IT Lecture 4
25 Channel Capacity Proof of the Weak Converse Feedback Capacity 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 25 / 56 I-Hsiang Wang IT Lecture 4
26 Channel Capacity Proof of the Weak Converse Feedback Capacity Channel Coding with Feedback: Problem Setup w Channel Encoder x N Noisy Channel y N Channel Decoder bw D 1 A ( 2 NR, N ) channel code consists of an encoding function (encoder) enc N : [1 : 2 K ] Y N 1 X N that maps each message w to a length N codeword x N, where K NR, and the k-th symbol x k is a function of ( w, y k 1) for all k [1 : N]. a decoding function (decoder) dec N : Y N [1 : 2 K ] { } that maps a channel output sequence y N to a reconstructed message ŵ or an error message. 2 The error probability is defined as P (N) e { } P W Ŵ. 3 A rate R is said to be achievable if there exist a sequence of ( 2 NR, N ) codes such that P (N) e 0 as N. The channel capacity is defined as C sup {R R : achievable}. 26 / 56 I-Hsiang Wang IT Lecture 4
27 Channel Capacity Proof of the Weak Converse Feedback Capacity Dependency Graph: Without vs. With Feedback X N X 1 p Y X Y N Y 1 enc N X 2 Y 2 dec N W cw X k Y k X N No Feedback Y N 27 / 56 I-Hsiang Wang IT Lecture 4
28 Channel Capacity Proof of the Weak Converse Feedback Capacity Dependency Graph: Without vs. With Feedback X N X 1 p Y X Y N Y 1 enc N X 2 Y 2 dec N W cw X k Y k X N With Feedback Y N 28 / 56 I-Hsiang Wang IT Lecture 4
29 Channel Capacity Proof of the Weak Converse Feedback Capacity Feedback Capacity Theorem 2 (Channel Coding Theorem for DMC with Feedback) The capacity of the DMC p (y x) with feedback is given by (1), the same as that without feedback. In other words, feedback does not increase the channel capacity for DMC. The proof is immediate because in the converse proof of channel coding theorem without feedback, we do not make use of the assumption that there is no feedback. In other words, the proof is identical even with feedback. Remark: Although feedback does not increase capacity, it does improve the reliability (error exponent) and finite-blocklength performance greatly. Furthermore, the design and the complexity of the coding scheme may also be greatly simplified and reduced due to feedback. The details are out of scope of this lecture. 29 / 56 I-Hsiang Wang IT Lecture 4
30 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 30 / 56 I-Hsiang Wang IT Lecture 4
31 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 31 / 56 I-Hsiang Wang IT Lecture 4
32 Overview In order to prove the achievability part of Theorem 1, we need to show the following mathematical statement: R < C, R 0, a sequence of ( 2 NR, N ) codes such that lim N P(N) e = 0. In general, to prove the existence of certain objects satisfying some desirable properties, there are two possible ways: 1 Explicitly construct an object and prove that the properties hold. 2 Assume that no objects can satisfy the properties, and show contradiction. The achievability proof presented in this lecture is more of the second flavor, and in fact belongs to the so-called probabilistic method. 32 / 56 I-Hsiang Wang IT Lecture 4
33 The Probabilistic Method What is the probabilistic method? Roughly speaking, in order to show the existence of certain objects satisfying some desirable properties, One first imposes particular probability distribution over the possible object space. Then, by showing that on average the properties hold or the properties hold with non-zero probability, one concludes the existence of such objects. Example 1 Given a set of n-dimensional unit vectors {v 1, v 2,..., v k }, show that x i {±1}, i = 1,..., k such that k i=1 x iv i k. 33 / 56 I-Hsiang Wang IT Lecture 4
34 pf: Let {X i } k i=1 be i.i.d. r.v. s with P {X i = 1} = P {X i = 1} = 1 2. Define V [ k i=1 X iv i. Compute E V 2] as follows: [ E V 2] = E [ V T V ] [( k = E X i v T i k = E i=1 j=1 i=1 k X i X j v T i v j = ) ( k )] X i v i k i=1 i=1 j=1 k E [X i X j ] v T i v j {X i } are mutually independent, E [X i X j ] = E [X i ] E [X j ] = 0 for i j. [ E V 2] = k i=1 E [ ] X 2 i vi 2 = k. Hence, x i {±1}, i = 1,..., k such that k i=1 x iv i k. Otherwise, [ E V 2] should be less than k, leading to contradiction. 34 / 56 I-Hsiang Wang IT Lecture 4
35 Paul Erdős ( ) 35 / 56 I-Hsiang Wang IT Lecture 4
36 Coding over Noisy Channel Before we prove the main theorem, let us set up a few notations related to coding over noisy channel. 1 Codebook c = { x N (1), x N (2)... x N ( 2 K)} consists of the 2 K codewords and is the range of the encoding function. 2 ML Decoder (maximum likelihood) is the optimal decoder that minimizes the probability of error P e (N) when the messages are uniformly chosen (uniform prior): ŵ ML = arg max w [1:2K ] p ( y N x N (w) ). } W 3 Probability of Error of Message m: λ m P {Ŵ W = m In principle, one can derive the ML decoding rule and compute P (N) e given codebook. But, there are some challenges toward proving the channel coding theorem. for a 36 / 56 I-Hsiang Wang IT Lecture 4
37 Challenges and Work-Arounds First, the expression of error probability of ML is usually intractable, and it is hard to obtain any insight regarding the asymptotic behaviors. Second, it is unclear how to construct the codebook and the corresponding decoding scheme. In summary, to prove the achievability part of the channel coding theorem, there are two main challenges we shall overcome: 1 How to show the existence of good codebooks? We circumvent the issue of explicit construction by using a random coding argument (a kind of the probabilistic method) 2 How to analyze the error probability? We circumvent the issue of ML decoding error analysis by using a suboptimal decoder and derive upper bounds on the probability of error of the chosen decoder. 37 / 56 I-Hsiang Wang IT Lecture 4
38 Proof Program 1 Random Codebook Generation: Generate an ensemble of codebooks according to certain probability distribution. Hence, codebook C becomes a random object. 2 Error Probability Analysis: Goal: Show that as N, E C [ P (N) e,ml (C) ] 0, and conclude that there must exist a codebook c such that the decoding error probability P (N) e,ml 0. To simplify analysis, we shall introduce suboptimal decoders and give a tractable upper bound of error probability using union of events bound. 38 / 56 I-Hsiang Wang IT Lecture 4
39 Random Codebook Generation A simple way is to i.i.d. generate 2 K codewords, and each codeword p ( x N) i=1 p X (x i ). In other words, if we stack all 2 K codewords together into a 2 K N matrix C, the elements of the matrix C will be i.i.d. distributed according to p X : (each row is a codeword) X 1 (1) X 2 (1) X N (1) X 1 (2) X 2 (2) X N (2) c = ( X ) ( 1 2 K X ) ( 2 2 K X ) N 2 K and p (c) P {C = c} = 2 K w=1 N i=1 p X (x i (w)). It turns out the symmetry in such codebook ensemble distribution helps simplify the analysis. 39 / 56 I-Hsiang Wang IT Lecture 4
40 Encoding and Decoding For a realization c of the codebook random ensemble C, we describe the encoding and decoding methods below. Encoding: for a message m [1 : 2 K ], choose the m-th row of the codebook c and send it out. Decoding: ideally one would like to use the following ML decoding rule: ŵ ML = arg max w [1:2K ] p ( y N x N (w) ). However, the performance of ML decoder is usually not tractable, as mentioned before. Instead, we introduce a suboptimal decoder based on typical sequences as follows: ŵ T = a unique w such that ( x N (w), y N) T ε (N) (X, Y). Note: there are some other suboptimal decoders can be used, such as threshold decoders. 40 / 56 I-Hsiang Wang IT Lecture 4
41 Error Probability Analysis (1) Since the ML decoder is optimal, we can analyze the performance of the typicality decoder and use it as[ an upper] bound. Hence, our goal is turned to proving lim N E C P (N) e,t (C) = 0. 1 The first step [ is to use] the symmetry of codebook ensemble to simplify E C P (N) e,t (C) and argue that we can focus on analyzing the error probability of the first codeword X N (1) averaged over C: [ ] [ E C P (N) e,t (C) = E C 2 ] K 2 K λ m (C) = 2 K E C [λ m (C)] m=1 m = 2 K m E C [λ 1 (C)] = E C [λ 1 (C)] = P {Error, averaged over C W = 1} 41 / 56 I-Hsiang Wang IT Lecture 4
42 Error Probability Analysis (2) 2 For notational simplicity, use E denote the text Error event and drop the averaged over C. Our next focus is to upper bound P {E W = 1} P 1 (E). The trick here is to distinguish into two different kinds of errors: { (X E a N (1), Y N) } / T ε (N) E = E a E t, { (X E t N (w), Y N) } T ε (N) for some w 1 The core is whether or not the joint sequence ( { X N (w), Y N) are (X ε-typical. Let us define A w N (w), Y N) } T ε (N). We can then rewrite E a = A c 1, E t = w 1 A w, and hence E = E a E t = A c 1 ( w 1A w ). 42 / 56 I-Hsiang Wang IT Lecture 4
43 Error Probability Analysis (3) 3 We are now ready to apply the union of events bound: P 1 {E} = P 1 {A c 1 ( w 1 A w )} P 1 {A c 1} + 2 K w=2 P 1 {A w }. Next, we shall develop upper bounds on the probability that the actual transmitted codeword X N (1) and the actual received signal Y N are not (jointly) typical. the probability that some other (random) codeword X N ( 1) and the actual received signal Y N are (jointly) typical. Lemma 1 (A Key Lemma) N(I(X ;Y ) δ(ε)) P 1 {A 1 } 1 ε for N large enough, and P 1 {A w } 2 for all w 1, where δ (ε) 0 as ε / 56 I-Hsiang Wang IT Lecture 4
44 Error Probability Analysis (4) 4 Finally, let us put all the above together and apply Lemma 1: [ ] E C P (N) e,t (C) P {E} = P {E W = 1} P 1 {E} P 1 {A c 1} + ε + 2 K w=2 2 K w=2 P 1 {A w } N(I(X ;Y ) δ(ε)) 2 N(I(X ;Y ) δ(ε) R) ε + 2 As long as R I (X ; Y ) δ(ε), we are able to make [ P {E} ] 2ε for N large enough, which is equivalent to lim N E C P (N) e,t (C) / 56 I-Hsiang Wang IT Lecture 4
45 Completion of the We have shown [ that as] long as R I (X ; Y ) δ(ε), lim N E C P (N) e,t (C) 0, and hence there must exist a realization of codebook c such that P (N) e,t (c) 0 as N. Finally, taking the codebook generating distribution p X = arg max p(x) I (X ; Y ), we conclude that R < C = max p(x) I (X ; Y ), R is achievable. 45 / 56 I-Hsiang Wang IT Lecture 4
46 Proof of Lemma 1 (1): Recap of Typicality Recall: by definition, an ε-typical (vector) sequence (x n, y n ) shall satisfy π (a, b x n, y n ) p X,Y (a, b) εp X,Y (a, b), (a, b) X Y. (Note: we can think of (X, Y) as a r.v. and apply the same definition of typicality!) Hence, if (X n, Y n ) n i=1 p X,Y (x i, y i ), then we have 0 (x n, y n ) T (n) ε (X, Y) = x n T (n) ε (X), y n T ε (n) (Y). 1 (x n, y n ) T ε (n) (X, Y), 1 n log p (xn, y n ) H (X, Y ) δ(ε), where δ(ε) = εh (X, Y ). ( ) 2 p T ε (n) (X, Y) 1 ε for n large enough. 3 T (n) ε (X, Y) 2 n(h(x,y )+δ(ε)). 4 T ε (n) (X, Y) (1 ε)2 n(h(x,y ) δ(ε)) for n large enough. 46 / 56 I-Hsiang Wang IT Lecture 4
47 Proof of Lemma 1 (2): Typical with Actual Codeword { (X Let us first consider P 1 {A 1 } = P N (1), Y N) } T (N) W = 1. We are averaging over a random codebook ensemble C, and the random codebook is generated element-by-element i.i.d. based on p X. DMC without feedback implies p ( y N x N) = N i=1 p Y X (y i x i ). Hence, given W = 1, ( X N (1), Y N) has the following joint distribution: p ( x N, y N) = p ( x N) p ( y N x N) = N = N i=1 p X,Y (x i, y i ) ε i=1 p X (x i ) N i=1 p Y X (y i x i ) By Property 2 (LLN), we see that for N large enough, { (X P 1 {A 1 } = P N (1), Y N) } T (N) W = 1 1 ε. ε 47 / 56 I-Hsiang Wang IT Lecture 4
48 Proof of Lemma 1 (3): Typical with a Wrong Codeword { (X Consider P 1 {A w } = P N (w), Y N) } T (N) W = 1 for w 1. Note that we are averaging over a random codebook ensemble C, and the random codebook is generated element-by-element i.i.d. based on p X. Hence, although X N (1) and X N (w) have the same marginal distribution p X, they are actually independent. Due to DMC, ( X N (1), Y N) X N (w). Hence, Y N X N (w), and P 1 {A w } = (x N,y N ) T (N) ε 2 N(1+ε)H(X,Y ) }{{} cardinality upper bound on typical set = 2 N(I(X ;Y ) δ(ε)), ε p ( x N) p ( y N) 2 N(1 ε)h(x) }{{} upper bound on prob. of a typical sequence where δ (ε) = ε (H (X, Y ) + H (X) + H (Y)) 0 as ε 0. 2 N(1 ε)h(y) }{{} upper bound on prob. of a typical sequence 48 / 56 I-Hsiang Wang IT Lecture 4
49 Some Reflections Reflection 1: Mutual independence of codewords. In the random coding argument of the proof, 2 K N elements of the codebook matrix C are generated i.i.d., and hence the 2 K rows { X N (1),..., X N ( 2 K)} are mutually independent. However, in the proof we only require pairwise independence: X N (1) X N (w), w 1. Reflection 2: Typicality decoder. We use typicality decoder other than the optimal ML decoder to find tractable upper bounds on the error probability. There are other suboptimal decoders can be used. For example, the following threshold decoder can also work: ( ŵ th a unique w such that i x N (w) ; y N) > β, where i ( x N ; y N) log p(xn,y N ) p(x N )p(y N ) = N k=1 log p Y X (y k x k ) p X (x k ), and β I (X ; Y ) ε. 49 / 56 I-Hsiang Wang IT Lecture 4
50 1 Channel Capacity and the Weak Converse Channel Capacity Proof of the Weak Converse Feedback Capacity 2 50 / 56 I-Hsiang Wang IT Lecture 4
51 Joint Source-Channel Coding: Problem Setup Source {S i } s Ns Nc x ync Channel Encoder p Y X Decoder bs Ns Destination Source model: discrete stationary ergodic with entropy rate H ({S i } ). Channel model: DMC p Y X with channel capacity C ( ) p Y X. ) 1 A ( S NcR, N c joint source-channel code consists of an encoding function (encoder) enc Nc : S Ns X Nc that maps each source sequence s Ns to a length N c codeword x Nc, N s N c R. a decoding function (decoder) dec Nc : Y N c S N s that maps a channel output sequence y N c to a reconstructed sequence ŝ N s. { } 2 The error probability is defined as P (N c) e P S Ns ŜN s. 3 A rate R is said ) to be achievable if there exist a sequence of ( S NcR, N c codes such that P (N c) e 0 as N c. 51 / 56 I-Hsiang Wang IT Lecture 4
52 Theorem Theorem 3 () C 1 If R < H({S i} ), then R is achievable, i.e., lossless reconstruction of source {S i } is possible via the noisy channel p Y X. C 2 Conversely, if R > H({S i } ), then R is not achievable, i.e., lossless reconstruction is impossible. Source s Ns Source Encoder b K Channel Encoder x Nc Binary Interface Noisy Channel Destination bs Ns Source Decoder b b K Channel Decoder y Nc 52 / 56 I-Hsiang Wang IT Lecture 4
53 Proof of Achievability pf: (Achievability Part): Choose a ( 2 N sr s, N s ) lossless source code with Rs = H ({S i } ) + ε s. Choose a ( 2 N cr c, N c ) channel code with Rc = C ε c. Due the the channel coding theorem, the binary sequence b K lives in the digital interface between the source and the channel coders can be decoded with vanishing error probability. Due to the lossless source coding theorem, the source sequences can be reconstructed with vanishing error probability as long as the bit sequence b K can be successfully decoded by the channel decoder. Concatenate the above two codes together, we see that as long as N s R s < N c R c N s N c < R c C ε R s = c H({S i } )+ε s, the separation scheme is able to reconstruct the source sequence with vanishing error probability. C Since ε s, ε c can be made arbitrarily small, as long as R < H({S i} ), it is achievable. 53 / 56 I-Hsiang Wang IT Lecture 4
54 Proof of Converse pf: (Converse Part): We shall prove that achievable R, R N s H ({S i } ) H ( S ) ) Ns = I (S Ns ; ŜNs ( I ( S N s ; Y N c ) + N c k=1 1 + P (N c) e I ( S N s ; Y k Y k 1 ) ( + ) + H (S Ns Ŝ Ns ) N s log S 1 + P (N c) e C H({S i } ). ) N s log S (7) (8) N c (C + ε Nc ), where ε Nc 0 as N c. (9) (7) is due to the property of entropy rate and chain rule. (8) is due to S Ns Y Nc ŜNs and Fano s inequality. (9) is due to similar steps as in the channel coding converse proof. Hence, R Ns N c C H({S i} ) if R is achievable. 54 / 56 I-Hsiang Wang IT Lecture 4
55 Summary 55 / 56 I-Hsiang Wang IT Lecture 4
56 Channel coding theorem: C = max p(x) I (X ; Y ), for DMC p Y X with or without feedback Weak converse: Fano s inequality, data processing inequality, and DMC assumption Achievability: random coding argument, typicality decoder Feedback does not increase the capacity of DMC. Symmetric channel capacity = log Y H (p ), where p permutes all rows of p Y X. Erasure channel capacity = (1 p) log X. Joint source-channel coding theorem: C R < = R is achievable; R > H({S i} ) C H({S i} ) = R is not achievable. Source-channel separation is optimal. 56 / 56 I-Hsiang Wang IT Lecture 4
Lecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
More informationLecture 5 Channel Coding over Continuous Channels
Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationLecture 5: Channel Capacity. Copyright G. Caire (Sample Lectures) 122
Lecture 5: Channel Capacity Copyright G. Caire (Sample Lectures) 122 M Definitions and Problem Setup 2 X n Y n Encoder p(y x) Decoder ˆM Message Channel Estimate Definition 11. Discrete Memoryless Channel
More informationMidterm Exam Information Theory Fall Midterm Exam. Time: 09:10 12:10 11/23, 2016
Midterm Exam Time: 09:10 12:10 11/23, 2016 Name: Student ID: Policy: (Read before You Start to Work) The exam is closed book. However, you are allowed to bring TWO A4-size cheat sheet (single-sheet, two-sided).
More informationCapacity of a channel Shannon s second theorem. Information Theory 1/33
Capacity of a channel Shannon s second theorem Information Theory 1/33 Outline 1. Memoryless channels, examples ; 2. Capacity ; 3. Symmetric channels ; 4. Channel Coding ; 5. Shannon s second theorem,
More informationEE 4TM4: Digital Communications II. Channel Capacity
EE 4TM4: Digital Communications II 1 Channel Capacity I. CHANNEL CODING THEOREM Definition 1: A rater is said to be achievable if there exists a sequence of(2 nr,n) codes such thatlim n P (n) e (C) = 0.
More informationLecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
More informationLECTURE 10. Last time: Lecture outline
LECTURE 10 Joint AEP Coding Theorem Last time: Error Exponents Lecture outline Strong Coding Theorem Reading: Gallager, Chapter 5. Review Joint AEP A ( ɛ n) (X) A ( ɛ n) (Y ) vs. A ( ɛ n) (X, Y ) 2 nh(x)
More informationEE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15
EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability
More informationLecture 6 Channel Coding over Continuous Channels
Lecture 6 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 9, 015 1 / 59 I-Hsiang Wang IT Lecture 6 We have
More informationAppendix B Information theory from first principles
Appendix B Information theory from first principles This appendix discusses the information theory behind the capacity expressions used in the book. Section 8.3.4 is the only part of the book that supposes
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationfor some error exponent E( R) as a function R,
. Capacity-achieving codes via Forney concatenation Shannon s Noisy Channel Theorem assures us the existence of capacity-achieving codes. However, exhaustive search for the code has double-exponential
More informationLecture 9 Polar Coding
Lecture 9 Polar Coding I-Hsiang ang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 29, 2015 1 / 25 I-Hsiang ang IT Lecture 9 In Pursuit of Shannon s Limit Since
More informationLecture 8: Channel and source-channel coding theorems; BEC & linear codes. 1 Intuitive justification for upper bound on channel capacity
5-859: Information Theory and Applications in TCS CMU: Spring 23 Lecture 8: Channel and source-channel coding theorems; BEC & linear codes February 7, 23 Lecturer: Venkatesan Guruswami Scribe: Dan Stahlke
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationLecture 15: Conditional and Joint Typicaility
EE376A Information Theory Lecture 1-02/26/2015 Lecture 15: Conditional and Joint Typicaility Lecturer: Kartik Venkat Scribe: Max Zimet, Brian Wai, Sepehr Nezami 1 Notation We always write a sequence of
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationLecture 8: Shannon s Noise Models
Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have
More informationNoisy channel communication
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 6 Communication channels and Information Some notes on the noisy channel setup: Iain Murray, 2012 School of Informatics, University
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 13, 2015 Lucas Slot, Sebastian Zur Shannon s Noisy-Channel Coding Theorem February 13, 2015 1 / 29 Outline 1 Definitions and Terminology
More informationX 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
More informationNational University of Singapore Department of Electrical & Computer Engineering. Examination for
National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed:
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More informationReliable Computation over Multiple-Access Channels
Reliable Computation over Multiple-Access Channels Bobak Nazer and Michael Gastpar Dept. of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, 94720-1770 {bobak,
More informationLecture 1: The Multiple Access Channel. Copyright G. Caire 12
Lecture 1: The Multiple Access Channel Copyright G. Caire 12 Outline Two-user MAC. The Gaussian case. The K-user case. Polymatroid structure and resource allocation problems. Copyright G. Caire 13 Two-user
More informationShannon s noisy-channel theorem
Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for
More informationFeedback Capacity of a Class of Symmetric Finite-State Markov Channels
Feedback Capacity of a Class of Symmetric Finite-State Markov Channels Nevroz Şen, Fady Alajaji and Serdar Yüksel Department of Mathematics and Statistics Queen s University Kingston, ON K7L 3N6, Canada
More informationLecture 14 February 28
EE/Stats 376A: Information Theory Winter 07 Lecture 4 February 8 Lecturer: David Tse Scribe: Sagnik M, Vivek B 4 Outline Gaussian channel and capacity Information measures for continuous random variables
More informationSecond-Order Asymptotics in Information Theory
Second-Order Asymptotics in Information Theory Vincent Y. F. Tan (vtan@nus.edu.sg) Dept. of ECE and Dept. of Mathematics National University of Singapore (NUS) National Taiwan University November 2015
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 2015 Abstract In information theory, Shannon s Noisy-Channel Coding Theorem states that it is possible to communicate over a noisy
More informationLecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
More informationECEN 655: Advanced Channel Coding
ECEN 655: Advanced Channel Coding Course Introduction Henry D. Pfister Department of Electrical and Computer Engineering Texas A&M University ECEN 655: Advanced Channel Coding 1 / 19 Outline 1 History
More informationLecture 10: Broadcast Channel and Superposition Coding
Lecture 10: Broadcast Channel and Superposition Coding Scribed by: Zhe Yao 1 Broadcast channel M 0M 1M P{y 1 y x} M M 01 1 M M 0 The capacity of the broadcast channel depends only on the marginal conditional
More informationComputing and Communications 2. Information Theory -Entropy
1896 1920 1987 2006 Computing and Communications 2. Information Theory -Entropy Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Entropy Joint entropy
More informationLecture 8: Channel Capacity, Continuous Random Variables
EE376A/STATS376A Information Theory Lecture 8-02/0/208 Lecture 8: Channel Capacity, Continuous Random Variables Lecturer: Tsachy Weissman Scribe: Augustine Chemparathy, Adithya Ganesh, Philip Hwang Channel
More informationLecture 2. Capacity of the Gaussian channel
Spring, 207 5237S, Wireless Communications II 2. Lecture 2 Capacity of the Gaussian channel Review on basic concepts in inf. theory ( Cover&Thomas: Elements of Inf. Theory, Tse&Viswanath: Appendix B) AWGN
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationA Graph-based Framework for Transmission of Correlated Sources over Multiple Access Channels
A Graph-based Framework for Transmission of Correlated Sources over Multiple Access Channels S. Sandeep Pradhan a, Suhan Choi a and Kannan Ramchandran b, a {pradhanv,suhanc}@eecs.umich.edu, EECS Dept.,
More informationEE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
More informationLecture 4: Proof of Shannon s theorem and an explicit code
CSE 533: Error-Correcting Codes (Autumn 006 Lecture 4: Proof of Shannon s theorem and an explicit code October 11, 006 Lecturer: Venkatesan Guruswami Scribe: Atri Rudra 1 Overview Last lecture we stated
More informationSolutions to Homework Set #3 Channel and Source coding
Solutions to Homework Set #3 Channel and Source coding. Rates (a) Channels coding Rate: Assuming you are sending 4 different messages using usages of a channel. What is the rate (in bits per channel use)
More informationNetwork coding for multicast relation to compression and generalization of Slepian-Wolf
Network coding for multicast relation to compression and generalization of Slepian-Wolf 1 Overview Review of Slepian-Wolf Distributed network compression Error exponents Source-channel separation issues
More informationEE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions
EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where
More informationPerformance-based Security for Encoding of Information Signals. FA ( ) Paul Cuff (Princeton University)
Performance-based Security for Encoding of Information Signals FA9550-15-1-0180 (2015-2018) Paul Cuff (Princeton University) Contributors Two students finished PhD Tiance Wang (Goldman Sachs) Eva Song
More informationECE Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE)
ECE 74 - Advanced Communication Theory, Spring 2009 Homework #1 (INCOMPLETE) 1. A Huffman code finds the optimal codeword to assign to a given block of source symbols. (a) Show that cannot be a Huffman
More informationInformation Theory. Lecture 10. Network Information Theory (CT15); a focus on channel capacity results
Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity results The (two-user) multiple access channel (15.3) The (two-user) broadcast channel (15.6) The relay channel
More informationAn Achievable Error Exponent for the Mismatched Multiple-Access Channel
An Achievable Error Exponent for the Mismatched Multiple-Access Channel Jonathan Scarlett University of Cambridge jms265@camacuk Albert Guillén i Fàbregas ICREA & Universitat Pompeu Fabra University of
More informationLecture 11: Quantum Information III - Source Coding
CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More informationCapacity of AWGN channels
Chapter 3 Capacity of AWGN channels In this chapter we prove that the capacity of an AWGN channel with bandwidth W and signal-tonoise ratio SNR is W log 2 (1+SNR) bits per second (b/s). The proof that
More informationLecture 11: Polar codes construction
15-859: Information Theory and Applications in TCS CMU: Spring 2013 Lecturer: Venkatesan Guruswami Lecture 11: Polar codes construction February 26, 2013 Scribe: Dan Stahlke 1 Polar codes: recap of last
More informationShannon s A Mathematical Theory of Communication
Shannon s A Mathematical Theory of Communication Emre Telatar EPFL Kanpur October 19, 2016 First published in two parts in the July and October 1948 issues of BSTJ. First published in two parts in the
More informationNoisy-Channel Coding
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/05264298 Part II Noisy-Channel Coding Copyright Cambridge University Press 2003.
More informationLECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem
LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser
More informationlossless, optimal compressor
6. Variable-length Lossless Compression The principal engineering goal of compression is to represent a given sequence a, a 2,..., a n produced by a source as a sequence of bits of minimal possible length.
More informationEquivalence for Networks with Adversarial State
Equivalence for Networks with Adversarial State Oliver Kosut Department of Electrical, Computer and Energy Engineering Arizona State University Tempe, AZ 85287 Email: okosut@asu.edu Jörg Kliewer Department
More informationThe Poisson Channel with Side Information
The Poisson Channel with Side Information Shraga Bross School of Enginerring Bar-Ilan University, Israel brosss@macs.biu.ac.il Amos Lapidoth Ligong Wang Signal and Information Processing Laboratory ETH
More informationDistributed Lossless Compression. Distributed lossless compression system
Lecture #3 Distributed Lossless Compression (Reading: NIT 10.1 10.5, 4.4) Distributed lossless source coding Lossless source coding via random binning Time Sharing Achievability proof of the Slepian Wolf
More informationExercise 1. = P(y a 1)P(a 1 )
Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a
More informationCapacity of the Discrete Memoryless Energy Harvesting Channel with Side Information
204 IEEE International Symposium on Information Theory Capacity of the Discrete Memoryless Energy Harvesting Channel with Side Information Omur Ozel, Kaya Tutuncuoglu 2, Sennur Ulukus, and Aylin Yener
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationUCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, Solutions to Take-Home Midterm (Prepared by Pinar Sen)
UCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, 2017 Solutions to Take-Home Midterm (Prepared by Pinar Sen) 1. (30 points) Erasure broadcast channel. Let p(y 1,y 2 x) be a discrete
More informationA Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback
A Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback Vincent Y. F. Tan (NUS) Joint work with Silas L. Fong (Toronto) 2017 Information Theory Workshop, Kaohsiung,
More information1 Introduction to information theory
1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through
More informationInformation Theory. M1 Informatique (parcours recherche et innovation) Aline Roumy. January INRIA Rennes 1/ 73
1/ 73 Information Theory M1 Informatique (parcours recherche et innovation) Aline Roumy INRIA Rennes January 2018 Outline 2/ 73 1 Non mathematical introduction 2 Mathematical introduction: definitions
More informationEE376A - Information Theory Final, Monday March 14th 2016 Solutions. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Final, Monday March 14th 216 Solutions Instructions: You have three hours, 3.3PM - 6.3PM The exam has 4 questions, totaling 12 points. Please start answering each question on
More informationSolutions to Homework Set #1 Sanov s Theorem, Rate distortion
st Semester 00/ Solutions to Homework Set # Sanov s Theorem, Rate distortion. Sanov s theorem: Prove the simple version of Sanov s theorem for the binary random variables, i.e., let X,X,...,X n be a sequence
More informationEE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes
EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check
More information5 Mutual Information and Channel Capacity
5 Mutual Information and Channel Capacity In Section 2, we have seen the use of a quantity called entropy to measure the amount of randomness in a random variable. In this section, we introduce several
More informationCan Feedback Increase the Capacity of the Energy Harvesting Channel?
Can Feedback Increase the Capacity of the Energy Harvesting Channel? Dor Shaviv EE Dept., Stanford University shaviv@stanford.edu Ayfer Özgür EE Dept., Stanford University aozgur@stanford.edu Haim Permuter
More informationMultiaccess Channels with State Known to One Encoder: A Case of Degraded Message Sets
Multiaccess Channels with State Known to One Encoder: A Case of Degraded Message Sets Shivaprasad Kotagiri and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame,
More informationApproaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes
Approaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes 1 Zheng Wang, Student Member, IEEE, Jie Luo, Member, IEEE arxiv:0808.3756v1 [cs.it] 27 Aug 2008 Abstract We show that
More informationMulticoding Schemes for Interference Channels
Multicoding Schemes for Interference Channels 1 Ritesh Kolte, Ayfer Özgür, Haim Permuter Abstract arxiv:1502.04273v1 [cs.it] 15 Feb 2015 The best known inner bound for the 2-user discrete memoryless interference
More information4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information
4F5: Advanced Communications and Coding Handout 2: The Typical Set, Compression, Mutual Information Ramji Venkataramanan Signal Processing and Communications Lab Department of Engineering ramji.v@eng.cam.ac.uk
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationRepresentation of Correlated Sources into Graphs for Transmission over Broadcast Channels
Representation of Correlated s into Graphs for Transmission over Broadcast s Suhan Choi Department of Electrical Eng. and Computer Science University of Michigan, Ann Arbor, MI 80, USA Email: suhanc@eecs.umich.edu
More informationSubset Universal Lossy Compression
Subset Universal Lossy Compression Or Ordentlich Tel Aviv University ordent@eng.tau.ac.il Ofer Shayevitz Tel Aviv University ofersha@eng.tau.ac.il Abstract A lossy source code C with rate R for a discrete
More informationDelay, feedback, and the price of ignorance
Delay, feedback, and the price of ignorance Anant Sahai based in part on joint work with students: Tunc Simsek Cheng Chang Wireless Foundations Department of Electrical Engineering and Computer Sciences
More information5958 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010
5958 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 12, DECEMBER 2010 Capacity Theorems for Discrete, Finite-State Broadcast Channels With Feedback and Unidirectional Receiver Cooperation Ron Dabora
More informationDigital Communications III (ECE 154C) Introduction to Coding and Information Theory
Digital Communications III (ECE 154C) Introduction to Coding and Information Theory Tara Javidi These lecture notes were originally developed by late Prof. J. K. Wolf. UC San Diego Spring 2014 1 / 8 I
More informationECE Information theory Final (Fall 2008)
ECE 776 - Information theory Final (Fall 2008) Q.1. (1 point) Consider the following bursty transmission scheme for a Gaussian channel with noise power N and average power constraint P (i.e., 1/n X n i=1
More informationBlock 2: Introduction to Information Theory
Block 2: Introduction to Information Theory Francisco J. Escribano April 26, 2015 Francisco J. Escribano Block 2: Introduction to Information Theory April 26, 2015 1 / 51 Table of contents 1 Motivation
More informationCovert Communication with Channel-State Information at the Transmitter
Covert Communication with Channel-State Information at the Transmitter Si-Hyeon Lee Joint Work with Ligong Wang, Ashish Khisti, and Gregory W. Wornell July 27, 2017 1 / 21 Covert Communication Transmitter
More informationSource and Channel Coding for Correlated Sources Over Multiuser Channels
Source and Channel Coding for Correlated Sources Over Multiuser Channels Deniz Gündüz, Elza Erkip, Andrea Goldsmith, H. Vincent Poor Abstract Source and channel coding over multiuser channels in which
More informationFrans M.J. Willems. Authentication Based on Secret-Key Generation. Frans M.J. Willems. (joint work w. Tanya Ignatenko)
Eindhoven University of Technology IEEE EURASIP Spain Seminar on Signal Processing, Communication and Information Theory, Universidad Carlos III de Madrid, December 11, 2014 : Secret-Based Authentication
More informationError Correcting Codes: Combinatorics, Algorithms and Applications Spring Homework Due Monday March 23, 2009 in class
Error Correcting Codes: Combinatorics, Algorithms and Applications Spring 2009 Homework Due Monday March 23, 2009 in class You can collaborate in groups of up to 3. However, the write-ups must be done
More informationStrong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach Silas Fong (Joint work with Vincent Tan) Department of Electrical & Computer Engineering National University
More informationThe Method of Types and Its Application to Information Hiding
The Method of Types and Its Application to Information Hiding Pierre Moulin University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ moulin/talks/eusipco05-slides.pdf EUSIPCO Antalya, September 7,
More information4 An Introduction to Channel Coding and Decoding over BSC
4 An Introduction to Channel Coding and Decoding over BSC 4.1. Recall that channel coding introduces, in a controlled manner, some redundancy in the (binary information sequence that can be used at the
More information(each row defines a probability distribution). Given n-strings x X n, y Y n we can use the absence of memory in the channel to compute
ENEE 739C: Advanced Topics in Signal Processing: Coding Theory Instructor: Alexander Barg Lecture 6 (draft; 9/6/03. Error exponents for Discrete Memoryless Channels http://www.enee.umd.edu/ abarg/enee739c/course.html
More informationExercises with solutions (Set B)
Exercises with solutions (Set B) 3. A fair coin is tossed an infinite number of times. Let Y n be a random variable, with n Z, that describes the outcome of the n-th coin toss. If the outcome of the n-th
More informationLecture 4 Capacity of Wireless Channels
Lecture 4 Capacity of Wireless Channels I-Hsiang Wang ihwang@ntu.edu.tw 3/0, 014 What we have learned So far: looked at specific schemes and techniques Lecture : point-to-point wireless channel - Diversity:
More informationECE Information theory Final
ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the
More informationMAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK. SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A
MAHALAKSHMI ENGINEERING COLLEGE QUESTION BANK DEPARTMENT: ECE SEMESTER: IV SUBJECT CODE / Name: EC2252 COMMUNICATION THEORY UNIT-V INFORMATION THEORY PART-A 1. What is binary symmetric channel (AUC DEC
More information