School of Computer and Communication Sciences. Information Theory and Coding Notes on Random Coding December 12, 2003.
|
|
- Maria Hall
- 5 years ago
- Views:
Transcription
1 ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE School of Computer and Communication Sciences Handout 8 Information Theor and Coding Notes on Random Coding December 2, 2003 Random Coding In this note we prove the achievabilit part of the channel coding theorem for discrete memorless channels without feedback. The discussion is largel based on chapter 5 of R. G. Gallager, Information Theor and Reliable Communication, Wile, Discrete Memorless Channels Without Feedback Throughout this note we will fix the channel we wish to communicate over. Let X and Y denote the input and output alphabets of this channel. We will assume that the channel is discrete, i.e., that X and Y are finite sets. The behavior of the channel will be completel described b specifing for each k the function P k : X k Y k R, (x k, k ) P k ( k x k, k ), which gives the probabilit of receiving the letter k at the output at time k, given the past and current inputs to the channel, and the past outputs of the channel. We make the two following assumptions: the channel is memorless and used without feedback. We will call a channel memorless if P k ( k x k, k ) = P ( k x k ), which in words means that the channel keeps no memor of the past inputs and outputs in determining the output of time k. Also note that the channel does not behave differentl at different times: the function P on the right hand side is not an explicit function of k. If a memorless channel is used without feedback, i.e., if P Xk X k,y k (x k x k, k ) = P Xk X k (x k x k ) (in words: if the channel inputs do not depend on the past channel outputs) then P Y n X n(n x n ) = P X n,y n(xn, n ) P X n (x n ) n k= = P X k,y k X k,y (x k k, k x k, k ) P X n (x n ) n k= = P X k X k,y (x k k x k, k )P Yk X k k,y ( k x k, k ) P X n (x n ) n k= = P X k X (x k k x k )P Yk X k ( k x k ) P X n (x n ) n = P ( k x k ) k=
2 where we use the memorless and without feedback conditions at the fourth equalit. From now on, we will restrict our attention of channels used without feedback. With some abuse of notation we will let P ( x) denote the probabilit of receiving the sequence = n at the output of the channel when the channel input is the sequence x = x n. If the channel is memorless, we see from above that 2. Block Codes P ( x) = n P ( k x k ). k= A block code with M messages and block length n is a mapping from a set of M messages {,..., M} to channel input sequences of length n. Thus, a block code is specified when we specif the M channel input sequences c = (c,,..., c,n ),..., c M = (c M,,..., c M,n ) the messages are mapped into. We will call c m the codeword for message m. To send message m with such a block code we simpl give the sequence c m to the channel as input. A decoder for such a block code is a mapping from channel output sequences Y n to the set of M messages {,..., M}. For a given decoder, let D m Y n denote the set of channel outputs which are mapped to message m. Since an output sequence is mapped to exactl one message, D m s form a collection of disjoint sets whose union is Y n. We define the rate of a block code with M messages and block length n as ln M n, and given such a code and a decoder we define P e,m = D m P ( c m ), the probabilit of a decoding error when message m is sent. Further define P e,ave = M M m= P e,m and P e,max = max m M P e,m as the average and maximal (both over the possible messages) error probabilit of such a code and decoder. Among man possible decoding methods, the rule that minimizes P e,ave is the maximum likelihood rule. Given a channel output sequence, the maximum likelihood rule decodes a message m for which P ( c m ) P ( c m ) for ever m m, and if there are more than one such m chooses one of them arbitraril. We will restrict ourselves in the following to the maximum likelihood rule. 3. Error probabilit for two codewords Consider now the case when M = 2, so the block code consists of two codewords, c and c 2. We will find a bound on P e,m for the maximum likelihood decoding rule. 2
3 Suppose message is to be sent. The channel input is then c, and the probabilit of receiving at the channel output is P ( c ). An error will occur if the received sequence is not in D. Since for ever not in D, P ( c 2 ) P ( c ), (but there ma be s for which P ( c 2 ) = P ( c ) and D ) we have P e, = D P ( c ) : P ( c 2 ) P ( c ) P ( c ) = P ( c ) P ( c2 ) P ( c ) :P ( c 2 ) P ( c ) P ( c ) P ( c 2) s P ( c ) s P ( c ) P ( c 2) s P ( c ) s for an s 0 = P ( c ) s P ( c 2 ) s The choice s = /2 gives us P ( c )P ( c 2 ), P e, and b smmetr, the same quantit also upper bounds P e,2. For a memorless channel P ( c m ) = n k= P ( k c m,k ) and we obtain P e,m P ( c )P ( c 2 ) = P ( c, )P ( c 2, )... P ( n c,n P ( n c 2,n ) n [ ] [ ] = P ( c, )P ( c 2, )... P ( n c,n )P ( n c 2,n ) = n [ ] P ( c,k )P ( c 2,k ). k= For example, for a binar smmetric channel with crossover probabilit ɛ, we see that { if c,k = c 2,k P ( c,k )P ( c 2,k ) = 2 ɛ( ɛ) else, and we obtain n P e,m [4ɛ( ɛ)] d/2 where d is the number of places c and c 2 are different (i.e., d = {k : c,k c 2,k } ). 4. Error Probabilit for two randoml chosen codewords Suppose now that the codewords c and c 2 are chosen randoml and independentl each from a distribution Q on X n. Observe that the codewords are random variables C and 3
4 C 2 and the probabilit that the block code is the particular block code with codewords c and c 2 is given b Q(c )Q(c 2 ). The error probabilities P e,m are now random variables P e,m since the are functions of C and C 2. Let P e,m denote the expectation of P e,m. We will now give an upper bound on P e,m in two different was. The first is the more straightforward, but the second wa will turn out to be conceptuall more useful. We will take m =, but is is clear b smmetr that P e, = P e,2. Method : Write P e, = E[P e, ] = c c 2 Q(c )Q(c 2 )E[P e, C = c, C 2 = c 2 ]. But when we are given c and c 2, P e, is no longer random, and we can use the bound of the last section on the probabilit of error to get P e, Q(c )Q(c 2 ) P ( c ) P ( c 2 ) c c 2 = [ Q(c ) P ( c )][ Q(c 2 ) P ( c2 )] c c 2 = [ Q(c) ] 2 P ( c) c where the last equalit follows b noting that the sums in the brackets in the next to last line are identical since the differ onl b the index of summation. Method 2: Write P e, = E[P e, ] = Q(c )P ( c )E[P e, C = c, Y = ]. c Note that here we are computing the expectation b first conditioning on the transmitted and received sequences. Now, observe that given C = c and the received sequence, an error is sure not to occur if C 2 is chosen such that P ( C 2 ) < P ( c ), and otherwise we can upper bound P e, b. Thus, given C = c and, we have { if P ( C 2 ) P ( c ) P e, 0 if P ( C 2 ) < P ( c ) Taking expectations we see that = P ( C2 ) P ( c ). E[P e, C = c, Y = ] Pr(B 2 ) where B 2 is the event that C 2 is chosen such that P ( C 2 ) P ( c ). We can bound Pr(B 2 ) b Pr(B 2 ) = c 2 Q(c 2 ) P ( c2 ) P ( c ) c 2 Q(c 2 ) P ( c 2) s P ( c ) s for an s 0. 4
5 Taking s = /2 and substituting back we obtain the same bound as before, P e, [ Q(c) 2. P ( c)] c For memorless channels the bound simplifies if Q(c) is chosen to be Q(c) = n k= Q(c k). In this case we obtain [ [ P e, Q(x) ] 2 n P ( x)]. 5. Average error probabilit of a randoml chosen code x Consider now a code with M codewords, each codeword c m chosen independentl according to the probabilit distribution Q. Just as in the above section, the probabilit that the block code constructed is a particular block code with codewords c,..., c M is M m= Q(c m). The error probabilities P e,m are also random variables, and again let P e,m denote the expectation of P e,m. B the smmetr with respect to the permutation of the codewords in the construction, we see that P e,m does not depend on m, and it will suffice to analze P e,. We will take the extension of method 2 in the previous section as our analsis method, and write P e, = c Q(c )P ( c )E[P e, C = c, Y = ]. Now, for a given c and define for each m 2, B m as the event that codeword C m is chosen such that P ( C m ) P ( c ), i.e., that codeword m is at least as likel as the transmitted codeword. Then just as in the previous section, ( M E[P e, C = c, Y = ] Pr m=2 { min, B m ) M m=2 } Pr(B m ) [ M ρ Pr(B m )] for all ρ [0, ]. m=2 The second inequalit above is just the union bound, the third inequalit is because for ρ [0, ], x x ρ when x [0, ], and x ρ when x. Observe now that Pr(B m ) = c m Q(c m ) P ( cm) P ( c ) c m = c Q(c m ) P ( c m) s P ( c ) s for an s 0 Q(c) P ( c)s P ( c ) s and thus E[P e, C = c, Y = ] [ (M ) c 5 Q(c) P ] ρ ( c)s. P ( c ) s
6 Substituting back we get P e, (M ) ρ [ ][ ] ρ. Q(c )P ( c ) sρ Q(c)P ( c) s c c Choosing now s = /( + ρ) (this choice in fact minimizes the bound) and observing that for this choice sρ = s, and that [ ] [ ] Q(c )P ( c ) sρ = Q(c)P ( c) s c since the two summations differ onl b the summation index, we get P e, (M ) [ ] +ρ. ρ Q(c)P ( c) /(+ρ) c If we now specialize this theorem to discrete memorless channels and if we choose Q(c) = n i= Q(c i), we get that for ever ρ [0, ], P e, (M ) ρ [ c [ ] ] +ρ n Q(x)P ( x) /(+ρ) If M = e nr, then (M ) e nr, and we can summarize the above as x Theorem. Given a discrete memorless channel described b P ( x), for an blocklength n, and an R 0 consider constructing a random block code with M = e nr codewords b choosing each letter of each codeword independentl according to a distribution Q on X. Then, the expected average error probabilit of this random code satisfies P e,ave exp { n max [E 0 (ρ, Q) ρr] } ρ [0,] where E 0 (ρ, Q) = ln [ ] +ρ. Q(x)P ( x) /(+ρ) x Since the expected error probabilit cannot be better than the error probabilit of the best code we also get Corollar. Given a discrete memorless channel described b P ( x), for an distribution Q on X, an blocklength n and an R > 0, there exists a code of block length n and rate at least R with P e,ave exp { ne r (R, Q) }. where E r (R, Q) = max [E 0 (ρ, Q) ρr]. ρ [0,] Note that the above corollar establishes the existence of codes of a certain rate with a guarantee on their P e,ave, but suggests no mechanism to find them. However, if one does carr out the experiment of constructing a code b randoml choosing its codewords, the probabilit that the code obtained will be much worse than the average is small, in particular, Markov inequalit tells us that the probabilit that a code constructed as such has P e,ave larger than α P e,ave is small: Pr[P e,ave α P e,ave ] /α for α >. 6
7 ρi(q) E 0 (ρ, Q) ρ Figure : E 0 (ρ, Q) Thus, the difficult in finding good practical codes is not that the good codes are rare, but the difficult is in find good codes for which there are practical methods to encode and decode. The corollar above makes guarantees on P e,ave, but for a code constructed b random coding the value of P e,max ma be much higher than P e,ave. Can we ensure the existence of codes with small P e,max? Theorem 2. Given a discrete memorless channel described b P ( x), for an distribution Q on X, an blocklength n and an R > 0, there exists a block code of length n and rate at least R with P e,max 4 exp { ne r (R, Q) }. Proof. Let M = 2e nr. We know that there is a code with M codewords and satisfies P e,ave = M M m= P e,m (M ) ρ exp { ne 0 (ρ, Q) } for ever ρ [0, ]. Since we see that for this code (M ) ρ 2 ρ e nρr 2e nρr, P e,ave = M M m= P e,m 2 exp { ne r (R, Q) }. Now, among the M numbers P e,,..., P e,m there cannot be more than M /2 which exceed twice the average value P e,ave. Thus, among these M numbers there exist at least e nr of them which satisf P e,m 2P e,ave 4 exp { ne r (R, Q) }. Keeping onl these codewords, and noticing that in the maximum likelihood decoder that corresponds to this code the decoding sets can onl be enlarged, we see that we have constructed a code with the desired properties. 5.. Properties of E 0 and E r The value of the existence theorems we just proved depend on whether the bound we have on the probabilit of error can be made arbitraril small for a set of rates of interest. Define I(Q) = x Q(x)P ( x) ln P ( x) x Q(x )P ( x ) 7
8 as the value of mutual information between the input and output when the input distribution is Q. We will now show that for ever rate R < I(Q), the exponent E r (R, Q) > 0, and thus for ever rate R < I(Q) we can find codes with arbitraril small probabilit of error (b taking n large enough). Since this statement holds for ever input distribution Q, it also follows that for ever rate R < max Q I(Q) = max px I(X; Y ) there exist codes of arbitraril small probabilit of error. This proves that an rate R below capacit is achievable (achievabilit part of the coding theorem). A tedious but straightforward calculation also ields that E 0 (ρ, Q) ρ = I(Q). ρ=0 Using the fact that whenever Q and A are nonnegative, the function t [0, ) [ x Q(x)A(x)/t] t is nonincreasing, it is eas to show that E0 (ρ, Q) is nondecreasing in ρ. We do not need it here, but it is also possible to show that E 0 (ρ, Q) is a concave function of ρ. Thus, Figure shows the tpical behavior of E 0 as a function of ρ. Observe now that since the function R E r (R, Q) is a maximum of the linear functions R E 0 (ρ, Q) ρr, (see Figure 2), we see that E r (R, Q) > 0 whenever for some ρ [0, ], E 0 (ρ, Q) ρr > 0 is satisfied. E 0 (, Q) E 0 ( 2, Q) E 0 ( 4, Q) E r (R, Q) I(Q) R Figure 2: E r (R) as a maximum of linear functions. It now follows that if R < E 0 (ρ, Q)/ρ for some ρ (0, ] and Q, then E 0 (ρ, Q) ρr > 0 and thus E r (R, Q) > 0. But, I(Q) = E 0(ρ, Q) E 0 (ρ, Q) E 0 (0, Q) E 0 (ρ, Q) ρ = lim = lim. ρ 0 ρ=0 ρ ρ 0 ρ where the last equalit follows from fact that E 0 (0, Q) = ln x, Q(x)P ( x) = ln = 0. Thus, b the definition of the limit above, for ever R < I(Q), we can find a ρ such that E 0 (ρ, Q)/ρ > R. Hence for ever R < I(Q) we have E r (R, Q) > 0. Since the above argument holds for an input distribution Q, it holds in particular for the Q that maximizes I(Q). This proves the existence of codes with arbitraril small probabilit of error for ever rate less than the capacit max Q I(Q). The approach we used here ields to the function E r (R, Q) called random coding error exponent. To prove the achievabilit part of the coding theorem it would suffice to show that for an rate R below capacit there exists a sequence of rate R codes of length n with corresponding error probabilit that tends to zero as n tends to infinit. The random coding error exponent argument in addition to prove it, also characterizes the deca of the probabilit of error as n tends to infinit. 8
Stochastic Interpretation for the Arimoto Algorithm
Stochastic Interpretation for the Arimoto Algorithm Serge Tridenski EE - Sstems Department Tel Aviv Universit Tel Aviv, Israel Email: sergetr@post.tau.ac.il Ram Zamir EE - Sstems Department Tel Aviv Universit
More informationLECTURE 10. Last time: Lecture outline
LECTURE 10 Joint AEP Coding Theorem Last time: Error Exponents Lecture outline Strong Coding Theorem Reading: Gallager, Chapter 5. Review Joint AEP A ( ɛ n) (X) A ( ɛ n) (Y ) vs. A ( ɛ n) (X, Y ) 2 nh(x)
More informationA bound for block codes with delayed feedback related to the sphere-packing bound
A bound for block codes with delaed feedback related to the sphere-packing bound Harikrishna R Palaianur Anant Sahai Electrical Engineering and Computer Sciences Universit of California at Berkele echnical
More informationComputation of Total Capacity for Discrete Memoryless Multiple-Access Channels
IEEE TRANSACTIONS ON INFORATION THEORY, VOL. 50, NO. 11, NOVEBER 2004 2779 Computation of Total Capacit for Discrete emorless ultiple-access Channels ohammad Rezaeian, ember, IEEE, and Ale Grant, Senior
More informationShannon s noisy-channel theorem
Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for
More informationUniversal Anytime Codes: An approach to uncertain channels in control
Universal Anytime Codes: An approach to uncertain channels in control paper by Stark Draper and Anant Sahai presented by Sekhar Tatikonda Wireless Foundations Department of Electrical Engineering and Computer
More information(each row defines a probability distribution). Given n-strings x X n, y Y n we can use the absence of memory in the channel to compute
ENEE 739C: Advanced Topics in Signal Processing: Coding Theory Instructor: Alexander Barg Lecture 6 (draft; 9/6/03. Error exponents for Discrete Memoryless Channels http://www.enee.umd.edu/ abarg/enee739c/course.html
More informationEE 4TM4: Digital Communications II. Channel Capacity
EE 4TM4: Digital Communications II 1 Channel Capacity I. CHANNEL CODING THEOREM Definition 1: A rater is said to be achievable if there exists a sequence of(2 nr,n) codes such thatlim n P (n) e (C) = 0.
More informationBounds on the Maximum Likelihood Decoding Error Probability of Low Density Parity Check Codes
Bounds on the Maximum ikelihood Decoding Error Probability of ow Density Parity Check Codes Gadi Miller and David Burshtein Dept. of Electrical Engineering Systems Tel-Aviv University Tel-Aviv 69978, Israel
More informationLecture 4 Noisy Channel Coding
Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem
More information6.451 Principles of Digital Communication II Wednesday, May 4, 2005 MIT, Spring 2005 Handout #22. Problem Set 9 Solutions
6.45 Principles of Digital Communication II Wednesda, Ma 4, 25 MIT, Spring 25 Hand #22 Problem Set 9 Solutions Problem 8.3 (revised) (BCJR (sum-product) decoding of SPC codes) As shown in Problem 6.4 or
More informationNotes on Convexity. Roy Radner Stern School, NYU. September 11, 2006
Notes on Convexit Ro Radner Stern School, NYU September, 2006 Abstract These notes are intended to complement the material in an intermediate microeconomic theor course. In particular, the provide a rigorous
More informationLecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
More informationfor some error exponent E( R) as a function R,
. Capacity-achieving codes via Forney concatenation Shannon s Noisy Channel Theorem assures us the existence of capacity-achieving codes. However, exhaustive search for the code has double-exponential
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationError Exponent Region for Gaussian Broadcast Channels
Error Exponent Region for Gaussian Broadcast Channels Lihua Weng, S. Sandeep Pradhan, and Achilleas Anastasopoulos Electrical Engineering and Computer Science Dept. University of Michigan, Ann Arbor, MI
More informationNational University of Singapore Department of Electrical & Computer Engineering. Examination for
National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed:
More informationLecture 11: Quantum Information III - Source Coding
CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationEE5139R: Problem Set 4 Assigned: 31/08/16, Due: 07/09/16
EE539R: Problem Set 4 Assigned: 3/08/6, Due: 07/09/6. Cover and Thomas: Problem 3.5 Sets defined by probabilities: Define the set C n (t = {x n : P X n(x n 2 nt } (a We have = P X n(x n P X n(x n 2 nt
More informationCovert Communication with Channel-State Information at the Transmitter
Covert Communication with Channel-State Information at the Transmitter Si-Hyeon Lee Joint Work with Ligong Wang, Ashish Khisti, and Gregory W. Wornell July 27, 2017 1 / 21 Covert Communication Transmitter
More informationSpotlight on the Extended Method of Frobenius
113 Spotlight on the Extended Method of Frobenius See Sections 11.1 and 11.2 for the model of an aging spring. Reference: Section 11.4 and SPOTLIGHT ON BESSEL FUNCTIONS. Bessel functions of the first kind
More informationRandom Access: An Information-Theoretic Perspective
Random Access: An Information-Theoretic Perspective Paolo Minero, Massimo Franceschetti, and David N. C. Tse Abstract This paper considers a random access system where each sender can be in two modes of
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationLecture 1: The Multiple Access Channel. Copyright G. Caire 12
Lecture 1: The Multiple Access Channel Copyright G. Caire 12 Outline Two-user MAC. The Gaussian case. The K-user case. Polymatroid structure and resource allocation problems. Copyright G. Caire 13 Two-user
More informationLecture 5: Channel Capacity. Copyright G. Caire (Sample Lectures) 122
Lecture 5: Channel Capacity Copyright G. Caire (Sample Lectures) 122 M Definitions and Problem Setup 2 X n Y n Encoder p(y x) Decoder ˆM Message Channel Estimate Definition 11. Discrete Memoryless Channel
More informationVariable Length Codes for Degraded Broadcast Channels
Variable Length Codes for Degraded Broadcast Channels Stéphane Musy School of Computer and Communication Sciences, EPFL CH-1015 Lausanne, Switzerland Email: stephane.musy@ep.ch Abstract This paper investigates
More informationMismatched Multi-letter Successive Decoding for the Multiple-Access Channel
Mismatched Multi-letter Successive Decoding for the Multiple-Access Channel Jonathan Scarlett University of Cambridge jms265@cam.ac.uk Alfonso Martinez Universitat Pompeu Fabra alfonso.martinez@ieee.org
More informationEE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions
EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where
More informationLecture 7. Union bound for reducing M-ary to binary hypothesis testing
Lecture 7 Agenda for the lecture M-ary hypothesis testing and the MAP rule Union bound for reducing M-ary to binary hypothesis testing Introduction of the channel coding problem 7.1 M-ary hypothesis testing
More informationLecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
More informationX 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
More informationChapter 6* The PPSZ Algorithm
Chapter 6* The PPSZ Algorithm In this chapter we present an improvement of the pp algorithm. As pp is called after its inventors, Paturi, Pudlak, and Zane [PPZ99], the improved version, pps, is called
More informationCapacity of a channel Shannon s second theorem. Information Theory 1/33
Capacity of a channel Shannon s second theorem Information Theory 1/33 Outline 1. Memoryless channels, examples ; 2. Capacity ; 3. Symmetric channels ; 4. Channel Coding ; 5. Shannon s second theorem,
More informationarxiv: v1 [cs.it] 5 Feb 2016
An Achievable Rate-Distortion Region for Multiple Descriptions Source Coding Based on Coset Codes Farhad Shirani and S. Sandeep Pradhan Dept. of Electrical Engineering and Computer Science Univ. of Michigan,
More informationLecture 18: Shanon s Channel Coding Theorem. Lecture 18: Shanon s Channel Coding Theorem
Channel Definition (Channel) A channel is defined by Λ = (X, Y, Π), where X is the set of input alphabets, Y is the set of output alphabets and Π is the transition probability of obtaining a symbol y Y
More informationI - Information theory basics
I - Information theor basics Introduction To communicate, that is, to carr information between two oints, we can emlo analog or digital transmission techniques. In digital communications the message is
More informationFrans M.J. Willems. Authentication Based on Secret-Key Generation. Frans M.J. Willems. (joint work w. Tanya Ignatenko)
Eindhoven University of Technology IEEE EURASIP Spain Seminar on Signal Processing, Communication and Information Theory, Universidad Carlos III de Madrid, December 11, 2014 : Secret-Based Authentication
More informationExact Random Coding Error Exponents of Optimal Bin Index Decoding
Exact Random Coding Error Exponents of Optimal Bin Index Decoding Neri Merhav Department of Electrical Engineering Technion - Israel Institute of Technology Technion City, Haifa 32000, ISRAEL E mail: merhav@ee.technion.ac.il
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More informationReliable Computation over Multiple-Access Channels
Reliable Computation over Multiple-Access Channels Bobak Nazer and Michael Gastpar Dept. of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, 94720-1770 {bobak,
More informationLinear programming: Theory
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analsis and Economic Theor Winter 2018 Topic 28: Linear programming: Theor 28.1 The saddlepoint theorem for linear programming The
More informationEE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes
EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check
More informationCapacity of AWGN channels
Chapter 3 Capacity of AWGN channels In this chapter we prove that the capacity of an AWGN channel with bandwidth W and signal-tonoise ratio SNR is W log 2 (1+SNR) bits per second (b/s). The proof that
More informationChannel Polarization and Blackwell Measures
Channel Polarization Blackwell Measures Maxim Raginsky Abstract The Blackwell measure of a binary-input channel (BIC is the distribution of the posterior probability of 0 under the uniform input distribution
More informationAn Improved Sphere-Packing Bound for Finite-Length Codes over Symmetric Memoryless Channels
POST-PRIT OF THE IEEE TRAS. O IFORMATIO THEORY, VOL. 54, O. 5, PP. 96 99, MAY 8 An Improved Sphere-Packing Bound for Finite-Length Codes over Symmetric Memoryless Channels Gil Wiechman Igal Sason Department
More informationFlows and Connectivity
Chapter 4 Flows and Connectivit 4. Network Flows Capacit Networks A network N is a connected weighted loopless graph (G,w) with two specified vertices and, called the source and the sink, respectivel,
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationEE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15
EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability
More informationUnequal Error Protection Querying Policies for the Noisy 20 Questions Problem
Unequal Error Protection Querying Policies for the Noisy 20 Questions Problem Hye Won Chung, Brian M. Sadler, Lizhong Zheng and Alfred O. Hero arxiv:606.09233v2 [cs.it] 28 Sep 207 Abstract In this paper,
More informationLecture 8: Channel and source-channel coding theorems; BEC & linear codes. 1 Intuitive justification for upper bound on channel capacity
5-859: Information Theory and Applications in TCS CMU: Spring 23 Lecture 8: Channel and source-channel coding theorems; BEC & linear codes February 7, 23 Lecturer: Venkatesan Guruswami Scribe: Dan Stahlke
More informationCapacity of the Discrete Memoryless Energy Harvesting Channel with Side Information
204 IEEE International Symposium on Information Theory Capacity of the Discrete Memoryless Energy Harvesting Channel with Side Information Omur Ozel, Kaya Tutuncuoglu 2, Sennur Ulukus, and Aylin Yener
More informationEquivalence for Networks with Adversarial State
Equivalence for Networks with Adversarial State Oliver Kosut Department of Electrical, Computer and Energy Engineering Arizona State University Tempe, AZ 85287 Email: okosut@asu.edu Jörg Kliewer Department
More informationEE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
More informationA Comparison of Superposition Coding Schemes
A Comparison of Superposition Coding Schemes Lele Wang, Eren Şaşoğlu, Bernd Bandemer, and Young-Han Kim Department of Electrical and Computer Engineering University of California, San Diego La Jolla, CA
More informationSource Coding with Lists and Rényi Entropy or The Honey-Do Problem
Source Coding with Lists and Rényi Entropy or The Honey-Do Problem Amos Lapidoth ETH Zurich October 8, 2013 Joint work with Christoph Bunte. A Task from your Spouse Using a fixed number of bits, your spouse
More informationOn MIMO Fading Channels with Side Information at the Transmitter
On MIMO Fading Channels with ide Information at the Transmitter EE850 Project, Ma 005 Tairan Wang twang@ece.umn.edu, ID: 340947 Abstract Transmission of information over a MIMO discrete-time flat fading
More informationIN [1], Forney derived lower bounds on the random coding
Exact Random Coding Exponents for Erasure Decoding Anelia Somekh-Baruch and Neri Merhav Abstract Random coding of channel decoding with an erasure option is studied By analyzing the large deviations behavior
More informationAlgebraic Geometry Codes. Shelly Manber. Linear Codes. Algebraic Geometry Codes. Example: Hermitian. Shelly Manber. Codes. Decoding.
Linear December 2, 2011 References Linear Main Source: Stichtenoth, Henning. Function Fields and. Springer, 2009. Other Sources: Høholdt, Lint and Pellikaan. geometry codes. Handbook of Coding Theory,
More informationIntermittent Communication
Intermittent Communication Mostafa Khoshnevisan, Student Member, IEEE, and J. Nicholas Laneman, Senior Member, IEEE arxiv:32.42v2 [cs.it] 7 Mar 207 Abstract We formulate a model for intermittent communication
More informationCan Negligible Cooperation Increase Network Reliability?
Can Negligible Cooperation Increase Network Reliability? Parham Noorzad, Student Member, IEEE, Michelle Effros, Fellow, IEEE, and Michael Langberg, Senior Member, IEEE Abstract arxiv:60.05769v2 [cs.it]
More informationUC San Francisco UC San Francisco Previously Published Works
UC San Francisco UC San Francisco Previousl Published Works Title Radiative Corrections and Quantum Chaos. Permalink https://escholarship.org/uc/item/4jk9mg Journal PHYSICAL REVIEW LETTERS, 77(3) ISSN
More informationLow-Density Parity-Check Codes
Department of Computer Sciences Applied Algorithms Lab. July 24, 2011 Outline 1 Introduction 2 Algorithms for LDPC 3 Properties 4 Iterative Learning in Crowds 5 Algorithm 6 Results 7 Conclusion PART I
More informationLECTURE 15. Last time: Feedback channel: setting up the problem. Lecture outline. Joint source and channel coding theorem
LECTURE 15 Last time: Feedback channel: setting up the problem Perfect feedback Feedback capacity Data compression Lecture outline Joint source and channel coding theorem Converse Robustness Brain teaser
More informationAn introduction to basic information theory. Hampus Wessman
An introduction to basic information theory Hampus Wessman Abstract We give a short and simple introduction to basic information theory, by stripping away all the non-essentials. Theoretical bounds on
More informationApproaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes
Approaching Blokh-Zyablov Error Exponent with Linear-Time Encodable/Decodable Codes 1 Zheng Wang, Student Member, IEEE, Jie Luo, Member, IEEE arxiv:0808.3756v1 [cs.it] 27 Aug 2008 Abstract We show that
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationLecture 6: Gaussian Channels. Copyright G. Caire (Sample Lectures) 157
Lecture 6: Gaussian Channels Copyright G. Caire (Sample Lectures) 157 Differential entropy (1) Definition 18. The (joint) differential entropy of a continuous random vector X n p X n(x) over R is: Z h(x
More informationLDPC Codes. Intracom Telecom, Peania
LDPC Codes Alexios Balatsoukas-Stimming and Athanasios P. Liavas Technical University of Crete Dept. of Electronic and Computer Engineering Telecommunications Laboratory December 16, 2011 Intracom Telecom,
More informationDispersion of the Gilbert-Elliott Channel
Dispersion of the Gilbert-Elliott Channel Yury Polyanskiy Email: ypolyans@princeton.edu H. Vincent Poor Email: poor@princeton.edu Sergio Verdú Email: verdu@princeton.edu Abstract Channel dispersion plays
More informationLecture 7. The Cluster Expansion Lemma
Stanford Universit Spring 206 Math 233: Non-constructive methods in combinatorics Instructor: Jan Vondrák Lecture date: Apr 3, 206 Scribe: Erik Bates Lecture 7. The Cluster Expansion Lemma We have seen
More informationLecture 12: November 6, 2017
Information and Coding Theory Autumn 017 Lecturer: Madhur Tulsiani Lecture 1: November 6, 017 Recall: We were looking at codes of the form C : F k p F n p, where p is prime, k is the message length, and
More informationA Half-Duplex Cooperative Scheme with Partial Decode-Forward Relaying
A Half-Duplex Cooperative Scheme with Partial Decode-Forward Relaying Ahmad Abu Al Haija, and Mai Vu, Department of Electrical and Computer Engineering McGill University Montreal, QC H3A A7 Emails: ahmadabualhaija@mailmcgillca,
More informationSolutions to Homework Set #3 Channel and Source coding
Solutions to Homework Set #3 Channel and Source coding. Rates (a) Channels coding Rate: Assuming you are sending 4 different messages using usages of a channel. What is the rate (in bits per channel use)
More informationThe Method of Types and Its Application to Information Hiding
The Method of Types and Its Application to Information Hiding Pierre Moulin University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ moulin/talks/eusipco05-slides.pdf EUSIPCO Antalya, September 7,
More informationA Formula for the Capacity of the General Gel fand-pinsker Channel
A Formula for the Capacity of the General Gel fand-pinsker Channel Vincent Y. F. Tan Institute for Infocomm Research (I 2 R, A*STAR, Email: tanyfv@i2r.a-star.edu.sg ECE Dept., National University of Singapore
More informationRelay Networks With Delays
Relay Networks With Delays Abbas El Gamal, Navid Hassanpour, and James Mammen Department of Electrical Engineering Stanford University, Stanford, CA 94305-9510 Email: {abbas, navid, jmammen}@stanford.edu
More informationEECS 750. Hypothesis Testing with Communication Constraints
EECS 750 Hypothesis Testing with Communication Constraints Name: Dinesh Krithivasan Abstract In this report, we study a modification of the classical statistical problem of bivariate hypothesis testing.
More informationRefined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities
Refined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities Maxim Raginsky and Igal Sason ISIT 2013, Istanbul, Turkey Capacity-Achieving Channel Codes The set-up DMC
More informationInterference Channels with Source Cooperation
Interference Channels with Source Cooperation arxiv:95.319v1 [cs.it] 19 May 29 Vinod Prabhakaran and Pramod Viswanath Coordinated Science Laboratory University of Illinois, Urbana-Champaign Urbana, IL
More informationMidterm Exam Information Theory Fall Midterm Exam. Time: 09:10 12:10 11/23, 2016
Midterm Exam Time: 09:10 12:10 11/23, 2016 Name: Student ID: Policy: (Read before You Start to Work) The exam is closed book. However, you are allowed to bring TWO A4-size cheat sheet (single-sheet, two-sided).
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 13, 2015 Lucas Slot, Sebastian Zur Shannon s Noisy-Channel Coding Theorem February 13, 2015 1 / 29 Outline 1 Definitions and Terminology
More information11.4. RECURRENCE AND TRANSIENCE 93
11.4. RECURRENCE AND TRANSIENCE 93 Similar arguments show that simple smmetric random walk is also recurrent in 2 dimensions but transient in 3 or more dimensions. Proposition 11.10 If x is recurrent and
More informationEE376A - Information Theory Final, Monday March 14th 2016 Solutions. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Final, Monday March 14th 216 Solutions Instructions: You have three hours, 3.3PM - 6.3PM The exam has 4 questions, totaling 12 points. Please start answering each question on
More informationMultimedia Communications. Mathematical Preliminaries for Lossless Compression
Multimedia Communications Mathematical Preliminaries for Lossless Compression What we will see in this chapter Definition of information and entropy Modeling a data source Definition of coding and when
More informationAccurate Statistical Analysis of High-Speed Links with PAM-4 Modulation
EPEPS IBIS Summit San Jose, California October 8, 05 Accurate Statistical Analsis of High-Speed Links with PAM-4 Modulation Vladimir Dmitriev-Zdorov Mentor Graphics Corp., vladimir_dmitriev-zdorov@mentor.com
More informationError Exponent Regions for Gaussian Broadcast and Multiple Access Channels
Error Exponent Regions for Gaussian Broadcast and Multiple Access Channels Lihua Weng, S. Sandeep Pradhan, and Achilleas Anastasopoulos Submitted: December, 5 Abstract In modern communication systems,
More informationLecture 3: Error Correcting Codes
CS 880: Pseudorandomness and Derandomization 1/30/2013 Lecture 3: Error Correcting Codes Instructors: Holger Dell and Dieter van Melkebeek Scribe: Xi Wu In this lecture we review some background on error
More informationCovert Timing Channel Analysis of Rate Monotonic Real-Time Scheduling Algorithm in MLS systems
Covert Tig Channel Analsis of Rate Monotonic Real-Time Scheduling Algorithm in MS sstems Joon Son, Jim Alves-Foss Abstract The modern digital battlesphere requires the development and deploment of multi-level
More informationA. Derivation of regularized ERM duality
A. Derivation of regularized ERM dualit For completeness, in this section we derive the dual 5 to the problem of computing proximal operator for the ERM objective 3. We can rewrite the primal problem as
More informationLecture 4: Proof of Shannon s theorem and an explicit code
CSE 533: Error-Correcting Codes (Autumn 006 Lecture 4: Proof of Shannon s theorem and an explicit code October 11, 006 Lecturer: Venkatesan Guruswami Scribe: Atri Rudra 1 Overview Last lecture we stated
More informationIntroduction to Convolutional Codes, Part 1
Introduction to Convolutional Codes, Part 1 Frans M.J. Willems, Eindhoven University of Technology September 29, 2009 Elias, Father of Coding Theory Textbook Encoder Encoder Properties Systematic Codes
More informationAn instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationLecture 15: Conditional and Joint Typicaility
EE376A Information Theory Lecture 1-02/26/2015 Lecture 15: Conditional and Joint Typicaility Lecturer: Kartik Venkat Scribe: Max Zimet, Brian Wai, Sepehr Nezami 1 Notation We always write a sequence of
More informationPractical Polar Code Construction Using Generalised Generator Matrices
Practical Polar Code Construction Using Generalised Generator Matrices Berksan Serbetci and Ali E. Pusane Department of Electrical and Electronics Engineering Bogazici University Istanbul, Turkey E-mail:
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More informationQUADRATIC GRAPHS ALGEBRA 2. Dr Adrian Jannetta MIMA CMath FRAS INU0114/514 (MATHS 1) Quadratic Graphs 1/ 16 Adrian Jannetta
QUADRATIC GRAPHS ALGEBRA 2 INU0114/514 (MATHS 1) Dr Adrian Jannetta MIMA CMath FRAS Quadratic Graphs 1/ 16 Adrian Jannetta Objectives Be able to sketch the graph of a quadratic function Recognise the shape
More informationSection 1.5 Formal definitions of limits
Section.5 Formal definitions of limits (3/908) Overview: The definitions of the various tpes of limits in previous sections involve phrases such as arbitraril close, sufficientl close, arbitraril large,
More informationThe PPM Poisson Channel: Finite-Length Bounds and Code Design
August 21, 2014 The PPM Poisson Channel: Finite-Length Bounds and Code Design Flavio Zabini DEI - University of Bologna and Institute for Communications and Navigation German Aerospace Center (DLR) Balazs
More information