UCSD ECE 255C Handout #14 Prof. Young-Han Kim Thursday, March 9, Solutions to Homework Set #5
|
|
- Olivia Cannon
- 5 years ago
- Views:
Transcription
1 UCSD ECE 255C Handout #14 Prof. Young-Han Kim Thursday, March 9, 2017 Solutions to Homework Set # Bounds on the quadratic rate distortion function. Recall that R(D) = inf F(ˆx x):e(x ˆX)2 DI(X; ˆX). Let ˆX = (1 D )(X +Z), P where Z = N(0,PD/(P D)). It is straightforward to check that for this choice of F(ˆx x), E((X ˆX) 2 ) D, and therefore I(X; ˆX) is an upper bound on R(D). But I(X; ˆX) = h( ˆX) h( ˆX X) ( = h( ˆX) 1 2 log 2πe(1 D/P) 2 PD ) P D ( 1 (2πeE( 2 log ˆX 2 ) ) 1 2 log 2πe(1 D/P) 2 fpd ) P D 1 2 log( 2πe(1 D/P) 2 (P +PD/(P D))) ) ( 1 2 log 2πe(1 D/P) 2 PD ) P D = 1 2 log(p D ). Thus, R(D) R(P/D). (Note that by the discretization method, any rate R > R(P/D) is achievable.) For the lower bound, consider R(D) = min h(x) h(x ˆX) = h(x) max h(x ˆX) = h(x) max h(x h(x) max h(x) 1 2 log(2πed). h(x ˆX) ˆX ˆX) Since the upper bound is equal to the rate distortion function of the Gaussian source of power P, this suggests that among all sources of the same power, the Gaussian source are the hardest to compress Lossy source coding from a noisy observation. Since the encoder has access only to Y n, the distortion 1
2 becomes Ed(X n,ˆx n (m(y n )))]. This distortion can be rewritten as follows. Ed(X n,ˆx n (m(y n )))] = 1 n E Xi,Y n d(x i,ˆx i (m(y n )))] (a) = y n p(y n ) 1 n (b) = y n p(y n ) 1 n (c) = y n p(y n ) 1 n = 1 n Ed(X i,ˆx i (m(y n ))) Y n = y n ] Ed(X i,ˆx i (m(y n ))) Y i = y i ] d (y i,ˆx i (m(y n ))) Ed (Y i,ˆx i (m(y n )))] = Ed (Y n,ˆx n (m(y n )))] where(a)followsfromthepropertyofconditionalexpectations; (b)followsfromthefactthat ˆx i (m(y n )) is a function of y n ; (c) follows from the definition of d (y,ˆx) in the hint. Hence, this DMS equivalent to the one with source Y and a new distortion measure d. Therefore, the rate distortion function of this DMC is given by R(D) = min I(Y; ˆX) p(ˆx y):e(d (Y, ˆX)) D = min I(Y; ˆX) p(ˆx y):e Y E(d(X, ˆX) Y)] D = min I(Y; ˆX). p(ˆx y):e(d(x, ˆX)) D 11.6 Side information with occasional erasures. We show that { p(1 H(D/p)) 0 D/p 1/2, R SI-D = R SI-ED = pr(d/p) = 0 otherwise, where R(D) is the rate distortion function without side information. For the proof of the converse, consider R SI-ED (D) = min p(ˆx x,y):e(d(x, ˆX)) D I(X; ˆX Y) and note that for p(ˆx x,y) such that we must have D Ed(X, ˆX)] ped(x, ˆX) Y = e], I(X; ˆX Y) = pi(x; ˆX Y = e) (a) pr(d/p), where (a) follows by noting that X {Y = e} p(x) and p ˆX X,Y (ˆx x,e) satisfies Ed(X, ˆX) Y = e] D/p, and concluding that I(X; ˆX Y = e) R(D/p). Thus, R SI-ED (D) pr(d/p). 2
3 For achievability, consider R SI-D (D) = and set p(u x) to attain the minimum of and ˆx(u,y) to satisfy Then, and Thus, R X (D/p) = min I(X;U Y), p(u x),ˆx(u,y):e(d(x, ˆX)) D ˆX = min I(X;U) p(u x):ed(x,u)] D/p { U Y = e, Y otherwise. I(X;U Y) = pi(x;u Y = e) = pr(d/p) Ed(X, ˆX)] = ped(x, ˆX) Y = e]+(1 p)ed(x, ˆX) Y = X] = p D p = D. R SI-D pr(d/p). Notethat this argumentdue toperron, Diggavi,and Telagar(2007)isapplicable to anydms X p(x) and the erasure side information { X w.p. 1 p, Y = e w.p. p Triangular cyclic network. (a) The cutset bound is the set of rate triples (R 1,R 2,R 3 ) such that R 1 1, R 2 1, R 3 1. (b) Achievability follows simply by routing. For the proof of the converse, consider any sequence of codes with lim n P e (n) = 0. Since, by assumption, M 3 can be recovered from M 12 and M 2 with high probability at node 2, M 3 can also be recovered at node 1 with high probability. By applying the similar arguments to other nodes, it can be easily shown that the capacity region is the same as the one for the new network depicted in Figure 1. Now the cutset bound for the new network is the set of rate triples (R 1,R 2,R 3 ) such that This completes the proof of the converse. R 1 +R 2 1, R 2 +R 3 1, R 3 +R Properties of the Gaussian relay channel capacity. The first property follows from the fact that C(P) C(g 2 31P), the capacity of the direct channel, which is strictly greater than zero for P > 0, and tends to infinity as P. The second property follows from the fact that the cutset bound, which is greater than or equal to C(P) tends to zero as P 0. To prove the third property, use the following 3
4 M 2 ˆM 3, ˆM ˆM 2, ˆM 3 M 3 M 1 ˆM1, ˆM 2 Figure 1: Equivalent triangular cyclic network. time-sharing argument. For any blocklength n, let k = αn and k = n k. For any P,P > 0 and ǫ > 0, there exists n such that C(P) < C (k) (P) + ǫ and C(P ) < C (k ) (P ) + ǫ. Consider the distribution F(x k 1 ) and functions {x 2i} k that satisfy the power constraint P and attains C(k) (P), and F (x k 1 ) and {x 1i }k that satisfy the power constraint P and attain C (k ) (P ). Now the distribution F(x k 1)F (x n k+1 ) and functions ({x 2i} n,{x 2i }n i=k+1 ) satisfy the power constraint αp +ᾱp and attain αc (k) (P)+ βc (k ) (P ). Hence αc(p)+ᾱc(p ) αc (k) (P)+ᾱC (k ) (P )+ǫ C (n) (αp +ᾱp )+ǫ C(αP +ᾱp )+ǫ. Taking ǫ 0 establishes the concavity. Note that C (k) (P) may not be concave for fixed k. Finally, that C(P) is strictly monotonically increasing in P follows from the first two parts and concavity. This completes the proofs of the properties of C(P) Directed information. (a) Consider (b) Since p(y n,x n ) = = n p(y i,x i y i 1,x i 1 ) n p(x i y i 1,x i 1 )p(y i y i 1,x i ) = p(x n y n 1 )p(y n x n ). I(X n Y n ) = I(X i ;Y i Y i 1 ) and each term in the summation is nonnegative, so is I(X n Y n ). Moreover, it is equal to zero iff each term is equal to 0, or equivalently, p(y i y i 1,x i ) = p(y i y i 1 ), i 1 : n]. In the causal conditioning notation, this can be rewritten as p(y n x n ) = p(y n ). 4
5 (c) Consider I(X n ;Y n ) = E log p(y n,x n ] ) p(y n )p(x n ) = E log p(y n X n )p(x n X n 1 ] ) = E log p(y n X n ) p(y n ) p(y n )p(x n ) ] +E log p(xn Y n 1 ) p(x n ) = I(X n Y n )+I(Y 0,Y n 1 X n ). ] (d) The inequality I(X n Y n ) I(X n ;Y n ) follows immediately from part (c). By a similar argument to part (b), the equality holds iff p(x n y n 1 ) = p(x n ) Gaussian two-way channel. Assume without loss of generality that ] 1 ρ K =. ρ 1 By the maximum differential entropy lemma, the outer bound on the capacity region in Proposition 17.3 simplifies as R 1 I(X 1 ;Y 2 X 2 ) = 1 2 log(1+g2 12Var(X 1 X 2 )) = 1 2 log(1+g2 12P), R 2 I(X 2 ;Y 1 X 1 ) = 1 2 log(1+g2 21Var(X 2 X 1 )) = 1 2 log(1+g2 21P), (1) which is attained by X 1 N(0,P) and X 2 N(0,P), independent of each other. By setting the same Gaussian (X 1,X 2 ) and Q = in the inner bound in Proposition 17.2 and taking the standard discretization argument, the rectangular rate region in (1) is achievable. Note that point-to-point capacities are achieved and that the noise correlation is irrelevant Common-message feedback capacity of broadcast channels. The common-message feedback capacity is C F = max p(x) min{i(x;y 1), I(X;Y 2 )}. Achievability follows by the case without feedback. For the converse, consider nr = H(M) I(M;Y1 n )+nǫ n = I(M;Y 1i Y1 i 1 )+nǫ n (a) (b) = I(M,Y i 1 1 ;Y 1i )+nǫ n I(M,Y i 1 2 ;Y 1i )+nǫ n I(M,Y i 1 2,X i ;Y 1i )+nǫ n I(X i,y 1i )+nǫ n, 5
6 where(a)followssincex i isafunctionof(m,y i 1 2 )and(b)followssincei(m,y i 1 2 ;Y 1i X i ) = 0 for all i. Following the usual procedure by introducing the time-sharing random variable, we have shown that R I(X;Y 1 ). Similarly, we have R I(X;Y 2 ). Thus, R min{i(x;y 1 ), I(X;Y 2 )} for some pmf p(x) Broadcasting over a diamond network. The capacity is C = max min{i(x 1,Y 2 ), I(X 1,Y 3 ), I(X 2,X 3 ;Y 4 )}. p(x 1),p(x 2,x 3) For achievability, consider a two-hop relaying scheme in which node 1 communicates to nodes 2 and 3 using common-message broadcasting at rate R 1 < min{i(x 1 ;Y 2 ), I(X 1 ;Y 3 )} and nodes 2 and 3, in turn, communicate to node 4 using cooperative multiple access at rate R 2 < I(X 2,X 3 ;Y 4 ). The minimum of R 1 and R 2 is certainly achievable by two-hop relaying over multiple blocks. For the proof of the converse, consider the cutset bound C = max p(x 1,x 2,x 3) min S:1 S,S {1,2,3,4} I(X(S);Y(Sc ) X(S c )) with S = {1,3,4}, S = {1,2,4}, and S = {1,2,3}. The corresponding mutual information terms can be upper bounded as C I(X 1,X 3 ;Y 2 X 2 ) I(X 1,X 2,X 3 ;Y 2 ) = I(X 1 ;Y 2 ), C I(X 1,X 2 ;Y 3 X 3 ) I(X 1,X 2,X 3 ;Y 3 ) = I(X 1 ;Y 3 ), C I(X 1,X 2,X 3 ;Y 4 ) I(X 1,X 2,X 3 ;Y 4 ) = I(X 2,X 3 ;Y 4 ). Noting that these bounds depend on p(x 1,x 2,x 3 ) only through the marginals p(x 1 ) and p(x 2,x 3 ) completes the proof. 6
UCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, Solutions to Take-Home Midterm (Prepared by Pinar Sen)
UCSD ECE 255C Handout #12 Prof. Young-Han Kim Tuesday, February 28, 2017 Solutions to Take-Home Midterm (Prepared by Pinar Sen) 1. (30 points) Erasure broadcast channel. Let p(y 1,y 2 x) be a discrete
More informationEE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions
EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where
More informationSolutions to Homework Set #1 Sanov s Theorem, Rate distortion
st Semester 00/ Solutions to Homework Set # Sanov s Theorem, Rate distortion. Sanov s theorem: Prove the simple version of Sanov s theorem for the binary random variables, i.e., let X,X,...,X n be a sequence
More informationLecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
More informationEE 4TM4: Digital Communications II. Channel Capacity
EE 4TM4: Digital Communications II 1 Channel Capacity I. CHANNEL CODING THEOREM Definition 1: A rater is said to be achievable if there exists a sequence of(2 nr,n) codes such thatlim n P (n) e (C) = 0.
More informationLecture 5 Channel Coding over Continuous Channels
Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From
More informationStrong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach Silas Fong (Joint work with Vincent Tan) Department of Electrical & Computer Engineering National University
More informationSolutions to Homework Set #3 Channel and Source coding
Solutions to Homework Set #3 Channel and Source coding. Rates (a) Channels coding Rate: Assuming you are sending 4 different messages using usages of a channel. What is the rate (in bits per channel use)
More informationEE/Stats 376A: Homework 7 Solutions Due on Friday March 17, 5 pm
EE/Stats 376A: Homework 7 Solutions Due on Friday March 17, 5 pm 1. Feedback does not increase the capacity. Consider a channel with feedback. We assume that all the recieved outputs are sent back immediately
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More information18.2 Continuous Alphabet (discrete-time, memoryless) Channel
0-704: Information Processing and Learning Spring 0 Lecture 8: Gaussian channel, Parallel channels and Rate-distortion theory Lecturer: Aarti Singh Scribe: Danai Koutra Disclaimer: These notes have not
More informationThe Gallager Converse
The Gallager Converse Abbas El Gamal Director, Information Systems Laboratory Department of Electrical Engineering Stanford University Gallager s 75th Birthday 1 Information Theoretic Limits Establishing
More informationLecture 5: Channel Capacity. Copyright G. Caire (Sample Lectures) 122
Lecture 5: Channel Capacity Copyright G. Caire (Sample Lectures) 122 M Definitions and Problem Setup 2 X n Y n Encoder p(y x) Decoder ˆM Message Channel Estimate Definition 11. Discrete Memoryless Channel
More informationSolutions to Homework Set #2 Broadcast channel, degraded message set, Csiszar Sum Equality
1st Semester 2010/11 Solutions to Homework Set #2 Broadcast channel, degraded message set, Csiszar Sum Equality 1. Convexity of capacity region of broadcast channel. Let C R 2 be the capacity region of
More informationCapacity Theorems for Relay Channels
Capacity Theorems for Relay Channels Abbas El Gamal Department of Electrical Engineering Stanford University April, 2006 MSRI-06 Relay Channel Discrete-memoryless relay channel [vm 7] Relay Encoder Y n
More informationRelay Networks With Delays
Relay Networks With Delays Abbas El Gamal, Navid Hassanpour, and James Mammen Department of Electrical Engineering Stanford University, Stanford, CA 94305-9510 Email: {abbas, navid, jmammen}@stanford.edu
More informationA Comparison of Superposition Coding Schemes
A Comparison of Superposition Coding Schemes Lele Wang, Eren Şaşoğlu, Bernd Bandemer, and Young-Han Kim Department of Electrical and Computer Engineering University of California, San Diego La Jolla, CA
More informationUCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, Homework Set #6 Due: Thursday, May 22, 2011
UCSD ECE153 Handout #30 Prof. Young-Han Kim Thursday, May 15, 2014 Homework Set #6 Due: Thursday, May 22, 2011 1. Linear estimator. Consider a channel with the observation Y = XZ, where the signal X and
More informationDistributed Lossless Compression. Distributed lossless compression system
Lecture #3 Distributed Lossless Compression (Reading: NIT 10.1 10.5, 4.4) Distributed lossless source coding Lossless source coding via random binning Time Sharing Achievability proof of the Slepian Wolf
More informationInteractive Hypothesis Testing with Communication Constraints
Fiftieth Annual Allerton Conference Allerton House, UIUC, Illinois, USA October - 5, 22 Interactive Hypothesis Testing with Communication Constraints Yu Xiang and Young-Han Kim Department of Electrical
More informationLecture 4 Noisy Channel Coding
Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem
More informationECE Information theory Final
ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the
More informationThe Role of Directed Information in Network Capacity
The Role of Directed Information in Network Capacity Sudeep Kamath 1 and Young-Han Kim 2 1 Department of Electrical Engineering Princeton University 2 Department of Electrical and Computer Engineering
More informationECE598: Information-theoretic methods in high-dimensional statistics Spring 2016
ECE598: Information-theoretic methods in high-dimensional statistics Spring 06 Lecture : Mutual Information Method Lecturer: Yihong Wu Scribe: Jaeho Lee, Mar, 06 Ed. Mar 9 Quick review: Assouad s lemma
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationSolutions to Homework Set #4 Differential Entropy and Gaussian Channel
Solutions to Homework Set #4 Differential Entropy and Gaussian Channel 1. Differential entropy. Evaluate the differential entropy h(x = f lnf for the following: (a Find the entropy of the exponential density
More informationEE376A - Information Theory Final, Monday March 14th 2016 Solutions. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Final, Monday March 14th 216 Solutions Instructions: You have three hours, 3.3PM - 6.3PM The exam has 4 questions, totaling 12 points. Please start answering each question on
More informationLecture 1. Introduction
Lecture 1. Introduction What is the course about? Logistics Questionnaire Dr. Yao Xie, ECE587, Information Theory, Duke University What is information? Dr. Yao Xie, ECE587, Information Theory, Duke University
More informationECE Information theory Final (Fall 2008)
ECE 776 - Information theory Final (Fall 2008) Q.1. (1 point) Consider the following bursty transmission scheme for a Gaussian channel with noise power N and average power constraint P (i.e., 1/n X n i=1
More informationLecture 14 February 28
EE/Stats 376A: Information Theory Winter 07 Lecture 4 February 8 Lecturer: David Tse Scribe: Sagnik M, Vivek B 4 Outline Gaussian channel and capacity Information measures for continuous random variables
More informationLecture 2. Capacity of the Gaussian channel
Spring, 207 5237S, Wireless Communications II 2. Lecture 2 Capacity of the Gaussian channel Review on basic concepts in inf. theory ( Cover&Thomas: Elements of Inf. Theory, Tse&Viswanath: Appendix B) AWGN
More informationCh. 8 Math Preliminaries for Lossy Coding. 8.4 Info Theory Revisited
Ch. 8 Math Preliminaries for Lossy Coding 8.4 Info Theory Revisited 1 Info Theory Goals for Lossy Coding Again just as for the lossless case Info Theory provides: Basis for Algorithms & Bounds on Performance
More informationExercises with solutions (Set D)
Exercises with solutions Set D. A fair die is rolled at the same time as a fair coin is tossed. Let A be the number on the upper surface of the die and let B describe the outcome of the coin toss, where
More informationKatalin Marton. Abbas El Gamal. Stanford University. Withits A. El Gamal (Stanford University) Katalin Marton Withits / 9
Katalin Marton Abbas El Gamal Stanford University Withits 2010 A. El Gamal (Stanford University) Katalin Marton Withits 2010 1 / 9 Brief Bio Born in 1941, Budapest Hungary PhD from Eötvös Loránd University
More informationarxiv: v2 [cs.it] 28 May 2017
Feedback and Partial Message Side-Information on the Semideterministic Broadcast Channel Annina Bracher and Michèle Wigger arxiv:1508.01880v2 [cs.it] 28 May 2017 May 30, 2017 Abstract The capacity of the
More informationEE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018
Please submit the solutions on Gradescope. EE376A: Homework #3 Due by 11:59pm Saturday, February 10th, 2018 1. Optimal codeword lengths. Although the codeword lengths of an optimal variable length code
More informationUCSD ECE250 Handout #27 Prof. Young-Han Kim Friday, June 8, Practice Final Examination (Winter 2017)
UCSD ECE250 Handout #27 Prof. Young-Han Kim Friday, June 8, 208 Practice Final Examination (Winter 207) There are 6 problems, each problem with multiple parts. Your answer should be as clear and readable
More informationPerformance-based Security for Encoding of Information Signals. FA ( ) Paul Cuff (Princeton University)
Performance-based Security for Encoding of Information Signals FA9550-15-1-0180 (2015-2018) Paul Cuff (Princeton University) Contributors Two students finished PhD Tiance Wang (Goldman Sachs) Eva Song
More informationInformation Theoretic Limits of Randomness Generation
Information Theoretic Limits of Randomness Generation Abbas El Gamal Stanford University Shannon Centennial, University of Michigan, September 2016 Information theory The fundamental problem of communication
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationUCSD ECE250 Handout #20 Prof. Young-Han Kim Monday, February 26, Solutions to Exercise Set #7
UCSD ECE50 Handout #0 Prof. Young-Han Kim Monday, February 6, 07 Solutions to Exercise Set #7. Minimum waiting time. Let X,X,... be i.i.d. exponentially distributed random variables with parameter λ, i.e.,
More informationCapacity bounds for multiple access-cognitive interference channel
Mirmohseni et al. EURASIP Journal on Wireless Communications and Networking, :5 http://jwcn.eurasipjournals.com/content///5 RESEARCH Open Access Capacity bounds for multiple access-cognitive interference
More informationLecture 6: Gaussian Channels. Copyright G. Caire (Sample Lectures) 157
Lecture 6: Gaussian Channels Copyright G. Caire (Sample Lectures) 157 Differential entropy (1) Definition 18. The (joint) differential entropy of a continuous random vector X n p X n(x) over R is: Z h(x
More informationAdvanced Topics in Information Theory
Advanced Topics in Information Theory Lecture Notes Stefan M. Moser c Copyright Stefan M. Moser Signal and Information Processing Lab ETH Zürich Zurich, Switzerland Institute of Communications Engineering
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More informationLecture 10: Broadcast Channel and Superposition Coding
Lecture 10: Broadcast Channel and Superposition Coding Scribed by: Zhe Yao 1 Broadcast channel M 0M 1M P{y 1 y x} M M 01 1 M M 0 The capacity of the broadcast channel depends only on the marginal conditional
More informationCan Feedback Increase the Capacity of the Energy Harvesting Channel?
Can Feedback Increase the Capacity of the Energy Harvesting Channel? Dor Shaviv EE Dept., Stanford University shaviv@stanford.edu Ayfer Özgür EE Dept., Stanford University aozgur@stanford.edu Haim Permuter
More informationLecture 15: Conditional and Joint Typicaility
EE376A Information Theory Lecture 1-02/26/2015 Lecture 15: Conditional and Joint Typicaility Lecturer: Kartik Venkat Scribe: Max Zimet, Brian Wai, Sepehr Nezami 1 Notation We always write a sequence of
More informationX 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
More informationInformation Theory for Wireless Communications. Lecture 10 Discrete Memoryless Multiple Access Channel (DM-MAC): The Converse Theorem
Information Theory for Wireless Communications. Lecture 0 Discrete Memoryless Multiple Access Channel (DM-MAC: The Converse Theorem Instructor: Dr. Saif Khan Mohammed Scribe: Antonios Pitarokoilis I. THE
More informationAn Outer Bound for the Gaussian. Interference channel with a relay.
An Outer Bound for the Gaussian Interference Channel with a Relay Ivana Marić Stanford University Stanford, CA ivanam@wsl.stanford.edu Ron Dabora Ben-Gurion University Be er-sheva, Israel ron@ee.bgu.ac.il
More informationRemote Source Coding with Two-Sided Information
Remote Source Coding with Two-Sided Information Basak Guler Ebrahim MolavianJazi Aylin Yener Wireless Communications and Networking Laboratory Department of Electrical Engineering The Pennsylvania State
More informationLecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
More informationSOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003
SOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003 SLEPIAN-WOLF RESULT { X i} RATE R x ENCODER 1 DECODER X i V i {, } { V i} ENCODER 0 RATE R v Problem: Determine R, the
More informationOn Capacity Under Received-Signal Constraints
On Capacity Under Received-Signal Constraints Michael Gastpar Dept. of EECS, University of California, Berkeley, CA 9470-770 gastpar@berkeley.edu Abstract In a world where different systems have to share
More information( 1 k "information" I(X;Y) given by Y about X)
SUMMARY OF SHANNON DISTORTION-RATE THEORY Consider a stationary source X with f (x) as its th-order pdf. Recall the following OPTA function definitions: δ(,r) = least dist'n of -dim'l fixed-rate VQ's w.
More informationLecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
More informationFeedback Capacity of a Class of Symmetric Finite-State Markov Channels
Feedback Capacity of a Class of Symmetric Finite-State Markov Channels Nevroz Şen, Fady Alajaji and Serdar Yüksel Department of Mathematics and Statistics Queen s University Kingston, ON K7L 3N6, Canada
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More informationSuperposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels
Superposition Encoding and Partial Decoding Is Optimal for a Class of Z-interference Channels Nan Liu and Andrea Goldsmith Department of Electrical Engineering Stanford University, Stanford CA 94305 Email:
More informationGeneralized Writing on Dirty Paper
Generalized Writing on Dirty Paper Aaron S. Cohen acohen@mit.edu MIT, 36-689 77 Massachusetts Ave. Cambridge, MA 02139-4307 Amos Lapidoth lapidoth@isi.ee.ethz.ch ETF E107 ETH-Zentrum CH-8092 Zürich, Switzerland
More informationSimultaneous Nonunique Decoding Is Rate-Optimal
Fiftieth Annual Allerton Conference Allerton House, UIUC, Illinois, USA October 1-5, 2012 Simultaneous Nonunique Decoding Is Rate-Optimal Bernd Bandemer University of California, San Diego La Jolla, CA
More informationFeedback Capacity of the Gaussian Interference Channel to Within Bits: the Symmetric Case
1 arxiv:0901.3580v1 [cs.it] 23 Jan 2009 Feedback Capacity of the Gaussian Interference Channel to Within 1.7075 Bits: the Symmetric Case Changho Suh and David Tse Wireless Foundations in the Department
More informationExample: Letter Frequencies
Example: Letter Frequencies i a i p i 1 a 0.0575 2 b 0.0128 3 c 0.0263 4 d 0.0285 5 e 0.0913 6 f 0.0173 7 g 0.0133 8 h 0.0313 9 i 0.0599 10 j 0.0006 11 k 0.0084 12 l 0.0335 13 m 0.0235 14 n 0.0596 15 o
More information3F1: Signals and Systems INFORMATION THEORY Examples Paper Solutions
Engineering Tripos Part IIA THIRD YEAR 3F: Signals and Systems INFORMATION THEORY Examples Paper Solutions. Let the joint probability mass function of two binary random variables X and Y be given in the
More informationExample: Letter Frequencies
Example: Letter Frequencies i a i p i 1 a 0.0575 2 b 0.0128 3 c 0.0263 4 d 0.0285 5 e 0.0913 6 f 0.0173 7 g 0.0133 8 h 0.0313 9 i 0.0599 10 j 0.0006 11 k 0.0084 12 l 0.0335 13 m 0.0235 14 n 0.0596 15 o
More informationChapter 8: Differential entropy. University of Illinois at Chicago ECE 534, Natasha Devroye
Chapter 8: Differential entropy Chapter 8 outline Motivation Definitions Relation to discrete entropy Joint and conditional differential entropy Relative entropy and mutual information Properties AEP for
More informationInformation Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
More informationAnalyzing Large Communication Networks
Analyzing Large Communication Networks Shirin Jalali joint work with Michelle Effros and Tracey Ho Dec. 2015 1 The gap Fundamental questions: i. What is the best achievable performance? ii. How to communicate
More informationLecture 17: Differential Entropy
Lecture 17: Differential Entropy Differential entropy AEP for differential entropy Quantization Maximum differential entropy Estimation counterpart of Fano s inequality Dr. Yao Xie, ECE587, Information
More informationLECTURE 3. Last time:
LECTURE 3 Last time: Mutual Information. Convexity and concavity Jensen s inequality Information Inequality Data processing theorem Fano s Inequality Lecture outline Stochastic processes, Entropy rate
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationSolutions to Set #2 Data Compression, Huffman code and AEP
Solutions to Set #2 Data Compression, Huffman code and AEP. Huffman coding. Consider the random variable ( ) x x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0. 0.04 0.04 0.03 0.02 (a) Find a binary Huffman code
More informationProblemsWeCanSolveWithaHelper
ITW 2009, Volos, Greece, June 10-12, 2009 ProblemsWeCanSolveWitha Haim Permuter Ben-Gurion University of the Negev haimp@bgu.ac.il Yossef Steinberg Technion - IIT ysteinbe@ee.technion.ac.il Tsachy Weissman
More informationBounds on Capacity and Minimum Energy-Per-Bit for AWGN Relay Channels
Bounds on Capacity and Minimum Energy-Per-Bit for AWG Relay Channels Abbas El Gamal, Mehdi Mohseni and Sina Zahedi Information Systems Lab Department of Electrical Engineering Stanford University, Stanford,
More informationOn the Capacity of Interference Channels with Degraded Message sets
On the Capacity of Interference Channels with Degraded Message sets Wei Wu, Sriram Vishwanath and Ari Arapostathis arxiv:cs/060507v [cs.it] 7 May 006 Abstract This paper is motivated by a sensor network
More informationChapter I: Fundamental Information Theory
ECE-S622/T62 Notes Chapter I: Fundamental Information Theory Ruifeng Zhang Dept. of Electrical & Computer Eng. Drexel University. Information Source Information is the outcome of some physical processes.
More informationUCSD ECE153 Handout #34 Prof. Young-Han Kim Tuesday, May 27, Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #34 Prof Young-Han Kim Tuesday, May 7, 04 Solutions to Homework Set #6 (Prepared by TA Fatemeh Arbabjolfaei) Linear estimator Consider a channel with the observation Y XZ, where the
More informationLECTURE 2. Convexity and related notions. Last time: mutual information: definitions and properties. Lecture outline
LECTURE 2 Convexity and related notions Last time: Goals and mechanics of the class notation entropy: definitions and properties mutual information: definitions and properties Lecture outline Convexity
More informationInformation Theory. Lecture 10. Network Information Theory (CT15); a focus on channel capacity results
Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity results The (two-user) multiple access channel (15.3) The (two-user) broadcast channel (15.6) The relay channel
More informationNational University of Singapore Department of Electrical & Computer Engineering. Examination for
National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed:
More informationAn Achievable Rate Region for the 3-User-Pair Deterministic Interference Channel
Forty-Ninth Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 8-3, An Achievable Rate Region for the 3-User-Pair Deterministic Interference Channel Invited Paper Bernd Bandemer and
More informationThe binary entropy function
ECE 7680 Lecture 2 Definitions and Basic Facts Objective: To learn a bunch of definitions about entropy and information measures that will be useful through the quarter, and to present some simple but
More informationSource Coding with Lists and Rényi Entropy or The Honey-Do Problem
Source Coding with Lists and Rényi Entropy or The Honey-Do Problem Amos Lapidoth ETH Zurich October 8, 2013 Joint work with Christoph Bunte. A Task from your Spouse Using a fixed number of bits, your spouse
More informationUCSD ECE 153 Handout #20 Prof. Young-Han Kim Thursday, April 24, Solutions to Homework Set #3 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE 53 Handout #0 Prof. Young-Han Kim Thursday, April 4, 04 Solutions to Homework Set #3 (Prepared by TA Fatemeh Arbabjolfaei). Time until the n-th arrival. Let the random variable N(t) be the number
More informationEE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15
EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability
More informationJoint Write-Once-Memory and Error-Control Codes
1 Joint Write-Once-Memory and Error-Control Codes Xudong Ma Pattern Technology Lab LLC, U.S.A. Email: xma@ieee.org arxiv:1411.4617v1 [cs.it] 17 ov 2014 Abstract Write-Once-Memory (WOM) is a model for many
More informationChapter 2: Entropy and Mutual Information. University of Illinois at Chicago ECE 534, Natasha Devroye
Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality
More informationUCSD ECE153 Handout #27 Prof. Young-Han Kim Tuesday, May 6, Solutions to Homework Set #5 (Prepared by TA Fatemeh Arbabjolfaei)
UCSD ECE53 Handout #7 Prof. Young-Han Kim Tuesday, May 6, 4 Solutions to Homework Set #5 (Prepared by TA Fatemeh Arbabjolfaei). Neural net. Let Y = X + Z, where the signal X U[,] and noise Z N(,) are independent.
More informationOn the Duality between Multiple-Access Codes and Computation Codes
On the Duality between Multiple-Access Codes and Computation Codes Jingge Zhu University of California, Berkeley jingge.zhu@berkeley.edu Sung Hoon Lim KIOST shlim@kiost.ac.kr Michael Gastpar EPFL michael.gastpar@epfl.ch
More informationThe Capacity of the Semi-Deterministic Cognitive Interference Channel and its Application to Constant Gap Results for the Gaussian Channel
The Capacity of the Semi-Deterministic Cognitive Interference Channel and its Application to Constant Gap Results for the Gaussian Channel Stefano Rini, Daniela Tuninetti, and Natasha Devroye Department
More informationCapacity of a Class of Semi-Deterministic Primitive Relay Channels
Capacity of a Class of Semi-Deterministic Primitive Relay Channels Ravi Tandon Sennur Ulukus Department of Electrical and Computer Engineering University of Maryland, College Park, MD 2742 ravit@umd.edu
More informationConcentration Inequalities
Chapter Concentration Inequalities I. Moment generating functions, the Chernoff method, and sub-gaussian and sub-exponential random variables a. Goal for this section: given a random variable X, how does
More informationInformation Theory Primer:
Information Theory Primer: Entropy, KL Divergence, Mutual Information, Jensen s inequality Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More informationInformation Theory and Communication
Information Theory and Communication Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/8 General Chain Rules Definition Conditional mutual information
More informationOn Common Information and the Encoding of Sources that are Not Successively Refinable
On Common Information and the Encoding of Sources that are Not Successively Refinable Kumar Viswanatha, Emrah Akyol, Tejaswi Nanjundaswamy and Kenneth Rose ECE Department, University of California - Santa
More informationEE376A: Homeworks #4 Solutions Due on Thursday, February 22, 2018 Please submit on Gradescope. Start every question on a new page.
EE376A: Homeworks #4 Solutions Due on Thursday, February 22, 28 Please submit on Gradescope. Start every question on a new page.. Maximum Differential Entropy (a) Show that among all distributions supported
More informationSource-channel coding for coordination over a noisy two-node network
New Jersey Institute of Technology Digital Commons @ NJIT Theses Theses and Dissertations Fall 2016 Source-channel coding for coordination over a noisy two-node network Sarah A. Obead New Jersey Institute
More informationDraft. On Limiting Expressions for the Capacity Region of Gaussian Interference Channels. Mojtaba Vaezi and H. Vincent Poor
Preliminaries Counterexample Better Use On Limiting Expressions for the Capacity Region of Gaussian Interference Channels Mojtaba Vaezi and H. Vincent Poor Department of Electrical Engineering Princeton
More informationCapacity of a Class of Cognitive Radio Channels: Interference Channels with Degraded Message Sets
Capacity of a Class of Cognitive Radio Channels: Interference Channels with Degraded Message Sets Wei Wu, Sriram Vishwanath and Ari Arapostathis Abstract This paper is motivated by two different scenarios.
More information