# Lecture 11: Quantum Information III - Source Coding

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that generates symbol x with probability p x, her goal is to send the information to Bob. Classically, what Alice can do is to encode the symbol x as a codeword of certain length, namely C x {0, } n, then sends it to Bob and finally Bob decodes it to figure out what x is. Let Y be the symbol Bob gets after decoding, then I(X; Y ) essentially measures how much information of X Bob can extract. It is clear that I(X; Y ) I(C X ; Y ) = H(C X ) H(C X Y ) where the first inequality follows by the data processing inequality. It means that if we want Y to contain many information about X, then we need a long codeword. But what if we have a quantum channel that allows us to transmit qubits? In this case, Alice can actually encode x using qubits rather than classical bits, and the codeword she can send is now a quantum state of dimension 2 n where n is the number of qubits. After receiving the quantum state, Bob can perform measurement on the state to get the variable Y and figure out what X. It is natural to ask whether we can encode X using a small number of qubits? Again we are interested in the quantity I(X; Y ). From the above discussion, we know that I(X; Y ) is upper bounded by the number of different states Alice can send. And the issue about classical bit is that a single bit can only represent 2 different states. However, a qubit is much more powerful since it can represent infinitely many different states! It seems that we can do much better in the quantum case. But after a moment of thought, we will realize that encoding X using one qubit does not solve the problem since Bob cannot distinguish the states perfectly unless the states are orthogonal! In general, the less qubits Alice uses, the more difficult Bob can distinguish them and hence extracting the information about X. So, there is a trade-off between the number qubit Alice use and how well Bob can distinguish the states. And how I(X; Y ) is related to the quantum states is not clear. It turns out that one can upper bound I(X; Y ) in terms of the von Neumann entropies of the some quantum states. Theorem. (Holevo s bound). Let X be a random variable such that Pr[X = x] = p x. Suppose Alice sends a state ρ x to Bob if X = x, then for any measurement described by POVM elements {E y } on the state Bob receive and let Y be the measurement outcome, I(X; Y ) S(ρ) x p x S(ρ x ). This theorem tells us what we can hope for if we want Y to contain much information about X. Indeed, since S(ρ) is at most the logarithm of the dimension of the quantum state, which is nothing but the number of qubits sent, it is now clear that quantum does not buy us much. Proof. The idea is to view everything as a quantum system consisting of three parts P, Q and M, where Q represents the quantum system ρ x Alice sends to Bob, and P, M represent the classical information X and Y. Then the result will follow by applying suitable inequalities for the von Neumann entropy. Formally, we consider the quantum state ρ P QM = x p x x x ρ x 0 0. }{{}}{{}}{{} P Q M The joint system P Q corresponds to the situation that Alice prepares the state ρ x with probability p x. Then Bob performs a measurement with POVM elements {E y } on the system Q and stores the measurement outcome into M without touching P. After this step the state would become ρ P Q M = p x x x Ey ρ x Ey y y. }{{}}{{}}{{} x P y Q M L-

2 Now consider the quantity S(P ; Q). It is clear that S(P ; Q) = S(P ; Q, M) since M is independent of P and Q. We also have S(P ; Q, M) S(P ; Q, M ) since the measurement only apply to the system QM, it does not increase the mutual information between P and QM. Finally observe that S(P ; Q, M ) S(P ; M ) since discarding system does not increase mutual information. So we have the inequality S(P ; Q) S(P ; M ). We claim that S(P ; M ) = I(X; Y ) and S(P ; Q) = S(ρ) x p xs(ρ x ). The first one follows by the fact ρ P M = x p x x x y p y x y y = x,y p x,y x x y y. For the second one, by definition we have S(P ; Q) = S(ρ P ) + S(ρ Q ) S(ρ P Q ). Observe that S(ρ P ) = H(X), S(ρ Q ) = S(ρ) and by the inequality for von Neumann entropy, we have S(ρ P Q ) = H(X) + x p xs(ρ x ). Putting all together yields the equality S(P ; Q) = S(ρ) x p xs(ρ x ) and hence complete the proof..2 Application: quantum random access codes The Holevo s bound says that sending n bits of classical information requires n qubits. However, in some cases we are not very interested in knowing every bit of X (for instance when we are querying some entries in a database). Note that Holeve s bound does not say we need many qubits as I(X; Y i ) is small (which is at most ) for each fixed i. Classically, it is known that the codeword must have at least ( H(ε))m bits where m is the length of the original message and ε is the success probability (this bound is almost tight as there exists classical random access code of length ( H(ε))m + O(log m)). Can we do better using quantum mechanics? It turns out that the answer is no. Nayak gave a very simple proof using Holevo s bound. First let us define what a quantum random access code is. Definition.2 (Quantum random access code). Let m, n be positive integers, ε > 0, a (m, n, ε)- quantum random access code is a function C that maps a m-bit string to a quantum state of dimension 2 n such that for each i m, there exists a measurement {Mi 0, M i } such that the probability that the measurement outcome is b is at least ε when the i-th bit of the string is b. Theorem.3 (Nayak s bound). Let ε [/2, ], if (m, n, ε)-quantum random access code exists, then n ( H(ε))m. Proof. We use the following lemma, which follows from Holevo s bound and Fano s inequality: Lemma.4. Let ρ = 2 (ρ 0 + ρ ), if there exists measurement {M 0, M } such that Tr(M b ρ b ) ε [/2, ], then S(ρ) ( H(ε)) + 2( S(ρ0 ) + S(ρ ) ). Now, suppose we have a (m, n, ε)-quantum random access code. Let ρ x be the encoding of x and ρ = x 2 ρ m x = 2 (ρ 0 + ρ ) where ρ b = x:x =b 2 ρ m x. By the definition of quantum random access code, there exists measurement {M 0, M } such that for every ρ x with x = b, we have Tr(M b ρ x ) ε. It follows by the linearity of trace that Tr(M b ρ b ) ε. So by Lemma.4, we have S(ρ) ( H(ε)) + 2( S(ρ0 ) + S(ρ ) ). And it is not difficult to see ρ b still satisfy the condition of Lemma.4 and we have S(ρ) 2( H(ε)) + 4( S(ρ00 ) + S(ρ 0 ) + S(ρ 0 ) + S(ρ ) ), and by induction we have S(ρ) m( H(ε)) + 2 m b S(ρ b) m( H(ε)), as desired. L-2

3 Proof of Lemma.4. Let X be a uniform random bit, and let Y be the measurement outcome after applying the measurement {M 0, M } to the state ρ = 2 (ρ 0 + ρ ). Then by Holevo s bound, we have S(ρ) I(X; Y ) + 2( S(ρ0 ) + S(ρ ) ). Since we have Tr(M b ρ b ) ε, Pr[X = Y ] ε and hence I(X; Y ) = H(X) H(X Y ) = H(X Y ). By Fano s inequality, let p = Pr[X Y ], we have H(X Y ) H(p) + p log 2 ( X ) H(ε) since X = 2 and p ε /2. It follows that I(X; Y ) H(ε) and the proof is completed. For completeness, we also give a proof of Fano s inequality. The following proof is quite standard that can be found in many textbooks on information theory such as [?]. Lemma.5 (Fano s inequality). Let X, Y be random variables with support X, if Pr[X Y ] = p, then H(X Y ) H(p) + p log 2 ( X ). Proof. Let Z be a random variable that Z = {, if X = Y, 0, if X Y. Now, let us consider H(X Y ). Since H(Z X, Y ) = 0, we have H(X Y ) = H(X, Z Y ) = H(Z Y ) + H(X Y, Z) H(Z) + H(X Y, Z) = H(p) + ph(x Y, Z = 0) + ( p)h(x Y, Z = ) = H(p) + ph(x Y, Z = 0) H(p) + p log 2 ( X ) where the first inequality follows since conditioning does not increase entropy, and the second inequality follows by the fact that if X Y then the number of possible outcomes of X is at most X..3 Source coding theorems.3. Classical source coding theorem Suppose Alice have an information source (a random variable) X that generates symbol x with probability p x. For the purpose of efficient storage or transmission, she would want to encode the information using some short codes. So that when she sends the codeword to Bob, Bob can decode it and extract the information back. Ideally, Alice would want to have such a encoding and decoding scheme so that the probability that Bob can get back the symbol x is arbitrarily close to. Intuitively, H(X) bits seem to be sufficient since the entropy of X is just H(X). However, for any fix X, it is impossible to achieve arbitrarily small error probability if the codeword is shorter than log 2 X. Nevertheless, if we consider the asymptotic version of this problem, i.e. Alice has n independent copies of X, namely X n, whose generates symbol x n = x... x n with probability n i= p x i, and she want to assign a codeword for each x n so that Bob can decode the codeword to get back x n with probability tends to as n. It turns out that we can achieve it with rate (defined by the length of the codeword divided by n) arbitrarily close to H(X). And moreover, it is also necessary. Theorem.6 (Shannon s source coding theorem). Let X n be n i.i.d. copies of X. Then the following holds: L-3

4 . If R > H(X), there exist C n : X n {0, } nr and D n : {0, } nr X n such that lim n Pr x n X n[d n(c n (x n )) = x n ] =. 2. If R < H(X), for any C n : X n {0, } nr and D n : {0, } nr X n, we have lim n Pr x n X n[d n(c n (x n )) = x n ] <. The proof of this theorem uses a notion called typicality. The idea is that, if we look at the sequence x n = x... x n generated by X n, since each x i is an independent copy of X, with very high probability (when n sufficiently large) the number of i such that x i = x for each x X is roughly p x. Sequences with such property is said to be typical. The main observation is that actually the set of typical sequence is roughly 2 nh(x). Thus we can simply encode all the typical sequences into distinct codewords, which takes around nh(x) bits, and encode all the other sequences into arbitrary codewords. Then the decoder can successfully decode the message if it is typical, which happens with very high probability. Lemma.7 (Typical sequence theorem). Let X n be i.i.d. copies of X with support X. Let ε, δ > 0, and Tε n = {x n X n : 2 n(h(x)+ε) p(x n ) 2 n(h(x) ε) }, then for n sufficiently large, the following holds:. Pr[X n T n ε ] δ, () 2. ( δ)2 H(X) ε) T n ε 2 n(h(x)+ε) (2) Proof. For the first item, by the definition of T n ε, we have Pr[X n Tε n ] = Pr[2 n(h(x) ε) p(x n ) 2 n(h(x)+ε) ] [ n ] = Pr log n p(x i= i ) H(X) ε [ n [ = Pr log n p(x i ) E ] ] log ε p(x) δ i= for sufficiently large n, where the inequality follows by the weak law of large number and the fact that log p(x i ) are i.i.d. For the second item, by (), we have δ Pr[X n T n ε ], which implies δ p(x n ). Now, it is immediate from the definition of T n ε x n T n ε that δ T n ε 2 n(h(x) ε) and as desired. T n ε 2 n(h(x)+ε), L-4

5 Proof of Theorem.6. For the first item, let ε > 0 such that R ε > H(X), the C n simply assign each sequence x Tε n a distinct nr bit-string other sequences an arbitrary nr bit-string. The decoder output the typical sequence in Cn (y n ). If x is typical, there is no error. So, the error probability is at most the probability that x / Tε n, which can be made to be arbitrary small by (). For the second item, let A n = {x n X n : D n (C n (x n )) = x n }. It is clear that A n 2 nr since A n D n ({0, } nr ). It suffices to show Pr[X n A n ] is small. Now, take ε > 0 such that R + ε < H(X). Consider Pr[X n A n ] = Pr[X n A n Tε n ]+Pr[X n A n \Tε n ]. Take δ > 0 small, then for n sufficiently large, the second term is at most δ by (), and the first term can be upper bounded by Pr[X n A n T n ε ] A n 2 n(h(x) ε) 2 n(h(x) R ε) which tends to 0 as n. It follows that the error probability does not tends to as n, and the proof is completed..3.2 Quantum source coding theorem In the quantum case, Alice are given a state ρ n, which is a product of n copies of ρ. And her goal is to encode it using as less qubits as possible while at the same Bob are able to get back a state which is close to ρ after performing measurement. Here the closeness is measured by fidelity. We have the following theorem: Theorem.8 (Schumacher s quantum source coding theorem). Let ρ H be an i.i.d. quantum source. Then the following holds:. If R > S(ρ), there exist C n : H n C 2nR and D n : C 2nR H n such that lim F n (ρ n, D n (C n (ρ n )) = 2. If R < S(ρ), for any C n : H n C 2nR and D n : C 2nR H n, we have lim F n (ρ n, D n (C n (ρ n )) < Its proof is similar to the proof of Theorem.6. The main difference is that we use the typical subspace theorem, which is an analogue of the typical sequence theorem. L-5

### Lecture 18: Quantum Information Theory and Holevo s Bound

Quantum Computation (CMU 1-59BB, Fall 2015) Lecture 1: Quantum Information Theory and Holevo s Bound November 10, 2015 Lecturer: John Wright Scribe: Nicolas Resch 1 Question In today s lecture, we will

### Entropies & Information Theory

Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information

### LECTURE 13. Last time: Lecture outline

LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to

### The Communication Complexity of Correlation. Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan

The Communication Complexity of Correlation Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan Transmitting Correlated Variables (X, Y) pair of correlated random variables Transmitting

### Entanglement Manipulation

Entanglement Manipulation Steven T. Flammia 1 1 Perimeter Institute for Theoretical Physics, Waterloo, Ontario, N2L 2Y5 Canada (Dated: 22 March 2010) These are notes for my RIT tutorial lecture at the

### Quantum Information Theory

Chapter 5 Quantum Information Theory Quantum information theory is a rich subject that could easily have occupied us all term. But because we are short of time (I m anxious to move on to quantum computation),

### EE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions

EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where

### PERFECTLY secure key agreement has been studied recently

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 45, NO. 2, MARCH 1999 499 Unconditionally Secure Key Agreement the Intrinsic Conditional Information Ueli M. Maurer, Senior Member, IEEE, Stefan Wolf Abstract

### Lecture 3: Channel Capacity

Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.

### (Classical) Information Theory II: Source coding

(Classical) Information Theory II: Source coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract The information content of a random variable

### EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15

EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability

### An introduction to basic information theory. Hampus Wessman

An introduction to basic information theory Hampus Wessman Abstract We give a short and simple introduction to basic information theory, by stripping away all the non-essentials. Theoretical bounds on

### Chapter 14: Quantum Information Theory and Photonic Communications

Chapter 14: Quantum Information Theory and Photonic Communications Farhan Rana (MIT) March, 2002 Abstract The fundamental physical limits of optical communications are discussed in the light of the recent

### EE5319R: Problem Set 3 Assigned: 24/08/16, Due: 31/08/16

EE539R: Problem Set 3 Assigned: 24/08/6, Due: 3/08/6. Cover and Thomas: Problem 2.30 (Maimum Entropy): Solution: We are required to maimize H(P X ) over all distributions P X on the non-negative integers

### Indeterminate-length quantum coding

Indeterminate-length quantum coding arxiv:quant-ph/0011014v1 2 Nov 2000 Benjamin Schumacher (1) and Michael D. Westmoreland (2) February 1, 2008 (1) Department of Physics, Kenyon College, Gambier, OH 43022

### Quantum Teleportation Pt. 1

Quantum Teleportation Pt. 1 PHYS 500 - Southern Illinois University April 17, 2018 PHYS 500 - Southern Illinois University Quantum Teleportation Pt. 1 April 17, 2018 1 / 13 Types of Communication In the

### The Adaptive Classical Capacity of a Quantum Channel,

The Adaptive Classical Capacity of a Quantum Channel, or Information Capacities of Three Symmetric Pure States in Three Dimensions Peter W. Shor 1 AT&T Labs Research Florham Park, NJ 07932 02139 1 Current

### Quantum Information Chapter 10. Quantum Shannon Theory

Quantum Information Chapter 10. Quantum Shannon Theory John Preskill Institute for Quantum Information and Matter California Institute of Technology Updated January 2018 For further updates and additional

### Information Theory. Lecture 10. Network Information Theory (CT15); a focus on channel capacity results

Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity results The (two-user) multiple access channel (15.3) The (two-user) broadcast channel (15.6) The relay channel

### Chapter 3 Source Coding. 3.1 An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code

Chapter 3 Source Coding 3. An Introduction to Source Coding 3.2 Optimal Source Codes 3.3 Shannon-Fano Code 3.4 Huffman Code 3. An Introduction to Source Coding Entropy (in bits per symbol) implies in average

### A Holevo-type bound for a Hilbert Schmidt distance measure

Journal of Quantum Information Science, 205, *,** Published Online **** 204 in SciRes. http://www.scirp.org/journal/**** http://dx.doi.org/0.4236/****.204.***** A Holevo-type bound for a Hilbert Schmidt

### Quantum Error Correcting Codes and Quantum Cryptography. Peter Shor M.I.T. Cambridge, MA 02139

Quantum Error Correcting Codes and Quantum Cryptography Peter Shor M.I.T. Cambridge, MA 02139 1 We start out with two processes which are fundamentally quantum: superdense coding and teleportation. Superdense

### Homework Set #2 Data Compression, Huffman code and AEP

Homework Set #2 Data Compression, Huffman code and AEP 1. Huffman coding. Consider the random variable ( x1 x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a Find a binary Huffman code

### FRAMES IN QUANTUM AND CLASSICAL INFORMATION THEORY

FRAMES IN QUANTUM AND CLASSICAL INFORMATION THEORY Emina Soljanin Mathematical Sciences Research Center, Bell Labs April 16, 23 A FRAME 1 A sequence {x i } of vectors in a Hilbert space with the property

### An Introduction to Quantum Computation and Quantum Information

An to and Graduate Group in Applied Math University of California, Davis March 13, 009 A bit of history Benioff 198 : First paper published mentioning quantum computing Feynman 198 : Use a quantum computer

### An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1

Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,

### Quantum Simultaneous Contract Signing

Quantum Simultaneous Contract Signing J. Bouda, M. Pivoluska, L. Caha, P. Mateus, N. Paunkovic 22. October 2010 Based on work presented on AQIS 2010 J. Bouda, M. Pivoluska, L. Caha, P. Mateus, N. Quantum

### Quantum Information Types

qitd181 Quantum Information Types Robert B. Griffiths Version of 6 February 2012 References: R. B. Griffiths, Types of Quantum Information, Phys. Rev. A 76 (2007) 062320; arxiv:0707.3752 Contents 1 Introduction

### 5. Communication resources

5. Communication resources Classical channel Quantum channel Entanglement How does the state evolve under LOCC? Properties of maximally entangled states Bell basis Quantum dense coding Quantum teleportation

### Quantum Information & Quantum Computation

CS90A, Spring 005: Quantum Information & Quantum Computation Wim van Dam Engineering, Room 509 vandam@cs http://www.cs.ucsb.edu/~vandam/teaching/cs90/ Administrative The Final Examination will be: Monday

### Lecture 10: Broadcast Channel and Superposition Coding

Lecture 10: Broadcast Channel and Superposition Coding Scribed by: Zhe Yao 1 Broadcast channel M 0M 1M P{y 1 y x} M M 01 1 M M 0 The capacity of the broadcast channel depends only on the marginal conditional

### Solutions to Set #2 Data Compression, Huffman code and AEP

Solutions to Set #2 Data Compression, Huffman code and AEP. Huffman coding. Consider the random variable ( ) x x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0. 0.04 0.04 0.03 0.02 (a) Find a binary Huffman code

### Multimedia Communications. Mathematical Preliminaries for Lossless Compression

Multimedia Communications Mathematical Preliminaries for Lossless Compression What we will see in this chapter Definition of information and entropy Modeling a data source Definition of coding and when

### Physics is becoming too difficult for physicists. David Hilbert (mathematician)

Physics is becoming too difficult for physicists. David Hilbert (mathematician) Simple Harmonic Oscillator Credit: R. Nave (HyperPhysics) Particle 2 X 2-Particle wave functions 2 Particles, each moving

### Exemple of Hilbert spaces with dimension equal to two. *Spin1/2particle

University Paris-Saclay - IQUPS Optical Quantum Engineering: From fundamentals to applications Philippe Grangier, Institut d Optique, CNRS, Ecole Polytechnique. Eemple of Hilbert spaces with dimension

### 18.2 Continuous Alphabet (discrete-time, memoryless) Channel

0-704: Information Processing and Learning Spring 0 Lecture 8: Gaussian channel, Parallel channels and Rate-distortion theory Lecturer: Aarti Singh Scribe: Danai Koutra Disclaimer: These notes have not

### Asymmetric Communication Complexity and Data Structure Lower Bounds

Asymmetric Communication Complexity and Data Structure Lower Bounds Yuanhao Wei 11 November 2014 1 Introduction This lecture will be mostly based off of Miltersen s paper Cell Probe Complexity - a Survey

### PERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY

PERFECT SECRECY AND ADVERSARIAL INDISTINGUISHABILITY BURTON ROSENBERG UNIVERSITY OF MIAMI Contents 1. Perfect Secrecy 1 1.1. A Perfectly Secret Cipher 2 1.2. Odds Ratio and Bias 3 1.3. Conditions for Perfect

### (each row defines a probability distribution). Given n-strings x X n, y Y n we can use the absence of memory in the channel to compute

ENEE 739C: Advanced Topics in Signal Processing: Coding Theory Instructor: Alexander Barg Lecture 6 (draft; 9/6/03. Error exponents for Discrete Memoryless Channels http://www.enee.umd.edu/ abarg/enee739c/course.html

### Quantum rate distortion, reverse Shannon theorems, and source-channel separation

Quantum rate distortion, reverse Shannon theorems, and source-channel separation ilanjana Datta, Min-Hsiu Hsieh, and Mark M. Wilde arxiv:08.4940v3 [quant-ph] 9 Aug 202 Abstract We derive quantum counterparts

### Introduction to Information Theory. Uncertainty. Entropy. Surprisal. Joint entropy. Conditional entropy. Mutual information.

L65 Dept. of Linguistics, Indiana University Fall 205 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission rate

### Information Theory CHAPTER. 5.1 Introduction. 5.2 Entropy

Haykin_ch05_pp3.fm Page 207 Monday, November 26, 202 2:44 PM CHAPTER 5 Information Theory 5. Introduction As mentioned in Chapter and reiterated along the way, the purpose of a communication system is

### Dept. of Linguistics, Indiana University Fall 2015

L645 Dept. of Linguistics, Indiana University Fall 2015 1 / 28 Information theory answers two fundamental questions in communication theory: What is the ultimate data compression? What is the transmission

### Lecture 2: Minimax theorem, Impagliazzo Hard Core Lemma

Lecture 2: Minimax theorem, Impagliazzo Hard Core Lemma Topics in Pseudorandomness and Complexity Theory (Spring 207) Rutgers University Swastik Kopparty Scribe: Cole Franks Zero-sum games are two player

### Capacity of AWGN channels

Chapter 3 Capacity of AWGN channels In this chapter we prove that the capacity of an AWGN channel with bandwidth W and signal-tonoise ratio SNR is W log 2 (1+SNR) bits per second (b/s). The proof that

### Chapter 2: Source coding

Chapter 2: meghdadi@ensil.unilim.fr University of Limoges Chapter 2: Entropy of Markov Source Chapter 2: Entropy of Markov Source Markov model for information sources Given the present, the future is independent

### On the Duality between Multiple-Access Codes and Computation Codes

On the Duality between Multiple-Access Codes and Computation Codes Jingge Zhu University of California, Berkeley jingge.zhu@berkeley.edu Sung Hoon Lim KIOST shlim@kiost.ac.kr Michael Gastpar EPFL michael.gastpar@epfl.ch

### Quantum state discrimination with post-measurement information!

Quantum state discrimination with post-measurement information! DEEPTHI GOPAL, CALTECH! STEPHANIE WEHNER, NATIONAL UNIVERSITY OF SINGAPORE! Quantum states! A state is a mathematical object describing the

### 6.895 PCP and Hardness of Approximation MIT, Fall Lecture 3: Coding Theory

6895 PCP and Hardness of Approximation MIT, Fall 2010 Lecture 3: Coding Theory Lecturer: Dana Moshkovitz Scribe: Michael Forbes and Dana Moshkovitz 1 Motivation In the course we will make heavy use of

### Amobile satellite communication system, like Motorola s

I TRANSACTIONS ON INFORMATION THORY, VOL. 45, NO. 4, MAY 1999 1111 Distributed Source Coding for Satellite Communications Raymond W. Yeung, Senior Member, I, Zhen Zhang, Senior Member, I Abstract Inspired

### Entropy in Classical and Quantum Information Theory

Entropy in Classical and Quantum Information Theory William Fedus Physics Department, University of California, San Diego. Entropy is a central concept in both classical and quantum information theory,

### Error Detection and Correction: Hamming Code; Reed-Muller Code

Error Detection and Correction: Hamming Code; Reed-Muller Code Greg Plaxton Theory in Programming Practice, Spring 2005 Department of Computer Science University of Texas at Austin Hamming Code: Motivation

### Squashed entanglement

Squashed Entanglement based on Squashed Entanglement - An Additive Entanglement Measure (M. Christandl, A. Winter, quant-ph/0308088), and A paradigm for entanglement theory based on quantum communication

### PART III. Outline. Codes and Cryptography. Sources. Optimal Codes (I) Jorge L. Villar. MAMME, Fall 2015

Outline Codes and Cryptography 1 Information Sources and Optimal Codes 2 Building Optimal Codes: Huffman Codes MAMME, Fall 2015 3 Shannon Entropy and Mutual Information PART III Sources Information source:

### An Extended Fano s Inequality for the Finite Blocklength Coding

An Extended Fano s Inequality for the Finite Bloclength Coding Yunquan Dong, Pingyi Fan {dongyq8@mails,fpy@mail}.tsinghua.edu.cn Department of Electronic Engineering, Tsinghua University, Beijing, P.R.

### Network Coding on Directed Acyclic Graphs

Network Coding on Directed Acyclic Graphs John MacLaren Walsh, Ph.D. Multiterminal Information Theory, Spring Quarter, 0 Reference These notes are directly derived from Chapter of R. W. Yeung s Information

### On Source-Channel Communication in Networks

On Source-Channel Communication in Networks Michael Gastpar Department of EECS University of California, Berkeley gastpar@eecs.berkeley.edu DIMACS: March 17, 2003. Outline 1. Source-Channel Communication

### Chapter 9 Fundamental Limits in Information Theory

Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For

### Information-theoretic treatment of tripartite systems and quantum channels. Patrick Coles Carnegie Mellon University Pittsburgh, PA USA

Information-theoretic treatment of tripartite systems and quantum channels Patrick Coles Carnegie Mellon University Pittsburgh, PA USA ArXiv: 1006.4859 a c b Co-authors Li Yu Vlad Gheorghiu Robert Griffiths

### 1 Introduction to information theory

1 Introduction to information theory 1.1 Introduction In this chapter we present some of the basic concepts of information theory. The situations we have in mind involve the exchange of information through

### Quantum Information and the PCP Theorem

Quantum Information and the PCP Theorem arxiv:quant-ph/0504075v1 10 Apr 2005 Ran Raz Weizmann Institute ran.raz@weizmann.ac.il Abstract We show how to encode 2 n (classical) bits a 1,...,a 2 n by a single

### Exponential Separation of Quantum Communication and Classical Information

Exponential Separation of Quantum Communication and Classical Information Dave Touchette IQC and C&O, University of Waterloo, and Perimeter Institute for Theoretical Physics jt. work with Anurag Anshu

### Lecture 6. Today we shall use graph entropy to improve the obvious lower bound on good hash functions.

CSE533: Information Theory in Computer Science September 8, 010 Lecturer: Anup Rao Lecture 6 Scribe: Lukas Svec 1 A lower bound for perfect hash functions Today we shall use graph entropy to improve the

### 3F1: Signals and Systems INFORMATION THEORY Examples Paper Solutions

Engineering Tripos Part IIA THIRD YEAR 3F: Signals and Systems INFORMATION THEORY Examples Paper Solutions. Let the joint probability mass function of two binary random variables X and Y be given in the

### Privacy Amplification Theorem for Noisy Main Channel

Privacy Amplification Theorem for Noisy Main Channel Valeri Korjik 1, Guillermo Morales-Luna 2, and Vladimir B. Balakirsky 3 1 Telecommunications, CINVESTAV-IPN, Guadalajara Campus Prol. López Mateos Sur

### Kolmogorov complexity and its applications

Spring, 2009 Kolmogorov complexity and its applications Paul Vitanyi Computer Science University of Amsterdam http://www.cwi.nl/~paulv/course-kc We live in an information society. Information science is

### ECE 587 / STA 563: Lecture 5 Lossless Compression

ECE 587 / STA 563: Lecture 5 Lossless Compression Information Theory Duke University, Fall 2017 Author: Galen Reeves Last Modified: October 18, 2017 Outline of lecture: 5.1 Introduction to Lossless Source

### Exercise 1. = P(y a 1)P(a 1 )

Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a

### COS597D: Information Theory in Computer Science September 21, Lecture 2

COS597D: Information Theory in Computer Science September 1, 011 Lecture Lecturer: Mark Braverman Scribe: Mark Braverman In the last lecture, we introduced entropy H(X), and conditional entry H(X Y ),

### An approach from classical information theory to lower bounds for smooth codes

An approach from classical information theory to lower bounds for smooth codes Abstract Let C : {0, 1} n {0, 1} m be a code encoding an n-bit string into an m-bit string. Such a code is called a (q, c,

### Entropy. Probability and Computing. Presentation 22. Probability and Computing Presentation 22 Entropy 1/39

Entropy Probability and Computing Presentation 22 Probability and Computing Presentation 22 Entropy 1/39 Introduction Why randomness and information are related? An event that is almost certain to occur

### 10-704: Information Processing and Learning Spring Lecture 8: Feb 5

10-704: Information Processing and Learning Spring 2015 Lecture 8: Feb 5 Lecturer: Aarti Singh Scribe: Siheng Chen Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal

### Tutorial on Quantum Computing. Vwani P. Roychowdhury. Lecture 1: Introduction

Tutorial on Quantum Computing Vwani P. Roychowdhury Lecture 1: Introduction 1 & ) &! # Fundamentals Qubits A single qubit is a two state system, such as a two level atom we denote two orthogonal states

### Entanglement and Quantum Teleportation

Entanglement and Quantum Teleportation Stephen Bartlett Centre for Advanced Computing Algorithms and Cryptography Australian Centre of Excellence in Quantum Computer Technology Macquarie University, Sydney,

### Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006

Classical Information Theory Notes from the lectures by prof Suhov Trieste - june 2006 Fabio Grazioso... July 3, 2006 1 2 Contents 1 Lecture 1, Entropy 4 1.1 Random variable...............................

### 3F1 Information Theory, Lecture 3

3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output

### Performance-based Security for Encoding of Information Signals. FA ( ) Paul Cuff (Princeton University)

Performance-based Security for Encoding of Information Signals FA9550-15-1-0180 (2015-2018) Paul Cuff (Princeton University) Contributors Two students finished PhD Tiance Wang (Goldman Sachs) Eva Song

### Ping Pong Protocol & Auto-compensation

Ping Pong Protocol & Auto-compensation Adam de la Zerda For QIP seminar Spring 2004 02.06.04 Outline Introduction to QKD protocols + motivation Ping-Pong protocol Security Analysis for Ping-Pong Protocol

### 4. Quantization and Data Compression. ECE 302 Spring 2012 Purdue University, School of ECE Prof. Ilya Pollak

4. Quantization and Data Compression ECE 32 Spring 22 Purdue University, School of ECE Prof. What is data compression? Reducing the file size without compromising the quality of the data stored in the

### MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK UNIT V PART-A. 1. What is binary symmetric channel (AUC DEC 2006)

MAHALAKSHMI ENGINEERING COLLEGE-TRICHY QUESTION BANK SATELLITE COMMUNICATION DEPT./SEM.:ECE/VIII UNIT V PART-A 1. What is binary symmetric channel (AUC DEC 2006) 2. Define information rate? (AUC DEC 2007)

### Lecture 2. Capacity of the Gaussian channel

Spring, 207 5237S, Wireless Communications II 2. Lecture 2 Capacity of the Gaussian channel Review on basic concepts in inf. theory ( Cover&Thomas: Elements of Inf. Theory, Tse&Viswanath: Appendix B) AWGN

### Information Theory, Statistics, and Decision Trees

Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010

### 6.1 Occupancy Problem

15-859(M): Randomized Algorithms Lecturer: Anupam Gupta Topic: Occupancy Problems and Hashing Date: Sep 9 Scribe: Runting Shi 6.1 Occupancy Problem Bins and Balls Throw n balls into n bins at random. 1.

### 1 Randomized Computation

CS 6743 Lecture 17 1 Fall 2007 1 Randomized Computation Why is randomness useful? Imagine you have a stack of bank notes, with very few counterfeit ones. You want to choose a genuine bank note to pay at

### Kolmogorov complexity and its applications

CS860, Winter, 2010 Kolmogorov complexity and its applications Ming Li School of Computer Science University of Waterloo http://www.cs.uwaterloo.ca/~mli/cs860.html We live in an information society. Information

### Lecture 14: Hamming and Hadamard Codes

CSCI-B69: A Theorist s Toolkit, Fall 6 Oct 6 Lecture 4: Hamming and Hadamard Codes Lecturer: Yuan Zhou Scribe: Kaiyuan Zhu Recap Recall from the last lecture that error-correcting codes are in fact injective

### 6.02 Fall 2012 Lecture #1

6.02 Fall 2012 Lecture #1 Digital vs. analog communication The birth of modern digital communication Information and entropy Codes, Huffman coding 6.02 Fall 2012 Lecture 1, Slide #1 6.02 Fall 2012 Lecture

### COS597D: Information Theory in Computer Science October 19, Lecture 10

COS597D: Information Theory in Computer Science October 9, 20 Lecture 0 Lecturer: Mark Braverman Scribe: Andrej Risteski Kolmogorov Complexity In the previous lectures, we became acquainted with the concept

### Lec 03 Entropy and Coding II Hoffman and Golomb Coding

CS/EE 5590 / ENG 40 Special Topics Multimedia Communication, Spring 207 Lec 03 Entropy and Coding II Hoffman and Golomb Coding Zhu Li Z. Li Multimedia Communciation, 207 Spring p. Outline Lecture 02 ReCap

### Introduction to Information Theory. B. Škorić, Physical Aspects of Digital Security, Chapter 2

Introduction to Information Theory B. Škorić, Physical Aspects of Digital Security, Chapter 2 1 Information theory What is it? - formal way of counting information bits Why do we need it? - often used

### X 1 : X Table 1: Y = X X 2

ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access

### The Poisson Channel with Side Information

The Poisson Channel with Side Information Shraga Bross School of Enginerring Bar-Ilan University, Israel brosss@macs.biu.ac.il Amos Lapidoth Ligong Wang Signal and Information Processing Laboratory ETH

### Source and Channel Coding for Correlated Sources Over Multiuser Channels

Source and Channel Coding for Correlated Sources Over Multiuser Channels Deniz Gündüz, Elza Erkip, Andrea Goldsmith, H. Vincent Poor Abstract Source and channel coding over multiuser channels in which

### arxiv:quant-ph/ v2 2 Jan 2007

Revisiting controlled quantum secure direct communication using a non-symmetric quantum channel with quantum superdense coding arxiv:quant-ph/06106v Jan 007 Jun Liu 1, Yan Xia and Zhan-jun Zhang 1,, 1

### Simultaneous Communication Protocols with Quantum and Classical Messages

Simultaneous Communication Protocols with Quantum and Classical Messages Oded Regev Ronald de Wolf July 17, 2008 Abstract We study the simultaneous message passing model of communication complexity, for

### Quantum Information Theory

Quantum Information Theory C. H. Bennett IBM Research Yorktown QIS Workshop Vienna, VA 23 Apr 2009 Nature and importance of Information Distinctive features of Quantum Information Superposition principle,

### Linear Codes, Target Function Classes, and Network Computing Capacity

Linear Codes, Target Function Classes, and Network Computing Capacity Rathinakumar Appuswamy, Massimo Franceschetti, Nikhil Karamchandani, and Kenneth Zeger IEEE Transactions on Information Theory Submitted:

### An Introduction to Quantum Information and Applications

An Introduction to Quantum Information and Applications Iordanis Kerenidis CNRS LIAFA-Univ Paris-Diderot Quantum information and computation Quantum information and computation How is information encoded