Communication Complexity 16:198:671 2/15/2010. Lecture 6. P (x) log
|
|
- Felicia Melton
- 5 years ago
- Views:
Transcription
1 Communication Complexity 6:98:67 2/5/200 Lecture 6 Lecturer: Nikos Leonardos Scribe: Troy Lee Information theory lower bounds. Entropy basics Let Ω be a finite set and P a probability distribution on Ω. The entropy of a random variable X distributed according to P is defined as H(X) = x Ω P (x) log P (x). Intuitively, entropy is a means to quantify the amount of uncertainty in a distribution If the distribution is focused on a single element the entropy is zero; for the uniform distribution on Ω the entropy is maximal at log Ω. These examples set the bounds for the range of entropy, 0 H(X) log Ω. Entropy can also be interpreted in terms of compression Shannon s source coding theorem states that the optimal expected code word length of elements of a random variable X is the entropy of X. We will also make use of the conditional entropy. First consider conditioning on a single outcome. H(X Y = y) = P (x y) log P (x y). x This quantity can actually be larger than H(X). Imagine the case where random bit if Y = 0 X = 0 if Y = Then H(X) < yet H(X Y = 0) =. The conditional entropy is the expectation of the last quantity over Y H(X Y ) = E y [H(X Y = y)] H(X). This quantity is at most the entropy of X. Finally, let the joint entropy of X, Y be H(X, Y ) = H(X) + H(Y X).
2 .2 Mutual Information For lower bounds, we will use mutual information. For two random variables Z, Π this is defined as I(Z; Π) = H(Z) H(Z Π) = H(Π) H(Π Z) = H(Π) + H(Z) H(Π, Z). In applications to communication complexity, typically Z will be a distribution over inputs X Y and Π will be a distribution over protocol transcripts. To see how this can work to show lower bounds, let us look at a warmup example, the index function. The index function is defined as Index : 0, } n [n] 0, } where Index(x, i) = x i. Consider the one-way complexity of the index function from Alice to Bob. Let Π(X, R) be a random variable over the messages of Alice which depends on the distribution X over Alice s inputs and R random coins of Alice. Notice that H(Π) will be a lower bound on the maximum length of a message of Alice. We will actually lower bound the potentially smaller quantity I(X; Π). Use the uniform distribution over Alice s inputs. Then we have I(X; Π) = H(X) H(X Π) = n H(X Π). Now it remains to upper bound H(X Π). Using the fact that H(Y, Z) = H(Y ) + H(Z Y ) we have H(X Π) H(X j Π). j As we are dealing with one-way communication, conditioning on a particular input to Bob does not change Alice s action. Thus we have H(X j Π) = H(X j Π, B = j) that is, given that Bob s input is actually j. Finally we have by correctness of the protocol that this entropy must be small, at most H(ɛ) if the error probability is ɛ. Putting everything together we have Rɛ A B (Index) I(X; Π) ( H(ɛ))n. This example comes from Ablayev, Lower bounds for one-way probabilistic communication complexity and their application to space complexity, Theoretical Computer Science, Vol. 57(2), pg , Set Intersection Lower bound We will now show a Ω(n) lower bound for the randomized two-party complexity of the set intersection problem. We will follow the proof of Bar-Yossef et al. An information statistics 2
3 approach to data stream and communication complexity, JCSS 68, pg , 200. We refer the reader there for full details. Consider again a random variable Π((X, Y ), R A, R B ) over transcripts. This variable depends on the distribution of inputs (X, Y ) and Alice s random bits R A and Bob s random bits R B. It is actually important for this framework that we work with private coin complexity. Define the information cost of the protocol Π as IC((X, Y ); Π). The information cost of a function is then IC(f) = min IC(Π). Π Πcorrect 2. Direct sum The idea for the proof will be to show a direct sum theorem for the information cost measure that the information cost for set intersection must be n times the information cost of the one-bit and function. To do this we will use the following fact. Fact. If Z = (Z,..., Z n ) are mutually independent then I(Z; Π) I(Z ; Π) I(Z n ; Π). We will define a distribution on inputs where (X, Y ),..., (X n, Y n ) are mutually independent. In this case, I((X, Y ); Π) I((X, Y ); Π) I((X n, Y n ); Π). The goal will now be to relate I((X i, Y i ); Π) to the information cost of the one-bit AND function. The distribution we will use on (X i, Y i ) is P (0, 0) = /2, P (, 0) = /, P (0, ) = /. Notice that this distribution has the property that it only gives weight to inputs which evaluate to zero on AND. This is still OK, as in the definition of information cost we just minimize over correct protocols thus the trivial protocol which always outputs zero, although it works for this distribution, is excluded. It will be useful to view this distribution another way, as a mixture of product distributions. Introduce another variable D i uniformly distributed over 0, }. Then we define 0 if D i = 0 X i = random bit D i = 0 if D i = Y i = random bit D i = 0 Now for a fixed value of D i we have a product distribution on (X i, Y i ) and the mixture of these two product distributions is the distribution defined above. The key to the direct sum property is the following claim. 3
4 Claim 2. I((X i, Y i ); Π D) IC(AND). Proof. We design a protocol for AND by simulating the protocol for SET INTERSECTION with variables outside of (X i, Y i ) fixed. For notational convenience, let i =. I((X, Y ); Π D) = E d2,...,d n [I((X, Y ); Π D, D 2 = d 2,..., D n = d n )] We will show that each term of this expectation is at least IC(AND) which will then give the claim. Consider a fixing D 2 = d 2,..., D n = d n. We design a protocol for AND(A, B) using the protocol Π. As D j is fixed for j =,..., n, the distribution over (X j, Y j ) is a product distribution and so can be simulated by Alice and Bob without communication using their private random bits. Alice and Bob then run the protocol Π on the input (A, B), (x 2, y 2 ),..., (x n, y n ). This claim gives that IC(SI) nic(and). Now we just have to show a lower bound on the information complexity of the AND function on one bit. 2.2 One-bit AND function We want to lower bound I((A, B); Π D) = I((A, B); Π D = 0) + I((A, B); Π D = ). As 2 2 these are symmetrical we can just focus on I((A, B); Π D = ) = I(A; Π(A, 0) D = ). If the distribution on transcripts is very different on Π(0, 0) and Π(, 0) then we will be able to determine A from looking at the transcripts, implying the information complexity is large. To do this, we will transform the problem from one about mutual information to one about a metric, the Hellinger distance. Instead of viewing Π(a, b) as a probability distribution, consider instead Ψ(a, b) the unit vector which is the entrywise square root of Π(a, b). With this transformation, Hellinger distance simply becomes a scaled version of Euclidean distance: h(ψ, Ψ 2 ) = 2 Ψ Ψ 2. We will need three key properties of Hellinger distance and its relation to mutual information. See the above paper for proof of these statements.. Mutual information and Hellinger distance: Let u, v 0, } 2 be two inputs to AND, and U R u, v}. As before let Ψ(u) be the unit vector formed by the entrywise square root of Π(u). I(U; Π) 2 Ψ(u) Ψ(v) Soundness: If AND(u) AND(v) and Π is a protocol with error at most ɛ, then 2 Ψ(u) Ψ(v) 2 2 ɛ. 3. Cut and paste: Let u = (x, y), v = (x, y ) and u = (x, y ), v = (x, y). Then Ψ(u) Ψ(v) = Ψ(u ) Ψ(v ).
5 Given these three properties we can quickly finish the proof. 2 Ψ(0, 0) Ψ(, 0)) 2 + Ψ(0, 0) Ψ(0, )) 2 ( Ψ(0, 0) Ψ(, 0) + Ψ(0, 0) + Ψ(0, ) )2 Ψ(, 0) Ψ(0, ) 2 = Ψ(0, 0) Ψ(, ) ( 2 ɛ). Where the first inequality follows by Cauchy-Schwarz, the second by triangle inequality, the third by cut and paste, and the last by soundness. 5
CS Foundations of Communication Complexity
CS 2429 - Foundations of Communication Complexity Lecturer: Sergey Gorbunov 1 Introduction In this lecture we will see how to use methods of (conditional) information complexity to prove lower bounds for
More informationThe Communication Complexity of Correlation. Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan
The Communication Complexity of Correlation Prahladh Harsha Rahul Jain David McAllester Jaikumar Radhakrishnan Transmitting Correlated Variables (X, Y) pair of correlated random variables Transmitting
More informationCOMPSCI 650 Applied Information Theory Jan 21, Lecture 2
COMPSCI 650 Applied Information Theory Jan 21, 2016 Lecture 2 Instructor: Arya Mazumdar Scribe: Gayane Vardoyan, Jong-Chyi Su 1 Entropy Definition: Entropy is a measure of uncertainty of a random variable.
More informationLecture Lecture 9 October 1, 2015
CS 229r: Algorithms for Big Data Fall 2015 Lecture Lecture 9 October 1, 2015 Prof. Jelani Nelson Scribe: Rachit Singh 1 Overview In the last lecture we covered the distance to monotonicity (DTM) and longest
More informationInformation Complexity and Applications. Mark Braverman Princeton University and IAS FoCM 17 July 17, 2017
Information Complexity and Applications Mark Braverman Princeton University and IAS FoCM 17 July 17, 2017 Coding vs complexity: a tale of two theories Coding Goal: data transmission Different channels
More informationLecture 16 Oct 21, 2014
CS 395T: Sublinear Algorithms Fall 24 Prof. Eric Price Lecture 6 Oct 2, 24 Scribe: Chi-Kit Lam Overview In this lecture we will talk about information and compression, which the Huffman coding can achieve
More informationLecture: Quantum Information
Lecture: Quantum Information Transcribed by: Crystal Noel and Da An (Chi Chi) November 10, 016 1 Final Proect Information Find an issue related to class you are interested in and either: read some papers
More informationAn Information Complexity Approach to the Inner Product Problem
An Information Complexity Approach to the Inner Product Problem William Henderson-Frost Advisor: Amit Chakrabarti Senior Honors Thesis Submitted to the faculty in partial fulfillment of the requirements
More informationNear-Optimal Lower Bounds on the Multi-Party Communication Complexity of Set Disjointness
Near-Optimal Lower Bounds on the Multi-Party Communication Complexity of Set Disjointness Amit Chakrabarti School of Mathematics Institute for Advanced Study Princeton, NJ 08540 amitc@ias.edu Subhash Khot
More informationAQI: Advanced Quantum Information Lecture 6 (Module 2): Distinguishing Quantum States January 28, 2013
AQI: Advanced Quantum Information Lecture 6 (Module 2): Distinguishing Quantum States January 28, 2013 Lecturer: Dr. Mark Tame Introduction With the emergence of new types of information, in this case
More informationAn exponential separation between quantum and classical one-way communication complexity
An exponential separation between quantum and classical one-way communication complexity Ashley Montanaro Centre for Quantum Information and Foundations, Department of Applied Mathematics and Theoretical
More informationLecture 11: Quantum Information III - Source Coding
CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that
More informationLecture 18: Quantum Information Theory and Holevo s Bound
Quantum Computation (CMU 1-59BB, Fall 2015) Lecture 1: Quantum Information Theory and Holevo s Bound November 10, 2015 Lecturer: John Wright Scribe: Nicolas Resch 1 Question In today s lecture, we will
More informationInformation Complexity vs. Communication Complexity: Hidden Layers Game
Information Complexity vs. Communication Complexity: Hidden Layers Game Jiahui Liu Final Project Presentation for Information Theory in TCS Introduction Review of IC vs CC Hidden Layers Game Upper Bound
More informationEE5585 Data Compression May 2, Lecture 27
EE5585 Data Compression May 2, 2013 Lecture 27 Instructor: Arya Mazumdar Scribe: Fangying Zhang Distributed Data Compression/Source Coding In the previous class we used a H-W table as a simple example,
More informationTight Bounds for Distributed Functional Monitoring
Tight Bounds for Distributed Functional Monitoring Qin Zhang MADALGO, Aarhus University Joint with David Woodruff, IBM Almaden NII Shonan meeting, Japan Jan. 2012 1-1 The distributed streaming model (a.k.a.
More informationComputing and Communications 2. Information Theory -Entropy
1896 1920 1987 2006 Computing and Communications 2. Information Theory -Entropy Ying Cui Department of Electronic Engineering Shanghai Jiao Tong University, China 2017, Autumn 1 Outline Entropy Joint entropy
More informationExponential Separation of Quantum Communication and Classical Information
Exponential Separation of Quantum Communication and Classical Information Dave Touchette IQC and C&O, University of Waterloo, and Perimeter Institute for Theoretical Physics jt. work with Anurag Anshu
More information14. Direct Sum (Part 1) - Introduction
Communication Complexity 14 Oct, 2011 (@ IMSc) 14. Direct Sum (Part 1) - Introduction Lecturer: Prahladh Harsha Scribe: Abhishek Dang 14.1 Introduction The Direct Sum problem asks how the difficulty in
More informationLecture 19 October 28, 2015
PHYS 7895: Quantum Information Theory Fall 2015 Prof. Mark M. Wilde Lecture 19 October 28, 2015 Scribe: Mark M. Wilde This document is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike
More informationLecture 5 - Information theory
Lecture 5 - Information theory Jan Bouda FI MU May 18, 2012 Jan Bouda (FI MU) Lecture 5 - Information theory May 18, 2012 1 / 42 Part I Uncertainty and entropy Jan Bouda (FI MU) Lecture 5 - Information
More informationTHE UNIVERSITY OF CHICAGO COMMUNICATION COMPLEXITY AND INFORMATION COMPLEXITY A DISSERTATION SUBMITTED TO
THE UNIVERSITY OF CHICAGO COMMUNICATION COMPLEXITY AND INFORMATION COMPLEXITY A DISSERTATION SUBMITTED TO THE FACULTY OF THE DIVISION OF THE PHYSICAL SCIENCES IN CANDIDACY FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
More informationClassification & Information Theory Lecture #8
Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing
More informationECE598: Information-theoretic methods in high-dimensional statistics Spring 2016
ECE598: Information-theoretic methods in high-dimensional statistics Spring 06 Lecture : Mutual Information Method Lecturer: Yihong Wu Scribe: Jaeho Lee, Mar, 06 Ed. Mar 9 Quick review: Assouad s lemma
More informationHomework 1 Due: Thursday 2/5/2015. Instructions: Turn in your homework in class on Thursday 2/5/2015
10-704 Homework 1 Due: Thursday 2/5/2015 Instructions: Turn in your homework in class on Thursday 2/5/2015 1. Information Theory Basics and Inequalities C&T 2.47, 2.29 (a) A deck of n cards in order 1,
More informationLecture 20: Lower Bounds for Inner Product & Indexing
15-859: Information Theory and Applications in TCS CMU: Spring 201 Lecture 20: Lower Bounds for Inner Product & Indexing April 9, 201 Lecturer: Venkatesan Guruswami Scribe: Albert Gu 1 Recap Last class
More informationCS Communication Complexity: Applications and New Directions
CS 2429 - Communication Complexity: Applications and New Directions Lecturer: Toniann Pitassi 1 Introduction In this course we will define the basic two-party model of communication, as introduced in the
More informationCommunication vs information complexity, relative discrepancy and other lower bounds
Communication vs information complexity, relative discrepancy and other lower bounds Iordanis Kerenidis CNRS, LIAFA- Univ Paris Diderot 7 Joint work with: L. Fontes, R. Jain, S. Laplante, M. Lauriere,
More informationLower bounds for Edit Distance and Product Metrics via Poincaré-Type Inequalities
Lower bounds for Edit Distance and Product Metrics via Poincaré-Type Inequalities Alexandr Andoni Princeton U./CCI andoni@mit.edu T.S. Jayram IBM Almaden jayram@almaden.ibm.com Mihai Pǎtraşcu AT&T Labs
More informationLecture 1: Introduction, Entropy and ML estimation
0-704: Information Processing and Learning Spring 202 Lecture : Introduction, Entropy and ML estimation Lecturer: Aarti Singh Scribes: Min Xu Disclaimer: These notes have not been subjected to the usual
More informationLecture 1: September 25, A quick reminder about random variables and convexity
Information and Coding Theory Autumn 207 Lecturer: Madhur Tulsiani Lecture : September 25, 207 Administrivia This course will cover some basic concepts in information and coding theory, and their applications
More information6.1 Main properties of Shannon entropy. Let X be a random variable taking values x in some alphabet with probabilities.
Chapter 6 Quantum entropy There is a notion of entropy which quantifies the amount of uncertainty contained in an ensemble of Qbits. This is the von Neumann entropy that we introduce in this chapter. In
More informationEntropy and Ergodic Theory Lecture 4: Conditional entropy and mutual information
Entropy and Ergodic Theory Lecture 4: Conditional entropy and mutual information 1 Conditional entropy Let (Ω, F, P) be a probability space, let X be a RV taking values in some finite set A. In this lecture
More informationDistributed Statistical Estimation of Matrix Products with Applications
Distributed Statistical Estimation of Matrix Products with Applications David Woodruff CMU Qin Zhang IUB PODS 2018 June, 2018 1-1 The Distributed Computation Model p-norms, heavy-hitters,... A {0, 1} m
More informationExtractors and the Leftover Hash Lemma
6.889 New Developments in Cryptography March 8, 2011 Extractors and the Leftover Hash Lemma Instructors: Shafi Goldwasser, Yael Kalai, Leo Reyzin, Boaz Barak, and Salil Vadhan Lecturer: Leo Reyzin Scribe:
More information1 Basic Information Theory
ECE 6980 An Algorithmic and Information-Theoretic Toolbo for Massive Data Instructor: Jayadev Acharya Lecture #4 Scribe: Xiao Xu 6th September, 206 Please send errors to 243@cornell.edu and acharya@cornell.edu
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationLecture 3: Oct 7, 2014
Information and Coding Theory Autumn 04 Lecturer: Shi Li Lecture : Oct 7, 04 Scribe: Mrinalkanti Ghosh In last lecture we have seen an use of entropy to give a tight upper bound in number of triangles
More informationLecture 2: August 31
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 2: August 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy
More informationQuantitative Information Flow. Lecture 7
Quantitative Information Flow Lecture 7 1 The basic model: Systems = Information-Theoretic channels Secret Information Observables s1 o1... System... sm on Input Output 2 Probabilistic systems are noisy
More informationOn Function Computation with Privacy and Secrecy Constraints
1 On Function Computation with Privacy and Secrecy Constraints Wenwen Tu and Lifeng Lai Abstract In this paper, the problem of function computation with privacy and secrecy constraints is considered. The
More informationMachine Learning. Lecture 02.2: Basics of Information Theory. Nevin L. Zhang
Machine Learning Lecture 02.2: Basics of Information Theory Nevin L. Zhang lzhang@cse.ust.hk Department of Computer Science and Engineering The Hong Kong University of Science and Technology Nevin L. Zhang
More informationarxiv: v5 [cs.cc] 10 Jul 2014
The space complexity of recognizing well-parenthesized expressions in the streaming model: the Index function revisited Rahul Jain Ashwin Nayak arxiv:1004.3165v5 [cs.cc] 10 Jul 014 March 6, 014 Abstract
More informationInformation Theory + Polyhedral Combinatorics
Information Theory + Polyhedral Combinatorics Sebastian Pokutta Georgia Institute of Technology ISyE, ARC Joint work Gábor Braun Information Theory in Complexity Theory and Combinatorics Simons Institute
More informationThe Shannon s basic inequalities refer to the following fundamental properties of entropy function:
COMMUNICATIONS IN INFORMATION AND SYSTEMS c 2003 International Press Vol. 3, No. 1, pp. 47-60, June 2003 004 ON A NEW NON-SHANNON TYPE INFORMATION INEQUALITY ZHEN ZHANG Abstract. Recently, K. Makarychev,
More informationMGMT 69000: Topics in High-dimensional Data Analysis Falll 2016
MGMT 69000: Topics in High-dimensional Data Analysis Falll 2016 Lecture 14: Information Theoretic Methods Lecturer: Jiaming Xu Scribe: Hilda Ibriga, Adarsh Barik, December 02, 2016 Outline f-divergence
More informationSeries 7, May 22, 2018 (EM Convergence)
Exercises Introduction to Machine Learning SS 2018 Series 7, May 22, 2018 (EM Convergence) Institute for Machine Learning Dept. of Computer Science, ETH Zürich Prof. Dr. Andreas Krause Web: https://las.inf.ethz.ch/teaching/introml-s18
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationHands-On Learning Theory Fall 2016, Lecture 3
Hands-On Learning Theory Fall 016, Lecture 3 Jean Honorio jhonorio@purdue.edu 1 Information Theory First, we provide some information theory background. Definition 3.1 (Entropy). The entropy of a discrete
More informationLecture 21: Quantum communication complexity
CPSC 519/619: Quantum Computation John Watrous, University of Calgary Lecture 21: Quantum communication complexity April 6, 2006 In this lecture we will discuss how quantum information can allow for a
More informationNondeterminism LECTURE Nondeterminism as a proof system. University of California, Los Angeles CS 289A Communication Complexity
University of California, Los Angeles CS 289A Communication Complexity Instructor: Alexander Sherstov Scribe: Matt Brown Date: January 25, 2012 LECTURE 5 Nondeterminism In this lecture, we introduce nondeterministic
More informationCSCI-B609: A Theorist s Toolkit, Fall 2016 Oct 4. Theorem 1. A non-zero, univariate polynomial with degree d has at most d roots.
CSCI-B609: A Theorist s Toolkit, Fall 2016 Oct 4 Lecture 14: Schwartz-Zippel Lemma and Intro to Coding Theory Lecturer: Yuan Zhou Scribe: Haoyu Zhang 1 Roots of polynomials Theorem 1. A non-zero, univariate
More informationEE376A - Information Theory Final, Monday March 14th 2016 Solutions. Please start answering each question on a new page of the answer booklet.
EE376A - Information Theory Final, Monday March 14th 216 Solutions Instructions: You have three hours, 3.3PM - 6.3PM The exam has 4 questions, totaling 12 points. Please start answering each question on
More informationProblem Set: TT Quantum Information
Problem Set: TT Quantum Information Basics of Information Theory 1. Alice can send four messages A, B, C, and D over a classical channel. She chooses A with probability 1/, B with probability 1/4 and C
More informationInaccessible Entropy and its Applications. 1 Review: Psedorandom Generators from One-Way Functions
Columbia University - Crypto Reading Group Apr 27, 2011 Inaccessible Entropy and its Applications Igor Carboni Oliveira We summarize the constructions of PRGs from OWFs discussed so far and introduce the
More information2 Natural Proofs: a barrier for proving circuit lower bounds
Topics in Theoretical Computer Science April 4, 2016 Lecturer: Ola Svensson Lecture 6 (Notes) Scribes: Ola Svensson Disclaimer: These notes were written for the lecturer only and may contain inconsistent
More informationInteractive information and coding theory
Interactive information and coding theory Mark Braverman Abstract. We give a high-level overview of recent developments in interactive information and coding theory. These include developments involving
More informationFourier analysis of boolean functions in quantum computation
Fourier analysis of boolean functions in quantum computation Ashley Montanaro Centre for Quantum Information and Foundations, Department of Applied Mathematics and Theoretical Physics, University of Cambridge
More information9. Distance measures. 9.1 Classical information measures. Head Tail. How similar/close are two probability distributions? Trace distance.
9. Distance measures 9.1 Classical information measures How similar/close are two probability distributions? Trace distance Fidelity Example: Flipping two coins, one fair one biased Head Tail Trace distance
More informationEE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018
Please submit the solutions on Gradescope. Some definitions that may be useful: EE376A: Homework #2 Solutions Due by 11:59pm Thursday, February 1st, 2018 Definition 1: A sequence of random variables X
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationCS 630 Basic Probability and Information Theory. Tim Campbell
CS 630 Basic Probability and Information Theory Tim Campbell 21 January 2003 Probability Theory Probability Theory is the study of how best to predict outcomes of events. An experiment (or trial or event)
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationInformation Theory Primer:
Information Theory Primer: Entropy, KL Divergence, Mutual Information, Jensen s inequality Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro,
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationLecture 15 - Zero Knowledge Proofs
Lecture 15 - Zero Knowledge Proofs Boaz Barak November 21, 2007 Zero knowledge for 3-coloring. We gave a ZK proof for the language QR of (x, n) such that x QR n. We ll now give a ZK proof (due to Goldreich,
More informationCS Introduction to Complexity Theory. Lecture #11: Dec 8th, 2015
CS 2401 - Introduction to Complexity Theory Lecture #11: Dec 8th, 2015 Lecturer: Toniann Pitassi Scribe Notes by: Xu Zhao 1 Communication Complexity Applications Communication Complexity (CC) has many
More informationMajority is incompressible by AC 0 [p] circuits
Majority is incompressible by AC 0 [p] circuits Igor Carboni Oliveira Columbia University Joint work with Rahul Santhanam (Univ. Edinburgh) 1 Part 1 Background, Examples, and Motivation 2 Basic Definitions
More information1 Review of The Learning Setting
COS 5: Theoretical Machine Learning Lecturer: Rob Schapire Lecture #8 Scribe: Changyan Wang February 28, 208 Review of The Learning Setting Last class, we moved beyond the PAC model: in the PAC model we
More informationCMSC 858F: Algorithmic Lower Bounds: Fun with Hardness Proofs Fall 2014 Introduction to Streaming Algorithms
CMSC 858F: Algorithmic Lower Bounds: Fun with Hardness Proofs Fall 2014 Introduction to Streaming Algorithms Instructor: Mohammad T. Hajiaghayi Scribe: Huijing Gong November 11, 2014 1 Overview In the
More informationLecture 11: Key Agreement
Introduction to Cryptography 02/22/2018 Lecture 11: Key Agreement Instructor: Vipul Goyal Scribe: Francisco Maturana 1 Hardness Assumptions In order to prove the security of cryptographic primitives, we
More informationComputational Systems Biology: Biology X
Bud Mishra Room 1002, 715 Broadway, Courant Institute, NYU, New York, USA L#8:(November-08-2010) Cancer and Signals Outline 1 Bayesian Interpretation of Probabilities Information Theory Outline Bayesian
More informationEXPONENTIAL SEPARATION OF QUANTUM AND CLASSICAL ONE-WAY COMMUNICATION COMPLEXITY
EXPONENTIAL SEPARATION OF QUANTUM AND CLASSICAL ONE-WAY COMMUNICATION COMPLEXITY ZIV BAR-YOSSEF, T. S. JAYRAM, AND IORDANIS KERENIDIS Abstract. We give the first exponential separation between quantum
More informationCS286.2 Lecture 8: A variant of QPCP for multiplayer entangled games
CS286.2 Lecture 8: A variant of QPCP for multiplayer entangled games Scribe: Zeyu Guo In the first lecture, we saw three equivalent variants of the classical PCP theorems in terms of CSP, proof checking,
More informationMath General Topology Fall 2012 Homework 1 Solutions
Math 535 - General Topology Fall 2012 Homework 1 Solutions Definition. Let V be a (real or complex) vector space. A norm on V is a function : V R satisfying: 1. Positivity: x 0 for all x V and moreover
More informationLecture 8: Channel Capacity, Continuous Random Variables
EE376A/STATS376A Information Theory Lecture 8-02/0/208 Lecture 8: Channel Capacity, Continuous Random Variables Lecturer: Tsachy Weissman Scribe: Augustine Chemparathy, Adithya Ganesh, Philip Hwang Channel
More informationA Tight Lower Bound for High Frequency Moment Estimation with Small Error
A Tight Lower Bound for High Frequency Moment Estimation with Small Error Yi Li Department of EECS University of Michigan, Ann Arbor leeyi@umich.edu David P. Woodruff IBM Research, Almaden dpwoodru@us.ibm.com
More informationLecture 26: Arthur-Merlin Games
CS 710: Complexity Theory 12/09/2011 Lecture 26: Arthur-Merlin Games Instructor: Dieter van Melkebeek Scribe: Chetan Rao and Aaron Gorenstein Last time we compared counting versus alternation and showed
More informationNoisy-Channel Coding
Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/05264298 Part II Noisy-Channel Coding Copyright Cambridge University Press 2003.
More informationTutorial on Quantum Computing. Vwani P. Roychowdhury. Lecture 1: Introduction
Tutorial on Quantum Computing Vwani P. Roychowdhury Lecture 1: Introduction 1 & ) &! # Fundamentals Qubits A single qubit is a two state system, such as a two level atom we denote two orthogonal states
More information3F1: Signals and Systems INFORMATION THEORY Examples Paper Solutions
Engineering Tripos Part IIA THIRD YEAR 3F: Signals and Systems INFORMATION THEORY Examples Paper Solutions. Let the joint probability mass function of two binary random variables X and Y be given in the
More informationInformation Theory. Coding and Information Theory. Information Theory Textbooks. Entropy
Coding and Information Theory Chris Williams, School of Informatics, University of Edinburgh Overview What is information theory? Entropy Coding Information Theory Shannon (1948): Information theory is
More informationCourse Introduction. 1.1 Overview. Lecture 1. Scribe: Tore Frederiksen
Lecture 1 Course Introduction Scribe: Tore Frederiksen 1.1 Overview Assume we are in a situation where several people need to work together in order to solve a problem they cannot solve on their own. The
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More informationMetric spaces and metrizability
1 Motivation Metric spaces and metrizability By this point in the course, this section should not need much in the way of motivation. From the very beginning, we have talked about R n usual and how relatively
More informationInformation Theory, Statistics, and Decision Trees
Information Theory, Statistics, and Decision Trees Léon Bottou COS 424 4/6/2010 Summary 1. Basic information theory. 2. Decision trees. 3. Information theory and statistics. Léon Bottou 2/31 COS 424 4/6/2010
More informationThe one-way communication complexity of the Boolean Hidden Matching Problem
The one-way communication complexity of the Boolean Hidden Matching Problem Iordanis Kerenidis CRS - LRI Université Paris-Sud jkeren@lri.fr Ran Raz Faculty of Mathematics Weizmann Institute ran.raz@weizmann.ac.il
More information1 Randomized Computation
CS 6743 Lecture 17 1 Fall 2007 1 Randomized Computation Why is randomness useful? Imagine you have a stack of bank notes, with very few counterfeit ones. You want to choose a genuine bank note to pay at
More informationLecture 8: Channel and source-channel coding theorems; BEC & linear codes. 1 Intuitive justification for upper bound on channel capacity
5-859: Information Theory and Applications in TCS CMU: Spring 23 Lecture 8: Channel and source-channel coding theorems; BEC & linear codes February 7, 23 Lecturer: Venkatesan Guruswami Scribe: Dan Stahlke
More informationUniformly Additive Entropic Formulas
Uniformly Additive Entropic Formulas Andrew Cross, Ke Li, Graeme Smith JILA and Department of Physics, University of Colorado Boulder QMATH2016 Georgia Tech October 9, 2016 Information theory: optimal
More informationAn approach from classical information theory to lower bounds for smooth codes
An approach from classical information theory to lower bounds for smooth codes Abstract Let C : {0, 1} n {0, 1} m be a code encoding an n-bit string into an m-bit string. Such a code is called a (q, c,
More informationAn instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if. 2 l i. i=1
Kraft s inequality An instantaneous code (prefix code, tree code) with the codeword lengths l 1,..., l N exists if and only if N 2 l i 1 Proof: Suppose that we have a tree code. Let l max = max{l 1,...,
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 2015 Abstract In information theory, Shannon s Noisy-Channel Coding Theorem states that it is possible to communicate over a noisy
More informationQB LECTURE #4: Motif Finding
QB LECTURE #4: Motif Finding Adam Siepel Nov. 20, 2015 2 Plan for Today Probability models for binding sites Scoring and detecting binding sites De novo motif finding 3 Transcription Initiation Chromatin
More informationOn the tightness of the Buhrman-Cleve-Wigderson simulation
On the tightness of the Buhrman-Cleve-Wigderson simulation Shengyu Zhang Department of Computer Science and Engineering, The Chinese University of Hong Kong. syzhang@cse.cuhk.edu.hk Abstract. Buhrman,
More informationAn Introduction to Quantum Computation and Quantum Information
An to and Graduate Group in Applied Math University of California, Davis March 13, 009 A bit of history Benioff 198 : First paper published mentioning quantum computing Feynman 198 : Use a quantum computer
More informationLecture 14 February 28
EE/Stats 376A: Information Theory Winter 07 Lecture 4 February 8 Lecturer: David Tse Scribe: Sagnik M, Vivek B 4 Outline Gaussian channel and capacity Information measures for continuous random variables
More informationLecture 13: Lower Bounds using the Adversary Method. 2 The Super-Basic Adversary Method [Amb02]
Quantum Computation (CMU 18-859BB, Fall 015) Lecture 13: Lower Bounds using the Adversary Method October 1, 015 Lecturer: Ryan O Donnell Scribe: Kumail Jaffer 1 Introduction There are a number of known
More informationInteractive Channel Capacity
Electronic Colloquium on Computational Complexity, Report No. 1 (2013 Interactive Channel Capacity Gillat Kol Ran Raz Abstract We study the interactive channel capacity of an ɛ-noisy channel. The interactive
More informationLecture 3: Lower bound on statistically secure encryption, extractors
CS 7880 Graduate Cryptography September, 015 Lecture 3: Lower bound on statistically secure encryption, extractors Lecturer: Daniel Wichs Scribe: Giorgos Zirdelis 1 Topics Covered Statistical Secrecy Randomness
More information