ESE 523 Information Theory

Transcription

1 ESE 53 Inforation Theory Joseph A. O Sullivan Sauel C. Sachs Professor Electrical and Systes Engineering Washington University 11 Urbauer Hall 10E Green Hall (Lynda Marha Answers) jao@wustl.edu

2 Outline Naes, introduction Course Description Tentative Schedule Topics

3 Course Description Textboo: Thoas M. Cover and Joy A. Thoas, Eleents of Inforation Theory, Second Edition, New Yor: Wiley and Sons, 006. Tie and place: TuTh 8:30-10:00 a.., Green Hall 0159 Office hours: TBD Catalog Description Discrete source and channel odel, definition of inforation rate and channel capacity, coding theores for sources and channels, encoding and decoding of data for transission over noisy channels. Corequisite: ESE 50 or equivalent. Proble Set Solutions Solutions will be ade available. Use of bac files of any ind in the solution of proble sets is strictly forbidden. See Course Policy Stateent.

4 Tentative Schedule Chapter 1 and preview, Lecture 1 Chapter : Entropy, relative entropy, and utual inforation 3 lectures Chapter 3: Asyptotic equipartition property, 1 lecture Chapter 4: Entropy rates of a stochastic process, lectures Chapter 5: Data copression, 3 lectures Chapter 6: Gabling and data copression, lectures Chapter 7: Channel capacity, 3 lectures Chapter 8: Differential entropy, 1 lecture Chapter 1: Maxiu entropy, 1 lecture Chapter 9: Gaussian channel capacity, lectures Chapter 10: Rate-distortion theory, lectures Chapter 11: Inforation theory and statistics, 4 lectures Chapter 13: Universal source coding, lectures Chapter 15: Networ inforation theory, 1 lecture Total: 8 lectures?!

5 Schedule Coents Mostly we wor straight through the boo. We sip soe sections. Chapter 6: Gabling is fun Chapter 14: Kologorov coplexity is hard Chapter 15: Networ inforation theory continues to be a hot research area J. L. Kelly, Jr.; See also Ed Thorp Andrey Kologorov

6 What is inforation theory? Inforation Theory contributes to and draws results fro a nuber of other fields including: Counication Theory Probability Theory Statistics: Estiation and Detection Theory Physics Coputer Science Econoics

7 Origins in Counication Theory Shannon The Matheatical Theory of Counication. BSTJ. Channel capacity: fundaental bound on bits per second; channel coding deals with approaching this rate Source encoding: bound on nuber of bits to copress a source Rate-distortion theory: bound on nuber of bits to copress a source subject to a distortion constraint Separation theore Source Source Encoder Channel Encoder Channel Modulator Channel User Source Decoder Channel Decoder Channel Deodulator

8 Coplexity Theory Tie-coplexity Space-coplexity Kologorov coplexity More closely related to inforation theory and equals the length of the sallest progra that can be written to reproduce the data. This is not coputable. For rando data, this quantity is approxiately equal to the entropy.

9 Physics In statistical physics (statistical echanics), the fundaental quantities studied are energy and entropy Second Law: Entropy of systes ust increase (subject to other constraints) Quantu Inforation Theory Inforation theory and blac holes: apparently inforation is preserved in the event horizon

10 Statistics The Fisher inforation deterines a lower bound on the variance of any estiator. Relative entropy deterines the exponent in the probability of error in optial detection algoriths as the data set grows. Large deviations quantifies the probability of rare events.

11 Cobinatorics Given a discrete alphabet with eleents n independent and identically distributed (i.i.d.) eleents are drawn (sapling with replaceent) How any ways are there to get 1,, of the eleents ( =n)? Call this a type. What is the probability of getting one outcoe in such a type? What is the probability of getting a type? What type is ost liely? What is the probability of getting a type far fro the ost liely?

12 Cobinatorics: Answers part 1 How any ways are there to get 1,, of the eleents ( =n)? What is the probability of getting one outcoe in such a type? Lea: 1 ln! ln 1 n n! !!...! ln! i1 1 ln ln ln 1 1 i1 i1 ln i ln 1 ln i ( 1) ln( 1) ( 1) 1 i1 ln i ln 1 o( ) x dx i x dx

13 Cobinatorics: Answers part 1 How any ways are there to get 1,, of the eleents ( =n)? What is the probability of getting one outcoe in such a type? Entropy of epirical distribution deterines the nuber of eleents in a type (exponent is the entropy) Probability of one outcoe is deterined August 30, 011 by the type and J. A. the O Sullivan, ESE 53 Lecture 1 probability of each outcoe n n! !!...! 1 1 n log log... log o( n) n n n n n n 1 nh,,..., no( n) n n n 1 h p, p,..., p p log p q q q i i i1 n p log q p log q... p log q 1 1 p1 p p n p1 log p log... plog h p1, p,..., p q1 q q 1 p1 p... p n n n

14 Cobinatorics : Answers part What is the probability of getting a type? Answer: deterined by relative entropy between type and probability What type is ost liely? Answer: p=q What is the probability of getting a type far fro the ost liely? Answer: deterined by the relative entropy to the closest type in the unliely set Relative entropy between epirical distribution and probability distribution deterines probability of a type (exponential in relative entropy) q 1 p 1 1 q nd p q n ind p 0 n p... q 1 1 n p log q 1 1 p no( n) p p q no( n) 1 p log q n n p... p log q p o( n) n

15 Cobinatorics : Experients Bernoulli p (binary, P(X i =1)=p 50 i.i.d. draws Repeat 1,000,000 ties See Matlab Code

16 Matlab Code function [histout]=binhist(p,n,) % histout=binhist(p,n,) % p=bernoulli probability % n=length of vector % =nuber of trials % Progra runs trials and copares the noralized log histogra % to the relative entropy that bounds the probability of large devaiations. % % This file was created by Joseph A. O'Sullivan % August 8, 003 % copyright 003 x=rand(n,); y=(sign(x-1+p)+1)/; y=su(y,1)/n; q=0:1/n:1; histout=hist(y,q); figure subplot(,,1) plot([1:],y) xlabel('trial Nuber') ylabel('fraction of Successes') continued

17 Matlab Code subplot(,,) plot(q,histout) xlabel('fraction of Successes') ylabel('histogra') titlestring=['probability: ', nustr(p), ' Length: ', nustr(n), ' Trials: ', nustr()]; title(titlestring) subplot(,,3) loghist=ax(in(log(p),log(1-p)),log(histout/)/n); plot(q,loghist) xlabel('fraction of Successes') ylabel('noralized Log-Histogra') subplot(,,4) dpq=q(:n).*log(q(:n)/p)+(1-q(:n)).*log((1-q(:n))/(1-p)); dpq=[-log(1-p) dpq]; dpq=[dpq -log(p)]; plot(q,dpq,'r') hold on plot(q,-loghist) xlabel('fraction of Successes') ylabel('d(q p) and Negative Log-Histogra')

18 Noralized Log-Histogra D(q p) and Negative Log-Histogra Fraction of Successes Histogra Exaple Result Trial Nuber Probability: 0.3 Length: 100 Trials: Fraction of Successes 1.5 Histogra of fraction of successes loos reasonable Probability of large deviations is exponentially sall Fraction of Successes Fraction of Successes