ELEMENTS O F INFORMATION THEORY

Size: px

Start display at page:

Download "ELEMENTS O F INFORMATION THEORY"

Marianna Cross
5 years ago
Views:

1 ELEMENTS O F INFORMATION THEORY THOMAS M. COVER JOY A. THOMAS

3 Preface to the Second Edition Preface to the First Edition Acknowledgments for the Second Edition Acknowledgments for the First Edition x v xvi i xx i xxii i 1 Introduction and Preview Preview of the Book 5 2 Entropy, Relative Entropy, and Mutual Information Entropy Joint Entropy and Conditional Entropy Relative Entropy and Mutual Information Relationship Between Entropy and Mutua l Information Chain Rules for Entropy, Relative Entropy, and Mutual Information Jensen's Inequality and Its Consequences Log Sum Inequality and Its Applications Data-Processing Inequality Sufficient Statistics Fano's Inequality 3 7 Summary 4 1 Problems 43 Historical Notes 54

4 3 Asymptotic Equipartition Property Asymptotic Equipartition Property Theorem Consequences of the AEP: Data Compression High-Probability Sets and the Typical Set 62 Summary 64 Problems 64 Historical Notes 69 4 Entropy Rates of a Stochastic Process Markov Chains Entropy Rate Example: Entropy Rate of a Random Walk on a Weighted Graph Second Law of Thermodynamics Functions of Markov Chains 84 Summary 87 Problems 8 8 Historical Notes Data Compression Examples of Codes Kraft Inequality Optimal Codes Bounds on the Optimal Code Length Kraft Inequality for Uniquely Decodabl e Codes Huffman Codes Some Comments on Huffman Codes Optimality of Huffman Codes Shannon-Fano-Elias Coding Competitive Optimality of the Shanno n Code Generation of Discrete Distributions from Fair Coins 134 Summary 14 1 Problems 142 Historical Notes 157

5 6 Gambling and Data Compression The Horse Race Gambling and Side Information Dependent Horse Races and Entropy Rate The Entropy of English Data Compression and Gambling Gambling Estimate of the Entropy of English 17 3 Summary 17 5 Problems 17 6 Historical Notes Channel Capacity Examples of Channel Capacity Noiseless Binary Channel Noisy Channel with Nonoverlappin g Outputs Noisy Typewriter Binary Symmetric Channel Binary Erasure Channel Symmetric Channels Properties of Channel Capacity Preview of the Channel Coding Theorem Definitions Jointly Typical Sequences Channel Coding Theorem Zero-Error Codes Fano's Inequality and the Converse to the Codin g Theorem Equality in the Converse to the Channel Coding Theorem Hamming Codes Feedback Capacity Source-Channel Separation Theorem 21 8 Summary 22 2 Problems 22 3 Historical Notes 240

6 8 Differential Entropy Definitions AEP for Continuous Random Variables Relation of Differential Entropy to Discrete Entropy Joint and Conditional Differential Entropy Relative Entropy and Mutual Information Properties of Differential Entropy, Relative Entropy, and Mutual Information 252 Summary 25 6 Problems 256 Historical Notes Gaussian Channel Gaussian Channel : Definitions Converse to the Coding Theorem for Gaussian Channels Bandlimited Channels Parallel Gaussian Channels Channels with Colored Gaussian Noise Gaussian Channels with Feedback 28 0 Summary 28 9 Problems 290 Historical Notes Rate Distortion Theory Quantization Definitions Calculation of the Rate Distortion Function Binary Source Gaussian Source Simultaneous Description of Independen t Gaussian Random Variables Converse to the Rate Distortion Theorem Achievability of the Rate Distortion Function Strongly Typical Sequences and Rate Distortion Characterization of the Rate Distortion Function 329

7 10.8 Computation of Channel Capacity and the Rate Distortion Function 33 2 Summary 33 5 Problems 33 6 Historical Notes Information Theory and Statistics Method of Types Law of Large Numbers Universal Source Coding Large Deviation Theory Examples of Sanov's Theorem Conditional Limit Theorem Hypothesis Testing Chernoff-Stein Lemma Chernoff Information Fisher Information and the Cramer-Ra o Inequality 39 2 Summary 39 7 Problems 399 Historical Notes Maximum Entropy Maximum Entropy Distributions Examples Anomalous Maximum Entropy Problem Spectrum Estimation Entropy Rates of a Gaussian Process Burg's Maximum Entropy Theorem 41 7 Summary 420 Problems 42 1 Historical Notes Universal Source Coding Universal Codes and Channel Capacity Universal Coding for Binary Sequences Arithmetic Coding 436

8 13.4 Lempel-Ziv Coding Sliding Window Lempel-Ziv Algorithm Tree-Structured Lempel-Ziv Algorithms Optimality of Lempel-Ziv Algorithms Sliding Window Lempel-Ziv Algorithms Optimality of Tree-Structured Lempel-Zi v Compression 44 8 Summary 45 6 Problems 45 7 Historical Notes Kolmogorov Complexity Models of Computation Kolmogorov Complexity: Definitions and Examples Kolmogorov Complexity and Entropy Kolmogorov Complexity of Integers Algorithmically Random and Incompressible Sequences Universal Probability Kolmogorov complexity Q Universal Gambling Occam's Razor Kolmogorov Complexity and Universa l Probability Kolmogorov Sufficient Statistic Minimum Description Length Principle 50 0 Summary 50 1 Problems 50 3 Historical Notes Network Information Theory Gaussian Multiple-User Channels 513

9 Single-User Gaussian Channel Gaussian Multiple-Access Channe l with in Users Gaussian Broadcast Channel Gaussian Relay Channel Gaussian Interference Channel Gaussian Two-Way Channel Jointly Typical Sequences Multiple-Access Channel Achievability of the Capacity Region for the Multiple-Access Channel Comments on the Capacity Region for th e Multiple-Access Channel Convexity of the Capacity Region of th e Multiple-Access Channel Converse for the Multiple-Acces s Channel m-user Multiple-Access Channels Gaussian Multiple-Access Channels Encoding of Correlated Sources Achievability of the Slepian-Wolf Theorem Converse for the Slepian-Wolf Theorem Slepian-Wolf Theorem for Man y Sources Interpretation of Slepian-Wol f Coding Duality Between Slepian-Wolf Encoding an d Multiple-Access Channels Broadcast Channel Definitions for a Broadcast Channel Degraded Broadcast Channels Capacity Region for the Degraded Broadcas t Channel Relay Channel Source Coding with Side Information Rate Distortion with Side Information 580

10 15.10 General Multiterminal Networks 58 7 Summary 594 Problems 59 6 Historical Notes Information Theory and Portfolio Theory The Stock Market: Some Definitions Kuhn-Tucker Characterization of the Log-Optima l Portfolio Asymptotic Optimality of the Log-Optima l Portfolio Side Information and the Growth Rate Investment in Stationary Markets Competitive Optimality of the Log-Optima l Portfolio Universal Portfolios Finite-Horizon Universal Portfolios Horizon-Free Universal Portfolios Shannon-McMillan-Breiman Theorem (General AEP) 644 Summary 65 0 Problems 65 2 Historical Notes Inequalities in Information Theory Basic Inequalities of Information Theory Differential Entropy Bounds on Entropy and Relative Entropy Inequalities for Types Combinatorial Bounds on Entropy Entropy Rates of Subsets Entropy and Fisher Information Entropy Power Inequality and Brunn-Minkowsk i Inequality Inequalities for Determinants 679

11 17.10 Inequalities for Ratios of Determinants 68 3 Summary 68 6 Problems 68 6 Historical Notes 68 7 Bibliography 68 9 List of Symbols 723 Index 727

ELEMENTS OF INFORMATION THEORY

ELEMENTS OF INFORMATION THEORY ELEMENTS OF INFORMATION THEORY Second Edition THOMAS M. COVER JOY A. THOMAS A JOHN WILEY & SONS, INC., PUBLICATION Copyright 2006 by John Wiley & Sons, Inc. All rights reserved.