Outline of the Lecture. Background and Motivation. Basics of Information Theory: 1. Introduction. Markku Juntti. Course Overview

Transcription

1 : Markku Juntti Overview The basic ideas and concepts of information theory are introduced. Some historical notes are made and the overview of the course is given. Source The material is mainly based on Chapter 1 of the course book [1]. Outline of the Lecture Background and motivation Introduction Application areas Basic concepts Self-information Entropy Mutual information Historical notes Summary Course overview Telecomm. Laboratory 1 Telecomm. Laboratory 2 Background and Motivation The discipline of information theory (IT) was originally created to explain the behavior of communication systems. Shannon s landmark paper 1948 [5] the start of information theory as a field of its own not just an spin-off of communication theory, but a discipline of its own. Applications also in many fields: source coding and data compression (in the original paper [5] already) statistics and statistical signal processing game theory stock market. Course Overview The emphasis on this course is on communication applications, but the other important fields are also covered. The course follows closely the structure of [1], which is the textbook and main exam requirement. Some additional material will be included from [3]. The trigger of the creation of information theory [5] illustrates the very origins of the discipline. Telecomm. Laboratory 3 Telecomm. Laboratory 4

2 Motivation (1) Information theory provides basic tools and understanding in many fields. Information theory is applied mathematics and statistics rather than pure engineering abstract, builds upon previously learned concepts engineering curriculum does not prepare very well for IT requires work to learn. Solving homework problems is necessary. Continuous reading is a must! Motivation (2) After learning very useful insight is achieved: There are a few key ideas and techniques that, when mastered, make the subject appear simple and provide great intuition on new questions [1, p. vii]. Information theory is still developing rapidly as a field of science. It is a key tool used in practical research and development of real engineering problems in several disciplines. Telecomm. Laboratory 5 Telecomm. Laboratory 6 Introduction Origins in analyzing the limits of communications. Two fundamental questions in communication theory: What is the ultimate data compression rate? Answer: entropy. What is the ultimate data transmission rate? Answer: channel capacity. compression limit H ( X ) ( X ; ˆ X ) min I max I ( X ; Y ) Rate distortion function Channel capacity transmission limit Application Areas Communication theory data compression limits: entropy data transmission limits: channel capacity suggestions, but no practical schemes to obtain the limits recent work: network or multiuser information theory several open problems still exist Coding theory schemes to obtain the limits source coding quantization and rate distortion theory channel coding multiuser coding theory is rather unsolved problem. Telecomm. Laboratory 7 Telecomm. Laboratory 8

3 Application Areas (2) Computer science complexity of presenting a string of data: Kolmogorov complexity Kolmogorov complexity is even more fundamental than Shannon entropy. Thermodynamics birthplace of entropy second law of thermodynamics Probability theory and statistics quantities of IT: functionals of probability distributions hypothesis testing and estimation Signal processing Application Areas (3) Philosophy of science The simplest explanation is the best. Investment (economics) Optimal investment strategy parallels IT. Computation larger computers out of smaller components computation limit and communication limit. Telecomm. Laboratory 9 Telecomm. Laboratory 10 Basic Concepts How to measure the information contents of a data source? How to find a tractable model for a data source? The model should be general enough to cover a variety of different data sources and to be useful for practice. voice data video images etc. Model the data and associated signals statistically. realizations as random variables. signals as stochastic processes. Properties of a Measure of Information Assume that the source output is a discrete time and discrete valued random variable (RV) X with probability mass function p(x). A good measure of information content should have the properties: The larger the more surprising the outcome (random variable realization) is functional of probability distribution proportional to the inverse of probability. Information content of two independent RV s is the sum of information contents (intuitively pleasing) logarithm of probability distribution. Telecomm. Laboratory 11 Telecomm. Laboratory 12

4 Self-Information The self-information of a realization x of RV X can be defined as 1 i ( x ) = log = log[ p ( x )]. p ( x ) The basis of logarithm can be any 2 information measured in binary units (bits) e information measured in natural units (nats). Example: Assume a source with M possible symbols. If M = 2 k, the symbols can be described by k = log 2 M bits. Example - M = 4: A=00, B=01, C=10, D=11. Entropy The average information of RV X is called the entropy of X H ( X ) = p ( x ) i ( x ) = p ( x ) log[ p ( x )]. x Entropy is a functional of the probability distribution independent of the outcomes or of the RV itself only the distribution matters. Note that 1 H ( X ) = E[ log p ( X )] = E log. ( ) p X H X 0 ( ). Entropy describes the ultimate data compression rate. x Telecomm. Laboratory 13 Telecomm. Laboratory 14 Example #1: Uniform RV Consider a uniform random variable with M = 32 = 2 5 possible outcomes. Five (5) bits suffice to describe each outcome: 00000, 00001, 00010,, 11110, Entropy: H ( X ) = p ( i ) log2[ p ( i )] = log i = i = = log ( 32) = 5. Entropy and the log(m) agree. 2 Example #2: Non-Uniform RV (1) Consider a non-uniform random variable with M = 8 = 2 3 possible outcomes 1, 2,, 8 and probabilities (,,,,,,, ) Three (3) bits suffice surely to describe each outcome: 1 000, 2 001, 3 010, The source code word length is log 2 (M) = log 2 (8) = 3 bits. Telecomm. Laboratory 15 Telecomm. Laboratory 16

5 Example #2: Non-Uniform RV (2) Entropy: H ( X ) 1 = log log log2 log log = 2 bits. Entropy and the log 2 (M) disagree: H(X) log 2 (M). The average source code word length can be made shorter than 3. Just an example: 1 0, 2 10, 3 110, , , , , Average length: = 2 bits = H ( X ) Mutual information Mutual Information ( ) p x, y I ( X ; Y ) = H ( X ) H ( X Y ) = p ( x, y ) log ( ) ( ) x y p x p y is a measure of the information one random variable (say, X) contains on the other (Y). Special case of relative entropy known also as Kullback-Leibler distance, cross entropy, information divergence, information for discrimination. Maximum mutual information is the ultimate data transmission rate or the channel capacity: C = max p ( x ) [ I ( X ; Y )].. Telecomm. Laboratory 17 Telecomm. Laboratory 18 Historical Notes General belief in early 1940 s: Increasing data transmission rate over a communication channel increases inavoidably the probability of error. Forward error control coding and its limits were not understood. Digital communication techniques were in their infancy. Claude Shannon surprised the scientific community and changed the whole basic paradigm by his landmark paper [5] in Birth of information theory one of the few disiplines with such a clear start. The same year as the invention of transistor! Communications Before s: telegraph by Morse Morse code is a digital and rather good one. 1876: telephone by Bell 1897: wireless telegraph by Marconi Early 1900 s: AM radio 1922: single-sideband modulation by Carson : television 1931: teletype 1936: frequency modulation by Armstrong : pulse-code modulation (PCM) by Reeves 1939: vocoder by Dudley 1940 s: spread spectrum Telecomm. Laboratory 19 Telecomm. Laboratory 20

6 Entropy, Relative Entropy and Channel Capacity The concept of entropy originates in thermodynamics from 1800 s: the second law. Boltzmann: connection between the macroscopic concept of entropy and the microscopic state of the system. 1924: Nyquist: rate ~ log of number of signal levels. 1928: logarithmic measure of information by Hartley: a logarithm of the alphabet size s: Shannon defined entropy as a logarithm of the probability, and derived the landmark results on data compression and transmission as well as rate distortion theory in 1948 appeared paper [5]. Some results were not yet fully proved therein. Relative entropy by Kullback and Leibler Some Consequences The birth of information theory launched development in the field itself. Boost in communication and coding theory. The whole modern information technology is based on the paradigm change provided by information theory. A key ingredient in the rapid development of the recent decades together with the invention of semiconductor technology. Telecomm. Laboratory 21 Telecomm. Laboratory 22 Summary Origins in analyzing the limits of communications. Two fundamental questions in communication theory: What is the ultimate data compression rate? Answer: entropy. What is the ultimate data transmission rate? Answer: channel capacity. compression limit H ( X ) ( X ; ˆ X ) min I max I ( X ; Y ) Rate distortion function Channel capacity transmission limit Telecomm. Laboratory 23 Course Overview Basic concepts and tools 1 Introduction 2 Entropy, relative entropy and mutual information 3 Asymptotic equipartition property 4 Entropy rates of a stochastic process Source coding or data compression 5 compression Channel capacity 8 Channel capacity 9 Differential entropy 10 The Gaussian channel Other applications 11 Maximum entropy and spectral estimation 13 Rate distortion theory 14 Network information theory Telecomm. Laboratory 24