A Simple Introduction to Information, Channel Capacity and Entropy

Transcription

1 A Simple Introduction to Information, Channel Capacity and Entropy Ulrich Hoensch Rocky Mountain College Billings, MT Friday, April 21, 2017

2 Introduction A frequently occurring concept in mathematics (e.g. in cryptography and dynamical systems) is that of entropy. Given a probability distribution p 1, p 2,..., p n, entropy is defined as H(p 1, p 2,..., p n ) = n p i log(p i ). (1) i=1 Thermodynamic Interpretation (not our approach) Information-theoretic Interpretation: Claude Shannon ( ) Worked in electrical engineering, cryptography Father of Information Theory A Mathematical Theory of Communication (1948) [3]

3 What s Confusing About Entropy? Not its mathematical definition: H(p 1, p 2,..., p n ) = n p i log(p i ). i=1 But its interpretation as a measure of: information (Shannon) missing information (Ben-Naim [1]) disorder, mixing etc. diversity (Shannon-Weaver Index) Entropy is usually introduced as having certain desirable mathematical properties related to any of the measures above.

4 Low Entropy Room (Ben-Naim [2]) Image retrieved from amazon.com

5 High Entropy Room (Ben-Naim [2]) Image retrieved from amazon.com

6 Example 1 Suppose we want to transmit a message consisting of the letters A, B, C and D over a channel by using only 0 s or 1 s. For example, suppose the message is CADBAACBAADABBABAABD (20 characters). We can code the symbols using two bits, for example as Then the coded message is A 00, B 01, C 10, D (40 bits). This means, we would need 2 bits per letter, and a text with N letters would then require 2N bits.

7 Example 1 Suppose the probabilities of the letters A, B, C and D are P(A) = 1 2, P(B) = 1 4, P(C) = 1 8, P(D) = 1 8. Use the coding P(A) = 1 2 P(B) = 1 4 P(C) = 1 8 P(D) = 1 8 first bit: 0 first bit: 1 A 0 2nd bit: 0 2nd bit: 1 B 10 3rd: 0 3rd: 1 C 110 D 111 To use as few bits as possible, letters that occur more frequently have shorter codes.

8 Example 1 Using this coding, the message CADBAACBAADABBABAABD (20 characters), is now coded as (36 bits). We can say that given the coding above, less information (number of bits) is required to code (and transmit) the same message. Question: How can we predict how many bits we will need to code a message?

9 Example 1 Let s compute the expected value of the random variable X : number of bits needed for each letter. Since P(X = 1) = 1 2, P(X = 2) = 1 4, P(X = 3) = 1 4, the expected value is E(X ) = = 7 4 = The central observation is the following. E(X ) = = 1 2 log log log log 2 8 = 1 2 log log log log 1 2 ( 8 1 = H 2, 1 4, 1 8, 1 ). 8

10 Example 1 So, entropy = expected number of bits If the distribution is uniform, the entropy is ( 1 H 4, 1 4, 1 4, 1 ) = log log log log = log 2 (4) = 2, so we need 2N bits. Note: If p i = 1/n, then the entropy is log 2 n, and it can be seen that maximal entropy occurs precisely in this situation.

11 Example 1 The redundancy of the distribution (p 1, p 2,..., p n ) is defined as R(p 1, p 2,..., p n ) = log 2 n H(p 1, p 2,..., p n ). (2) log 2 n In our example the redundancy is ( 1 R 2, 1 4, 1 8, 1 ) = log 2 4 H ( 1 2, 1 4, 1 8, 1 ) 8 8 log 2 4 = = This means we need 12.5% less channel capacity using the adapted coding, when compared to the coding that assumes each letter occurs with the same frequency.

12 so we should expect to need only about 0.8N bits for an N-letter text. Example 2 Consider the following distribution for the two-letter alphabet {A, B}: P(A) = 3 4, P(B) = 1 4. Suppose we use the coding A 0, B 1. Given the message AABAABAAAAAAAAABAABAAABBABABAA (30 characters), the coded message is (30 bits). The number of bits needed to code an N-letter text is N, but according to equation (1), the entropy is ( 3 H 4, 1 ) = log 2(3/4) 1 4 log 2(1/4) ,

13 Example 2 To reduce the channel capacity, consider all digrams and their probabilities. If we assume that the letters occur independently, then P(AA) = 9 3, P(AB) = P(BA) = 16 16, P(BB) = We can provide a coding for digrams as follows: P(AA) 8 16 = 1 2 P(AB) 4 16 = 1 4 P(BA) 2 16 = 1 8 P(BB) 2 16 = 1 8 So: AA 0, AB 10, BA 110, BB 111.

14 Example 2 first bit: 0 first bit: 1 AA 0 2nd bit: 0 2nd bit: 1 AB 10 3rd: 0 3rd: 1 BA 110BB 111 The message AABAABAAAAAAAAABAABAAABBABABAA (30 characters), is then coded as (25 bits).

15 Example 2 The expected number of bits for the digrams is E(X ) = = = This translates into a channel capacity of (N/2) 0.84N for an N-letter text using the characters A and B. By using a coding for digrams, we have reduced the channel capacity from N bits for an N-letter text to about 0.84N bits which is closer to the ideal channel capacity of N bits. It is expected that by coding e.g. trigrams, 4-grams, etc., the channel capacity will approach the value given by the entropy of the probability distribution. This is shown in [3].

16 Single-letter frequencies in the English language [5] Letter Relative Frequency Letter Relative Frequency A N B O C P D Q E R F S G T H U I V J W K X L Y M Z

17 Entropy and Redundancy of the English Language We get the entropy H(p 1,..., p 26 ) = , (3) and have a smaller entropy than ( 1 H 26,..., 1 ) = log 26 2 (26) = which assumes uniform distribution of all letters. Indeed, using the probabilities in the table above, which gives a redundancy of R = As mentioned before, we can interpret this number as saying that with an efficient coding, we can achieve an 11% reduction in the channel capacity.

18 Entropy and Redundancy of the English Language In order to achieve entropy of with a uniform distribution, we would only need = letters, not 26 (since log 2 ( ) = ). We are really considering a very simple model for the English language by only taking single-letter frequencies into account. In fact, under this model, any word (for example QWYURG ) is possible. More precise models using digrams or trigrams or multiple words indicate a redundancy of 50% for the English language (see [3]).

19 Conclusion Given entropy = expected number of bits Hopefully, entropy isn t that confusing any more; it is clearer as to why it is used as a measure of information. Questions?

20 References A. Ben-Naim, A Farewell to Entropy: Statistical Thermodynamics Based on Information, World Scientific Publishing Company, A. Ben-Naim, Discover Entropy and the Second Lay of Thermodynamics: A Playful Way of Discovering a Law of Nature, World Scientific Publishing Company, C. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, Vol. 27, pp , , July, October, C. Shannon, Communication Theory of Secrecy Systems, Bell System Technical Journal, Vol. 28, pp , R. E. Lewand, Cryptological Mathematics MAA Press, 2000.