Example: Letter Frequencies

Size: px

Start display at page:

Download "Example: Letter Frequencies"

Paula Hamilton
6 years ago
Views:

1 Example: Letter Frequencies i a i p i 1 a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y z Figure 2.1. Probability distribution over the 27 outcomes for a randomly selected letter in an English language document (estimated from The Frequently Asked Questions Manual for Linux ). The picture shows the probabilities by the areas of white squares. Book by David MacKay

2 Example: Letter Frequencies i a i p i 1 a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y z Figure 2.1. Probability distribution over the 27 outcomes for a randomly selected letter in an English language document (estimated from The Frequently Asked Questions Manual for Linux ). The picture shows the probabilities by the areas of white squares. x a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y z Figure 2.2. The probability distribution over the possible bigrams xy in an English language document, The Frequently Asked Questions Manual for Linux. Book by David MacKay y

Example: Surprisal Values from http://www.umsl.edu/~fraundorfp/egsurpri.html i a i p i h(p i ) 1 a.0575 4.1 2 b.0128 6.3 3 c.0263 5.2 4 d.0285 5.1 5 e.0913 3.5 6 f.0173 5.9 7 g.0133 6.2 8 h.0313 5.

3 Example: Surprisal Values from i a i p i h(p i ) 1 a b c d e f g h i j k l m n o p q r s t u v w x y z i p i log 2 1 p i 4.1 Table 2.9. Shannon information contents of the outcomes a z. Book by David MacKay

4 MacKay s Mnemonic convex concave

5 MacKay s Mnemonic convex concave

6 MacKay s Mnemonic convex concave

7 MacKay s Mnemonic convex convec-smile concave

8 MacKay s Mnemonic convex convec-smile concave conca-frown

9 Examples: Convex & Concave Functions x Book by David MacKay

10 Examples: Convex & Concave Functions x Book by David MacKay

11 Examples: Convex & Concave Functions x 2 e x Book by David MacKay

12 Examples: Convex & Concave Functions x 2 e x log 1 x Book by David MacKay

13 Examples: Convex & Concave Functions x 2 e x log 1 x x log x Book by David MacKay

14 Examples: Convex & Concave Functions x 2 e x log 1 x x log x Book by David MacKay

15 Binary Entropy Function H 2 (x) x Figure 1.3. The binary entropy function. Book by David MacKay

16 Order These in Terms of Entropy ECE 534 by Natasha Devroye

17 Order These in Terms of Entropy ECE 534 by Natasha Devroye

18 Mutual Information and Entropy Theorem: Relationship between mutual information and entropy. I(X; Y ) = H(X) H(X Y ) I(X; Y ) = H(Y ) H(Y X) I(X; Y ) = H(X)+H(Y ) H(X, Y ) I(X; Y ) = I(Y ; X) (symmetry) I(X; X) = H(X) ( self-information ) H(X) H(Y) H(Y) H(X Y) H(X) H(Y) I(X;Y) I(X;Y) I(X;Y) ECE 534 by Natasha Devroye

19 Chain Rule for Entropy Theorem: (Chain rule for entropy): (X 1,X 2,..., X n ) p(x 1,x 2,..., x n ) H(X1) H(X2) H(X 1,X 2,..., X n )= n i=1 H(X i X i 1,..., X 1 )! H(X3) H(X1) H(X2 X1) H(X1,X2,X3) = + + H(X3 X1,X2) ECE 534 by Natasha Devroye

20 Chain Rule for Mutual Information Theorem: (Chain rule for mutual information) I(X 1,X 2,..., X n ; Y )= n i=1 I(X i ; Y X i 1,X i 2,..., X 1 ) H(X) H(Y) H(X) H(Y) H(X) H(Y) I(X,Y;Z) = I(X,;Z) + I(Y,;Z X) H(Z) H(Z) H(Z) ECE 534 by Natasha Devroye

21 What are the Grey Regions? H(X) H(Y) H(X) H(Y) H(Z) H(Z) ECE 534 by Natasha Devroye

Example: Letter Frequencies

Example: Letter Frequencies i a i p i 1 a 0.0575 2 b 0.0128 3 c 0.0263 4 d 0.0285 5 e 0.0913 6 f 0.0173 7 g 0.0133 8 h 0.0313 9 i 0.0599 10 j 0.0006 11 k 0.0084 12 l 0.0335 13 m 0.0235 14 n 0.0596 15 o