Physics 239/139 Spring 2018 Assignment 3 Solutions

Transcription

1 University of California at San Diego Department of Physics Prof. John McGreevy Physics 239/139 Spring 2018 Assignment 3 Solutions Due 12:30pm Monday, April 23, Mutual information bounds correlations. Consider ( again the distribution on two binary variables from last homework: 0 a p yx =, where y = 1, 1 is the row index and x = 1, 1 is the column b b yx index (so yx are like the indices on a matrix. Normalization implies xy p xy = a + 2b = 1, so we have a one-parameter family of distributions, labelled by b. (a I ve changed the labels on the variables from, to 1, 1 so that we can consider correlation functions, such as the connected two-point function C xy c xy x y where A xy p yxa. Compute C as a function of b. C = xy ( ( p xy xy p x x p y y = a+a(a 2b a=1 2b = 4b(2b 1. x y (b Compute the mutual information between X and Y I(X : Y = xy p yx log p yx p y p x. I(X : Y = xy p xy log p ( xy = 0 + b log p x p y a a(a + b + a log b 2b + b log 2 b a=1 2b. (a + b2b (c Check that I(X : Y 1 2 C2 for every value of b (for example, plot both functions. 1

2 (d [Bonus] The inequality I quoted in lecture, and which we will prove in the more general quantum case later, is I(X : Y 1 O X O Y 2 c 2 O X 2 O Y where the norms are defined (in the classical case by O X 2 sup p x px=1 { x O xo x p x }. Show that in the above example, the operators x, y are normalized, in the sense that x = y = 1. The functions were ±1, so their maximum absolute squares are just Strong subadditivity, the classical case. [From Barnett] Prove strong subaddivity of the Shannon entropy: for any distribution on three random variables, H(ABC + H(B H(AB + H(BC. (The corresponding statement about the von Neumann entropy is not so easy to show. Hint: q(a, b, c p(a,bp(b,c p(b ABC. is a perfectly cromulent probability distribution on What is the name for the situation when equality holds? Write the condition for equality in terms of the conditional mutual information I(A : C B. The relative entropy is positive: 0 D(p(ABC q(abc = abc p(abc (log p(abc log p(abp(bc + log p(b 2

3 = H(ABC ab p(ab log p(ab bc log p(bc + b log p(b = H(ABC + H(AB + H(BC H(B. The ease with which that just happened contrasts stiffly with the quantum case. 3. Symbol coding problem. You are a mad scientist, but a sloppy one. You have 127 identical-looking jars of liquid, and you have forgotten which one is the poison one. You have at your disposal 7 rats on whom your poor moral compass will allow you to test the liquids. However (the rats have a strong social network and excellent spies you only get one shot: the rats must drink all at once (or they will catch on to what is happening and revolt. You may mix the liquids in separate containers; any rat that drinks any amount of poison will turn bright orange. Design a protocol to uniquely identify the poison jar. Number the jars 1 to 127, and write these numbers in binary, so 1 = , 2 = , = Number the rats 1 to 8. Feed a little bit of the liquid from jar number N = s 1..s 8 to the rats i with s i Huffman code. Make the Huffman code for the probability distribution p(x =.5,.2,.15,.1,.05. Compare the average word length to the Shannon entropy. Bonus: what property of the distribution determines the deviation from optimality? Using the coarse-graining steps (time goes to the left : I find the codewords 0, 11, 100, 1010, My conventions are: the less probable element gets the 1, I sort the list at each step, if there s a tie I do not switch the order. Different conventions will lead to different codewords. The Shannon entropy is and the average code length is Huffman code decryption problem. [Optional, but fun.]

4 ? , ( : Hint: I used the letter frequencies from The Origin of Species. You might want to use Mathematica to do this problem! do you think it is ethical for me to assign a homework problem that is just a string of bits? think of it like a treasure hunt. although the huffman code is an optimal symbol code, stream codes (mackay chapter six are better than any symbol code because they are more flexible. research problem: find an algorithm for solving quantum many body systems inspired by arithmetic codes and lempel-ziv codes. 6. Analogy with strong-disorder RG. [open ended, more optional question] Test or decide the following consequence suggested by the analogy between Huffman coding and strong-disorder RG: The optimality of the Huffman code is better 4

5 when the distribution is broader. A special case is the claim that the Huffman code is worst when all the probabilities are the same. Note that the outcome of the Huffman algorithm in this case depends on the number of elements of the alphabet. Measure the optimality by l H[p] (or maybe l H[p] H[p]?. It is possible to make some pretty pictures this way. I find that the general trend is in the direction predicted by the SDRG intuition, but with lots of numbertheoretic features which deviate from it. Here is deviation from optimal average codeword length for random distributions on four letters, 10 5 on five letters, on eight letters: and on 13 and 16 letters each: Powers of two are special because then it is possible to saturate the bound with a Huffman code, as in the initial Shannon example I showed in lecture. This was a successful problem if only in that these pictures look like murmurations. 7. Binary symmetric channel. For the binary symmetric channel ABE defined in lecture, with a, b, e {0, 1}, and p(a = (p, 1 p a, p(e = (q, 1 q e, and b = (a + e 2, find all the quantities p(a, b, p(b, p(b a, p(a b and H(B, H(B A, I(B : A, I(B : A E. Find the channel capacity. I m sorry for switching the convention that q is the error probability. We have: P (b = ae p(aq(eδ b,(a+e2. 5

6 where p(a = (p, 1 p a, q(e = (q, 1 q e. P (b = 0 = pq + (1 p(1 q, P (b = 1 = p(1 q + (1 pq. H(B = H 2 (pq + (1 p(1 q. ( q 1 q p(b a = 1 q q ab ( H(B A = a p(ah(b A = a = a p(a b p(b log p(b (1 = p (H 2 (q + (1 p H 2 (1 q = H }{{} 2 (q. =H 2 (q I(B : A = H(B H(B A = H 2 (pq + (1 p(1 q H 2 (q. The channel capacity is obtained by maximizing this function of p (for each fixed q: which happens at p = 1/2, since p I(B : A = (2q 1 log ( p + q 2pq which vanishes for generic q only if the argument of the log is 1 = 1 The value at the maximum is: p + q 2pq C(q = 1 H 2 (q. 6 p = 1 2.

7 To find I(B : A E, I found it easiest to do the following: Then to find H(B E, use: I(B : A E = H(B E H(B AE. }{{} =0 p(abe = δ b,a+e p(aq(e p(be = a p(abe = (pδ b,e + (1 pδ b,e+1 q(e p(b e = p(be p(e = pδ b,e + (1 pδ b,e+1. This distribution on b has Shannon entropy H 2 (p. Therefore as claimed in lecture. I(B : A E = H(B E H(B AE = H }{{} 2 (p =0 7