PHYS Statistical Mechanics I Assignment 4 Solutions

Transcription

1 PHYS Statistical Mechanics I Assignment 4 Solutions 1. The Shannon entropy is S = d p i log 2 p i. The Boltzmann entropy is the same, other than a prefactor of k B and the base of the log is e. Neither of those things are important for this proof, so let s work with the Shannon entropy for convenience. We maximize the entropy by asking that S/ p i = 0. The problem is that we don t know how to take derivatives of logs, only of lns. So first, write this in terms of the ln, in which case it is almost identical to the Boltzmann entropy: S = 1 ln(2) d p i lnp i. If d = 2, then Maximizing the entropy gives S 2 = 1 ln(2) [plnp+(1 p)ln(1 p)]. 1 [lnp+1 ln(1 p) 1] = 0 ln(p) = ln(1 p), ln(2) which has the solution p = 1 2, as expected. For a d-dimensional coin, the probabilities are p 1,p 2,...,p d where p d = 1 p 1 p 2... p = 1 p i. So the entropy is [ ( S = 1 p 1 lnp 1 +p 2 lnp ln(2) ) ( )] p i ln 1 p i. Now, we need to maximize the entropy for each of the p s, but all of them have exactly the same equation, that is [ ( ] ( ) 1 lnp j +1 ln 1 p i ) 1 = 0 ln(p j ) = ln 1 p i ln(2) ( 1 ) ln p i = 0. p j This means that each p j = 1 p i, that is the difference between 1 and the sum of all the others. But this means that each p j is identical, or p j = 1 ()p j, which is equivalent to dp j = 1 or p j = 1/d.

2 PHYS Assignment 4 Solutions 2 2. This is done in C, and the results are attached at the end. I ll summarize the results here. (a) First off, looking at the actual data one finds that there are exactly zero pixels with the bit value 0. Is this white or black? Given that the number of pixels with value 255 is large, and the picture is fairly dark, I would guess that bit value 0 is actually white, not black. Anyhow, it doesn t change the analysis. Calculating the various probabilities of obtaining the bit values, one can calculate the Shannon entropy. I obtain S 7.463, so that the image needs on average bits per pixel to represent the data. Given that there are pixels in the image, this suggests that I could compress the image to a minimum size of bits or bytes (there are 8 bits in byte) which is 13 Mb. Considering that the original file is 14 Mb, this is not much of a savings given lossless compression. (b) Interestingly, gzip compressed the figure to only bytes, so this algorithm uses something fancier than straight Huffman coding. Likely it is making use of improved encodings where (for example) contiguous strings of the same bit value are reduced to two integers (the bit value and the number of instances). Windows zip produced a file that was slightly larger at bytes. A straight conversion of the tif gives a gif that is bytes, a jpeg that is bytes, and a png that is bytes. Given that both gzip and zip are lossless, and close to optimal, we can infer that both gif and png are lossless image formats, but that jpeg is lossy. A quick scan of the literature confirms this. (c) The mean bit length for the data using the Huffman coding was found to be approximately 7.506, which is larger than the Shannon entropy (thank goodness, otherwise it would be wrong!) but very close. The encoding that I found uses a maximum of 17 bits and is 99.43% efficient. Not bad! (d) The variance of the bit length about the mean was found to be approximately 1.398, which gives a standard deviation of only 1.182, which is only just larger than a single bit. Because the variance is the difference between the mean of the square and the square of the mean, we can subtract the square of the mean to obtain b 2 = ( b) 2 + b 2 = so that the RMS value is approximately bits. This is, unsurprisingly, very close to the mean bitlength because the variance is so comparably tiny. 3. (a) {0,10,11 This cannot be a Huffman encoding for any probabilities. If the smallest probability is assigned bitstring 0, then necessarily the other two probabilities must connect to the tree on the final branch labeled 1, so that the last bit for both must be 1. (b) {00,01,10,110 This also can t be a Huffman encoding. With four variables, the only two possible trees are perfectly balanced (i.e. all probabilities are close) or totally unbalanced (like for the tree produced with probabilities 1/2, 1/4, 1/8 and 1/8. The first gives an encoding like {00,01,10,11 while the second gives something like {0, 01, 011, 111. (c) {01,10. This one is fine. It doesn t violate any rules. It is simply stupid because it employs two bits when you could get away with using only one.

3 // This code performs a binary Huffman encoding #include <stdio.h> #include <stdlib.h> #include <math.h> #define elements 256 FILE *fin; main() { int i,j,c,nc; int symbol[elements]; double prob,entropy,total; char filename[50]; // parameters for the Huffman encoding algorithm int which1,which2,tree,max_bits; int bits[elements],branch[elements]; unsigned long bin[elements][elements]; double min,min1,min2,mean_bits,variance; double problist[elements]; // first read in the data from a text file (formatted input) fin=fopen("histogram.txt","r"); if (fin == NULL) { printf("unable to open file\n"); exit(1); for (i=0;i<elements;i++) fscanf(fin,"%d\t%d\n",&j,&(symbol[i])); fclose(fin); // print out to check it's read in properly // for (i=0;i<elements;i++) printf("%d %d\n",i,symbol[i]); // calculate the probabilities and the Shannon entropy nc=0; for (i=0;i<elements;i++) nc += symbol[i]; printf("total number of pixels = %d\n",nc); entropy=0.0; total=0.0; prob=(double)(symbol[i])/(double)nc; problist[i]=prob; // printf("%d %lf\n",i,prob); if (prob > 0.0) entropy -= prob*log(prob)/log(2.0); total += prob; printf("total probability = %lf\n",total); printf("minimum entropy = %lf\n",entropy); // Next determine the encoding that approximates this max_bits=0; for (j=0;j<elements;j++) bin[i][j]=0; bits[i]=0; branch[i]=i; tree=elements-1; for (;;) { min=2.0;

4 if (problist[i] <= min) { min=problist[i]; which1=i; min1=min; min=2.0; if (problist[i] <= min && problist[i] > min1) { min=problist[i]; which2=i; min2=min; // printf("%d: %lf, %d: %lf\n",which1+1,min1,which2+1,min2); ++tree; if (branch[i] == branch[which2]) { bin[i][bits[i]] += 1; if (branch[i] == branch[which1] branch[i] == branch[which2]) { branch[i]=tree; problist[i]=min1+min2; /* printf("element %d: branch %d, %d bits, probability %lf,",i+1, branch[i],bits[i]+1,problist[i]); printf("binary = "); for (j=elements-1;j>=0;j--) printf("%d",bin[i][j]); printf("\n"); */ ++(bits[i]); if (bits[i] > max_bits) max_bits = bits[i]; if (fabs(problist[which1]-1.0) < 0.01) break; // now parse the results mean_bits=0.0; printf("maximum number of bits used = %d\n",max_bits); prob=(double)(symbol[i])/(double)nc; printf("greyscale value %2d: bits = %2d, val = ",i,bits[i]); for (j=max_bits-1;j>=0;j--) printf("%d",bin[i][j]); printf(", prob = %lf\n",prob); mean_bits += prob*bits[i]; printf("mean bitlength = %lf\n",mean_bits); printf("efficiency = %lf\%\n",entropy/mean_bits); variance=0.0; prob=(double)(symbol[i])/(double)nc; variance += prob*(bits[i]-mean_bits)*(bits[i]-mean_bits); // printf("%d %lf\n",i+1,prob*(bits[i]-mean_bits)*(bits[i]-mean_bits)); printf("variance = %lf\n",variance);

5 total number of pixels = total probability = minimum entropy = maximum number of bits used = 17 greyscale value 0: bits = 17, val = , prob = greyscale value 1: bits = 17, val = , prob = greyscale value 2: bits = 16, val = , prob = greyscale value 3: bits = 15, val = , prob = greyscale value 4: bits = 14, val = , prob = greyscale value 5: bits = 13, val = , prob = greyscale value 6: bits = 12, val = , prob = greyscale value 7: bits = 11, val = , prob = greyscale value 8: bits = 10, val = , prob = greyscale value 9: bits = 9, val = , prob = greyscale value 10: bits = 8, val = , prob = greyscale value 11: bits = 8, val = , prob = greyscale value 12: bits = 8, val = , prob = greyscale value 13: bits = 7, val = , prob = greyscale value 14: bits = 7, val = , prob = greyscale value 15: bits = 7, val = , prob = greyscale value 16: bits = 7, val = , prob = greyscale value 17: bits = 7, val = , prob = greyscale value 18: bits = 7, val = , prob = greyscale value 19: bits = 7, val = , prob = greyscale value 20: bits = 7, val = , prob = greyscale value 21: bits = 7, val = , prob = greyscale value 22: bits = 7, val = , prob = greyscale value 23: bits = 7, val = , prob = greyscale value 24: bits = 7, val = , prob = greyscale value 25: bits = 7, val = , prob = greyscale value 26: bits = 7, val = , prob = greyscale value 27: bits = 7, val = , prob = greyscale value 28: bits = 7, val = , prob = greyscale value 29: bits = 8, val = , prob = greyscale value 30: bits = 8, val = , prob = greyscale value 31: bits = 8, val = , prob = greyscale value 32: bits = 8, val = , prob = greyscale value 33: bits = 8, val = , prob = greyscale value 34: bits = 8, val = , prob = greyscale value 35: bits = 8, val = , prob = greyscale value 36: bits = 8, val = , prob = greyscale value 37: bits = 8, val = , prob = greyscale value 38: bits = 8, val = , prob = greyscale value 39: bits = 8, val = , prob = greyscale value 40: bits = 8, val = , prob = greyscale value 41: bits = 8, val = , prob = greyscale value 42: bits = 8, val = , prob = greyscale value 43: bits = 8, val = , prob = greyscale value 44: bits = 8, val = , prob = greyscale value 45: bits = 8, val = , prob = greyscale value 46: bits = 8, val = , prob = greyscale value 47: bits = 7, val = , prob = greyscale value 48: bits = 7, val = , prob = greyscale value 49: bits = 7, val = , prob = greyscale value 50: bits = 7, val = , prob = greyscale value 51: bits = 7, val = , prob = greyscale value 52: bits = 8, val = , prob = greyscale value 53: bits = 8, val = , prob = greyscale value 54: bits = 8, val = , prob = greyscale value 55: bits = 8, val = , prob = greyscale value 56: bits = 8, val = , prob = greyscale value 57: bits = 8, val = , prob = greyscale value 58: bits = 8, val = , prob = greyscale value 59: bits = 8, val = , prob = greyscale value 60: bits = 8, val = , prob = greyscale value 61: bits = 8, val = , prob =

6 greyscale value 62: bits = 8, val = , prob = greyscale value 63: bits = 8, val = , prob = greyscale value 64: bits = 8, val = , prob = greyscale value 65: bits = 8, val = , prob = greyscale value 66: bits = 8, val = , prob = greyscale value 67: bits = 8, val = , prob = greyscale value 68: bits = 8, val = , prob = greyscale value 69: bits = 8, val = , prob = greyscale value 70: bits = 8, val = , prob = greyscale value 71: bits = 8, val = , prob = greyscale value 72: bits = 8, val = , prob = greyscale value 73: bits = 8, val = , prob = greyscale value 74: bits = 8, val = , prob = greyscale value 75: bits = 8, val = , prob = greyscale value 76: bits = 8, val = , prob = greyscale value 77: bits = 8, val = , prob = greyscale value 78: bits = 8, val = , prob = greyscale value 79: bits = 8, val = , prob = greyscale value 80: bits = 8, val = , prob = greyscale value 81: bits = 8, val = , prob = greyscale value 82: bits = 8, val = , prob = greyscale value 83: bits = 8, val = , prob = greyscale value 84: bits = 8, val = , prob = greyscale value 85: bits = 9, val = , prob = greyscale value 86: bits = 9, val = , prob = greyscale value 87: bits = 9, val = , prob = greyscale value 88: bits = 9, val = , prob = greyscale value 89: bits = 9, val = , prob = greyscale value 90: bits = 9, val = , prob = greyscale value 91: bits = 9, val = , prob = greyscale value 92: bits = 9, val = , prob = greyscale value 93: bits = 9, val = , prob = greyscale value 94: bits = 9, val = , prob = greyscale value 95: bits = 9, val = , prob = greyscale value 96: bits = 9, val = , prob = greyscale value 97: bits = 9, val = , prob = greyscale value 98: bits = 9, val = , prob = greyscale value 99: bits = 9, val = , prob = greyscale value 100: bits = 9, val = , prob = greyscale value 101: bits = 9, val = , prob = greyscale value 102: bits = 9, val = , prob = greyscale value 103: bits = 9, val = , prob = greyscale value 104: bits = 9, val = , prob = greyscale value 105: bits = 9, val = , prob = greyscale value 106: bits = 9, val = , prob = greyscale value 107: bits = 9, val = , prob = greyscale value 108: bits = 9, val = , prob = greyscale value 109: bits = 9, val = , prob = greyscale value 110: bits = 9, val = , prob = greyscale value 111: bits = 10, val = , prob = greyscale value 112: bits = 10, val = , prob = greyscale value 113: bits = 10, val = , prob = greyscale value 114: bits = 9, val = , prob = greyscale value 115: bits = 9, val = , prob = greyscale value 116: bits = 9, val = , prob = greyscale value 117: bits = 9, val = , prob = greyscale value 118: bits = 8, val = , prob = greyscale value 119: bits = 7, val = , prob = greyscale value 120: bits = 7, val = , prob = greyscale value 121: bits = 7, val = , prob = greyscale value 122: bits = 7, val = , prob = greyscale value 123: bits = 6, val = , prob = greyscale value 124: bits = 6, val = , prob = greyscale value 125: bits = 6, val = , prob = greyscale value 126: bits = 6, val = , prob = greyscale value 127: bits = 6, val = , prob =

7 greyscale value 128: bits = 6, val = , prob = greyscale value 129: bits = 6, val = , prob = greyscale value 130: bits = 6, val = , prob = greyscale value 131: bits = 6, val = , prob = greyscale value 132: bits = 6, val = , prob = greyscale value 133: bits = 6, val = , prob = greyscale value 134: bits = 6, val = , prob = greyscale value 135: bits = 6, val = , prob = greyscale value 136: bits = 6, val = , prob = greyscale value 137: bits = 6, val = , prob = greyscale value 138: bits = 6, val = , prob = greyscale value 139: bits = 6, val = , prob = greyscale value 140: bits = 7, val = , prob = greyscale value 141: bits = 7, val = , prob = greyscale value 142: bits = 7, val = , prob = greyscale value 143: bits = 7, val = , prob = greyscale value 144: bits = 7, val = , prob = greyscale value 145: bits = 7, val = , prob = greyscale value 146: bits = 7, val = , prob = greyscale value 147: bits = 7, val = , prob = greyscale value 148: bits = 7, val = , prob = greyscale value 149: bits = 7, val = , prob = greyscale value 150: bits = 7, val = , prob = greyscale value 151: bits = 7, val = , prob = greyscale value 152: bits = 7, val = , prob = greyscale value 153: bits = 7, val = , prob = greyscale value 154: bits = 7, val = , prob = greyscale value 155: bits = 7, val = , prob = greyscale value 156: bits = 8, val = , prob = greyscale value 157: bits = 8, val = , prob = greyscale value 158: bits = 8, val = , prob = greyscale value 159: bits = 8, val = , prob = greyscale value 160: bits = 8, val = , prob = greyscale value 161: bits = 8, val = , prob = greyscale value 162: bits = 8, val = , prob = greyscale value 163: bits = 8, val = , prob = greyscale value 164: bits = 8, val = , prob = greyscale value 165: bits = 8, val = , prob = greyscale value 166: bits = 8, val = , prob = greyscale value 167: bits = 8, val = , prob = greyscale value 168: bits = 9, val = , prob = greyscale value 169: bits = 9, val = , prob = greyscale value 170: bits = 9, val = , prob = greyscale value 171: bits = 9, val = , prob = greyscale value 172: bits = 9, val = , prob = greyscale value 173: bits = 9, val = , prob = greyscale value 174: bits = 9, val = , prob = greyscale value 175: bits = 9, val = , prob = greyscale value 176: bits = 9, val = , prob = greyscale value 177: bits = 9, val = , prob = greyscale value 178: bits = 9, val = , prob = greyscale value 179: bits = 9, val = , prob = greyscale value 180: bits = 9, val = , prob = greyscale value 181: bits = 10, val = , prob = greyscale value 182: bits = 9, val = , prob = greyscale value 183: bits = 9, val = , prob = greyscale value 184: bits = 9, val = , prob = greyscale value 185: bits = 9, val = , prob = greyscale value 186: bits = 9, val = , prob = greyscale value 187: bits = 9, val = , prob = greyscale value 188: bits = 10, val = , prob = greyscale value 189: bits = 10, val = , prob = greyscale value 190: bits = 10, val = , prob = greyscale value 191: bits = 11, val = , prob = greyscale value 192: bits = 11, val = , prob = greyscale value 193: bits = 11, val = , prob =

8 greyscale value 194: bits = 11, val = , prob = greyscale value 195: bits = 11, val = , prob = greyscale value 196: bits = 11, val = , prob = greyscale value 197: bits = 11, val = , prob = greyscale value 198: bits = 11, val = , prob = greyscale value 199: bits = 11, val = , prob = greyscale value 200: bits = 11, val = , prob = greyscale value 201: bits = 11, val = , prob = greyscale value 202: bits = 11, val = , prob = greyscale value 203: bits = 11, val = , prob = greyscale value 204: bits = 11, val = , prob = greyscale value 205: bits = 11, val = , prob = greyscale value 206: bits = 11, val = , prob = greyscale value 207: bits = 11, val = , prob = greyscale value 208: bits = 11, val = , prob = greyscale value 209: bits = 11, val = , prob = greyscale value 210: bits = 11, val = , prob = greyscale value 211: bits = 11, val = , prob = greyscale value 212: bits = 11, val = , prob = greyscale value 213: bits = 11, val = , prob = greyscale value 214: bits = 11, val = , prob = greyscale value 215: bits = 11, val = , prob = greyscale value 216: bits = 11, val = , prob = greyscale value 217: bits = 11, val = , prob = greyscale value 218: bits = 11, val = , prob = greyscale value 219: bits = 10, val = , prob = greyscale value 220: bits = 10, val = , prob = greyscale value 221: bits = 10, val = , prob = greyscale value 222: bits = 10, val = , prob = greyscale value 223: bits = 10, val = , prob = greyscale value 224: bits = 10, val = , prob = greyscale value 225: bits = 10, val = , prob = greyscale value 226: bits = 10, val = , prob = greyscale value 227: bits = 10, val = , prob = greyscale value 228: bits = 10, val = , prob = greyscale value 229: bits = 11, val = , prob = greyscale value 230: bits = 11, val = , prob = greyscale value 231: bits = 11, val = , prob = greyscale value 232: bits = 11, val = , prob = greyscale value 233: bits = 11, val = , prob = greyscale value 234: bits = 10, val = , prob = greyscale value 235: bits = 10, val = , prob = greyscale value 236: bits = 10, val = , prob = greyscale value 237: bits = 11, val = , prob = greyscale value 238: bits = 11, val = , prob = greyscale value 239: bits = 11, val = , prob = greyscale value 240: bits = 11, val = , prob = greyscale value 241: bits = 11, val = , prob = greyscale value 242: bits = 11, val = , prob = greyscale value 243: bits = 11, val = , prob = greyscale value 244: bits = 11, val = , prob = greyscale value 245: bits = 11, val = , prob = greyscale value 246: bits = 11, val = , prob = greyscale value 247: bits = 11, val = , prob = greyscale value 248: bits = 11, val = , prob = greyscale value 249: bits = 11, val = , prob = greyscale value 250: bits = 11, val = , prob = greyscale value 251: bits = 11, val = , prob = greyscale value 252: bits = 11, val = , prob = greyscale value 253: bits = 11, val = , prob = greyscale value 254: bits = 10, val = , prob = greyscale value 255: bits = 7, val = , prob = mean bitlength = efficiency = variance =