Chapter 2 Source Models and Entropy. Any information-generating process can be viewed as. computer program in executed form: binary 0

Size: px

Start display at page:

Download "Chapter 2 Source Models and Entropy. Any information-generating process can be viewed as. computer program in executed form: binary 0"

Rosamond French
5 years ago
Views:

1 Part II Information Theory Concepts Chapter 2 Source Models and Entropy Any information-generating process can be viewed as a source: { emitting a sequence of symbols { symbols from a nite alphabet text: ASCII symbols computer program in executed form: binary 0 and 1 n-bit image: 2 n symbols 1

2 Discrete Memoryless Sources (DMS) successive symbols statistically independent S = fs 1 ;s 2 ;:::;s n g fp(s 1 );p(s 2 );:::;p(s n )g I(s i ), the information revealed by the occurrence of a certain source symbol, is dened as I(s i ) = log 2 1 p(s i ) Average Information per source symbol, entropy H(s) = P p(si )I(s i )= P p(s i ) log 2 p(s i ) bits/symbol 2

3 Extensions of a Discrete Memoryless Source DMS S with an alphabet of size n the output of the source grouped into blocks of N symbols S N with an alphabet of size n N : the Nth extension of the source S For a memoryless source, the probability of a symbol i =(s i1 ;s i2 ;:::;s in ) from S N is given by p( i )=p(s i1 )p(s i2 ):::p(s in ) H(S N )=NH(S) 3

4 Markov Sources DMS too restrictive In general, the previous part of a message inuences the probabilities for the next symbol, the source has memory. In English text, the letter Q is almost always followed by the letter U. In digital images, the probability of a given pixel taking on a particular code value is dependent on the surrounding pixel values. 4

5 Such a source can be modeled as a Markov source. An mth-order Markov source: p(s i js j1 ;:::;s jm ) s j1 ;:::;s jm preceding to s i i; j k (k =1;2;:::;m)=1;2;:::;n (s j1 ;:::;s jm ): a state for the mth-order Markov source, a total of n m states For an ergodic Markov source, 9 a unique probability distribution over the set of states: stationary or equilibrium distribution. 5

6 H(Sjs j1 ;:::;s jm )= P ip(s i js j1 ;:::;s jm ) log p(s i js j1 ;:::;s jm ) H(S) = P S mh(sjs j 1 ;:::;s jm ) p(s j1 ;:::;s jm ) = P S p(s m+1 j 1 ;:::;s jm )p(s i js j1 ;:::;s jm ) log p(s i js j1 ;:::;s jm ) = P S p(s m+1 j 1 ;:::;s jm ;s i ) log p(s i js j1 ;:::;s jm ) 6

7 p(0 0,0)=0.8 0,0 p(1 0,0)=0.2 p(0 0,1)=0.5 p(0 1,0)=0.5 1,0 0,1 p(1 0,1)=0.5 p(1 1,0)=0.5 p(0 1,1)=0.2 1,1 p(1 1,1)=0.8 p(0; 0) = p(1; 1) = 5 2, p(0; 1) = p(1; 0) = H(S) = 0:801 bit/symbol 7

8 Extensions of a Markov Source and Adjoint Sources The Nth extension of a Markov source, S N,isathorder Markov source with symbols dened as blocks of N symbols from the original source, where = dm=ne As in the case of a DMS, H(S N )=NH(S) 8

9 The Nth extension of a Markov source, S N, with source symbols f 1 ; 2 ;:::; n Ngand stationary probabilities fp( 1 );p( 2 ); ;p( n N)g: a DMS with the same alphabet and the same symbol probabilities is called the adjoint source of S N and denoted by S N. { The adjoint source ignores the conditional probabilities which describe the dependence between the extended symbols. { H( S N ) H(S N ) { H N (S) = H( S N ) N!H(S) 9

10 The Noiseless Source Coding Theorem { S an ergodic source with an alphabet of size n and an entropy H(S) { encoding blocks of N source symbols at a time into binary codewords { For any >0, it is possible, by choosing N large enough, to construct a code so that the average number of bits per original source symbol, L, satises H(S) L H(S)+ 10

11 Chapter 3 Variable-Length Codes Variable-length codes with source extensions to achieve the entropy of a source { a DMS S = fs 1 ;s 2 ;s 3 ;s 4 ; p(s 1 )=0:60, p(s 2 )=0:30, p(s 3 )=0:05, p(s 4 )= 0:05g { each codeword in the sequence is instantaneously decodeable without reference to the succeeding codewords i no codeword be a prex of some other codeword (called by a prex condition code) 11

12 { entropy H(S) = P n p(s i=1 i)i(s i ) with I(s i )= log 2 p(s i ) { average codeword length or average length of the code, L P = n p(s i=1 i)l(s i ) with L(s i ) being the length of the codeword for s i { TohaveLH(S), we need L(s i ) log 2 p(s i )= 1 log bits 2 p(s i ) 1 or L(s i )=dlog e (bits) 2 p(s i ) { Shannon-Fano coding 12

13 Symbol Probability Code I Code II s s s s H(S) = 1:40 bits/symbol L 1 L 2 = 2:0 bits/symbol = 1:5 bits/symbol Acodeis compact (for a given source) if it has the smallest possible average codeword length. 13

14 Code Eciency and Source Extensions { Code II compact on S, its average codeword length is still far greater than H(S) { the code eciency: = H(S) L Code II: = 1:4 1:5 =0:93 { Extension to S 2 of 16 symbols formed as pairs of symbols from S. Table 3.2 shows a compact code of S 2, L =2:86 bits/extended symbol 14

15 = 1:43 bits/original source symbol = 1:40 1:43 =0:98 Human Codes for constructing compact codes { The Human code for a source fs 1 ;s 2 ghas trivial codewords \0" and \1". { Consider S = fs 1 ;s 2 ;:::;s n g (n>2) Let s n 1 ;s n be least probable symbols of this source. 15

16 Let Human code for fs 1 s 2 ; ;s n 2 ;fs n 1 ;s n gg be constructed and the codeword for fs n 1 ;s n g be w. Then Human code for fs 1 ; ;s n 1 ;s n g will be Human code for s 1 ; ;s n 2 and w0 for s n 1, w1 for s n. understanding Fig. 3 16

17 Modied Human Codes { Frequently, most of symbols in a large symbol set have very small probabilities. { Lump the less probable symbols into a symbol called \Else" and design a Human code for the reduced symbol set: the modied Human code. { Whenever a symbol in the ELSE category needs to be encoded, the encoder transmits the codeword for ELSE followed 17

18 by extra bits needed to identify the actual message within the ELSE category. the loss in coding eciency very small the storage requirements and the decoding complexity substantially reduced Group 3 international digital facsimile coding standards: { each binary image scan line: a sequence of alternating black and white runs which are encoded with separable variable-length code tables { A run is the number of times a particular value occurs consecutively along a scanline. 18

19 { 1728 pixels for each scanline { each Human table should have 1728 entries { greatly simplied by taking advantage of the fact that the longer runs are highly improbable { The rst 64 entries in each table represent the Human code for runs 0 to 63 { All other runs 64N + M (1 N 27, 0 M 64): entries for 64 to 90 encode N entries for 0 to 63 encode M 19

20 { a run of 213: N = 3 and M = 21 its Human code the entry 67(64 + 3) for N =3 the entry 21 for M =21 {simplifying the search for decoding Limitations of Human Coding { The ideal binary codeword length for a source symbol s i from a DMS is log 2 p(s i ), this condition is met only if p(s i )= 1. 2 k 20

21 { Otherwise, direct encoding of the individual source symbols may result in poor code eciency. p(s 1 ) 1, p(s 2 )=::: =p(s n )0 H(S) = p(s 1 ) log 2 p(s 1 ) P k2 p(s k ) log 2 p(s k ) (n 1)p(s 2 ) log 2 p(s 2 ) = (n 1) (1 p(s 1 )) (1 p(s log 1 )) (n 1) 2 (n 1) = (1 p(s 1 )) log 2 (1 p(s 1 )) (n 1)! 0 as p(s 1 )! 1 L 1 since the shortest codeword length for each individual symbol is one 21

22 { S = f0; 1g The Human codewords for \0" and \1" are \0" and \1", thus L = 1, regardless of the symbol probabilities. { Encoding an extended source may improve the coding eciency, but convergence to the source entropy could be slow. { The numberofentries in the Human code table grows exponentially with the block size. 22

23 { For an mth order Markov source, the conditional probabilities p(s i js i1 ;:::;s im ) vary as the state (s i1 ;:::;s im )changes. Thus, a separable Human table is needed for each state. { The coding eciency may still be low if the symbol conditional probabilities deviate from the ideal case. { Using an extended source and encoding the adjoint, its entropy H N may get close to the entropy H of the Markov source but the block size must be large. 23

24 { The Human coding cannot eciently adapt to changing source statistics { Arithmetic coding is more complex than Human coding, but it can overcome the limitations of Human coding 24

3F1 Information Theory, Lecture 3

3F1 Information Theory, Lecture 3 Jossy Sayir Department of Engineering Michaelmas 2013, 29 November 2013 Memoryless Sources Arithmetic Coding Sources with Memory Markov Example 2 / 21 Encoding the output