Draft of a lecture with exercises

Transcription

1 Draft of a lecture with exercises Piotr Chołda March, 207 Introduction to the Course. Teachers: Lecture and exam: Piotr Chołda, Ph.D., Dr.habil. Room: 3, pav. D5 (first floor). Office hours: please contact me by . Phone: (67 ) piotr.cholda@agh.edu.pl. WWW: course webpage. Exercise classes and projects: Andrzej Kamisiński, M.Sc. andrzejk@agh.edu.pl. 2. How to get a credit (a positive final grade for the course): Exercise classes: with support of Matlab; three tests K-K3 (3 20 = 60 pts.); short quizzes (0 4 = 40 pts.); up to two correction (revision) tests are allowed provided some thresholds are met; activity during classes: scored with single points, no negative points; additional points can be obtained by solving exercises or performing tasks denoted with asterisk(s); exercise classes grade is calculated according to the AGH Bylaw 3.; precise rules related to the exercise classes are present at the course webpage in the document Rules related to the exercise classes, which is the integral part of the course rules. Project: implementation of a code or a procedure related to the information theory. The precise rules related to the project are present at the course webpage in the document Information on the project and the proposed topics, which is the integral part of the course rules. Page

2 Oral exam (contents of the lecture): the exam can be taken only by the students who have obtained a credit for the exercise classes and the project. The final grade is calculated as weighted arithmetic mean of the exercise classes (30%), project (30%) and the exam grades (40%), while the exam grade takes into account all failed terms. The grade is found according to the AGH Bylaw Simplified version of the exam: presentation of a lecture during the semester, exam: limited to the topic of the presented lecture, detailed conditions: topics and dates given at Google Docs, except for the topic, also the main problems to be covered and the suggested literature is given, required format: L A TEX, beamer class: the template is given at the course webpage, deadlines: it is necessary to provide the teacher with the slides two weeks prior to the scheduled presentation, the suggestions given by the teacher with respect to the initial version of the presentation should be taken into account, presence during the lectures: at least 50% (i.e., 7 lectures). 3. Presence/absence: If at least two students attend the lectures, the lectures are not obligatory. Starting from the lecture, where the attendance is smaller than two students, the lectures become obligatory and the presence is taken into account (no more than 40% of absences is allowed). Due to the AGH Bylaw.3, presence at exercise classes is obligatory. 4. There are no presentations available only lecture drafts are given before a lecture (at WWW, sometimes modifications can be introduced after a lecture). 2 Bibliography In case you have problems with finding the books, you can borrow them from the teacher. Dominic Welsh. Codes and Cryptography. Clarendon Press, Oxford, UK, 988. Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, Inc., New York, NY, 99. Gareth A. Jones and J. Mary Jones. Information and Coding Theory. Springer-Verlag London Ltd., London, UK, Page 2

3 Todd K. Moon. Error Correction Coding. John Wiley & Sons, Inc., Hoboken, NJ, Stefan M. Moser and Po-Ning Chen. A Student s Guide to Coding and Information Theory. Cambridge University Press, Cambridge, UK, Lecture I (March 7, 207): entropy, channels, capacity a primer to the information theory 3. Sources of information and entropy. Digital communication system: 2. Noise: open, closed. external (thunders, radiation), internal (thermal/johnson, shot noise). 3. Message sources: set of messages: discrete, continuous, distributions: deterministic, random (probabilistic, stochastic), 4. Information capacity/denstity (of a message): if message x i is generated with probability Pr{x i } = p, its information capacity (zawartość informacyjna) is expressed as: I(x i ) = I(p) = log r p = log r p. Typical base for our course is r = Entropy of a random variable (source) X with a discrete probability distriburion Pr{X = x i } = Pr{x i } = p i ( p i = ) can be characterized with the following entropy (entropia zmiennej losowej ): H(X) = i H(p, p 2,... ) = p i lg p i [bit(s)]. Entropy is the average value of i information capacity for the random variable (information source), E[I]. The more random data, the larger entropy. 6. For a discrete probability distribution of random variable X, with the cardinality of the realization set equal to N (we can identify this distribution with the probability distribution of message generation for a particular information source): Limit for non-zero probabilities lim p 0 p lg p = 0. Upper and lower bounds 0 H(X) lg N. Page 3

4 Equiprobable distribution For probability distribution Pr{X = x i } = N, i N entropy reaches its maximum for all N-element distributions: H max (X) = H( N, N,..., N ) = lg N. 7. L-th extension of a source: consider a discrete source X, generating N possible messages x, x 2,..., x N. Now, let us consider a sequence of messages of source X of length L: s i = x i() x i(2)... x i(l). A discrete source generating sequences s i is called the L-th extension of a source (L-krotne rozszerzenie źródła). If X L is the L-th extension of a memoryless source X, then: H(X L ) = L H(X). 8. Joint/mutual entropy: for two random variables X, Y with joint probability distribution Pr{X = x i, Y = y j } = Pr{x i, y j } = p(i, j) = p ij, p(i, j) = ) joint entropy (entropia łączna) is defined as follows: i j H(X, Y ) = p(i, j) lg p(i, j). i j 9. Markov source (źródło marko[wo]wskie) of order L is a source generating N messages x,..., x N, that is described by probabilities Pr {s i } and conditional probabilities Pr {x j s i }, where s i is a message sequence genereted prior to x j : s i = x i()... x i(l). The entropy of such Markov source is defined as: H L (X) = = N N L Pr{x j, s i } lg Pr{x j s i } j= i= N N j= i = N Pr{x j, x i,..., x il } lg Pr{x j x i,..., x il }. i L = We must not confuse source extensions (memoryless) and Markov sources (that always have a memory)! 0. Conditional entropy: for joint random variables X i Y conditional entropy H(Y X) (entropia warunkowa) is defined as the average value of entropy of random variable Y given the occurence of X: H(Y X) = Pr{x i } Pr{y j x i } lg Pr{y j x i }. i j. Mutual information (transinformation) (informacja wzajemna, transinformacja) is defined as follows: I(X; Y ) = M N Pr{x i, y j } lg i= j= Pr{x i,y j} Pr{x i} Pr{y j}. Mutual information quantifies the reduction in uncertainty due to another random variable. We use a semicolon I(X; Y ), not a comma (sometimes used notion I(X, Y ) denotes the joint entropy!). There is not a minus in the definition formula!. 3.2 Channels. Channels can be: discrete vs. continuous, Page 4

5 x = 0 y = x y 3 = E x 2 = y 2 = x 2 Figure : Binary erasure channel. binary, memoryless, stationary, symmetric. 2. Discrete memoryless channel is defined with: a set of input messages: X = {x, x 2,..., x M }; a set of output messages: Y = {y, y 2,..., y N }; a set of M N transition probabilities: Pr{y j x i } (represented by P N M matrix). 3. Noiseless channel (purely theoretical construction): Pr{y k x k = y k } =. 4. Binary erasure channel (Figure ). 5. Binary symmetric channel (BSC, binarny kanał symetryczny): Pr{0 } BSC = Pr{ 0} BSC = BER (Bit Error Rate). Due to simplified modeling, we eagerly use the memoryless channels, but this modeling is not exact for large bitrates (where burst errors take place). 6. Information channel is a very broad notion: transmission channel (radio, fibre, etc.), storage device, packed file Capacity. We are interested in the amount of information that can be transferred via a transmission channel (defined by X, Y, Pr{y j x i }). Channel capacity (przepustowość kanału) for a discrete memoryless channel is defined as maximum transinformation I(X; Y ) that can be carried by the channel at once, while optimization goes through all possible probability distributions of the input messages: C = max P X max [H(Y ) H(Y X)]. P X 3.4 Exercises I(X; Y ) = max P X [H(X) H(X Y )] = The best way to deal with the exercises (it is only a piece of advice, this is not a homework): before the classes, look through all the exercises; Page 5

6 try to solve at least one exercise from each group; during the classes declare the exercises you recognize as very difficult to solve (we will try to solve them) typically, during the classes the students decide which exercises are to be solved; solve all the exercises after the classes (if there are some issues not covered during the classes, you can ask the teacher during his office hours).. Entropy calculation: (a) Which of the following sequences carries more information: i. 0 letters of the Latin alphabet (it is assumed that each of 32 letter generation is equally possible, we do not care about uppercase or lowercase letters), ii. 32 numerical digits (it is assumed that generation of each of ten digits is equally possible)? (b) What can be the maximum uncertainty related to the 6 9 cm 2 photography, if a size of a single pixel equals cm 2, and each pixel can be characterized with one of the three values: white, gray and black? (c) A colleague tosses a coin three times and informs us how many heads (or tails) were obtained. What is the uncertainty about the result of the first toss? 2. Complex entropy calculations: (a) Information source generates messages according to the following scheme: in each time step, 0 or is randomly chosen (with probability p 0 = p and p = p, respectively). Such a drawing finishes when the first is picked. The source sends the information in which step this has happened. Find entropy of the source. (b) Source Z is defined as follows: () we have two sources X and Y with known entropies: H(X) and H(Y ), respectively; (2) messages generated at every moment by these two sources are different (e.g., X generates letters, and Y generates digits); (3) at every moment, source Z selects with known probability p to send a message just generated by X (equivalently, we can also say that it selects with probability p to send a message just generated by Y ). Find entropy of Z. (c) The inhabitants of a certain village are divided into two groups A and B. Half the people in group A always tell the truth, three tenths always lie, and two tenths always refuse to answer. In group B, threetenths of the people are truthful, half are liars, and two tenths always refuse to answer. Let p be the probability that a person selected at random will belong to group A. Let I(p) be the information conveyed about a person s truth-telling status by specifying his group membership. Find the maximum possible value of I and the percentage of people in group A for which the maximum occurs. 3. Properties of various entropy measures: Page 6

7 (a) Show that for independent random variables X, Y the following relationship holds for mutual entropy: H(X, Y ) = H(X) + H(Y ). (b) Prove that the following property takes place (the so-called chain rule): H(X, Y ) = H(X) + H(Y X). (c) Show that for transinformation, the following relations are the truth: I(X; Y ) = I(Y ; X) = H(X) H(X Y ) = H(Y ) H(Y X) = H(X) + H(Y ) H(X, Y ). (d) Random variable X can be equiprobably realized as an integer value from the set, 2, 3,..., 2N. On the other hand, random variable Y is defined as equal to 0, when the value of X is even, while Y =, when the value of X is odd. Prove that in such a case the following relationship is true for conditional entropy: H(X Y ) = H(X). Then show that the following relation is also the truth: H(Y X) = 0. (e) Messages of sources W and F are emitted in a mutually independent way. We know that the entropy of W is equal to eight bits and F twelve bits. Plot the relationship H(W F) = f (H(F W)), i.e, H(W F) as a function of variable H(F W) (from its minimum to its maximum value). (f) X a discrete random variable is given. Another random variable is defined by the following function: Y = g(x). Which of the following is the general relationship between H(Y ) and H(X): H(Y ) H(X), H(Y ) H(X)? What are the conditions related to g( ) to have the equality? 4. Sources with memory: (a) We are observing a source emitting two messages: $ and. We note that the probabilities of generation of two-message blocks are given in Table. Check if this source is memoryless. (b) A source generates two messages: 0 and. Calculate entropy understood in the classical way and entropy of a Markov source for two cases: Page 7

8 Table : Frequencies of the two message blocks occurence for exercise 4a $$ $ $ 2/5 /0 /0 2/5 p p N p N x 0 x x i x N p N Figure 2: A sample Markov source for exercise 4d. i. messages are equiprobable, there is a source with memory, and: Pr{0 } = Pr{ 0} = 3 ; ii. there is a source with memory, but for this case: Pr{0} = 3 4, Pr{0 0} = 2, Pr{0 } =. 3 (c) Draw a diagram of a Markov source of the 2 nd order, if the source is binary and conditional probabilities are given below: Pr{0 00} = Pr{ } = 0.8; Pr{ 00} = Pr{0 } = 0.2; Pr{0 0} = Pr{0 0} = Pr{ 0} = Pr{ 0} = 0.5. Then, find entropy of this source if we know that: Pr{00} = Pr{} = 5 2, Pr{0} = Pr{0} = 4 4. (d) With the help of the state diagram, Figure 2 shows S, a Markov source of the first order. Lack of arrow indicated that the related conditional probability equals zero. If a probability is not given next to an arrow, it means that this probability can be inferred somehow. For x, x 2,..., x N all transitions are analogous to x i. Sketch plots of H(S) and H (S) as functions of p. 5. Channels: (a) A discrete memoryless binary channel along with probabilities of the input and output messages is given in Figure??. Find the range of values of the joint probability of having simultaneously $ at the input and at the output. Page 8

9 3 $ a b 3 4 (b) Check if a cascade of binary symmetric channels (they are connected serially: an output of one is connected to an input of the next one; the channels do not necessarily have the same probabilistic characteristics) is also a binary symmetric channel. (c) We have a number of (not necessarily identical) binary symmetric channels characterized with the probabilities of error, given as: 2 p i <. How can we, using (not necessarily all of) them, construct a binary symmetric channel characterized with: 0 < BER 2? (d) Information consisting of N binary symbols is transmitter through binary symmetric channel with BER equal to p. What is the expected value of the number of bits transmitted without errors in this information? (e) A binary channel is represented with the following matrix: [ ] a b, a b where at the input and the output we have symbols from set {0, }; and a b. The receiver (taking signals from the channel) is given symbols 0 and with the same frequency. Find the probability distribution at the input of this channel (at the output of the transmitter) and show that the entropy at the channel output is larger than the one at its input. (f) Find the mutual information I(X; Y ) for a channel with two inputs x, x 2 X and three outputs y, y 2, y 3 Y given in Figure 3 if Pr{x } = Pr{x 2 } = Capacity calculation: (a) A binary channel is characterized with the following transition probabilities: p(y x ) =, p(y x 2 ) = p(y 2 x 2 ) = 2. We observe that one is present at the output of the channel thirteen times more than zero. Find capacity of this channel. Page 9

10 x = 0 x 2 = y = 0 y 3 = E y 2 = Figure 3: Example memoryless channel for exercise 5f. x = 0 y = x y 3 = E x 2 = y 2 = x 2 Figure 4: Binary erasure channel for exercise 6b. (b) A binary erasure channel (given in Figure 4) is characterized with the following probabilities: α β α α α β. β β Show that the capacity of this channel is given by the following formula: C = ( β)[ lg( β)] + ( α β) lg( α β) + α lg α. (c) Find the channel capacity for a binary erasure channel shown in Figure 5. (d) Find the capacity of the channel given in Figure 6 with the transition probabilities p(y j x i ) = 2 : p(y x 2 ) = p(y x 5 ) = p(y 2 x ) = p(y 2 x 3 ) = p(y 3 x 2 ) = p(y 3 x 4 ) = p(y 4 x 3 ) = p(y 4 x 5 ) = p(y 5 x 4 ) = p(y 5 x ) = 2 (the channel is symmetric). 0 p p 2 p p 3 Figure 5: Binary erasure channel for exercise 6c Page 0

11 x y x 2 y 2 x 3 y 3 x 4 y 4 x 5 y 5 Figure 6: Exemplary symmetric channel for exercise 6d Figure 7: Simplified keypad for exercise 6f. (e) Let us observe the M-ary input, M-ary output, and symmetric channel for which Pr{y j x i } = Ps M (if i j) and 0 < P s <. Find this channel capacity. (f) A simplified numerical keypad is given in Figure 7. It has four keys (in two rows, each row containts two keys). A user planning to push key x, will push another key in the same row with probability α or another key in the same column with α. That means, that the user with push the intended key x with probability 2α. Such a keypad can be described as a transmission channel with the input being the user s intention and the output the actually pushed key. Find the matrix describing this channel and its capacity. (g) N identical binary symmetric channels, each with bit error rate 0 < p <, is connected in cascade. Show that the capacity of such a cascade is equal to: C N = + p N lg p N + ( p N ) lg ( p N ), [ gdzie p N = ] 2 + ( 2p) N. (h) A weather forecast output for some town is given in Table 2. A man noted that the forecast is accurate only in 3 /4 cases (show why). He also noted that he would be right in 3 /6 of all cases, if he always foresees that it will be sunny. He suggested to a local radio station that it is better to always say in a weather forecast that it will be sunny. They rejected his proposal, using the reasoning based on the information theory concept of a channel capacity. Reconstruct this reasoning. 7. Various (*): (a) On the basis of the following property of the logarithm: { x if x = ln x = y < x if x Page

12 Table 2: Example weather forecast results for exercise 6h Reality Forecast Rains Sunny Rains /8 3/6 Sunny /6 0/6 prove the following theorem: When a source generates N messages (,..., N), each of which is generated with the same probability p i, then entropy of this source can be characterized with the following property: H(X) = H(p, p 2,..., p N ) lg N, The equality holds iff i p i = N. (b) Show that information source X which generates messages x 0, x 2,..., x N following the binomial distribution (Bernoulli s distribution), i.e., for 0 i N, 0 < p < and q = p: ( ) N Pr{X = x i } = p i q N i, i can be characterized with the following entropy: H(X) N(p log 2 p + q log 2 q). A few useful relationships are going to be given in the following form in a test sheet: H(X, Y ) = H(X) + H(Y X) (chain rule) I(X; Y ) = H(X) H(X Y ) = H(Y ) H(Y X) = H(X) + H(Y ) H(X, Y ) X, Y independent: H(X, Y ) = H(X) + H(Y ) lg Bibliography The contents of this lecture is based on the following books: Dominic Welsh. Codes and Cryptography. Clarendon Press, Oxford, UK, 988: chapters, 2., , 3.4, , 6, appendices -2. Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. John Wiley & Sons, Inc., New York, NY, 99: chapters 2, 4, 8, 6. Gareth A. Jones and J. Mary Jones. Information and Coding Theory. Springer-Verlag London Ltd., London, UK, 2000: chapters 2.6, , , , Todd K. Moon. Error Correction Coding. John Wiley & Sons, Inc., Hoboken, NJ, 2005: chapters.-.3, Stefan M. Moser and Po-Ning Chen. A Student s Guide to Coding and Information Theory. Cambridge University Press, Cambridge, UK, 202: chapters.-.3, , 5.8, 6. Page 2