Entropies & Information Theory

Transcription

1 Etropies & Iformatio Theory LECTURE I Nilajaa Datta Uiversity of Cambridge,U.K. For more details: see lecture otes (Lecture 1- Lecture 5) o

2 Quatum Iformatio Theory Bor out of Classical Iformatio Theory Mathematical theory of storage, trasmissio & processig of iformatio Quatum Iformatio Theory: how these tasks ca be accomplished usig quatum-mechaical systems as iformatio carriers (e.g. photos, electros, ) Quatum Physics Quatum Iformatio Theory Iformatio Theory

3 The uderlyig quatum mechaics distictively ew features These ca be exploited to: improve the performace of certai iformatio-processig tasks as well as accomplish tasks which are impossible i the classical realm!

4 He posed 2 questios: Classical Iformatio Theory: 1948, Claude Shao (Q1) What is the limit to which iformatio ca be reliably compressed? relevace: there is ofte a physical limit to the amout of space available for storage iformatio/data e.g. i mobile phoes (Q2) What is the maximum amout of iformatio that ca be trasmitted reliably per use of a commuicatios chael? relevace: biggest hurdle i trasmittig ifo is presece of oise i commuicatios chaels, e.g. cracklig telephoe lie, iformatio = data =sigals= messages = outputs of a source

5 He posed 2 questios: Classical Iformatio Theory:1948, Claude Shao (Q1) What is the limit to which iformatio ca be reliably compressed? (A1) Shao s Source Codig Theorem: data compressio limit = Shao etropy of the source (Q2) What is the maximum amout of iformatio that ca be trasmitted reliably per use of a commuicatios chael? (A2) Shao s Noisy Chael Codig Theorem: maximum rate of ifo trasmissio: give i terms of the mutual iformatio

6 Shao: iformatio What is iformatio? ucertaity Iformatio gai = decrease i ucertaity of a evet measure of iformatio measure of ucertaity Surprisal or Self-iformatio: Cosider a evet described by a radom variable (r.v.) X p( x) (p.m.f); x J (fiite alphabet) A measure of ucertaity i gettig outcome : ( x): log p( x) a highly improbable outcome is surprisig! rarer a evet, more ifo we gai whe we kow it has occurred px ( ) oly depeds o -- ot o values take by cotiuous; additive for idepedet evets x x log log 2 X

7 Shao etropy = average surprisal Def: Shao etropy H ( X ) of a discrete r.v. X p( x), H ( X) ( ( X)) px ( )log px ( ) xj Covetio: 0log01 lim wlog w 0 w0 log log 2 (If a evet has zero probability, it does ot cotribute to the etropy) H ( X ) : a measure of ucertaity of the r.v. also quatifies the amout of ifo we gai o average whe we lear the value of X X x J H ( X) H( p ) H p( x) X p ( ) X p x x J

8 Example: Biary Etropy X p( x) J {0,1} p(0) p; p(1) 1 p; H ( X) plog p(1 p)log(1 p) h( p) Properties p p 0 x 1 h( p) 0 hp ( ) 1 x0 o ucertaity p 0.5 : h( p) 1 Cocave fuctio of Cotiuous fuctio of maximum ucertaity p p

9 Operatioal Sigificace of the Shao Etropy = optimal rate of data compressio for a classical memoryless (i.i.d.) iformatio source Classical Iformatio Source Outputs/sigals : sequeces of letters from a fiite set J : source alphabet (i) biary alphabet J {0,1} (ii) telegraph Eglish : 26 letters + a space (iii) writte Eglish : 26 letters i upper & lower case + puctuatio J

10 Simplest example: a memoryless source successive sigals: idepedet of each other characterized by a probability distributio O each use of the source, a letter u J pu Modelled by a sequece of i.i.d. radom variables U p( u) ( ) u J emitted with prob U, U,..., U u J 1 2 i pu ( ) PU ( u), uj1 k. k p( u) Sigal emitted by uses of the source: u ( u, u,..., u ) u 1 2 p( u) P( U u, U u,..., U u ) p( u ) p( u )... p( u ) Shao etropy of the source: HU ( ): pu ( )log pu ( ) uj

11 (Q) Why is data compressio possible? (A) There is redudacy i the ifo emitted by the source -- a ifo source typically produces some outputs more frequetly tha others: I Eglish text e occurs more frequetly tha z. --durig data compressio oe exploits this redudacy i the data to form the most compressed versio possible Variable legth codig: -- more frequetly occurrig sigals (e.g e ) assiged shorter descriptios (fewer bits) tha the less frequet oes (e.g. z ) Fixed legth codig: -- idetify a set of sigals which have high prob of occurrece: typical sigals -- assig uique fixed legth (l) biary strigs to each of them -- all other sigal (atypical) assiged a sigle biary strig of same legth (l)

12 Typical Sequeces Def: Cosider a i.i.d. ifo source : 0, U, U,... U ; p( u) ; u J 1 2 For ay sequeces 1 2 u: ( u, u,... u ) J for which ( H( U) ) ( H( U) ) 2 pu ( 1, u2,... u ) 2, where HU ( ) are called typical sequeces Shao etropy of the source ( T ) : typical set = set of typical sequeces Note: Typical sequeces are almost equiprobable u ( T ), pu ( ) 2 H U ( )

13 u ( T ), pu ( ) 2 H U ( ) U1 U2 U,,... ; pu ( ) ; u J (Q) Does this agree with our ituitive otio of typical sequeces? (A) Yes! For a i.i.d. source : U, U,... U ; U p( u) ; u J 1 2 i A typical sequece u u1 u2 u : (,,... ) of legth p( u ) u, u J is oe which cotais approx. copies of, Probability of such a sequece is approximately give by uj p( u)log p( u) p( u) p( u)log p( u) ( ) = 2 2 u J uj pu ( ) 2 H U

14 Properties of the Typical Set T T Let : umber of typical sequeces PT : probability of the typical set Typical Sequece Theorem: Fix the ad large eough, 0, 0, PT 1 J T A (1 )2 T 2 (disjoit uio) ( H( U) ) ( H( U) ) sequeces i the atypical set rarely occur ( P A ) atypical set typical sequeces are almost equiprobable

15 Operatioal Sigificace of the Shao Etropy (Q) What is the optimal rate of data compressio for such a source? [ mi. # of bits eeded to store the sigals emitted per use of the source] (for reliable data compressio) Optimal rate is evaluated i the asymptotic limit umber of uses of the source Oe requires p error 0 ; (A) optimal rate of data compressio = HU ( ) Shao etropy of the source

16 Compressio-Decompressio Scheme U, U,... U ; U p( u) ; u J Suppose is a i.i.d. iformatio 1 2 i Shao etropy source A compressio scheme of rate Whe is this a compressio scheme? Decompressio: Average probability of error: H ( U ) R : E : u: ( u1, u2,... u ) 1 2 J D Compr.-decompr. scheme reliable if m : 0,1 p av x : ( x, x,... x m ) lim av m J R m 0,1 pup ( ) D( E( u)) u u p 0 as

17 Shao s Source Codig Theorem: U, U,... U ; U p( u) ; u J Suppose is a i.i.d. iformatio 1 2 i Shao etropy source H ( U ) R Suppose : the there exists a reliable compressio scheme of rate H( U) R for the source. R H( U) If the ay compressio scheme of rate will ot be reliable. R

18 Shao s Source Codig Theorem: U, U,... U ; U p( u) ; u J Suppose is a i.i.d. iformatio 1 2 i Shao etropy source H ( U ) R Suppose : the there exists a reliable compressio scheme of rate H( U) R for the source. Sketch of proof (achievability)

19 R Shao s Source Codig Theorem (proof cotd.) H( U) If the ay compressio scheme of rate will ot be reliable. Proof follows from: S (coverse) u u1 u2 u R S 2 R H( U) Lemma: Let ( ) be a set of sequeces of legth of size, where u Each sequece is produced with prob. 0, The for ay ad sufficietly large, S u S pu ( ) pu R : (,,... ) ( ) 2 R R H( U) is fixed. if is a set of at most sequeces with, the with a high probability the source will produce sequeces which will ot lie i this set. Hece ecodig sequeces reliable data compressio 2 R

20 Etropies for a pair of radom variables Cosider a pair of discrete radom variables X p( x) ; x JX ad Y p( y) ; y JY Give their joit probabilities & their coditioal probabilities P( X x, Y y) p( x, y) ; P( Y y X x) p( y x) ; Joit etropy: H( X, Y): p( x, y) log p( x, y) xj X yj Y Coditioal etropy: HY ( X): pxhy ( ) ( X x) p( xy, ) log py ( x) xj X xj X yj Y Chai Rule: H( X, Y) H( Y X) H( X)

21 Etropies for a pair of radom variables Relative Etropy: Measure of the distace betwee two probability distributios p p( x) ; q q( x) xj p( x) D( p q): p( x)log xj qx ( ) xj covetio: 0 u 0log 0 ; ulog u 0 u 0 D( p q) 0 D( p q) 0 if & oly if p q BUT ot a true distace ot symmetric; does ot satisfy the triagle iequality

22 Etropies for a pair of radom variables Mutual Iformatio: Measure of the amout of ifo oe r.v. cotais about aother r.v. X p( x), Y p( y) pxy (, ) I( X, Y): p( x, y)log xy, p( x) p( y) I( X : Y) D( p p p ) XY X Y p pxy (, ) ; p px ( ) ; p py ( ) XY xy, X x Y y Chai rules: I( X : Y) H( X) H( Y) H( X, Y) H( X) H( X Y) HY ( ) HY ( X)

23 Properties of Etropies Let X p( x), Y p( y) be discrete radom variables: The, H( X) 0, with equality if & oly if is determiistic H ( X) log J, if x J Subadditivity: H( X, Y) H( X) H( Y), p X Cocavity: if & are 2 prob. distributios, H ( p (1 ) p ) H( p ) (1 ) H( p ), X Y X Y HY ( X) 0, or equivaletly H( X, Y) H( Y), p Y X I( X : Y) 0 with equality if & oly if & X are idepedet Y

24 So far. Classical Data Compressio: aswer to Shao s questio (Q1) What is the limit to which iformatio ca be reliably compressed? (A1) Shao s Source Codig Theorem: data compressio limit = Shao etropy of the source 1 st Classical etropies ad their properties

25 Shao s 2 d questio (Q2) What is the maximum amout of iformatio that ca be trasmitted reliably per use of a commuicatios chael? The biggest hurdle i the path of efficiet trasmissio of ifo is the presece of oise i the commuicatios chael Noise distorts the iformatio set through the chael. iput oisy chael output output iput To combat the effects of oise: use error-correctig codes

26 To overcome the effects of oise: istead of trasmittig the origial messages, -- the seder ecodes her messages ito suitable codewords -- these codewords are the set through (multiple uses of) the chael Alice Bob Alice s message ecodig E codeword iput N uses of N output decodig D Bob s iferece Error-correctig code: C : ( E, D ):

27 The idea behid the ecodig: To itroduce redudacy i the message so that upo decodig, Bob ca retrieve the origial message with a low probability of error: The amout of redudacy which eeds to be added depeds o the oise i the chael

28 Memoryless biary symmetric chael (m.b.s.c.) 0 1 p p 1-p 1-p Ecodig: it trasmits sigle bits effect of the oise: to flip the bit with probability p codewords the 3 bits are set through 3 successive uses of the m.b.s.c. Suppose m.b.s.c. codeword Example Decodig : (majority votig) Repetitio Code (Bob receives) (Bob ifers)

29 Probability of error for the m.b.s.c. : without ecodig = p with ecodig = Prob (2 or more bits flipped) := q iferece possible iputs output of 3 uses of a m.b.s.c. 010 Prove: q < p if p < 1/2 -- i this case ecodig helps! (Ecodig Decodig) : Repetitio Code.

30 Iformatio trasmissio is said to be reliable if: -- the probability of error i decodig the output vaishes asymptotically i the umber of uses of the chael Aim: to achieve reliable iformatio trasmissio whilst optimizig the rate the amout of iformatio that ca be set per use of the chael The optimal rate of reliable ifo trasmissio: capacity

31 Discrete classical chael N J X iput alphabet; JY output alphabet uses of N iput x ( ) x J X N output ( ) ( ) ( py x ) y y J Y py ( x ) coditioal probabilities ; kow to seder & receiver

32 Correspodece betwee iput & output sequeces is ot 1-1 x ( ) J X y ( ) J Y ( ) x ' Shao proved: it is possible to choose a subset of iput sequeces-- such that there exists oly : 1 highly likely iput correspodig to a give iput Use these iput sequeces as codewords

33 Trasmissio of ifo through a classical chael Alice s Alice M : m M Alice s message codeword: output: fiite set of messages ecodig E x iput N oisy chael N uses of x ( x1, x2,..., x ); y ( y, y,..., y ); 1 2 N y output N ( ) : decodig D p y Bob m M Bob s iferece ( x ) Error-correctig code: C : ( E, D ):

34 m M Alice s message If m m ecodig E x iput N the a error occurs! y output decodig Ifo trasmissio is reliable if: Prob. of error 0 D as m M Bob s iferece Rate of ifo trasmissio = umber of bits of message trasmitted per use of the chael Aim: achieve reliable trasmissio whilst maximizig the rate Shao: there is a fudametal limit o the rate of reliable ifo trasmissio ; property of the chael Capacity: maximum rate of reliable iformatio trasmissio

35 Shao i his Noisy Chael Codig Theorem: -- obtaied a explicit expressio for the capacity of a memoryless classical chael ( ) ( i i) i1 p y x p y x Memoryless (classical or quatum) chaels actio of each use of the chael is idetical ad it is idepedet for differet uses -- i.e., the oise affectig states trasmitted through the chael o successive uses is assumed to be ucorrelated.

36 Classical memoryless chael: a schematic represetatio X px ( ) N iput x output y x J, p( y x) X y J Y, Y chael: a set of coditioal probs. p( y x) Capacity iput distributios C( N ) max I( X : Y) px ( ) I( X : Y) H( X) H( Y) H( X, Y) Shao Etropy mutual iformatio H( X) px ( )log px ( ) x

37 Shao s Noisy Chael Codig Theorem: For a memoryless chael: X px ( ) iput N p( y x) Y output Optimal rate of reliable ifo trasmissio capacity C( N ) max I( X : Y) px ( ) Sketch of proof