ESE 523 Information Theory

Transcription

1 ESE 53 Iformato Theory Joseph A. O Sullva Samuel C. Sachs Professor Electrcal ad Systems Egeerg Washgto Uversty Urbauer Hall 0E Gree Hall jao@wustl.edu 09// J. A. O'Sullva, ESE 53, Lecture -6

2 Etropy Bary Etropy Fucto 0.8 Outle p, p H p p log p plog p Probablty of Oe Etropy Jot Etropy Codtoal Etropy Relatve Etropy Mutual Iformato H p log p H, Y p, ylog p, y yy H Y p, ylog p y yy p D p q p log q p, y I ; Y p, ylog p p y yy 09// J. A. O'Sullva, ESE 53, Lecture -6

3 Notato : Radom varable R.V. Alphabet dscrete: Probablty mass fucto: P = p p p p log=log 0, p = {,, } 09// J. A. O'Sullva, ESE 53, Lecture -6 Based co flp: ={h,t}; p=p,-p Two dce: ={,3,4,5,6,7,8,9,0,,}; p=,,3,4,5,6,5,4,3,,/36 Powerball: 59 p , 49, 054 3

4 Measure of Iformato: Etropy The etropy of, H s: H p log p Uts are bts Measure of ucertaty of a R.V. H E log p log E p the eerly self-referetal epectato Cover ad Thomas, p. 4 09// J. A. O'Sullva, ESE 53, Lecture -6 4

5 Etropy Eample : Determstc R.V. p ad p j 0 H 0 j No formato gaed from observg the outcome log log 0 lm log 0 0 Proof uses l'hoptal's ˆ rule: log/ / lm log lm lm log e 0 / / // J. A. O'Sullva, ESE 53, Lecture -6 5

6 Etropy Eample : Flp a far co = {h,t}; ph pt H log log bt 09// J. A. O'Sullva, ESE 53, Lecture -6 6

7 Etropy Eample 3: Flp a far co tmes = {h,h, h,h,h, t, t,t, t} p,, H log bts 09// J. A. O'Sullva, ESE 53, Lecture -6 7

8 Etropy Eample 4: Powerball, or ay other uform dstrbuto. = {,,..., M}; p, for all M M H log M log M M H Powerball log // J. A. O'Sullva, ESE 53, Lecture -6 8

9 Etropy Eample 5: Flp a far co tmes ad add the umber of heads = {0,,}; p0 p, p 4 H log log log bts 09// J. A. O'Sullva, ESE 53, Lecture -6 9

10 Propertes ad Remarks Etropy s the epected umber of bary questos oe eeds to ask to determe the value of a R.V. Last eample: o average how may yes-o questos to determe outcome? Aswer:.5 questos Etropy s oegatve Base chage: other uts H b Elog b p log b a Elog a p ats for base e at = log e bts =.44 bts 09// J. A. O'Sullva, ESE 53, Lecture -6 0

11 Etropy Bary Etropy Fucto = {0,}; p p H plog p plog p H p Bary Etropy Fucto Probablty of Oe 09// J. A. O'Sullva, ESE 53, Lecture -6

12 Matlab Fucto etropy.m fucto et=etropyp p=szep; f legthp>, p=reshapep,prodp,; ed p=fdadp>0,p<; pp=pp/sumpp; hhp=-pp.*logpp; et=sumhhp; 09// J. A. O'Sullva, ESE 53, Lecture -6

13 Matlab Fucto plotbetropy.m p=0.005:0.005:-0.005; oep=-p; et=-p.*logp-oep.*logoep; et=[0 et 0]; p=[0 p ]; fgure=fgure; aes = aes'fotsze',6,'paret',fgure; ttleaes,'bary Etropy Fucto'; labelaes,'probablty of Oe'; ylabelaes,'etropy'; boaes,'o'; holdaes,'all'; plotp,et,'lewdth', 09// J. A. O'Sullva, ESE 53, Lecture -6 3

14 Eample 6: Etropy as Aswer to Combatorcs Questo, Lecture Assume = m. There are trals. How may ways are there to get k, k, k m of the elemets k +k + +k m =? Operatoal role of etropy for a combatorcs questo.! kk... km k! k!... km! k k k k km km log log... log o k k km h,,..., o Theorem: log h p, p,..., p kk... k m k k km f p, p,..., p m m 09// J. A. O'Sullva, ESE 53, Lecture -6 4

15 Deftos The jot etropy of R.V. s ad Y s: H,Y yy E log p,y p,ylog p,y The codtoal etropy of Y gve s: HY p py log py yy E log py 09// J. A. O'Sullva, ESE 53, Lecture -6 5

16 Etropes Theorem: Proof: p,y ppy H,Y H HY HY H Y log p,y log p log py p,ylog p,y p,ylog p p,ylog py,y,y,y p,ylog p,y plog p p,ylog py,y H,Y H HY,y 09// J. A. O'Sullva, ESE 53, Lecture -6 6

17 Defto The relatve etropy betwee probablty dstrbuto fuctos p ad q s: p p D p q p log Ep log q q Not a true dstace: Dp q Dq p 09// J. A. O'Sullva, ESE 53, Lecture -6 7

18 Matlab Fucto reletropy.m fucto relet=reletropyp,q p=szep; q=szeq; f p~=q, errormess='matlab fucto reletropy error: dm msmatch' retur ed f legthp>, p=reshapep,prodp,; q=reshapeq,prodp,; ed p=fdadp>0,q>0; rpq=pp.*logpp./qp; % Gves the wrog aswer f qk=0 ad pk~=0 relet=sumrpq; 09// J. A. O'Sullva, ESE 53, Lecture -6 8

19 Relatve Etropy Dp q Matlab Fucto plotreletropy.m fucto re=plotreletropyq; p=0.005:0.005:-0.005; oep=-p; re=p.*logp/q+oep.*logoep/-q; re=[-log-q re -logq]; p=[0 p ]; fgure=fgure; aes = aes'fotsze',6,'paret',fgure; ttleaes,strcat'bary Relatve Etropy q=',umstrq; labelaes,'probablty p'; ylabelaes,'relatve Etropy Dp q'; boaes,'o'; 0.8 holdaes,'all'; plotp,re,'lewdth', 0.6 grd 0.4 Bary Relatve Etropy q= // J. A. O'Sullva, ESE 53, Lecture Probablty p

20 Image of Relatve Etropy Fucto 09// J. A. O'Sullva, ESE 53, Lecture -6 0

21 Defto The mutual formato betwee ad Y s: I ; Y D p, y p p y yy p, y p, ylog p p y Some Propertes: I ; Y H H Y H Y H Y I ; Y H H Y H, Y 3 I ; Y I Y; 4 I ; Y 0 09// J. A. O'Sullva, ESE 53, Lecture -6

22 Propertes of Mutual Iformato Proof: I;Y H H Y HY HY p, Y I ; Y E log p p Y E log E log p Y p Elog E log p p Y H H Y 09// J. A. O'Sullva, ESE 53, Lecture -6

23 Matlab Fucto mutualformato.m fucto fo=mutualformatop p=p/sumsump; p=sump,; py=sump,; fo=etropyp+etropypy-etropyp; I ; Y H H Y H H, Y H Y H H Y H, Y 09// J. A. O'Sullva, ESE 53, Lecture -6 3

24 Etropy Last Class Outle p, p H p p log p plog p Bary Etropy Fucto Probablty of Oe Etropy Jot Etropy Codtoal Etropy Relatve Etropy Mutual Iformato H p log p H, Y p, ylog p, y yy H Y p, ylog p y yy p D p q p log q p, y I ; Y p, ylog p p y yy 09// J. A. O'Sullva, ESE 53, Lecture -6 4

25 Eample: Etropy ad Mutual Iformato p, y 0 ; values of colums, y rows p ; p y H H Y log log log log H, Y 4 log log // J. A. O'Sullva, ESE 53, Lecture -6 I ; Y H H Y H, Y log

26 Telescopg Sums: Etropy Theorem: Proof: E log E log E p Commets: p log Geeralzato of two varable case Eample for three varables, H,,..., H,... p,... p 09// J. A. O'Sullva, ESE 53, Lecture -6,..., p 3,... p,..., 6 H, H H H,, H H 3 H, 3

27 09// J. A. O'Sullva, ESE 53, Lecture -6 7 Telescopg Sums: Mutual Iformato Defto: Theorem: Proof:, ; Z Y H Z H Z Y I Y I Y I,... ; ;,...,,,...,...,,,...,...,, Y H Y H H H

28 l-+ l Towards Jese s Iequalty: Cove ad Cocave Fuctos Defto: A fucto f s cove over a,b f for ay, Є a,b ad λ Є [0,], f f f A fucto f s cocave f f s cove. f s strctly cove or cocave f the equaltes are strct for λ 0 or..4 Jeses Iequalty Jeses Iequalty // J. A. O'Sullva, ESE 53, Lecture -6 8

29 09// J. A. O'Sullva, ESE 53, Lecture -6 9 Towards Jese s Iequalty: Suffcet Codto for Covety Theorem: Suppose that f s twce cotuously dfferetable. If d f/d s oegatve postve everywhere, the f s cove strctly cove. Proof: Usg a Taylor seres appromato, where * s some value betwee ad 0. Take * 0 0 o o d f d d df f f f f f f d df f f d df f f

30 Fucto Values: l, -, -/, ep-- Cove ad Cocave Fucto Eamples f = log s strctly cocave. Proof: df/d = loge/ ; d f/d = - loge/ < 0 f = - log s strctly cocave. Proof: df/d = - log-loge ; d f/d = - loge/ < 0 Commet: Ths mples cocavty of etropy H=-Σ plogp m for m s cove for > 0 e s strctly cove Commet: several formato equaltes ca be derved from Relatve etropy s oegatve l p q D p q Ep log log ee p 0 q p 09// J. A. O'Sullva, ESE 53, Lecture Cove ad Cocave Fucto Eamples Varable t 30

31 Jese s Iequalty Theorem Jese s Iequalty: If f s cove over a,b ad s a radom varable takg values a,b, the E f f E If f s strctly cove, the equalty mples that =E[] wth probablty oe. 09// J. A. O'Sullva, ESE 53, Lecture -6 3

32 Proof of Jese s Iequalty Proof by ducto. Let,,. The 3 09// J. A. O'Sullva, ESE 53, Lecture -6 pf p f f p p, by defto. Assume k ad that for ay set of cardalty k- the theorem holds. The k p f p f p k k k k p f k p pk f k pk f, by ducto hypothess pk k p f pk k pk f E[ ], by defto. p k p k

33 Relatve Etropy s Noegatve Theorem: Dp q 0 wth equalty f ad oly f ff p = q. Proof uses Jese s equalty. The fucto log s strctly cocave so log s strctly cove. p q D p q Ep log Ep log q p q log Ep log 0 p Refemet: Need to restrct sums to A p 0 09// J. A. O'Sullva, ESE 53, Lecture -6 33

34 Relatve Etropy s Noegatve Theorem: Dp q 0 wth equalty ff p = q. Proof uses Jese s equalty. log s strctly cove. p q D p q Ep log p log q p A q log p log q p A A log 0 a b where A p 0, a follows from Jese's Iequalty, ad b follows from A 09// J. A. O'Sullva, ESE 53, Lecture -6 q. 34

35 Start Here Sept. 8, 0 Outle Cocavty of etropy Log-sum equalty Covety of relatve etropy Codtog reduces etropy Covety ad cocavty of mutual formato toward optmzato Data processg equalty Chapter 3: Asymptotc equpartto property 09// J. A. O'Sullva, ESE 53, Lecture -6 35

36 Etropy s Cocave ad Bouded Theorem: Etropy s cocave ad bouded above by the log of the cardalty of the set, wth equalty ff the radom varable s uformly dstrbuted. Proof: Cocavty follows from cocavty of log. p H Ep log log Ep log log p p p equalty ff costat p 09// J. A. O'Sullva, ESE 53, Lecture -6 36

37 Log Sum Iequalty Theorem: For ay oegatve umbers a, a,..., a ad b, b,..., b, a alog a log b b 09// J. A. O'Sullva, ESE 53, Lecture -6 0 wth b 0. Assume that f b 0 the a 0 0 log 0. The 0 Proof: By Jese's equalty b b b b b l b l bl l tlog t s cove l l Epected value a b a a b a b a log log. Equalty ff a /b = costat 37

38 Covety of Relatve Etropy Theorem: Dp q s cove the par p,q. Proof: By the log sum equalty p p D p p q q p p log q q p p p log p log = q q D p q D p q Log-sum equalty 09// J. A. O'Sullva, ESE 53, Lecture -6 38

39 Cocavty of Etropy Revsted Let u be a uform dstrbuto. The H p log D p u Proof: p D p u p log log p log p log H p Covety of relatve etropy mples cocavty of etropy. 09// J. A. O'Sullva, ESE 53, Lecture -6 39

40 Jese s Iequalty Summary Theorem Jese s Iequalty: If f s cove over a,b ad s a radom varable takg values a,b, the If f s strctly cove, the equalty mples that =E[] wth probablty oe. Corollary: Dp q 0 wth equalty ff p = q. Corollary: I;Y 0, wth equalty ff p,y = ppy; that s, ff ad Y are depedet. Corollary: Codtog reduces etropy. I;Y 0 H H Y. Commet: We ofte use ths corollary proofs. 09// J. A. O'Sullva, ESE 53, Lecture -6 E f f E 40

41 Mutual Iformato Cocavty ad Covety Motvato Chael capacty ad ts computato Mamze mutual formato over put probablty dstrbuto Mamzato problems are better-behaved for cocave fuctos To show: mutual formato s cocave the put probablty dstrbuto Rate-dstorto fuctos ad ther computato Mmze mutual formato over chael trasto probabltes Mmzato problems are better-behaved for cove fuctos To show: mutual formato s cove the chael probabltes Computatos ad propertes of mutual formato multtermal formato theory Curret research problems 09// J. A. O'Sullva, ESE 53, Lecture -6 4

42 Mutual Iformato Vew mutual formato as a fucto of p ad of py. The mutual formato s a cocave fucto of p for py fed ad a cove fucto of py for p fed. Proofs follows from cocavty of etropy ad covety of relatve etropy. I ; Y H Y p H Y I ; Y D p, y p p y Cosder p y = p y p y p, y p p y p y p, y p, y p y p y p y Cocavty of HY cocavty wrt p 09// J. A. O'Sullva, ESE 53, Lecture -6 Covety of relatve etropy log-sum equalty D p, y p p y D p y p p p y D p, y p p y 4

43 09// J. A. O'Sullva, ESE 53, Lecture Data Processg Iequalty Defto: The radom varables, Y, ad Z form a Markov cha that order f pz,y = pz y. The p,y,z = ppy pz y. Also, ad Z are codtoally depedet gve Y. Wrte Y Z.,, y z p y p y p y z p y p y z p

44 09// J. A. O'Sullva, ESE 53, Lecture Data Processg Iequalty Theorem: If Y Z, the I;Y I;Z. Note that ths says Y gves more formato about tha Z does. Proof: But I;Z Y = 0, so I;Y I;Z. Commet: ; ; ; ;, ; Z Y I Z I Y Z I Y I Z Y I Y H Z H Z H H Y H H

45 Chapter 3: Asymptotc Equpartto Property Strog law of large umbers weak law Asymptotc equpartto property AEP All hghly lkely sequeces are equally lkely The set of hghly lkely sequeces s the typcal set The cardalty of the typcal set s determed by etropy Data compresso result: Number of bts requred to represet sequeces o average equals etropy tmes the legth of the sequece Number of bts per symbol, o average, equals etropy 09// J. A. O'Sullva, ESE 53, Lecture -6 46

46 09// J. A. O'Sullva, ESE 53, Lecture Theorem Strog Law of Large Numbers Let,,, be a sequece of..d. RVs. Let f: R be a arbtrary fucto such that E[ f ] s fte. The wth probablty oe. If the varace of f s fte, ths covergece s the mea also. Commet: I ether evet, we get covergece probablty I fact 3. lm f E f var where 0 as f f E f P f E f P

47 Theorem If,,, are..d. wth dstrbuto p, the log p,,..., H probablty. Proof: p,,, = p p p Set f = -log p the prevous theorem. E p f E log H. Commet: Aga log p s a fucto of the realzato. 09// J. A. O'Sullva, ESE 53, Lecture -6 48

48 Typcal Sets Defto: The typcal set s A,,..., : log p,,..., H Commet: Ths s the set of sequeces whose ormalzed log-probablty s close to etropy. Theorem H H p A for P A A A H for suffcetly large 09// J. A. O'Sullva, ESE 53, Lecture -6 H for suffcetly large 49

49 Typcal Sets 09// Defto: The typcal set s A,,..., : log p,,..., H Commet: Ths s the set of sequeces whose ormalzed log-probablty s close to etropy. Theorem H H p A for P A A A H for suffcetly large H for suffcetly large J. A. O'Sullva, ESE 53, Lecture -6 50

50 Proof Frst le s defto of typcal set. Secod le follows from prevous theorem. Thrd ad fourth les: p p A A H 09// J. A. O'Sullva, ESE 53, Lecture -6 H A A H For the fourth le, P A for suffcetly large H p A A 5

51 Typcal Sets ad the AEP Theorem Commets: H H p A for P A A A H for suffcetly large H The typcal set has probablty arbtrarly close to. The log-cardalty of the typcal set s upper bouded by etropy plus ε for suffcetly large The log-cardalty s lower bouded by etropy mus ε for large eough H log A H log H 09// J. A. O Sullva, ESE 53, Lectures -5 5

52 Data Compresso Idea: Partto all outcomes to the typcal ad otypcal sets for some ε. Desg a reasoable code for the typcal set ad do aythg else for the rest. Defto: A bary code s a mappg from to bary sequeces. Theorem: Let be..d. wth probablty dstrbuto p ad let ε>0. The there ests a bary code that s oe-to-oe ad E l H for suffcetly large, where l s the legth of a bary codeword assged to. 09// J. A. O'Sullva, ESE 53, Lecture -6 53

53 Proof To every sequece the typcal set, assg a codeword of legth less tha or equal to H+ε+. To every sequece ot the typcal set, assg a codeword of legth less tha or equal to log + The the epected legth satsfes E l H log H log 09// J. A. O'Sullva, ESE 53, Lecture -6 54

54 Chapter 4 Outle Etropy Rates of Stochastc Processes Two epressos: equal for statoary processes Markov chas etropy rates Net Class: Markov chas decreasg codtoal etropy secod law of thermodyamcs 09// J. A. O'Sullva, ESE 53, Lecture -6 55

55 Etropy Rates of a Stochastc Process Etropy rates bts per symbol Stochastc process:,,,, a radom sequece s a RV; ; possbly cofusg otato P Structure of the radom sequece must be assumed to make progress Defto: A stochastc process,,,, s statoary f the jot dstrbuto s varat to shfts; for all l 0, P,..., P,,...,, l l l 09// J. A. O'Sullva, ESE 53, Lecture -6 56

56 Etropy Rates Defto: The etropy rate of a stochastc process { } s H lm H,,..., whe the lmt ests. Proposto: If are..d., the Proof: H Commets:,,..., H H If are depedet, but ot detcally dstrbuted, the frst equalty holds. However the lmt lm H may or may ot est H H A secod possble defto for etropy rate s H lm H,,...,, whe the lmt ests. 09// J. A. O'Sullva, ESE 53, Lecture -6 57

57 Etropy Rates Theorem: For a statoary stochastc process, H ad H est ad are equal. Proof: There are three parts: H ests; a techcal result Cesáro mea; ad H ests ad equals H. H ests: 0 H H H H,,,... by statoarty Thus s a ocreasg sequece of oegatve umbers. Thus t has a lmt. 09// J. A. O'Sullva, ESE 53, Lecture -6,,...,... 3,,... codtog reduces etropy 58

58 Proof cotued Cesáro mea: If a a ad b =a +a + a /, the b a. Completo: H,,..., H,,..., Thus, H lm H,,..., lm H,,..., H 09// J. A. O'Sullva, ESE 53, Lecture -6 b a 59

59 09// J. A. O'Sullva, ESE 53, Lecture Applcatos All results from Chapter 3 hold ths cotet, cludg deftos of typcal sets, the AEP, ad the data compresso. Also,...,,,...,,,...,,,...,,,...,, H H H H H

60 Outle September 5, 0 Markov Cha Propertes, Classfcato Etropy rate of Markov chas Markov chas Decreasg codtoal etropy Secod law of thermodyamcs 09// J. A. O'Sullva, ESE 53, Lecture -6 6

61 Iformato Dverso of the Day James Gleck: The Iformato: A Hstory, a Theory, a Flood 03/0/the-formato-by-james-gleckrevew-by-cholas-carr.html revew/book-revew-the-formato-byjames-gleck.html?pagewated=all 09// J. A. O'Sullva, ESE 53, Lecture -6 The Iformato s so ambtous, llumatg ad sely theoretcal that t wll amout to aspratoal readg for may of those who have the mettle to tackle t. Do t make the mstake of readg t quckly. Image luuratg o a W-F-equpped desert slad wth Mr. Gleck s book, a search ege ad o dstractos. The Iformato s to the ature, hstory ad sgfcace of data what the beach s to sad. Jaet Masl, The New York Tmes 6

62 Markov Chas Defto: A stochastc process { } s a Markov cha f P,..., P For a Markov cha, p,,..., p p... p Defto: A Markov cha s tme-varat f the trasto probabltes do o deped o. S 0 S S 09// J. A. O'Sullva, ESE 53, Lecture -6 63

63 09// J. A. O'Sullva, ESE 53, Lecture Markov cha otato: tme-varat case s called the state at tme. If =m s fte, the probablty trasto matr s statoary stochastc process. a the the M arkovcha s,, for all If a statoary dstrbuto. s the, If... m j P P P P p p p P p p p P S 0 S S

64 Markov Cha Propertes S 0 S S Defto: If for all ad j, there s a k such that k P 0, j the Markov cha s rreducble coected. If there s a k such that k P 0, j for all ad j, the Markov cha s strogly coected rreducble ad aperodc. Commet: strogly coected rreducble coected 09// J. A. O'Sullva, ESE 53, Lecture -6 66

65 Three-State Eample q=r=rreducble, perodc wth perod 3 q=;r=0.5 rreducble ad aperodc strogly coected S 0 S S 0 q q P r 0 r q r P 0 0 ; P P P 0 0 ; P 0 0 ; q P ; r P 0 ; P 0 ; ; P ; P // J. A. O'Sullva, ESE 53, Lecture

66 Two-State Eample Let P If ether α=0 or β=0, the Markov cha s ot coected. For α 0 ad β 0, the statoary dstrbuto s If α=β=, the Markov cha s coected, but ot strogly coected. 09// J. A. O'Sullva, ESE 53, Lecture -6 68

67 Etropy rates of Markov chas Theorem: Let { } be a statoary Markov cha. The the etropy rate s H P logp Proof: j j j H lm H,..., H P P j log P j j j P logp j j 09// J. A. O'Sullva, ESE 53, Lecture -6 69

68 Etropy rates of Markov chas Theorem: Let { } be a tme-varat Markov cha that s rreducble ad aperodc. The the etropy rate s H Pj logpj j where μ s the statoary dstrbuto. Proof: H lm H,..., lm H lm P P log P j j j 09// J. A. O'Sullva, ESE 53, Lecture -6 70

69 log log log Thus,. ad. or, as m j j j m j m j m j m j m j m j j j j D P p P P P p P P p D P p p P P p p p 09// J. A. O'Sullva, ESE 53, Lecture -6 7 Proof cotued All that remas to be show s that Note that The equalty s the log sum equalty; get equalty ff μ P j =p - P j for all ad j, or μ =p - for P strogly coected. Ths s a formatotheoretc proof of covergece.

70 Markov Chas ad Tme Let,,, be a Markov cha. Suppose that p - does ot deped o tme. Let μ be a dstrbuto at tme. The. The relatve etropy betwee two dstrbutos decreases wth. The relatve etropy betwee a dstrbuto ad a statoary dstrbuto decreases wth 3. Etropy creases wth f the statoary dstrbuto s uform d Law of Thermodyamcs 09// J. A. O'Sullva, ESE 53, Lecture -6 7

71 The relatve etropy betwee two dstrbutos decreases wth. Suppose two possble probablty dstrbutos at tme are gve p P ad. There are two correspodg probablty dstrbutos at tme m p j p P ad j P. j j The goal s to prove that D p D p. To show ths, 09// J. A. O'Sullva, ESE 53, Lecture -6 m m m j j j j j Pj D p P P p P log p log m m Pj p D p j D p P P j j D p p P p Pj m m p Pj p j p j p j log log j p j P j j j 73

72 The relatve etropy betwee a dstrbuto ad a statoary dstrbuto decreases wth Let be the statoary dstrbuto. The j P ad from the prevous j j result, D p D p. m 09// J. A. O'Sullva, ESE 53, Lecture -6 74

73 d Law of Thermodyamcs: Etropy creases wth f the statoary dstrbuto s uform Let = be the statoary dstrbuto. The p D p p log log H p D p D p H p H p 09// J. A. O'Sullva, ESE 53, Lecture -6 75