1 p(x) = E log 1. H(X) = plog 1 p +(1 p)log 1 1 p.

Size: px

Start display at page:

Download "1 p(x) = E log 1. H(X) = plog 1 p +(1 p)log 1 1 p."

Cameron James
5 years ago
Views:

1 LECTURE 2 Iformatio Measures 2. ENTROPY LetXbeadiscreteradomvariableoaalphabetX drawaccordigtotheprobability mass fuctio (pmf) p() = P(X = ), X, deoted i short as X p(). The ucertaity about the outcome of X, or equivaletly, the amout of iformatio gaied by observig X, is measured by its etropy H(X) = p() log p() = Elog p(x). X Bycotiuity,weusethecovetio 0log0 = 0itheabovesummatio. Sometimeswe deoteh(x)byh(p()),highlightigthefactthath(x)isafuctioalofthepmf p(). Eample.. If X is a Beroulli radom variable with parameter p = P{X = } [0,] (ishort X Ber(p)),the H(X) = plog p +( p)log p. With a slight abuse of otatio, we deote this quatity by H(p) ad refer to it as the biary etropy fuctio. Figure. plots the biary etropy fuctio. 0 /2 Figure.. The biary etropy fuctio H(p).

2 2 Iformatio Measures Eample.. If X is a geometric radom variable with parameter p (0,]) (i short X Geom(p)),the H(X) = i= p( p) i log = log p + p p log p( p) i p. The etropy H(X) satisfies the followig properties.. H(X) 0.. H(X)isacocavefuctioi p().. H(X) log X.. H(X)isivariatuderayoe-to-oetrasformatio o X,i.e.,H(X) = H(f(X)), provided that f is ijective. Iparticular, H(X) = H(X +a) ad H(X) = H(aX) for a = 0. Thefirstpropertyistrivial. Theproofofthesecodpropertyisleftasaeercise. Forthe proof of the third property, we recall the followig. Lemma. (Jese s iequality). If f() is cove, the E(f(X)) f(e(x)). If f()iscocave,the E(f(X)) f(e(x)). Now by the cocavity of the logarithm fuctio ad Jese s iequality, H(X) = Elog where the last iequality follows sice loge log X, p(x) p(x) E = p(x) : p() =0 p() p() = {: p() = 0} X. Let (X,Y) be a pair of discrete radom variables. The the coditioal etropy of Y give X isdefiedas H(Y X) = Elog p(y X), (. )

3 2. Etropy 3 where p(y ) = p(, y)/p() is the coditioal pmf of Y give {X = }. We sometimes use the otatio H(Y X = ) = H(p(y )), X. I this otatio, H(Y X) = p()p(y ) log y = p()h(y X = ). p(y ) (. ) More geerally, if X is a arbitrary radom variable ady {X = } p(y ) is discrete forevery X,thethecoditioaletropycabedefiedasi(. )ad(. ): H(Y X) = Elog p(y X) = H(Y X = )dp() IfY isdiscrete,thebythecocavityofh(p(y))i p(y)adjese siequality, H(p(y ))dp() Hp(y )dp() = H(p(y)), where the iequality holds with equality if p(y ) p(y), or equivaletly, X ad Y are idepedet. We summarize this relatioship betwee the coditioal ad ucoditioal etropies as follows. Coditioig reduces etropy. H(Y X) H(Y) (. ) withequalityif X ady areidepedet. Let(X,Y) p(, y)beapairofdiscreteradomvariables. Theirjoitetropyis H(X,Y) = Elog p(x,y). Bythechairuleofprobability p(, y) = p()p(y ) = p(y)p( y),wehavethechairule of etropy H(X,Y) = Elog p(x) +Elog p(y X) = H(X)+H(Y X) = H(Y)+H(X Y). More geerally, for a -tuple of radom variables X = (X,X 2,...,X ), we have the followig.

4 4 Iformatio Measures Chai rule of etropy. H(X ) = H(X )+H(X 2 X )+H(X X ) = i= H(X i X i ), where X 0 issettobeauspecifiedcostatbycovetio. Bythechairulead(. ),wecaupperboudthejoitetropyas H(X ) i= H(X i ) withequalityif X,...,X aremutually idepedet. 2.2 DIFFERENTIAL ENTROPY Let X be a cotiuous radom variable draw accordig to the probability desity fuctio(pdf) dp(x ) p() =, Ă, d deotedishortas X p(). Itsdifferetialetropyisdefiedas h(x) = p()log p() d = Elog p(x). Eample2.3. If X is a uiform radom variable over the iterval [a,b] (i short X Uif[a,b]),the b h(x) = log(b a)d a b a = E[log(b a)] = log(b a). Eample2.4. If X isagaussiaradomvariable withmea advariace σ 2 (ishort X N(,σ 2 )),the ( ) 2 h(x) = 2πσ 2e 2 2 log2πσ 2 e ( )2 2 2 d = 2 log(2πσ2 )+ 2σ 2 E[(X )2 ] = 2 log(2πeσ2 ).

5 2.2 Differetial Etropy 5 The differetial etropy h(x) satisfies the followig properties.. h(x)cabeegative.. h(x)isacocavefuctioi p().. h(x) is ivariat uder ay trasformatio that preserves the Lebesgue measure. For eample, h(x) = h( X)ad h(x) = h(x +a).. h(ax) = h(x)+log a fora = 0. If X isaarbitraryradomvariable ady {X = }iscotiuouswithpdf p(y )for every X,thethecoditioaldifferetialetropyofY give X isdefiedas H(Y X) = Elog p(y X) = p(y )log = H(Y X = )dp(). p(y ) dydp() whereh(y X = ) = H(p(y )) deotes thedifferetial etropy of p(y ) forafied. Suppose that Y is cotiuous. As i the discrete case, by cocavity ad Jese s iequality, we have the followig. Coditioig reduces differetial etropy. h(y X) h(y) (. ) withequalityif X ady areidepedet. Let(X,Y) p(, y)beapairofcotiuous radomvariables. Theirjoitdifferetial etropy is h(x,y) = Elog p(x,y) = Elog p(x) +Elog p(y X) = h(x)+h(y X) = h(y)+ h(x Y). Asithediscretecase,fora-tupleofcotiuousradomvariablesX = (X,X 2,...,X ), we have the followig.

6 6 Iformatio Measures Chai rule of differetial etropy. h(x ) = i= h(x i X i ), where X 0 issettobeauspecifiedcostatbycovetio. Bythechairulead(. ),wecaupperboudthejoitetropyas h(x ) i= h(x i ) (. ) withequalityif X,...,X aremutually idepedet. Eample.. Let X = (X,...,X ) be joitly Gaussia radom variables with mea vector ad covariace matri K. The, their differetial etropy is h(x ) = 2 log((2πe) K ), (. ) where K deotes the determiat of K. By (. ), h(x ) i= h(x i ) = 2 log(2πek ii). (. ) By comparig(. ) ad(. ), we ca easily establish the famous Hadamard iequality 2.3 RELATIVE ENTROPY K i= K ii. Let p() ad q() be a pair of pmfs o X. The etet of discrepacy betwee p() ad q() is measured by their relative etropy(also referred to as Kullback Leibler divergece) D(pq) = D(p()q()) = E p log p(x) q(x) = p()log p() q(). X (. ) where the epectatio is take w.r.t. X p(). Note that this quatity is well defied oly whe p() is absolutely cotiuous w.r.t. q(), amely, p() = 0 wheever q() = 0. Otherwise, we defie D(pq) =, which follows by adoptig the covetio /0 =

7 2.4 f -Divergece 7 as well. The relative etropy of two pdfs p() ad q() is similarly defied, with the summatio i(. ) replaced by a itegral. The relative etropy D(pq) satisfies the followig properties.. D(pq) 0withequalityifadolyif(iff)p q,amely,p() = q()forevery X.. D(pq) is ot symmetric, i.e., D(pq) = D(qp) i geeral.. D(pq) iscovei(p,q),i.e.,foray(p,q ),(p 2,q 2 ),ad λ, λ = λ [0,], λd(p q )+ λd(p 2 q 2 ) D(λp + λp 2 λq + λq 2 ).. Chairule.Foray p(, y)adq(, y), D(p(, y)q(, y)) = D(p()q()) +p()d(p(y )q(y )) = D(p()q()) + E p D(p(y X)q(y X)). Theproof ofthefirstthreeproperties isleft asa eercise. For thefourthproperty,cosider D(p(, y)q(, y)) = E p log p(x,y) q(x,y) = E plog p(x) q(x) +E plog p(y X) q(y X). The otio of relative etropy ca be eteded to arbitrary probability measures P ad Qdefiedothesamesamplespaceadsetofevetsas D(PQ) = log dp dq dp, where dp/dqistherado Nikodym derivative of Pw.r.t. Q. (If Pisotabsolutelycotiuousw.r.t. Q,theD(PQ) =.) Iparticular,if Pad Qhaverespectivedesities p adq w.r.t.aσ-fiitemeasure ox,the D(P Q) = D(pq) = p()log p() q() d (). (. ) This defiitio geeralizes our earlier defiitio of the relative etropy for pmfs ad pdfs sice they ca be viewed as desities w.r.t. coutig ad Lebesgue measures, respectively. 2.4 f -DIVERGENCE Wedigressabittogeeralizetheotioofrelativeetropy.Let f : [0, ) Ă Ăbecove with f() = 0. Thethe f-divergecebetweeapairofdesitiespadqw.r.t. isdefied as D f (pq) = q()f p() q() d () = E qf p(x) q(x).

8 8 Iformatio Measures Eample2.6. Let f(u) = ulogu. The D f (pq) = q() p() p() log q() q() d () = p()log p() q() d () = D(pq). Eample2.7. Nowlet f(u) = logu. The D f (pq) = q()log p() q() d () = q()log q() p() d () = D(qp). Eample2.8. Combiig theabovetwocases,let f(u) = (u )logu. The whichissymmetrici(p,q). D f (pq) = D(pq)+D(qp), May basic distace fuctios o probability measures ca be represeted as f-divergeces; see, for eample, Liese ad Vajda( ). 2.5 MUTUAL INFORMATION Let (X,Y) be a pair of discrete radom variables with joit pmf p(, y) = p()p(y ). Theamoutofiformatioaboutoeprovidedbytheotherismeasuredbytheirmutual iformatio I(X;Y) = D(p(, y)p()p(y)) p(, y) = p(, y)log,y p()p(y) = p()p(y )log p(y ) y p(y) = p()d(p(y )p(y)). The mutual iformatio I(X; Y) satisfies the followig properties.. I(X;Y)isaoegativefuctioof p(, y).. I(X;Y) = 0iff X ady areidepedet,i.e., p(, y) p()p(y).

9 2.5 Mutual Iformatio 9. Asafuctioof(p(), p(y )), I(X;Y)iscocavei p()forafied p(y ),adcovei p(y )forafied p().. Mutual iformatio ad etropy. ad. Variatioal characterizatio. I(X;X) = H(X) I(X;Y) = H(X) H(X Y) = H(Y) H(Y X) = H(X)+H(Y) H(X,Y). I(X;Y) = mi q(y) p()d(p(y )q(y)), (. ) wherethemiimumisattaiedbyq(y) p(y). Theproofofthefirstfourpropertiesisleftasaeercise. Forthefifthproperty,cosider I(X;Y) = p(, y)log p(y ),y p(y) = p()p(y )log p(y ) q(y),y q(y) p(y) = p()d(p(y )q(y)) p(y)log p(y) y q(y) p()d(p(y )q(y)), where the last iequality follows sice the subtracted term is equal to D(p(y)q(y)) 0, adholdswithequalityiff p(y) q(y). The otio of mutual iformatio ca be geeralized to a arbitrary pair of radom variables. Let(X,Y)be draw accordig totheprobability distributio P X,Y, adlet P X ad P Y bethemargialdistributiosof X ady,respectively. Thethemutualiformatiobetwee X ady isdefiedas I(X;Y) = D(P X,Y P X P Y ). (. ) Recallthatthejoitdistributio P X,Y isalwaysabsolutelycotiuousw.r.t.theproductof themargialdistributios, P X P Y. Thismutualiformatiocabeepressedequivaletly as I(X;Y) = sup I( (X); ỳ(y)), (),ỳ(y) where the supremum is over all fiite-valued fuctios () ad ỳ(y).

10 0 Iformatio Measures Asaspecialcase,supposethatXisdiscretewithpmfp()adY {X = }iscotiuous with probability desity fuctio(pdf) p(y ). The Y is cotiuous with margial pdf p(y) = p()p(y ) ad the coditioal distributio of X give Y (or the posterior) is p( y) = p()p(y ) p(y) = p()p(y ) p( )p(y ). The mutual iformatio epressio i(. ) ca be ow writte as I(X;Y) = H(X) H(X Y). Flippigtherolesof X ady,thismutual iformatiocabealsowritteas I(X;Y) = h(y) h(y X). 2.6 CAPACITY Sometimes we are iterested i the maimum mutual iformatio ma p() I(X;Y) of a coditioal pmf p(y ), which is referred to as the iformatio capacity (or the capacity i short). By the variatioal characterizatio i (. ), the iformatio capacity ca be epressed as mai(x;y) = ma mi p() p() q(y) p()d(p(y )q(y)), whichcabeviewedasagamebetweetwoplayers,oechoosig p()firstadtheother choosig q(y) et, with the payoff fuctio f(p(),q(y)) = p()d(p(y )q(y)). Usig the followig fudametal result i game theory, we show that the order of plays ca be echaged without affectig the outcome of the game. Miima theorem(sio ). Suppose that U ad V are compact cove subsets of the Euclidea space, ad that a real-valued cotiuous fuctio f(u,) o U V is cocavei uforeach adcovei foreachu. The ma mi f(u,) = mi ma f(u,). u U V V u U

11 2.7 Etropy Rate Sice f(p(),q(y)) is liear (thus cocave) i p() ad cove i q(y) (recall Property of relative etropy), we ca apply the miima theorem ad coclude that mai(x;y) = ma mi p() p() q(y) = mi q(y) ma p() = mi ma q(y) p()d(p(y )q(y)) p()d(p(y )q(y)) D(p(y )q(y)), where the last equality follows by otig that the maimum epectatio is attaied by puttig all the weights o the value that maimizes D(p(y )q(y)). Furthermore, if p ()attaisthemaimum, thebytheoptimalitycoditioof (. ) attais the miimum. 2.7 ENTROPY RATE q (y) p ()p(y ) Let X = (X ) = bearadomprocessoafiitealphabetx. Theamout ofucertaity persymbol ismeasuredbyitsetropyrate if the limit eists. H(X) = lim Ă H(X,...,X ), Eample 2.9. If X is a aperiodic irreducible Markov chai, the the limit eists ad H(X) = lim Ă H(X X ) = π( )H(p( 2 )), where π is the uique statioary distributio of the chai. Eample2.0. If X,X 2,...areidepedetadidetically distributed(i.i.d.), the H(X) = H(X ). Eample2.. LetY = (Y ) = beastatioarymarkovchaiadx = f(y ), =,2,... Theresultigradomprocess X = (X ) = ishiddemarkovaditsetropyratesatisfies H(X X,Y ) H(X) H(X X ) ad H(X) = lim Ă H(X X,Y ) = lim Ă H(X X ).

12 2 Iformatio Measures Eample 2.2. If X is statioary, amely, P{X,X 2 2,...,X } = P{X +m,x 2+m 2,...,X +m } foreverym,adevery, 2,..., X,thetheetropyrateis H(X) = lim Ă H(X X ). 2.8 ERGODICITY Let X = (X ) N be a radom process o X with idices i N ad (, F, P) be the associated probability space, where = X N is the sample space (the set of outcomes), F = σ((x ) N ) is the σ-algebra geerated by X (the set of evets), ad P is the probability measure. Note that X (ω) = ω, where the latter deotes the -th coordiate of ω. The ide set N is typically oe of Ă = {,2,...}, Ă + = {0,,...}, ad Ă = {...,,0,,...},adisomittedwheitisirrelevatorclearfromthecotet. LetT beatimeshiftoperatoro,thatis,if the ω = (...,ω, ω 0,ω,...), Tω = (...,ω 0, ω,ω 2,...). Here the boes deote the zeroth coordiate. Ithisotatio,theprocess X = (X )isstatioaryif P(T A) = P({ω: Tω A}) = P(A) forevery A F. Iftheprocessisdouble-sided (i.e., N = Ă), statioarityis equivaletly defiedby P(A) = P(TA). Foreample, ifa = {ω: ω 0 = },the T A = {ω: (Tω) 0 = } = { = }, TA = {Tω: ω 0 = } = { = }. Thus P(A) = P(X 0 = )while P(T A) = P(X = )ad P(TA) = P(X = ). AevetAissaidtobeshift-ivariatifA = T A. Theprocess X = (X )issaidto beergodicifeveryshift-ivariatevetaistrivial, amely, P(A) = 0or. Eample2.3. Let withprobability (w.p.) /2, X = w.p. /2. The P(T A) = P(A) for every A. Thus, X is statioary. Moreover, A = T A implies A = or. Thus, P(A) {0,}ad X isergodic.

13 2.8 Ergodicity 3 Eample2.4. Let X bedefiedasieample., Z beaidepedetcopyof X,ad Y = (X,Z). AsbeforeY isstatioary. Cosidertheevet A = {y = (,z): i = z i forall i}, whichis shift-ivariat. However, P(A) = P(X = Z) = /2adY isotergodic. Thus, a composite of two idepedet ergodic processes is ot ecessarily ergodic. Eample2.5. Let X,X 2,... be i.i.d. The X = (X ) = is ergodic. To prove this, first observethatayshift-ivariatamustbeithetailσ-algebrat = Ȃ = σ(x,x +,...); seeproblem.. Sice X,X 2,...areidepedet,bytheKolmogorov - law, P(A) = 0 orad X isergodic. Eample2.6. Let /3 w.p. /2, Θ = 2/3 w.p. /2, adgive {Θ = θ}, X,X 2,... be i.i.d. Ber(θ). Note that X,X 2,...are ucoditioally depedet. Cosider the shift-ivariat evet A = ω: limsup Ă = ω: limsup Ă Thebythestroglawoflarge umbers, i= + i=2 X i (ω) 2 X i (ω) 2 = T A. ifθ = /3, P(A Θ = θ) = 0 otherwise ad P(A) = 2 P(A Θ = /3)+ 2 P(A Θ = 2/3) = 2. Thus,Xisotergodic. Igeeral,amitureofergodicprocessesisotergodic. However, every statioary process ca be viewed as a miture of statioary ergodic processes. The strog law of large umbers states that the time average of a i.i.d. radom sequece coverges to the esemble average with probability oe. This determiistic behavior arises more geerally for ergodic processes. Theorem. (Poitwiseergodictheorem(Birkhoff )). Let X = (X ) = be statioaryadergodicwith E( X ) <. The lim Ă i= X = E(X ) a.s. (. )

14 4 Iformatio Measures If E(X 2 ) <,thethecovergecei(. )alsoholdsithel 2 sese,whichisofte referred to as vo Neuma s mea ergodic theorem. The followig result is a aalog of the ergodic theorem for iformatio theory. is sta- Theorem. (Shao,McMilla,Breima ). If X = (X ) = tioary ad ergodic, the lim Ă log p(x = H(X) a.s. ) Roughlyspeakig,thetheoremstatesthateveryrealizedsequece hasaprobability earlyequalto2 H(X) (asymptoticequipartitioproperty). Fortheproofofthetheorem, refer to Cover ad Thomas(, Sectio. ). 2.9 RELATIVE ENTROPY RATE Let Pad QbetwoprobabilitymeasuresoX with-thorderdesitiesp( )adq( ), respectively. The ormalized discrepacy betwee them is measured by their relative etropy rate D(PQ) = lim Ă D(p( )q( )) if the limit eists. Eample 2.7. If P is statioary, Q is statioary fiite-order Markov, ad P is absolutely cotiuous w.r.t. Q, the the limit eists ad D(PQ) = lim Ă p( )D(p( )q( ))d. (. ) See, for eample, Barro( ) ad Gray(, Lemma. ). Eample 2.8. Similarly, if P ad Q are statioary hidde Markov ad P is absolutely cotiuous w.r.t. Q, the the limit eists ad(. ) holds(juag ad Rabier, Lerou, Ephraim ad Merhav ). The followig geeralizes the Shao McMilla Breima theorem to the relative etropy rate betwee desities. Theorem. (Barro ). If P is a statioary ergodic probability measure, Q is a statioary ergodic Markov probability measure, ad P is absolutely cotiuous w.r.t. Q, the lim Ă log p(x ) q(x = D(PQ) P a.s. )

15 Problems 5 PROBLEMS.. Prove Property of etropy i Sectio. ad fid the equality coditio for Property... Prove Properties through of relative etropy i Sectio.... Etropy ad relative etropy. Let X be a fiite alphabet ad q() be the uiform pmfox. Showthatforaypmf p()ox, D(pq) = log X H(p())... The total variatio distace betwee two pmfs p() ad q() is defied as δ(p,q) = 2 p() q(). X (a) Showthatthisdistaceisa f-divergecebyfidigthecorrespodig f. (b) Show that δ(p,q) = ma A X P(A) Q(A), where Pad Qarecorrespodigprobabilitymeasures,e.g., P(A) = A p()... Pisker s iequality. Show that δ(p,q) 2loge D(pq), where the logarithm has the same base as the relative etropy.(hit: First cosider thecasethatx isbiary.).. Let p(, y) be a joit pmf o X Y. Show that p(, y) is absolutely cotiuous w.r.t. p()p(y)... Prove Properties through of mutual iformatio i Sectio.... Let X = (X ) = beastatioaryradomprocess. Showthat H(X) = lim Ă H(X X )... LetY = (Y ) = beastatioarymarkovchaiadx = {д(y )}beahiddemarkov process. Show that ad coclude that H(X X,Y ) H(X) H(X X ) H(X) = lim Ă H(X X,Y ) = lim Ă H(X X ).

16 6 Iformatio Measures.. Recurrecetime. LetX 0,X,X 2,...bei.i.d.copiesofX p(),adletn = mi{ : X = X 0 }bethewaitigtimetotheetoccurreceof X 0. (a) Showthat E(N) = X. (b)showthat E(logN) H(X)... Thepastadthefuture. Let X = (X ) = bestatioary. Showthat lim Ă I(X,...,X ;X +,...,X 2 ) = 0... Variable-duratio symbols. A discrete memoryless source has the alphabet {, 2}, where symbol has duratio ad symbol has duratio. Let X = (X ) = be the resultig radom process. (a) FiditsetropyrateH(X)itermsoftheprobability pofsymbol. (b) Fid the maimum etropy by optimizig over p... Shift-ivariat sets. Show that the collectio I of shift-ivariat sets is a σ-algebra... Tailσ-algebra. LetX = (X ) = bearadomprocessad T = Ȃ σ(x,x +,...). (a) Showthateveryshift-ivariatA F isi T,i.e., I T. (b)doesthecoversehold,thatis, T I?.. Multiple time shifts. Let X = (X ) = be ergodic ad Y = X 2, =,2,... Is Y = (Y ) = ergodic? Proveorprovideacoutereample... Miture of ergodic processes. Let U = (U ) ad V = (V ) be two statioary ergodic processes o the same alphabet X, but with differet etropy rates H(U) ad H(V), respectively. Let X = (X ) be a radom process that is eitheru orv uiformly at radom, i.e., (a) Is X statioary? U w.p. /2, X = V w.p. /2. (b)fiditsetropyrateh(x)itermsofh(u)adh(v). (c) Does the radom sequece log p(x ) coverge almost surely? If so, characterize the limitig radom variable... Cesàro summatio. Let {a } be a sequece of real umbers. We say that a coverges iflim Ă a = a < ;thata covergesithestrogcesàroseseif lim Ă i= a a = 0;

17 Problems 7 adthata covergesithecesàroseseif lim Ă i= a i = a. (a) Show that covergece implies strog Cesàro covergece, which, i tur, implies Cesàro covergece. (b)usigpart(a)toshowthatstrogmiigimpliesweakmiig,which,itur, implies ergodicity.

19 Bibliography Barro, A. R.( ). The strog ergodic theorem for desities: Geeralized Shao McMilla Breima theorem. A. Probab., ( ),. [ ] Birkhoff, G. D. ( ). Proof of the ergodic theorem. Proc. Natl. Acad. Sci. USA, ( ),. [ ] Breima, L. ( ). The idividual ergodic theorem of iformatio theory. A. Math. Statist., ( ),. Correctio ( ). ( ),. [ ] Cover,T.M.adThomas,J.A. ( ). ElemetsofIformatio Theory. ded.wiley,newyork. [ ] Ephraim, Y. ad Merhav, N. ( ). Hidde Markov processes. IEEE Tras. If. Theory, ( ),. [ ] Gray, R. M.( ). Etropy ad iformatio theory. Spriger, New York. [ ] Juag, B.-H. F. ad Rabier, L. R. ( ). A probabilistic distace measure for hidde Markov models. AT&T Tech. J., ( ),. [ ] Lerou, B. G.( ). Maimum-likelihood estimatio for hidde Markov models. Stoc. Proc. Appl., ( ),. [ ] Liese, F. ad Vajda, I.( ). O divergeces ad iformatios i statistics ad iformatio theory. IEEE Tras. If. Theory, ( ),. [ ] McMilla, B.( ). The basic theorems of iformatio theory. A. Math. Statist., ( ),. [ ] Shao, C. E.( ). A mathematical theory of commuicatio. Bell Syst. Tech. J., ( ),, ( ),. [ ] Sio, M.( ). O geeral miima theorems. Pacific J. Math.,,. [ ]

H(X) = plog 1 p +(1 p)log 1 1 p. With a slight abuse of notation, we denote this quantity by H(p) and refer to it as the binary entropy function.

H(X) = plog 1 p +(1 p)log 1 1 p. With a slight abuse of notation, we denote this quantity by H(p) and refer to it as the binary entropy function. LECTURE 2 Information Measures 2. ENTROPY LetXbeadiscreterandomvariableonanalphabetX drawnaccordingtotheprobability mass function (pmf) p() = P(X = ), X, denoted in short as X p(). The uncertainty about