3. Basic Concepts: Consequences and Properties

Size: px

Start display at page:

Download "3. Basic Concepts: Consequences and Properties"

Alfred Reeves
6 years ago
Views:

1 : 3. Basc Cocepts: Cosequeces ad Propertes Markku Jutt Overvew More advaced cosequeces ad propertes of the basc cocepts troduced the prevous lecture are derved. Source The materal s maly based o Sectos.6.8 ad.0. of the course book []. Telecomm. Laboratory Course Overvew Basc cocepts ad tools troducto Etropy relatve etropy ad mutual formato 3 Asymptotc equpartto property 4 Etropy rates of a stochastc process Source codg or data compresso 5 Data compresso Chael capacty 8 Chael capacty 9 Dfferetal etropy 0 The Gaussa chael Other applcatos Mamum etropy ad spectral estmato 3 Rate dstorto theory 4 Network formato theory Telecomm. Laboratory Outle of the Lecture Revew of the last lecture troducto Jese s equalty The log-sum equalty Data processg equalty Suffcet statstcs Fao s equalty Summary Data source Revew of Last Lecture Dscrete valued radom varable X Realzato Probablty mass fuctos (PMF s): ( ) = Pr[ X ] p = ( y ) = Pr[ X = Y y ] p = X X y Y X deotes the cardalty (the umber of elemets) of set X. Telecomm. Laboratory 3 Telecomm. Laboratory 4

2 Summary of Basc Cocepts () Summary of Basc Cocepts () ( ) ( ) [ ( )]. Etropy X = p log p measure of ucertaty X o-egatve upper-bouded by uform dstrbuto. Base ca be ay the default assumpto s. Relatve etropy p ( ) ( ) ( ) p X D p q = p log E log ( ) = p q q X measure of smlarty of dstrbutos o-egatve dstace-lke measure ( ) ( ). Mutual formato ( X Y ) = p ( y ) specal case of relatve etropy o-egatve smlarty measure of jot ad product PMF s Cha rules etropy X X K X = X X mutual formato = ( y ) ( ) p ( ) p ; log y p y ( X ; Y ) = ( X ) ( X Y ) = ( Y ) ( Y X ) ( Y ; X ). = relatve etropy ( ) ( X K X ). ( X X K X ; Y ) ( X ; Y X X K X ). = = ( p ( y ) q ( y )) = D ( p ( ) q ( )) D ( p ( y ) q ( y )). D + Telecomm. Laboratory 5 Telecomm. Laboratory 6 (X) ucertaty of X llustratos (XY) jot ucertaty of X ad Y (X Y) ucertaty of X whe Y kow (X;Y) (Y X) ucertaty of Y whe X kow formato X carres o Y ad vce versa (Y) ucertaty of X Telecomm. Laboratory 7 troducto Basc deftos: etropy relatve etropy mutual formato Basc tools: relatoshps cha rules Jese s equalty log-sum equalty data processg equalty Fao s equalty Basc deftos ad tools are gve to be used later chapters. Telecomm. Laboratory 8

3 Jese s equalty Jese s equalty s used very wdely formato theory. The very basc theorems are based o Jese s equalty. Used also wdely optmzato ad other felds. Covety Fucto f() s cove over (ab) f ( a b ) 0 λ : f ( λ + ( λ) ) λ f ( ) + ( λ) f ( ). Fucto f() s strctly cove over (ab) f t s cove ad ( a b ) 0 λ : f ( λ + ( λ) ) = λf ( ) + ( λ) f ( ) λ = 0 λ =. Cove fucto f() les below ay chord. f() Telecomm. Laboratory 9 Telecomm. Laboratory 0 Eamples of Cove Fuctos Cocavty Fucto f() s cocave over (ab) f -f() s cove or a b 0 λ : ( ) ( λ + ( λ) ) λ ( ) + ( λ) ( ) f f f. Fucto f() s strctly cocave over (ab) f -f() s strctly cove. Cocave fucto f() les above ay chord. f() Telecomm. Laboratory Telecomm. Laboratory

4 Eamples of Cocave Fuctos Test for Covety ad Cocavty f fucto f() has a secod dervatve f () whch s o-egatve (postve) everywhere the f() s cove (strctly cove). Proof: See the tetbook. Telecomm. Laboratory 3 Telecomm. Laboratory 4 Jese s equalty For a cove fucto f() ad radom varable X: E [ f ( X )] f ( E( X )). For a strctly cove fucto f() ad radom varable X: [ f ( X )] = f ( E( X )) Pr[ X = E( X )]. E = Proof: See the tetbook. Cosequeces: formato equalty Jese s equalty formato equalty: D D ( p q ) 0 ( p q ) = 0 p ( ) = q ( ). No-egatvty of mutual formato: ( X ; Y ) 0 ( X ; Y Z ) 0 wth equalty f ad oly f X ad Y are (codtoally) depedet. Proofs: See the tetbook. Telecomm. Laboratory 5 Telecomm. Laboratory 6

5 Cosequeces: Mamum Etropy Dstrbuto Uform PMF mamzes the etropy: ( X ) log( X ) ( X ) = log( X ) p ( ) = u ( ) =. X Proof: p ( ) ( ) ( ) D p u = p log = p ( ) log p ( ) p ( ) logu ( ) u ( ) = log( X ) ( X ). By o-egatvty of relatve etropy: 0 D ( p u ) = log( X ) ( X ) ( X ) log( X ). Note the coecto to the secod law of thermodyamcs. Cosequeces: Codtog Reduces Etropy Codtog reduces etropy: because depedece boud: ( X Y ) ( X ) ( X ) ( X ). 0 ( X; Y ) = Y ( X X K X ) ( X ). = wth equalty ff X are depedet. Telecomm. Laboratory 7 Telecomm. Laboratory 8 Jot PMF: Eample #: Codtog Reduces Etropy Y\X 0 3/4 /8 /8 ( X ) = ( 7) 8 8 ( X Y = ) = 0 log( ) ( X Y = ) = log( ) ( ) 3 X Y = ( X Y = ) + ( X Y = ) 4 4 Log-Sum equalty For o-egatve umbers a a a ad b b b : a a = a log a log = b = b = ( X ) = bts ( X Y = ) = 0 bts ( X Y = ) = bt ( X Y ) = 0.5 bts Etropy of X decreases gve Y=. Etropy of X creases gve Y=. Etropy of X decreases o average. wth equalty ff Proof: See the tetbook. a = costat. b Telecomm. Laboratory 9 Telecomm. Laboratory 0

6 D Cosequeces Covety of relatve etropy: D(p q) s cove the par (pq): ( λp + ( λ) p λq + ( λ) q ) D ( p q ) + ( λ) D ( p ). q Proof: See the tetbook. Cocavty of etropy: (p) s a cocave fucto of p. Proof: D ( p u ) = log( X ) ( p ) ( p ) = log( X ) D ( p u ). Cocavty follow ow from covety of D. Mutual formato s a cocave fucto of p() for fed p(y ) cove fucto of p(y ) for fed p(). Data Processg equalty Radom varables X Y Z form a Markov cha (X Y Z ) f p y z = p p y p z y Cosequeces: ( z y ) ( ) ( ) ( ) ( ). ( y z ) p ( y ) p ( y ) p ( y ) ( y ) p ( z y ) p ( y ) p ( z ) p ( y ) p p p = = = y X Y Z Z Y X Z = f ( Y ) X Y Z. Telecomm. Laboratory Telecomm. Laboratory Data Processg equalty Data processg equalty: X Y Z ( X ; Y ) ( X ; Z ). (A clever) mapulato of data caot crease ts formato. Corollares: Z = g ( Y ) ( X ; Y ) ( X ; g ( Y )) ( X ; Y Z ) ( X ; Y ). Data mapulato caot crease the depedece of the orgal RV s. Note that ths apples to Markov chas oly. New observatos ca crease formato. Suffcet Statstcs Cosder a famly of PMF s {p θ ()} parameterzed by θ. Parameter θ s ofte a ukow parameter to be estmated. Form statstcs T(X) based o the observed RV X. θ X T(X) ( θ; X ) ( θ; T ( X )). Suffcet statstcs f ( θ ; X ) = θ; T ( X ) ( ). Mapulato of data does ot decrease ts formato may eable data compresso. Mmal suffcet statstcs provde the best compresso. Telecomm. Laboratory 3 Telecomm. Laboratory 4

7 Data source Fao s equalty Y Observe dscrete valued radom varable Y. Estmate a correlated RV X based o Y. ( ). Estmate: ˆ X = g Y Probablty of error: P Pr ˆ e = ( X X ). Fao s equalty: Pe + Pe log X X Y A weaker form: ( ) ( ) ( X Y ) + Pe log X X Y Pe. log( X ) Note: P e = 0 ( X Y ) = 0 as s tutvely pleasg. Proof: See the tetbook. ( ) ( ) ( ). Summary Jese s equalty. For a cove fucto: [ ( )] ( ( )). E f X f E X Log-sum equalty: a log log a = a a b = = b = Data processg equalty: ( ). ( ) ( ). X Y Z X ; Y X ; Z Suffcet statstcs f ( θ ; X ) = ( θ; T ( X )). Fao s equalty: ( P ) + P log( X ) ( X Y ). e e Telecomm. Laboratory 5 Telecomm. Laboratory 6

Chain Rules for Entropy

Chain Rules for Entropy Cha Rules for Etroy The etroy of a collecto of radom varables s the sum of codtoal etroes. Theorem: Let be radom varables havg the mass robablty x x.x. The...... The roof s obtaed by reeatg the alcato