Chain Rules for Entropy

Transcription

1 Cha Rules for Etroy The etroy of a collecto of radom varables s the sum of codtoal etroes. Theorem: Let be radom varables havg the mass robablty x x.x. The The roof s obtaed by reeatg the alcato of the two-varable exaso rule for etroes. Codtoal Mutual Iformato We defe the codtoal mutual formato of radom varable ad gve as: log ; E I z y x Mutual formato also satsfy a cha rule: I I... ; ;...

2 Covex Fucto We recall the defto of covex fucto. A fucto s sad to be covex over a terval ab f for every x x a.b ad 0 λ f λx + λ x λf x + λ f x A fucto f s sad to be strctly covex f equalty holds oly f λ0 or λ. Theorem: If the fucto f has a secod dervatve whch s o-egatve ostve everywhere the the fucto s covex strctly covex. Jese s Iequalty If f s a covex fucto ad s a radom varable the Ef f E Moreover f f s strctly covex the equalty mles that E wth robablty.e. s a costat.

3 Iformato Iequalty Theorem: Let x qx x χ be two robablty mass fucto. The Wth equalty f ad oly f D q 0 x q x for all x. Corollary: No egatvty of mutual formato: For ay two radom varables I ; 0 Wth equalty f ad oly f ad are deedet Bouded Etroy We show that the uform dstrbuto over the rage χ s the maxmum etroy dstrbuto over ths rage. It follows that ay radom varable wth ths rage has a etroy o greater tha log χ. Theorem: log χ where χ deotes the umber of elemets the rage of wth equalty f ad oly f has a uform dstrbuto over χ. Proof: Let ux / χ be the uform robablty mass fucto over χ ad let x be the robablty mass fucto for. The x D q xlog log χ u x ece by the o-egatvty of the relatve etroy 0 D u log χ 3

4 Codtog Reduces Etroy Theorem: wth equalty f ad oly f ad are deedet. Proof: 0 I ; Itutvely the theorem says that kowg aother radom varable ca oly reduce the ucertaty. Note that ths s true oly o the average. Secfcally y may be greater tha or less tha or equal to but o the average y y y Examle Let have the followg jot dstrbuto 0 3/4 /8 /8 The /8 7/80544 bts 0 bts ad bt. We calculate 3/4 +/4 0.5 bts. Thus the ucertaty s creased f s observed ad decreased f s observed but ucertaty decreases o the average. 4

5 Ideedece Boud o Etroy Let are radom varables wth mass robablty x x x. The:... Wth equalty f ad oly f the are deedet. Proof: By the cha rule of etroes: Where the equalty follows drectly from the revous theorem. We have equalty f ad oly f s deedet of for all.e. f ad oly f the s are deedet. Fao s Iequalty Suose that we kow a radom varable ad we wsh to guess the value of a correlated radom varable. Fao s equalty relates the robablty of error guessg the radom varable to ts codtoal etroy. It wll be crucal rovg the coverse to Shao s chael caacty theorem. We kow that the codtoal etroy of a radom varable gve aother radom varable s zero f ad oly f s a fucto of. eceweca estmate from wth zero robablty of error f ad oly f 0. Extedg ths argumet we exect to be able to estmate wth a low robablty of error oly f the codtoal etroy s small. Fao s equalty quatfes ths dea. Suose that we wsh to estmate a radom varable wth a dstrbuto x. We observe a radom varable that s related to by the codtoal dstrbuto y x. 5

6 Fao s Iequalty From we calculate a fucto g ^ where ^ s a estmate of ad takes o values ^. We wll ot restrct the alhabet ^ to be equal to ad we wll also allow the fucto g to be radom. We wsh to boud the robablty that ^. We observe that ^ forms a Markov cha. Defe the robablty of error: Pe Pr{^ }. Theorem: P + P log χ e e + P e log χ The equalty ca be weakeed to: P e log χ Remark: Note that P e 0 mles that 0 as tuto suggests. 6