Differential Entropy

Transcription

1 School o Iormatio Sciece Dieretial Etropy Course - Iormatio Theory - Tetsuo Asao ad Tad matsumoto {t-asao matumoto}@jaist.ac.jp Japa Advaced Istitute o Sciece ad Techology Asahidai - Nomi Ishikawa 93-9 Japa I the last Chapters we leared.. We skip the trasmitter ad receiver!!!! School o Iormatio Sciece We assumed that the chael iput ad output both take discrete values: Chael Ecoder Chael Decoder Biary/No-biary iite alphabet Noise Error source We derived Chael Codig Theorem: There eists a R rate R code such that the maimum error probability λ ca be made arbitrarily small i the code rate is lower tha the capacity R<C. Coversely ay R rate R code that ca achieve arbitrarily small λ must satisy R<C.

2 School o Iormatio Sciece I this ad the et Chapters we derive.. Chael Codig Theorem or the chael where iput ad output o the chael both take aalog values. For this we elimiate the iite alphabet assumptio. Chael Ecoder Chael Decoder Aalog values Noise However still we igore the trasmitter ad receiver. The impact o havig to use physical trasmitter ad receiver is ivestigated i aother course Chael Codig. Outlie School o Iormatio Sciece. Dieretial Etropy - Deiitio - Some Eamples - Asymptotic Property. Relatio o Dieretial ad Discrete Etropies 3. Joit ad Coditioal Dieretial Etropy 4. Mutual Iormatio - Some Importat Properties

3 Dieretial Etropy Deiitio 9..: Dieretial Etropy h log d S School o Iormatio Sciece For cotiuous radom variable with a probability desity uctio the dieretial etropy is deied as: where S is the set where is deied. Eample: Uiorm Distributio I a radom variable is distributed over a uiorm distributio / a 0 a the dieretial etropy o is: a h log d log a 0 a a Property: Value Rage The dieretial etropy ca take egative values as see i the above eample. Dieretial Etropy School o Iormatio Sciece Eample: Normal Distributio I a radom variable is distributed over a Normal distributio N0σ the dieretial etropy o is: h log d with ep σ σ Ater mior mathematical maipulatios we have: E h l σ l σ l e l σ σ l eσ [ ats] log eσ [ bits] Theorem 9..: Asymptotic Property Let. be a sequece o i.i.d. radom variable ollowig a desity uctio. The the ollowig holds: log L E log h

4 School o Iormatio Sciece Relatio o Dieretial ad Discrete Etropies Property: There eists a value i such that the probability that a radom variable takes a value betwee i ad i is give by: i Pr i i i d i Where is the desity uctio o. Area i i i School o Iormatio Sciece Relatio o Dieretial ad Discrete Etropies Deiitio 9..: Deie a discrete radom variable i i i i Etropy o this radom variable is give by: H which leads to: H pi log pi with pi log i i log i log i i d i i log i i log i 443

5 School o Iormatio Sciece Relatio o Dieretial ad Discrete Etropies 3 Theorem 9..: Discrete Cotiuous log H i log i h h as 0 Eample: As we saw previously i a eample dieretial etropy h o a radom variable uiormly distributed over [0 a] is log a ad with a h0. I we quatize this radom variable ollowig distributio [0 ] with - bit A/D coverter Thereore log H log H 0 h H H which meas that bits are eough to epress the cotiguous radom variable while keepig -bit accuracy. School o Iormatio Sciece Joit ad Coditioal Dieretial Etropy Deiitio 9.3.: Joit Dieretial Etropy The dieretial etropy o a set o radom variables ollowig the desity uctio is deied as: h L L log L dd Ld Deiitio 9.3.: Coditioal Dieretial Etropy The coditioal dieretial etropy o radom variables ad Y is deied as: y h Y ylog y ddy ylog ddy h Y h Y y where Y is the joit desity o the radom variable ad Y.

6 School o Iormatio Sciece Joit ad Coditioal Dieretial Etropy Theorem 9.3.: Normal Distributio The dieretial etropy o radom variables ollowig the multivariate ormal desity uctio / ep T L bits e N h h log L where ad E is give by [ ] T E E L M Proo: d d h T / l log School o Iormatio Sciece Joit ad Coditioal Dieretial Etropy 3 Proo Cotiued: / l T E / l bits e ats e e log l l l / Eercise: Provide more detailed ad more cocrete proo or this mathematical maipulatio.

7 Mutual Iormatio Deiitio 9.4.: ullback Leibler Distace School o Iormatio Sciece ullback Leibler distace D g betwee two desity uctios is deied as: D g log S g with 0log 0/00 ad S beig the regio where the radom variables are deied. Deiitio 9.4.: Mutual Iormatio The mutual iormatio IY betwee the two cotiuous radom variables ad Y with the joit desity uctio y is deied as y I ; Y ylog ddy y Property 9.4.:. y I ; Y h h Y h Y h Y D y Mutual Iormatio School o Iormatio Sciece Theorem 9.4.: No Negativity o ullback Leibler Distace D g 0 ad equality holds i ad oly i g. g g Proo: D g log log log g log 0 S 443 due to Jese' s uequality Jese s iequality or cocave uctios states that the equality holds i ad oly o g. Theorem 9.4.: No Negativity o Mutual Iormatio I ; Y 0 ad equality holds i ad oly ad Y are idepedet. Theorem 9.4.3: owledge Decreases Ucertaity h Y h ad equality holds i ad oly ad Y are idepedet. S S

8 Theorem 9.4.4: Chai Rule Theorem 9.4.5: Mutual Iormatio 3 L h i i i h L h i i h L ad equality holds i ad oly ad Y are idepedet. School o Iormatio Sciece Theorem 9.4.6: h c h i.e. traslatio does ot chage the dieretial etropy. Mutual Iormatio 4 School o Iormatio Sciece Theorem 9.4.7: Normal Distributio Maimize Etropy The Normal distributio maimizes the etropy amog those distributios havig the same variace. Proo: Let be a zero mea radom variable with variace σ ollowig the desity uctio g. Also let φ be a zero mea Normal radom variable with variace σ. The g 0 D g φ g l d h g g lφd h g l σ gd φ σ h g l σ gd σ However sice g ad have the same variace gd φd Thereore 0 D g φ h g l σ gd h g σ φd σ l σ h g φ lφd h g h φ The equality holds i g φ

9 Mutual Iormatio 5 School o Iormatio Sciece Theorem 9.4.8: Normal Distributio Maimize Etropy - Multi-variable s Case - The -dimesioal zero-mea Normal distributio maimizes the etropy amog other -dimesioal zero-mea distributios havig the same covariace matri { ij }. Proo: Eercise Summary School o Iormatio Sciece We have visited... Dieretial Etropy - Deiitio - Some Eamples - Asymptotic Property. Relatio o Dieretial ad Discrete Etropies 3. Joit ad Coditioal Dieretial Etropy 4. Mutual Iormatio - Some Importat Properties