: Markku Jutt Overvew Te basc cocepts o ormato teory lke etropy mutual ormato ad EP are eeralzed or cotuous-valued radom varables by troduc deretal etropy ource Te materal s maly based o Capter 9 o te course book [] Telecomm Laboratory Course Overvew Basc cocepts ad tools Itroducto Etropy relatve etropy ad mutual ormato 3 symptotc equpartto t property 4 Etropy rates o a stocastc process ource cod or data compresso 5 Data compresso Cael capacty 8 Cael capacty 9 Deretal etropy 0 Te Gaussa cael Oter applcatos Mamum etropy ad spectral estmato 3 Rate dstorto teory 4 Network ormato teory Telecomm Laboratory Outle o te Lecture Revew o te last lecture Itroducto Detos Te EP or cotuous RV s Relato o deretal etropy ad dscrete etropy Jot ad codtoal deretal etropy Relatve etropy ad mutual ormato Propertes ummary Revew o Last Lecture Te udametal questos bed ormato teory: Wat s te ultmate data compresso rate? swer: etropy Wat s te ultmate data trasmsso rate? swer: cael capacty commucates wt B: pyscal acts o duce desred pyscal state B ad B sould aree o wat was set Te mamum umber o dstusable sals cael capacty = data rate lmt or relable commucato B Telecomm Laboratory 3 Telecomm Laboratory 4
Te Cael Cod Teorem ll rates below cael capacty C are acevable For all rate R < C: a sequece o ( R ) codes so tat 0 as (Weak) coverse: ay sequece o acevable ( R ) codes: R C tro coverse: or R < C P () e 0 ad R > C P () e epoetally Capacty wt eedback: supremum o acevable rates wt eedback codes C FB : C FB C ma I ; Y p Feedback does ot crease te capacty ource-cael Cod Teorem a source-cael code wt P () e 0 H(V)<C were te probablty o error s ˆ Pe PrV V pv p y v y v y v were Vˆ s te estmate o V s true (o - zero) () s a dcator ucto: 0 s alse (zero) Coverse: I H(V)>C probablty o error s bouded away rom zero ad relable trasmsso s ot possble Telecomm Laboratory 5 Telecomm Laboratory 6 ummary Itroducto Iormato cael capacty = operatoal cael capacty: C ma I; Y p ll rates below te capacty (R<C) are acevable: 0 as No rate above te capacty (R>C) s acevable Feedback does ot crease te capacty Feedback ca elp acev te capacty source wt etropy rate below te capacty ca be trasmtted relably source wt etropy rate above te capacty caot be trasmtted relably Dscrete-valued radom varables ave bee cosdered so ar Eteso or cotuous RV s eeded deretal etropy tratorward eteso o most oter cocepts partcular te asymptotc equpartto property ome mportat dereces tou Deretal etropy ca be eatve Dscrete etropy s always postve Mamum deretal etropy uder a covarace costrat by ormal (Gaussa) dstrbuto Dscrete etropy wt o costrat mamzed by uorm dstrbuto Telecomm Laboratory 7 Telecomm Laboratory 8
Detos Let be a RV wt cumulatve dstrbuto ucto (CDF) F() = Pr( ) I F() s cotuous te radom varable s sad to be cotuous Let df ' F I s ormalzed so tat d d () s called probablty desty ucto (PDF) or Te set o suc values o were () > 0 s called te support set (or smply support) o (or ucto ()) Deretal Etropy Deretal etropy ( te teral ests): d were s te support set o te radom varable () = E[()] Depeds o te PDF oly as does te etropy te dscrete case eeralzato o te etropy H p p marked also as () = () Telecomm Laboratory 9 Telecomm Laboratory 0 Eample #: Uorm Dstrbuto 0 a a /a 0 oterwse a a d a a a 0 d Note #: I a = (a) = 0 () = 0 Note #: I a < (a) < 0 Deretal etropy ca be eatve derece compared to te etropy However () = a = a s te volume o te support set wc always o-eatve Eample #: Normal Dstrbuto () Let te RV ollow ormal (e Gaussa) dstrbuto: ~ () () were deotes te PDF o real Gaussa dstrbuto wt mea zero ad varace : e Due to te epoet ucto t s smplest to calculate te deretal etropy ats Te deretal etropy bts ca be oud va cae o bases Telecomm Laboratory Telecomm Laboratory
Eample #: Normal Dstrbuto () l d l d E E l l l e l e ats e e l e bts e bts e e bts Te EP or Cotuous RV s Remd te asymptotc equpartto property or dscrete RV s: p H probablty Most o te probablty s cotaed te typcal set H : p : so tat p H p Pr or lare eou 3 H 3 Te same ca be sow or cotuous RV s Telecomm Laboratory 3 Telecomm Laboratory 4 EP ad Typcal et For a sequece o IID RV s: E probablty Proo: Follows drectly rom te weak law o lare umbers Typcal set: were Volume o set: Vol d d d R : Telecomm Laboratory 5 Propertes o te Typcal et Pr( () ) > or sucetly lare Vol( () ) (()+) or all 3 Vol( () ( ) (() ) ) or sucetly lare Proo: ee te tetbook [ ect 9] Te typcal set s te set wt te smallest volume satsy probablty bl ( ) to rst order o epoet Proo: mlar to te dscrete case Te smallest set tat cotas most o te probablty as appromately volume o Te correspod sde let s () = (sde let o m set wt most prob) Telecomm Laboratory 6
Relato o Deretal Etropy ad Dscrete Etropy Cosder a RV wt PDF () Let us dvde te rae o to bs o wdt ssume () s cotuous wt te bs By te mea value teorem: () : d Quatze: = <(+) Pr p d Telecomm Laboratory 7 Dervato Te etropy o te quatzed RV : H p p d d d I ()() ) s Rema terable d 0 Telecomm Laboratory 8 Teorem: Te Relato betwee te Deretal ad Dscrete Etropes I te PDF () s Rema terable te H 0 Te etropy o a -bt quatzato o a cotuous RV s appromately () + Eamples: I ~ U[0] ad = te =0adH( )= bts suce to descrbe to bt accuracy I ~ U[0/8] ad = te = 3 ad H( ) = Te rst 3 bts to te rt o decmal pot must be 0 3 bts suce to descrbe to bt accuracy I eeral () + s te averae umber o bts Te deretal etropy o a dscrete RV ca be cosdered to be result te volume o support set be zero Telecomm Laboratory 9 Jot ad Codtoal Deretal Etropy Jot deretal etropy: d d d Codtoal deretal etropy: Y y y d dy ( Y ) ( ) ( Y ) Telecomm Laboratory 0
Jot Deretal Etropy o a Multvarate Gaussa Radom Vector Te Gaussa PDF: ep T μ μ were = det() deotes te determat o te covarace matr Te deretal etropy o real-valued Gaussa RV s wt mea vector ad covarace matr : e Proo: ee te tetbook [ ect 94] Telecomm Laboratory Relatve Etropy ad Mutual Iormato Relatve etropy or ullback-lebler dstace: D Mutual ormato: y I ; Y y d dy y D y y Y Y Y Relato to dscrete mutual ormato: I ; Y H H Y Y I ;Y Telecomm Laboratory Propertes No-eatvty: D 0 D 0 almost everywere Proo by Jese s equalty: D 0 Furter propertes: I ; Y 0 I ; Y 0 y y Y Y y y Telecomm Laboratory 3 Propertes () Ca rule or deretal etropy: Idepedece boud: Hadamard equalty: For ~N(0): Traslatos ad rotatos: c a a Telecomm Laboratory 4
Te Gaussa PDF Mamzes Deretal Etropy Multvarate ormal (Gaussa) dstrbuto mamzes te deretal etropy over all dstrbutos wt te same covarace Let te real -dmesoal radom vector ave zero mea ad covarace = E(( T ) or j = E( ) j j Te e wt equalty ad oly ~N(0) Proo Let () be ay PDF satsy te covarace costrat jd j j Let be te PDF o N(0) or Now T ep ep 0 D * Telecomm Laboratory 5 Telecomm Laboratory 6 tep * te Proo T d e d T e T e d e e e d d ummary Eteso o ormato teoretc cocepts to cotuous RV s Deretal etropy: d symptotc equpartto property ad te typcal set: E probablty : Telecomm Laboratory 7 Telecomm Laboratory 8
ummary () Relatve etropy: D Mutual ormato: y I ; Y y d dy y D y y Y Y Y Multvarate ormal (Gaussa) dstrbuto mamzes te deretal etropy over all dstrbutos wt te same covarace: e Telecomm Laboratory 9