Channel Capacity Course - Information Theory - Tetsuo Asano and Tad matsumoto {t-asano,

Transcription

1 School of Iformato Scc Chal Capacty Cours - Iformato Thory - Ttsuo Asao ad Tad matsumoto Emal: {t-asao matumoto}@jast.ac.jp Japa Advacd Isttut of Scc ad Tchology Asahda - Nom Ishkawa 93-9 Japa I th last Chaptrs w lard.. School of Iformato Scc Iformato Sourc + Ecodr Tx Rx Dcodr Dstato Nos ow to ffctvly cod th sourc wth th kowldg of sourc apparac probablty. hat th thortcal asymptotc proprts of th sourc codg tchqus ar wth th kowldg of th probablty.

2 School of Iformato Scc I ths Chaptrs w lar.. Commucato systms us chal as mdum for formato trasfr. - Th qusto arss that what s th maxmum capablty of th chal. - ow ca th commucato systms us th chal s capablty. - hat ar thortcal asymptotc proprts of th chal codg tchqus for rror protcto gv th kowldg of th chal charactrstcs. skp th trasmttr ad rcvr!!!! Assumptos: Chal Ecodr + Chal Dcodr Bary/No-bary ft alphabt Nos Error sourc Th trasmttr ad rcvrs ar gord Chal output s drctly coctd to th chal dcodr. Th both chal codr output ad os tak form of bary or o-bary ft alphabt. Not: Thos assumptos ar to b lmatd th xt Chaptr. Outl School of Iformato Scc. Chal Capacty - Dfto - Som Exampls - Som Proprts. Radom Codg 3. Chal Codg Thorm - Proof for Suffccy 4. Fao s Iqualty for Extso - Proof for Ncssty

3 Chal Modl: Rvw School of Iformato Scc Chal Ecodr + Chal Dcodr Bary/No-bary ft alphabt Nos Error sourc Chal put squc: 0. Ft Alphabt: {x x x 3. x q }whr q s th alphabt sz. Chal output squc: 0. Ft Alphabt: {y y y 3. y r }whr r s th alphabt sz. Lt th codtoal jot probablty of 0 y 0 y - y - codtod upo 0 x 0 x - x - b dotd as P y y... y x x x Chal Capacty Dfto 8..0: Chal Matrx School of Iformato Scc Th chal matrx T{t j } s gv as a matrx of whch try s dfd as th probablty that th trasmttd symbol x s rcvd as y j as: y t y t M M yr tr y Tx t t t r L O L t q x t q x M M t r q xq x x M x q... T { t j }... y y M y r Dfto 8..: Chal Capacty Th formato chal capacty C of mmory-lss chal s dfd as: C max { I ; p x whr th maxmzato s tak ovr all possbl put dstrbutos px.

4 School of Iformato Scc Chal Capacty Exampl: Bary Symmtrc Chal -q 0 0 q q -q I ; x 0 p x x p x q q q x 0 whr th qualty holds f th put dstrbuto s uform. Thrfor w ca coclud that C q bts. Exampl: Bary Erasur Chal -α 0 0 α α Erasur Rx ca ot dcd - α Chal Capacty 3 School of Iformato Scc Df th rasur vt by E. Sc th probablts of o-rror ad rasur vts ar dstctv Dot that Prπ E E + E π α π α + πα π α π α α π α α + α π c C max { α max { α π + α α α wth π /. p x π

5 School of Iformato Scc Chal Capacty 4 Exampl: Nosy Chal wth o Ovrlappg Outputs 0 -α - β 3 β 4 I ; p y y y α Ths s bcaus th rcvr ca dtfy whch of 0 or was trasmttd oly by lookg at th rcvd symbol. Sc th trasmttd symbols ar bary. wth PrPr0/. C max I ; max bt 0 Exrcs: Nosy Typ rtr Chal Capacty 5 School of Iformato Scc A B C D / / / / / / A B C D Calculat th capacty of ths osy typ wrtr assumg that all charactrs A-Z appars wth qual probablty. / Z / / / Z

6 School of Iformato Scc Chal Capacty 6 Dfto 8..: Symmtrc Chal Th chal s calld symmtrc f th rows of th chal trasto matrx ar prmutato of ach othr ad so ar th colums. Exampl: A chal havg th followg trasto matrx s symmtrc p y x Ths chal has th capacty: I ; r log r whr r s a row of th chal trasto matrx. Th qualty holds f th output dstrbuto s uform. owvr f px s uform p y p y x p x p y x ad hc py s also uform. x x Chal Capacty 7 School of Iformato Scc Proprty 8..: Th chal capacty s o gatv.. C 0 bcaus C max I ; I ; 0 C log bcaus C max I ; max log 3 C log bcaus C max I ; max log 4 I ; s a cotuous ad cocav fucto of px. S Thorm 4.3.4

7 School of Iformato Scc Chal Capacty 8 Burst Error Chal: Sc thr s o chac that w chag Error th apparac probablty of th alphabt Sourc from thsourc capacty s: C-E E Errors whr E E{p:ProbEp} + Iput wth x-xlog Output x--xlog -x. Mmory Chal Statoary Stat Probablts: p 0 ad p 0/0.9 /0. /0.8 S 0 S 0/0. Etropy of th Error Sourc: p0 0.9 p p 0.8 p 0 wth p 0 + p p 0 /3 ad p /3 E/3 0. +/3 0.8 / / Capacty: C-E bts/symbol Chal Capacty 9 School of Iformato Scc Bt Error Rat: P b 0. p p /3 Mmory-lss Chal P b /3 Capacty: C-E-/ bts/symbol Capacty wth Mmory Chal >>> Capacty wth Mmory-lss Chal Fadg chal wthout Itrlavg: Mmory Chal Itrlavd chals: Mmory-lss Chal

8 School of Iformato Scc Chal Codg Thorm: Prparato Assum that a mssag to b trasmttd ovr th chal s draw from th dx st { M}.. ach mssag s dxd by a umbr from ths st. Th followg summarzs how th commucato systm w aalyz works: Th mssag s th codd to a lgth block of symbols x x x. Th codd mssag s dotd as. Th trasmttd mssag x x x suffrs from os th chal ad rcvd as a radom squc y y y by th rcvr. 3 Th chal has ts trasto matrx as dscrbd by a codtoal probablty py x py y y x x x. 4 Th rcvr ams to rtrv th trasmttd mssag by a stmato rul g. ˆ whr g. 5 Th rcvr maks a rror f ˆ. Dfto 8..: Commucato Systm Th commucato systm that follows th rul dscrbd abov s dotd as: p y x Obvously accordg to ths dfto th chal s th -th xtso of th mmory-lss chal p y x. School of Iformato Scc Chal Codg Thorm: Prparato Dfto 8..: Mmory-lss Chal Th chal accordg to Dfto 8.. s mmory-lss f k k p yk x y p yk xk k L Th trasto fucto for th mmory-lss chal rducs to: p y x p y x Dfto 8..3: Chal Cod A M cod s dfd for th chal p y x whr M s th umbr of th mssags to b trasmttd ad s th lgth of th codd squc. Th Rols of th codr ad dcodr ar dfd as follows: Ecodr maps th M mssags to thr corrspodg squcs. M. Th st of th cod words s calld cod book. Th dcodr fucto g. maps th rcvd squc of radom varabl o to th most lkly mssag stmat g L M. { }

9 School of Iformato Scc Chal Codg Thorm: Prparato 3 Dfto 8..3: Probablty of Error Th codtoal probablty λ gv th mssag dxd by has b trasmttd s gv by: λ Pr{ g } p y x I g y y whr I. s th dcator fucto whch taks valu f th argumt s satsfd ad othrws t taks valu 0. Dfto 8..4: Maxmal Probablty of Error Th maxmal rror probablty λ for a M cod s dfd as: λ max λ { { L M } Dfto 8..5: Avrag Probablty of Error Th avrag rror probablty P for a M cod s dfd as: P M M λ School of Iformato Scc Chal Codg Thorm: Prparato 4 Proprty 8..: Avrag Probablty of Error Th followg proprts hold: P P Pr g λ f th mssag trasmttd s chos uformly from th st. Dfto 8..6: Cod Rat Th rat R of a M cod s dfd as: R log M

10 Radom Codg School of Iformato Scc Cosdr R cod. Grat a cod at radom accordg to th probablty dstrbuto px of th symbols x. Sc thr ar R cod words radomly slctd from th codbook C whch s dotd as: x C M R x x x M R L O L x M R x whr th argumt of. dcats th cod word dx ad th subscrpt dcats th symbol dx. Sc th apparac of th symbols s dpdt p x p R { x w } for w... Sc ach cod word s gratd radomly ths codbook C tslf s radom varabl. Th probablty of gratg a partcular cod word s gv by: PrC R w p x w Radom Codg School of Iformato Scc Th cod C s kow to th both trasmttr ad rcvr. Thy ar assumd to also kow th chal trasto matrx py x. A mssag s chos qu-probably. Thrfor R R Pr w w... Assum that th w-th cod word w corrspodg to th w-th row of C s trasmttd. A squc rcvd by th rcvr dotd as follows th dstrbuto: P y x w p ˆ arg max P y w { y x w } Th rcvr gusss whch of th R squcs s most lkly to hav b st by comparg th codtoal probablts of th possbl cod words ad slcts th o satsfyg: If ˆ thr s a dcodg rror. x w Th codbook C s updatd at vry trasmsso tmg. Thrfor ths tchqu s calld radom codg.

11 Chal Codg Thorm Thorm 8.3.: Chal Codg Thorm School of Iformato Scc Thr xsts a R rat R cod such that th maxmum rror probablty λ ca b mad arbtrarly small f th cod rat s lowr tha th capacty R<C. Covrsly ay R rat R cod that ca achv arbtrarly small λ must satsfy R<C. Proof of : Th proof uss th proprts of radom codg. Th dtald proof s far byod th xpctd lvl of ths cours ad thrfor a proof outl s dscrbd stad blow: Bfor rcvr rcvs y t has th oly kowldg that thr ar radomly slctd squcs to b st from th trasmttr. Th rcvr rcvs y. Rcvr crass kowldg about th cod word x by rcvg th squc y. owvr stll thr rmas ucrtaty whch s avragd ovr all possbl cod words. Th avragd ucrtaty s xprssd by th codtoal tropy. Chal Codg Thorm School of Iformato Scc Proof of Cotud: R Cod ords Possbl Squcs w Iput to th Chal Possbl squcs Output from th Chal Possbl Squcs y Ths mas that thr ar caddat squcs wth whch rcvd squc s most probably y ad th probablty that th othr cod words ar rcvd th form of th squc y ca b mad arbtrarly small f R<C as show blow: Th probablty that s slctd squc slctd from amog th all possbl lgth- Squcs s a cod word of th rat R cod s R / th avrag probablty that ay cod word othr tha w s NOT slctd s gv by: P g y w x w P g y w x w R R

12 Proof of Cotud: Chal Codg Thorm R School of Iformato Scc whr C- bcaus of th cod radomss maxmzato wth rspct to px dos t hav to b tak. Sc th probablty dscrbd abov dcats that th trasmttd cod word s dcodd corrctly th rror probablty s gv by { CR } P g y w x w c th avrag rror probablty ca b mad arbtrarly small by makg th cod lgth larg ough f R<C. Th probablty that ca b mad arbtrarly small s avragd rror probablty ad thrfor roughly spakg th bst half of th cod words hav a maxmal rror probablty lss tha -C-R ad aothr half s hghr. Throwg away th cod words havg rror probablty hghr tha -C-R ad R- cod words rmas. Ths mas that th cod rat s chagd to: R-/ bcaus R- R-/. Ths rat loss s glgbl f s larg ough. Th th maxmal rror probablty satsfs: { CR } λ { R } { CR } Chal Codg Thorm 3 Proof of : Th proof s comprsd of th followg stps: A P 0 mpls R C B P 0 mpls R C School of Iformato Scc Proof of A: Assum that w us R rat R cod. Thr ar cod words that ar slctd qu-probably ad st to th rcvr; Assum that th mssag s to b st. Th th tropy of s: R + I ; owvr by th assumpto that g 0. c R I ; I ; I ; C a b a: From th data procssg qualty whr Markov cha holds. b: To b prov th xt slds Thorm 8.4. c: By dfto of Capacty c for ay cods achvg P 0 R C. c

13 School of Iformato Scc Fao s Iqualty for Extso Proof of B rqurs Fao s Iqualty for Extso. Lt s df th vt: ˆ ˆ 0 ˆ g wth f f E Th by usg th cha rul a E E + b E E E a: Bcaus E s a fucto of ad. b: Bcaus E s bary-valud. Furthrmor E E P E E P E log ˆ Pr 0 ˆ Pr + R P whr ˆ Pr P Combg all w hav R P + Ths s Fao s qualty. owvr bcaus s a fucto of Th w hav Fao s qualty for xtso: R P + School of Iformato Scc Fao s Iqualty for Extso Thorm 8.4.: Proof: ; x p ay for C I ; I L bcaus th chal s mmory-lss. owvr Thrfor C I I ; ; th ths rsult th proof of A-of- of th chal codg thorm s compltd.

14 School of Iformato Scc Fao s Iqualty for Extso 3 Proof of B ow kow that R + I ; But w do t assum th fact g s kow ths cas. c >0. R + I ; + I ; + P R + I ; + P R + C whr w hav usd Fao s qualty for xtso. Dvdg th both sds by R P R by assumpto 0 { + C Fally w kow that R C has to b satsfd. C also kow that P R R Summary School of Iformato Scc hav vstd... Chal Capacty - Dfto - Som Exampls - Som Proprts. Radom Codg 3. Chal Codg Thorm - Proof for Suffccy 4. Fao s Iqualty for Extso - Proof for Ncssty