5. Data Compression. Review of Last Lecture. Outline of the Lecture. Course Overview. Basics of Information Theory: Markku Juntti

Transcription

1 : Markku Jutt Overvew The deas of lossless data copresso ad source codg are troduced ad copresso lts are derved. Source The ateral s aly based o Sectos of the course book []. Teleco. Laboratory Course Overvew Basc cocepts ad tools Itroducto Etropy, relatve etropy ad utual forato Asyptotc equpartto property 4 Etropy rates of a stochastc process Source codg or data copresso 5 ata copresso Chael capacty 8 Chael capacty 9 fferetal etropy The Gaussa chael Other applcatos Mau etropy ad spectral estato Rate dstorto theory 4 Network forato theory Teleco. Laboratory Outle of the Lecture Revew of the last lecture Itroducto efto of a source code Kraft equalty Optal codes Bouds o optal codelegth Kraft equalty for uquely decodable codes Suary Revew of Last Lecture Two tools ad cocepts were troduced. The asyptotc equpartto property (AEP) s the aalog of the law of large ubers forato theory. If rado varables for a statoary stochastc process, ts etropy rate s the eteso of the AEP. Teleco. Laboratory Teleco. Laboratory 4

2 Asyptotc Equpartto operty Loosely Speakg AEP: Iplcatos to ata Copresso ( X, X, X ) H probablty. log p X Most of the probablty s cotaed the typcal set A () ε { } ( ) H ( X ) Aε =,, X : p,, so that ( ). (,, ) A ε logp (,, ) H. X. ( A ε ), for large eough.. ( ) H ( X ) Aε. Ay hgh probablty set B δ () wth (B δ () ) has sgfcat tersecto wth A ε (). Teleco. Laboratory 5 Coplete observato set X wth X eleets. Atypcal set. escrpto wth code words of log X + bts. Typcal set A ε () wth (H+ε) eleets. escrpto wth code words of (H+ε) + bts. Teleco. Laboratory 6 H Etropy Rate Etropy rate of a statoary stochastc process: ( X) = H ( X, X, X ) = H ' ( X) = l H ( X X, X, X ). l It ca be show by AEP for ay statoary ergodc stochastc process that ( X, X, X ) H probablty. logp X The AEP equvalet for stochastc processes. Etropy rate for statoary Markov processes: = µ P logp. H X, j j j Itroducto The fudaetal questos behd forato theory: What s the ultate data copresso rate? Aswer: etropy. What s the ultate data trassso rate? Aswer: chael capacty. ata copresso bulds flesh aroud the skeleto provded by etropy ad other tools. Etropy proves out to be the data copresso lt codes achevg etropy are optal several eags the uber of bts eeded rado uber geerato. Teleco. Laboratory 7 Teleco. Laboratory 8

3 ata source Source Codg oble screte valued rado varable X Realzato oble: esg a lossless source code to ze the average codeword legth. Lossless copresso: o forato lost. Itutve soluto proposal: Allocate the shortest code words to the ost probable outcoes of RV X, ad the evtably loger oes to less lkely outcoes of RV X. Well kow eaple: Morse code. efto of a Source Code Source code C: C : X : C ( ), where s the set of fte legth strgs of sybols fro a -ary alphabet. The legth of codeword C() s l(). Wthout loss of geeralty: = {,,,, -}. Eaple: X = {red, blue}, = {, }, C(red) =, C(blue) =. The epected legth of a source code: L = X Eaple above: L(C) =. ( C ) p ( ) l ( ). Teleco. Laboratory 9 Teleco. Laboratory Eaple #: Source Code wth H(X) = L(C) ( X = ) =, C = L( C ) = ( X = ) =, 4 C ( ) = ( X = ) =, 8 C ( ) = ( X = 4 ) =, C ( 4 ) = = = 8 H(X) =.75 H(X) = L(C). Code s uquely decodable. = 7 4 =.75 bts. Eaple..: bts. 8 Eaple #: Source Code wth H(X) L(C) ( X = ) =, C = 5 L C = + = =.67 bts. ( X = ) =, C ( ) = H ( X ) = log ( X = ) =, C ( ) = = log( ) =.58 bts. H(X) < L(C). Code s aga uquely decodable. Teleco. Laboratory Teleco. Laboratory

4 No-sgular code: eftos (). j C C j Uabguous for sgle sybol, for a strea eeds coas. A eaple of o-sgular codes: For bary-valued RV X, C( ) =, C( ) =. A eaple of sgular codes: For bary-valued RV X, C( ) =, C( ) =. Eteso of a code: X ( L ) = C ( ) C ( ) LC ( ). : C Eaple: C( ) =, C( ) =, C( ) =. eftos () A uquely decodable code: eteso s osgular oly oe possble source strg. A eaple of a uquely decodable code: C( ) =, C( ) =. A eaple of a ot uquely decodable code: C( ) =, C( ) =, C( ) =. C( ) = = C( ). A pref-free code = stataeous code: o codeword s a pref of ay other codeword ca be decoded wthout referece to the future desrable, desg goal. Teleco. Laboratory Teleco. Laboratory 4 Classes of Codes All codes No-sgular codes Uquely decodable codes Istataeous codes Eaple #: Classes of Codes X Sgular No-sgular Uq. dec. ef ot uquely dec. ot pref 4 The shortest codewords caot be assged for all sybols a pref code. Teleco. Laboratory 5 Teleco. Laboratory 6

5 Kraft Iequalty For ay stataeous code (pref code) the codeword legths l, l,, l satsfy the Kraft equalty: =. where s the uber of codewords. Coverse: for codeword legths satsfyg above, there ests a stataeous code. Eteded Kraft equalty for ay coutably fte set of codewords: Coverse also holds. = l l. oof of Kraft Iequalty Cosder a -ary tree, each ode wth chldre. The braches of the tree represet the sybols of the codeword. A codeword s represeted by a leaf of the tree. A path through a tree traces out the sybols of the codeword. ef codto o codeword s a acestor of ay other codeword o the tree. Let a l be the legth of the logest codeword. At level a l, soe of the odes are codewords descedats of codewords, or oe of the above. Teleco. Laboratory 7 Teleco. Laboratory 8 Root oof of Kraft Iequalty: A Eaple Code Tree wth =, l a = A codeword at level l has ^(l a l) descedats at level a l. The descedat sets ust be dsjot. = = a a a l l l l : l. Coverse by tree costructo. Optal Codes Kraft equalty code optzato: L = subject to,, pl l, l K l over tegers l, l,, l. Soluto over real l, l,, l : zg codelegths l u average p =, l = log ( p ) code legth L = pl = p log ( p ) = H ( X ). For tegers: L = H L H ( X ), l ( X ) = p. l -adc dstrbuto Teleco. Laboratory 9 Teleco. Laboratory

6 Bouds o Optal Codelegth Optzato l L = p l l subject to l, l, l real valued solutos: l = log p Straghtforward teger approato: Shao code. Sce. log ( ) log ( ) p l = = p =. Teleco. Laboratory log. p Kraft equalty satsfed a stataeous code ests. log ( ) log ( l < ) + p p p H X L < H X + p. Codewords over Multple Sybols Cocateate sybols to for a codeword wth legth l(,,, ) ad epected legth L [ l ( X, X, X )] = p (,, ) l (,, = E By prevous result: H X, X, X E l X, X, X [ ] < H ( X, X, X ) H X, X, X H X, X, X L < For a statoary process: L H ( X). For II RV s: H + ( X ) L < H ( X ) Teleco. Laboratory strbuto Msatch Mu codelegth wth erroeous PMF assupto: true PMF: p() PMF assued by source ecoder: q() Shao code: l ( ) = log. q( ) E H oof: p ( p ) + ( p q ) E [ l ( X )] < H ( p ) + ( p q ) +. [ l ( X )] = log p q ( ) = p = p < [ log( p ) + ] q p [ log( p ) + ] = log( ) + log( p p ) q p q p ( ) ( p q ) + H ( p ) +. Teleco. Laboratory + Kraft Iequalty for Uquely ecodable Codes Istataeous codes uquely decodable codes. Surprsgly: The Kraft equalty ad the coverse stll hold for all uquely decodable codes. Corollary for fte source alphabets holds as well. Bouds hold also for uquely decodable codes. Uquely decodable codes provde o advatage over stataeous codes codeword legth. Istataeous codes are spler to decode cosder those the sequel. Teleco. Laboratory 4

7 Suary Source codes C : X : C ( ), o-sgular uquely decodable stataeous. Kraft equalty for l. stataeous codes = uquely decodable codes. Cocetrate o stataeous codes. Etropy bouds the epected codelegth: H ( X ) L < H( X ) +. Teleco. Laboratory 5