ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE

Transcription

1 ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE School of Computer ad Commuicatio Scieces Hadout Iformatio Theory ad Sigal Processig Compressio ad Quatizatio November 0, 207 Data compressio Notatio Give a set A we deote by A the set of all fiite sequeces {(a,, a ) : 0, a i A} (icludig the ull sequece λ of legth 0) I particular {0, } = {λ, 0,, 00, 0, 0,, 000, } Cosider the problem of assigig biary sequeces (also called biary strigs) to elemets of a fiite set U Such a assigmet c : U {0, } is called a biary code for the set U The biary strig c(u) is called the codeword for u The collectio {c(u) : u U} is thus the set of codewords Defiitio A code c is called ijective if for all u v we have c(u) c(v) Defiitio 2 A code c is called prefix-free if c(u) is ot a prefix of c(v) for all u v I particular, if c is prefix-free the c is ijective (To be clear: a strig a a m is a prefix of a strig b b if m ad a i = b i for i =,, m Thus, the ull strig is a prefix of ay strig, ad each strig is a prefix of itself) Lemma Suppose c : U {0, } is ijective The, u 2 legth(c(u)) log 2 ( + U ) Proof Without loss of geerality, we ca assume that wheever k = legth(c(u)) for some u, the for every biary strig b of legth i < k there is a v with b = c(v) (Otherwise, there is a b with legth(b) < k which is ot a codeword, ad replacig c(u) with b will preserve the ijectiveess of c ad icrease the left had side of the iequality) For such a code c, with k deotig the legth of the logest codeword, the set of codewords is the uio of k i=0 {0, }i with a o-empty subset of {0, } k With r 2 k deotig the cardiality of this last subset, we have U = 2 k + r ad u 2 legth(c(u)) = k + r2 k As log 2 ( + U ) = k + log 2 ( + r2 k ) ad 0 < r2 k, all we eed to show is x log 2 ( + x) for 0 < x As equality obtais for x = 0 ad x =, the iequality follows from the cocavity of log Lemma 2 Suppose c : U {0, } is prefix-free The, u 2 legth(c(u)) Coversely, if l : U {0,, 2, } with u 2 l(u), the there exists a prefix-free code c : U {0, } with legth(c(u)) = l(u) Proof Give a biary sequece a = a a m, let p(a) = m a i2 i deote the ratioal umber whose biary expasio is 0a a m With this otatio, a biary sequece a = a a m is a prefix of a biary sequece b = b b if ad oly the p(b) lies i the iterval I(a) = [p(a), p(a) + 2 m ) For the first claim, observe that c beig prefix-free thus implies that the itervals I(c(u)) are disjoit As I(c(u)) is of size 2 legth(c(u)) ad all of the itervals are icluded i [0, ), the iequality follows For the secod claim, order the elemets of U as u,, u K such that l := l(u ) l K := l(u K ) Let p k = i<k 2 l i ad set I k = [p k, p k + 2 l k ) Observe that the itervals I,, I K are disjoit, ad I k [0, ) Furthermore, for each k, 2 l k pk is a iteger, thus p k ca be expressed i biary as 0b (k) with b (k) a biary strig of legth l k The code c(u k ) = b (k) ow has the required properties it beig prefix free a cosequece of the disjoitess of the collectio itervals I k

2 Lemma 3 Suppose P Π(U) is a probability distributio o U ad U is radom variable with distributio P The, with H(U) = u P (u) log 2 P (u) deotig the etropy of U, (i) for ay prefix-free c : U {0, }, E[legth(c(U))] H(U); (ii) there exists a prefix-free c : U {0, } with E[legth(c(U))] H(U) + ; (iii) for ay ijective c : U {0, }, E[legth(c(U))] H(U) log 2 log 2 ( + U ), (iv) there exists a ijective c : U {0, } with E[legth(c(U))] H(U) Proof For (i) ad (iii) let Q(u) = 2 legth(c(u)) ad observe that H(U) E[legth(c(U))] = u P (u) log 2 Q(u) P (u) log 2 Q(u), where the iequality is because log is cocave Whe c is prefix-free u Q(u) by Lemma 2, ad whe c is ijective u Q(u) log 2( + U ) by Lemma The iequalities (i) ad (iii) thus follow For (ii) set l(u) = log 2 P (u) As 2 l(u) P (u), we see that u 2 l(u) ad by Lemma 2 there exists a prefix-free code c with legth(c(u)) = l(u) As l(u) < log 2 P (u)+, (ii) follows For (iv) order the elemets of U as u,, u K with P (u ) P (u K ) Let c(u k ) = b k where b k is the kth elemet of the sequece λ, 0,, 00, 0, 0,, 000, 00,, (eg, b = λ, b 2 = 0, b 3 =, b 4 = 00,, b 9 = 00, ) Observe that legth(b k ) = log 2 k log 2 k Also ote that k P (u i) kp (u k ), ad thus log 2 k log 2 P (u k ) Cosequetly, for this c, E[legth(c(U))] k P (u k) log 2 P (u k ) = H(U) Corollary Suppose U, U 2, is a stochastic process The for ay sequece c : U {0, } of ijective codes lim if E[legth(c (U ))] lim if ad there exists a sequece c of prefix-free codes for which lim sup H(U ), E[legth(c (U ))] lim sup H(U ) I particular, if r = lim H(U ) exists, all faithful represetatios of the process U, U 2, with bits will asymptotically require at least r bits per letter, ad there is a represetatio that asymptotically requires exactly as much Proof The first iequality follows from otig that E[legth(c (U ))] H(U ) log 2 log 2 ( + U ) ad observig that lim log 2 log 2 (+ U ) = 0 The secod iequality follows from otig that there exist prefix-free c with ad that lim / = 0 E[legth(c (U ))] H(U ) + u 2

3 Remark Lemma 2 gives evidece of a strog coectio betwee prefix-free codes ad probability distributios O the oe had, give a prefix-free code c, oe ca costruct a probability distributio Q that assigs the letter u the probability Q(u) = 2 legth(c(u)) By the lemma, u Q(u) ; if equality holds Q is ideed a probability distributio, otherwise, we ca assig u Q(u) as the probability Q(u 0) of a fictitious symbol u 0 U If U is a radom radom variable with distributio P, we the have (by assigig P (u 0 ) = 0 if ecessary), E[legth(c(U))] H(U) = u P (u)[legth(c(u)) + log P (u)] = D(P Q) O the other had, give a distributio Q Π(U), by Lemma 2 we ca costruct a prefix-free code c : U {0, } with legth(c(u)) = log 2 Q(u) As log 2 Q(u) legth(c(u)) < log 2 Q(u) +, we see that E[legth(c(U))] H(U) = u P (u)[legth(c(u)) + log 2 P (u)] is bouded from below by D(P Q), ad from above D(P Q) + These observatios give the divergece D(P Q) a iterpretatio as the expected umber of excess bits (beyod the miimum possible H(U)) a code based o Q requires whe describig a radom variable with distributio P Cosequetly, if we are give S Π ad told that the distributio P of a radom variable U belogs to S, a reasoable strategy to desig a code c is to look for a distributio Q Π such that sup D(P Q) P S is small (eg, by fidig the Q that miimizes this quatity) ad costruct a code c based o Q as above Example To illustrate the remark above, suppose we are told that U, U 2, are biary ad iid radom variables The distributio of U ca be parametrized by θ = Pr(U = ), ad is give by Pr(U = u ) = P θ (u ) = ( θ) 0(u ) θ (u ) where 0 (u ) ad (u ) are the umber of zeros ad oes i the sequece u u With this otatio, S = {Pθ : 0 θ } is the class of distributios that we are told the distributio of U belogs to Cosider ow a sequece of coditioal distributios Q Uk+ U k(u uk ) = u(u k ) + k + 2 where u (u k ) is as above, deotig the umber of u s i u u k Note that Q U (0) = Q U () = /2 Defie Q (u ) = Q Ui U i (u i u i ) Oe ca prove by iductio o, that for ay ad ay u {0, }, Q (u ) ( 0 (u ) + 3 ) 0 (u )( (u ) ) (u )

4 If U,, U are iid with commo distributio P θ, [ D(Pθ Q ) = E log P θ (U ) ] Q (U ) [ log( + ) + E log [ = log( + ) + E 0 (U ) log log( + ) + ( θ) log Pθ (U ) ] ( 0 (U )/) 0(U ) ( (U )/) (U ) ( θ) 0 (U ) + (U ) log ( θ) θ + θ log ( θ) θ θ (U ) ] = log( + ), where the iequality i the last lie is because x x log[/x] is cocave ad E[ 0 (U )] = ( θ), ad E[ (U )] = θ Cosequetly, we see that sup P S D(P Q ) log( + ) If Q were used to costruct a prefix-free code c : {0, } {0, }, by the remark above, c will satisfy E[legth(c (U ))] H(P ) [log( + ) + ] wheever U is iid with distributio P As the right had side vaishes as gets large, we would be justified to call the sequece of codes c asymptotically uiversal for the class of biary iid data I the exercises we will see aother choice of Q which improves the upper boud o D(P Q ) to log 2 Note that, had we chose Q to be a member of S, say Q = Pθ 0 for some θ 0, the D(Pθ Q ) would have grow liearly i for ay θ θ 0 Thus, eve if we kow that the true distributio P is i S, choosig Q outside of S (as we have doe above) may lead to a better code costructio Remark 2 The example above also illustrates a coectio betwee compressio ad predictio (Oe ca also use the term learig istead of predictio) Suppose we have a family S of distributios o U, ad we are give a prefix-free code c : U {0, } performs well, i the sese that sup P S E P [legth(c (U ))] H(U ) is small Costruct the distributio Q associated with the code c, ie, Q(u ) = 2 legth(c(u )) ad factorize it as Q(u ) = Q(u i u i ) As the code c performs well, D(P Q) is small for all P S But D(P Q) = P (u ) log P (u ) Q(u ) u = P (u ) log P (ui ui ) Q(u u i u i ) = P (u i ) log P (ui ui ) Q(u i u i ) u i = P (u i ) P (u i u i ) log P (ui ui ) Q(u u i u i u i ) i = P (u i )D(P ( u i ) Q( u i )), u i 4

5 so we coclude that for a large fractio of i s i,,, ad for a set of u i s with large P probability, the quatity D(P ( u i ) Q( u i )) is small Which is to say, o matter what P from S is the true distributio of the data, if after observig u i we predicted the distributio of the ext symbol u i to be Q( u i ), our predictio will be close to the true distributio P ( u i ) for most i s ad for a high probability set of u i s 2 Uiversal data compressio with the Lempel-Ziv algorithm I the example i the previous sectio we saw a compressio method that was uiversal over the class of biary iid processes We will ow see a much more powerful method that is uiversal over all statioary processes The method was iveted by Ziv ad Lempel i 977, the versio we preset here is a variat due to Welch from 984 Give a alphabet U = {a,, a K }, the method ecodes a ifiite sequece u u 2 from this alphabet to biary as follows: Set a dictioary D = U Deote the dictioary etries as d(0) = a,, d(s ) = a K, with s = K beig the size of the dictioary Set i = 0 (the umber of iput letters read so far) 2 Fid the largest l such that w = u i+ u i+l is i D 3 With 0 j < s deotig the idex of w i D, output the log 2 s bit biary represetatio of j 4 Add the word wu i+l+ to D, ie, set d(s) = wu i+l+, ad icremet s by Icremet i by l Goto step 2 For example, with U = {a, b}, the iput strig abbbbaaab will lead to the executio steps D at 2 w output at 3 added-word at 4 a b a 0 ab a b ab b 0 bb a b ab bb bb bbb a b ab bb bbb b 00 ba a b ab bb bbb ba a 000 aa a b ab bb bbb ba aa aa 0 aab The first questio we eed to aswer is if we ca recover the iput sequece u u 2 from the output of the algorithm The questio is aswered i the affirmative i Lemma 4 below Note that the algorithm parses the sequece u u 2 ito a sequece of words w, w 2, foud at step 2 of the algorithm So, recovery of u u 2 is equivalet to the recovery of these words Let j, j 2, the dictioary idices that appear at step 3, ad d, d 2, the words added to the dictioary i step 4 As the dictioary size s icreases by each time a dictioary word is parsed, the bitstream that is output by the algorithm ca be parsed ito the idices j, j 2 Lemma 4 From j,, j i we ca determie w,, w i I other words, we ca recover the iput u u 2 from the output of the algorithm To be cocrete, if D(P Q) is less tha ɛ, the, except for a ɛ/3 fractio of the i s, we have u P i (ui )D(P ( u i ) Q( u i )) < ɛ 2/3, ad except for a set of P probability ɛ /3 of u i s, we have D(P ( u i ) Q( u i )) < ɛ /3 5

6 Proof From j, we ca determie w, ad so the claim is true for i = We proceed by iductio Suppose ow we observe j,, j i+ By the iductio hypothesis, j,, j i determies w,, w i Sice d k is the cocateatio of w k with the first letter of w k+, we kow d,, d i, ad, except for its last letter, d i We eed to show that we ca recostruct w i+ from the additioal iformatio obtaied by j i+ Note that j i+ refers to a word i the dictioary formed by augmetig U with the words d,, d i If j i+ refers to ay word other tha d i, we already determie w i+ Otherwise j i+ refers to d i But i this case the last letter of d i equals the first letter of d i which is already kow, ad we ca agai determie w i+ Next, we will obtai a upper boud o the umber of bits per letter the algorithm uses to describe a sequece u u 2 The aswer will be give i Theorem below Lemma 5 A word w ca appear i the sequece w, w 2, at most U times Proof As the algorithm always looks for the logest word w i the dictioary that matches the start of the as-yet-uprocessed segmet of the iput, wu i+l+ is ot i the dictioary before its additio to the dictioary i step 4 Thus the words added to the dictioary are distict For each occurrece of a word w i the parsig a word of the form wu with u U is added to the dictioary Sice these are distict, w caot appear more tha U times i the parsig Lemma 6 Suppose u = u u is parsed ito m(u ) words w w m by the algorithm The lim m(u )/ = 0 Proof There are U i words of legth i, ad by the previous lemma, each ca appear at most U times i the list w,, w m As the algorithm does ot parse the ull strig, at most F (k) = U k U i words i the list are of legth k or less, ad each of the remaiig words i the list has legth k or more Thus k[m F (k)] Cosequetly, lim sup m(u ) As k is arbitrary, the lemma follows lim sup /k + F (k) = /k Theorem Let l(u ) deote the umber of bits produced by the algorithm after readig u The, lim sup l(u )/ lim sup m(u ) log m(u ) Proof As the dictioary size icreases by at each iteratio of the algorithm, with m = m(u ), l(u ) = m i=0 log 2( U + i) Thus l(u ) m log 2 ( U + m ) + m = m log 2 m + m log 2 ( + ( U )/m) + m, ad the lemma follows from lim m(u ) = 0 We ow have a upper boud to the umber of bits per letter the LZW algorithm requires to describe a sequece u u 2 Next, we will derive a lower boud to the umber of bits per letter produced by ay iformatio loss fiite state machie that maps u u 2 to a sequece of bits (the terms fiite state machie ad iformatio lossless will be defied formally i the paragraphs that follow) The lower boud will eve apply to machies that may have bee desiged with prior kowledge of the sequece u u 2, ad most remarkably, will match the upper boud just derived above That is to say, for ay u u 2, LZW competes well agaist ay iformatio lossless fiite state machie I particular, each prefix-free ecoder c that appears i Corollary ca be implemeted by a fiite state iformatio lossless machie Cosequetly, oce we obtai the lower boud, we will have proved: 6

7 Theorem 2 If U, U 2 is a statioary ad ergodic stochastic process, the the umber of bits per letter emitted by LZW whe its iput is U U 2 approaches lim H(U )/ with probability oe 2 Fiite state iformatio lossless ecoders For our purposes, a fiite state machie is a device that reads the iput sequece oe symbol at a time Each symbol of the iput sequece belogs to a fiite alphabet U with U symbols The machie is i oe of a fiite umber s of states before it reads a symbol, ad goes to a ew state determied by the old state ad the symbol read We will assume that the machie is i a fixed, kow state z before it reads the first iput symbol The machie also produces a fiite strig of biary digits (possibly the ull strig) after each iput This output strig is agai a fuctio of the old state ad the iput symbol That is, whe the ifiite sequece u = u u 2 is give as the iput, the ecoder produces y = y y 2, while visitig a ifiite sequece of states z = z z 2, give by y k = f(z k, u k ), k z k+ = g(z k, u k ), k where the fuctio f takes values o the set {0, } of fiite biary strigs, so that each y k is a (perhaps ull) biary strig A fiite segmet x k x k+ x j of a sequece x = x x 2 will be deoted by x j k, ad by a abuse of the otatio, the fuctios f ad g will be exteded to idicate the output sequece ad the fial state Thus, f(z k, u j k ) will deote yj k ad g(z k, u j k ) will deote z j+ Without loss of geerality we will assume that ay state z is reachable from the iitial state z ie, that some iput sequece will take the machie from state z to z To make the questio of compressibility meaigful oe has to require some sort of a ivertibility coditio o the fiite state ecoders Give the descriptio of the fiite state machie that ecoded the strig, ad the startig state z, but (of course) without the kowledge of the iput strig, it should be possible to recostruct the iput strig u from the output of the ecoder y A weaker requiremet tha this is the followig: for ay state z ad two distict iput sequeces v m ad ṽ, either f(z, v m ) f(z, ṽ ) or g(z, v m ) g(z, ṽ ) A ecoder satisfyig this coditio will be called iformatio lossless (IL) It is clear that if a ecoder is ot IL, the there is o hope to recover the iput from the output, ad thus every ivertible ecoder is IL 2 We will eed to the followig fact: Lemma 7 Suppose v,, v m are biary strigs, with o strig occurrig more tha k times The, writig m = j i=0 k2i + r with 0 r < k2 j, we have m legth(v i) k j i=0 i2i + rj Proof The set of biary strigs ordered i icreasig legth cosists of: strig of legth 0, 2 strigs of legth,, 2 i strigs of legth i, The shortest total legth for the v i s will be attaied if the v i s are chose by traversig the set of all biary strigs i icreasig legth, each strig repeated k times, util all strigs of legth j or less are repeated k times, ad we are left to fid 0 r < k2 j strigs, which are chose from the set of strigs 2 However, as illustrated i Figure, a IL ecoder is ot ecessarily uiquely decodable Startig from state S, two distict iput sequeces will leave the ecoder i distict states if they have differet first symbols, otherwise they will lead to differet output sequeces Thus, the above ecoder is IL Nevertheless, o decoder ca distiguish betwee the iput sequeces aaaa ad bbbb by observig the output 000 7

8 a / 0 b / a / λ A S b / λ B b / 0 a / A fiite state machie with three states S, A ad B The otatio i /output meas that the machie produces output i respose to the iput i λ deotes the ull output Figure : A IL ecoder which is ot uiquely decodable of legth j The lower boud i the lemma is precisely the total legth of this optimal collectio Lemma 8 Suppose v,, v m are biary strigs, with o strig occurrig more tha k times The, m m legth(v i ) m log 2 8k () Proof Notig that j i=0 2i = 2 j ad j i=0 i2i = (j 2)2 j + 2, the previous lemma states: writig m = k2 j k + r with 0 r < k2 j, the total legth of the v i s is lower bouded by k((j 2)2 j + 2) + rj = (j 2)m + kj + 2r (j 2)m As r < k2 j, we have m < k(2 j+ ) Rearragig, we get 2 j+ > + m/k > m/k, ad thus j 2 > log m 8k Now we ca state the followig Lemma 9 For ay IL-ecoder with s states, legth(y ) m(u ) log 2 m(u ) 8 U s 2 (2) where, as before, m(u ) is the umber of words i the parsig of u by LZW Proof Let u = w w m be the parsig of the iput by LZW Let z i be the state the IL machie is i at just before it reads w i, ad z i+ be the state just after it has read w i Let t i be the biary strig output by the IL machie while it reads w i, so that y = t m No biary strig t ca occur i t,, t m more tha k = s 2 U times If it did, there will be a state-pair (z, z ) that occurs amog (z i, z i+ ) more tha U times with t i = t By Lemma 5 the there will be w i w j with t i = t j = t ad (z i, z i+ ) = (z j, z j+ ) = (z, z ) But this 8

9 cotradicts the IL property of the machie Thus the output of the IL machie t t m is a cocateatio of m biary strigs with o strig occurrig more tha k = s 2 U times, so their total legth is at least m log m 8k Usig the lemma just proved, ad by Lemma 6 we have Theorem 3 For ay fiite state iformatio lossless ecoder, the umber of output bits Let l(u ) produced by the ecoder after readig u satisfies 3 Quatizatio lim sup l(u m(u ) )/ lim sup log m(u ) Ofte it is ot ecessary to represet data exactly; a approximate represetatio suffices, eg, we may be cotet if a sequece u of bits areare reproduced as a sequece v of bits as log as v ad u differ oly at a few idices We formulate this state of affairs as follows: our data is a sequece of radom variables U, each U i takig values i a alphabet U We are give a represetatio alphabet V ad also a distortio measure d : U V R that gives the distortio caused by represetig a data letter u by v For defie d (u, v ) = d(u i, v i ) A quatizer cosists of two maps: f : U {,, M} ad g : {,, M} V The data sequece u is mapped by the quatizer to f (u ), which is a log M bit represetatio of the data, ad recostructed from this represetatio as v = g (f (u )) We measure the quality of the the quatizer by two criteria: its rate R = (log M)/ (the umber of bits per data letter), ad its expected distortio = E[d (U, V )] where V = g (f (U )) The followig lemma says that oe caot have too small rate for a give level of distortio Lemma 0 Suppose d is a distortio measure, U, U 2, are iid with distributio P U, ad we have a quatizer with rate R ad expected distortio The, there exists a distributio P UV such that R I(U; V ) ad = E[d(U, V )] Proof Let P Ui V i deote the joit distributio of (U i, V i ) ad set P UV = P U i V i We will show that P UV satisfies the coditios claimed i the lemma First ote that the U- margial of each P Ui V i equals P U, so P UV ideed has U margial P U Also, P V = i P V i Now, by the data processig theorem for mutual iformatio, R I(U ; V ) As the U i s are idepedet, H(U ) = i H(U i) Also by the chai rule ad removig terms from the coditioig H(U V ) i H(U i V i ) Thus I(U ; V ) = H(U ) H(U V ) ad cosequetly R I(U i ; V i ) = H(U i ) H(U i V i ) = I(U i ; V i ), D(P Ui V i P U P Vi ) D(P UV P U P V ) = I(U; V ) where the last iequality is due to the covexity of D( ) Furthermore, E[d(U, V )] = u,v P UV (u, v)d(u, v) = P Ui V i (u, v)d(u, v) = E[d (U, V )] = 9

10 The lemma says that the possible rate distortio pairs whe quatizig a iid source with distributio P lie i the followig regio of R 2 : { } (R, ) : R I(U; V ), = E[d(U, V )] P UV :P U =P We will ow show that there are quatizers whose performace closely approximates the boud give i the lemma Theorem 4 Give P U, a distortio measure d ad a joit distributio P UV, for ay ɛ > 0, there is a quatizer (f, g ) with rate R < I(U; V ) + ɛ ad expected distortio satisfyig E[d(U, V )] < ɛ whe it is used to quatize iid data U with distributio P U To prove the theorem we will make use of the followig lemma Lemma For all δ > 0, for all large eough, for every joit type P UV Π (U V), there is a subset S := S(P UV ) T (P V ) with S 2 (I(U;V )+δ) so that for every u T (P U ) there is a v S for which (u, v ) T (P UV ) Proof Let M = 2 (I(U;V )+δ), ad choose V (), V (M) idepedetly ad uiformly from T (P V ) Let S be the (radom) set {V (),, V (M)} For each u T (P U ), let Z(u ) = M {(u, V (m)) T (P UV )} m= Note that the S has the claimed property if ad oly if Z = u T (P U ) Z(u ) = 0 We will prove the existece of the set S with the property, by showig that E[Z] < To that ed, ote that for a give u T (P U ), the M radom variables {(u, V (m)) T (P UV )} are idepedet, with expectatio T (P UV ) T (P U ) T (P V ) Cosequetly, ( ) + U V 2 [H(UV ) H(U) H(V )] 2 [I(U;V )+δ/2] U V E[Z(u )] ( 2 (I(U;V )+δ/2) ) M exp( M2 (I(U;V )+δ/2) ) exp( 2 δ/2 ) As this expectatio is doubly expoetially small i, ad sice T (P U ) is oly expoetially large, we coclude that for large eough we will have E[Z] < Proof of the theorem Give P UV, set R 0 = I(U; V ) ad 0 = E[d(U, V )] Fix δ > 0, ad let Ω the set of all joit distributios Q o U V such that I(U; V ) < R 0 + δ ad E[d(U, V )] 0 < δ whe (U, V ) has joit distributio Q The distributio PUV belogs to Ω, thus Ω is a o-empty ope set of joit distributios Let Ω 0 be the set of U margials of the distributios i Ω Note that Ω 0 is also ope ad o-empty as it cotais P U As a cosequece D := if P Ω0 D(P P U ) = D(P P U ) for some P Ω 0, ad thus D > 0 sice P ca t equal P U Ω 0 Let S(Q) be as i the lemma above ad set S = S(Q) where the uio is over all Q Ω Π (U V) For large eough M = S 2 (R0+3δ), sice each S(Q) has S(Q) 2 (R0+2δ) ad the uio cotais oly Π (which is a polyomial i ) such sets Assig to each v i S a uique idex i {,, M}, ad for i =,, M, let g (i) be the elemet of S with idex i 0

11 By the costructio of the set S, for every P Ω 0 Π (U) ad every u T (P ) we ca fid a v i S such that the type of (u, v ) belogs to Ω, ad thus d (u, v ) 0 < δ For u P Ω 0 T (P ) defie f (u ) to be the idex of such a v i S For u ot i this uio defie f (u ) i ay arbitrary fashio, eg, by settig f (u ) = For those u, with v = g (f (u )), we do ot ecessarily have a small value for d (u, v ) 0, but evertheless we ca say d (u, v ) 0 < 2 max u,v d(u, v) Whe U,, U are iid with distributio P U, the probability that the type of U does ot belog to Ω 0 decays expoetially to zero as 2 D Cosequetly, with V = g (f (U )) η = Pr ( d (U, V ) 0 + δ ) decays to zero like 2 D Thus the quatizer (f, g ) has rate log M R 0 + 3δ, ad distortio = E[d (U, V )] satisfyig 0 E [ d(u, V ) 0 ] < δ + η 2 max d(u, v) < 2δ u,v for large eough As δ > 0 is arbitrary, the theorem follows I the light of the Theorem above ad Lemma 0 that precedes it, it makes sese to defie Defiitio 3 Give source ad reproductio alphabets U ad V, a distortio fuctio d : U V R, ad a distributio P U o U, defie the rate distortio fuctio R( ) ad distortio rate fuctio (R) as R( ) = if{i(u; V ) : E[d(U, V )] }, (R) = if{e[d(u, V ) : I(U; V ) R} where the ifima are over all joit distributios P UV with U margial P U Lemma 0 says that ay quatizer with rate at most R must have expected distortio at least (R), ad ay quatizer with expected distortio at most must have rate at least R( ) The theorem says that these bouds are achievable arbitrarily closely As a example cosider U = V = {0, }, d(u, v) = {u v}, ad P U () = p /2 It is easy to show that { 0 p R( ) = h 2 (p) h 2 ( ) 0 < p} where h 2 (q) = q log 2 q ( q) log 2 ( q) is the biary etropy fuctio I particular, for p = /2 ad = 0, R( ) 053, so it is possible to compress biary data by almost a factor half ad still be 90 percet accurate The aive approach for achievig the same accuracy would have retaied 80 percet of the data bits (the remaig 20 percet ca the be guessed i ay fashio; the guesses would be right with probability /2, resultig i 90 percet overall accuracy)