VARIABLE-RATE VQ (AKA VQ WITH ENTROPY CODING) Varable-Rate VQ = Quatzato + Lossless Varable-Legth Bary Codg A rage of optos -- from smple to complex A. Uform scalar quatzato wth varable-legth codg, oe dex at a tme. 1100 1101 100 00 01 101 1110 1111 1 w 2 N w w B. Nouform scalar quatzato wth varable-legth codg -- oe dex at a tme. C. Scalar quatzato wth hgher-order varable-legth codg -- ether block codg of dces at a tme or th-order codtoal codg of the dces. D. k-dme'l VQ wth varable-legth codg, oe dex at a tme. E. k-dme'l VQ wth hgher-order varable-legth codg -- ether block codg of dces at a tme or th-order codtoal codg of the dces. G. k-dme'l VQ wth other types of lossless codg We focus maly o E. wth block codg. A.-D. are specal cases of E. Codtoal codg s just a slght varato of E.. VQ-EC-1 D. VQ WITH 1ST-ORDER LOSSLESS CODING samples dex bts dex reproductos X 1...X k I 1 k partto bary bary I Y...Y codebook ecoder decoder ecoder decoder Key characterstcs: k-dmesoal VQ wth partto S = {S 1,..., S M }, codebook C = {w 1,..., w M }, quatzato rule Q, ad bary prefx codebook B = C b = {c 1,..., c M } wth legths {L 1,...,L M } Decompose ecoder to "partto" ad "bary ecoder" Gve x, the partto produces dex whe x S The bary ecoder outputs bary codeword c wth legth l Decompose decoder to "bary decoder" ad "codebook" The bary decoder decodes the bts to the dex. The codebook outputs w. Rate = R = 1 k x L(x) p(x) = 1 k rate of bary ecoder we ofte assume R = 1 k H(I) = 1 K H(Q(X)) ("VQ wth etropy codg") Dstorto = D = E 1 k X-Q(X) 2 (ot affected by choce of lossless coder) VQ-EC-2
E. VQ WITH BLOCK LOSSLESS CODING source vectors dces X... X I 1... I partto bary ecoder bary codeword 1 c Y 1... Y codebook I 1... I bary decoder reproducto vectors k-dme'l VQ wth varable-legth codg of dces blocks of. Decompose ecoder to "partto" ad "bary ecoder": "Partto" successve k-dm'l source vectors X 1,,X to dces I 1,,I, where X j = (X j,1,,x j,k ). Losslessly ecode dces at oce, (I 1,,I ), usg FVL block lossless code wth prefx codebook C b cotag M bary codewords. Decompose decoder to "bary decoder" ad "codebook": Decode bary codeword to dces I 1,,I. Output correspodg quatzato vectors w I1,,w I as reproductos of X 1,,X, respectvely. "Block lossless bary codg" s a easy to aalyze paradgm for studyg the beefts of varable-legth codg (a.k.a. etropy codg). We usually assume source s statoary, so that X j has the same pdf for all j, whch wll be deoted f X (x), f(x) or f k (x). VQ-EC-3 SUMMARY OF CHARACTERISTICS Quatzer = k-dmesoal VQ k = dmeso M = sze (ot eces'ly power of 2, ot so mportat, may be fte) S = {S 1,,S M } = k-dmesoal partto C = {w 1,,w M } = k-dmesoal codebook Q(x) = quatzato rule Bary Ecoder: = order of bary ecoder (.e. put blocklegth) source vectors dces bary codeword X 1... X I 1... I partto bary c ecoder Y 1... Y codebook I 1... I bary decoder reproducto vectors C b = {v : I } = bary prefx codebook -- oe cdwrd for each seq. of dces where I = set of cell dex -tuples = { = ( 1,, ) : 1 1 M,, 1 M } v = (v,1,,v,l ) = bary codeword of legth L for Dervatve Characterstcs: quatzato rule: Q(x j ) = w whe x j S ecodg rule: α(x 1,,x ) = v, whe = ( 1,, ) ad x j S j, j = 1,..., decodg rule: β(c ) = (w 1,w 1,,w 1 ) VQ-EC-4
Dstorto (same as usual) PERFORMANCE M D = 1 k E X - Q(X) 2 = 1 k x-y 2 f k (x) dx =1 S where X = (X 1,,X k ) ad f k (x) s ts desty. Dst' depeds o S ad C but ot C b. Rate R = 1 k L = 1 k P L bts/sample where L = average legth of bary codewords P = probablty of bary codeword v = Pr(X 1 S 1,,X S ), I From lossless codg theorem H(I) L * H(I) + 1 where L * = least avg. legth of prefx code for gve VQ & H(I) = - P log 2 P = etropy of I (or of (Y 1,,Y )) From ow o we assume (uless otherwse stated) that R = 1 k H(I) = 1 k H (I) = 1 k H(Y 1,,Y ) = H k (Y) We call ths "VQ wth th-order etropy codg (EC)". Note: The sze of the quatzer has o drect relato to ts rate. VQ-EC-5 IMPLEMENTATION AND COMPLEXITY Quatzer -- same ssues as wth fxed-rate codg Lossless Coder -- table lookup s the brute force method + Table stores M bary codewords of varous legths + M = 2 kr f where R f = 1 k log 2 M s "fxed-rate" rate + Complexty of brute force mplemetato of etropy creases expoetally wth k R f. OPTA fuctos we seek OPTIMAL PERFORMANCE δ(k,,r) = least MSE of k-dm'l VQ w. th-order etropy codg ad rate R or less S(k,,R) = max SNR of k-dm'l VQ's w. th-order etropy codg ad rate R or less δ(r) = f k, δ(k,,r) = least MSE of VQ wth EC ad rate R or less (ay k,) S(R) = sup k, δ(k,,r) = max SNR of VQ wth EC ad rate R or less (ay k,) VQ-EC-6
HIGH-RESOLUTION ANALYSIS As before, assume the VQ has mostly small cells, eglgble overload dstorto, large M, eghborg cells wth smlar szes & shapes, pot desty approx'ly Λ(x), ertal profle approx'y m(x) Sce quatzer sze s umportat (e.g. ts ot related to rate), ad ca eve be fte, we use uormalzed pot desty, Λ(x), whch s a fucto such that 1. A Λ(x) dx umber of codevectors (or cells) rego A 2. If A s small, but much larger tha the cells the vcty of x, Λ(x) A # pots\cells A 3. Λ(x) 0, Λ(x) dx = M = total umber of quatzato pots (ca be ) 4. Ordarly Λ(x) s a smooth or pecewse smooth fucto. 5. Λ(x) 1 S whe x S DISTORTION: BENNETT'S INTEGRAL Uder hgh-resoluto codtos, a dervato lke that for the orgal Beett shows D m(x) Λ 2/k (x) f k(x) dx VQ-EC-7 RATE: ASYMPTOTIC ENTROPY FORMULA Fact: If X 1,,X are detcally dstrbuted, the uder hgh-resoluto codtos, where R = 1 k H(I) h k + 1 k f k (x) log 2 Λ(x) dx I = (I 1,...,I ) x = (x 1,,x k ) h k = 1 k h(x 1,,X k ) = k-th order dfferetal etropy We'll derve ths shortly. = - 1 k f k (x 1 x k ) log 2 f k (x 1 x k ) dx 1 dx k Note: Dfferetal etropy s ot the same thg as etropy 1. For a cotuous radom varable or vector etropy s fte. 1 Sometmes people use the term etropy whe they really mea dfferetal etropy. VQ-EC-8
Most Importat Example: Uform scalar quatzer wth step sze ad ftely may levels Λ(x) 1 The from the approxmate rate formula R h k + 1 k f k (x) log 2 Λ(x) dx = h - log = h - 1 2 log 12 2 h - 1 2 log 12 D 12 Equvaletly D 1 12 22h 2-2R VQ-EC-9 DERIVATION OF ASYMPTOTIC FORMULA FOR H(I) Frst case: = 1 (for smplcty) H(I) P log P, where P = Pr(X 1 S ) = S f k (x) dx f k (x) dx log f k (x) dx S S - (f k (w ) S ) log (f k (w ) S ) because cells are small (f k (w ) log f k (w )) S - 1 f k (w ) log Λ(w ) S ) recall Λ(x) 1 S whe x S - f k (x) log f(x) dx + f k (x) log Λ(x) dx = h k + 1 k f k (x) log Λ(x) dx VQ-EC-10
Geeral case: 1 H(I) P log P f k (x) dx log f k (x) dx, S S - where x = (x 1 x k ), S = (S 1 S 2 S ) (f k (w ) S ) log (f k (w ) S ), where w = (w 1, w 2,, w ) (f k (w ) log f k (w )) S - (f k (w ) log S ) S The frst summato above ca be approxmated by the tegral - f k (x) log f k (x) dx = k h k ( * ) VQ-EC-11 Before approxmatg the secod sum, ote that log ISj - log Λ(wj ) log S = log ( S 1 IS 2 IS ) = Substtute the above to the secod summato: - (f k (w ) log IS ) IS (f k (w ) (- log Λ(wj ))) IS f k (x 1 x ) log Λ(xj ) dx 1 dx k = f k (x 1 x ) log Λ(x j ) dx 1 dx k = f k (x j ) log Λ(x j ) dx j = f k (x 1 ) log Λ(x) dx 1 because X 1,...,X are detcal ( ** ) Substtutg ( * ) ad ( ** ) to the expresso for H(I) gves 1 k H(I) 1 k (k h k + f k (x 1 ) log Λ(x) dx 1 ) = h k + 1 k f k (x) log 2 Λ(x) dx VQ-EC-12