Asymptotic Quantization: A Method for Determining Zador s Constant

Transcription

1 Asymptotc Quantzaton: A Method for Determnng Zador s Constant Joyce Shh Because of the fnte capacty of modern communcaton systems better methods of encodng data are requred. Quantzaton refers to the methods by whch analog sgnals are converted nto dgtal representatons and compressed thereby mang them sutable for storage. The asymptotc optmal performance of vector quantzers of fxed dmenson and large rate was frst developed n a rgorous fashon by Paul Zador. Ths paper descrbes a Lagrangan formulaton of Zador s quantzaton results and apples t to estmate Zador s constant. Knowledge of Zador s constant may mprove current quantzer desgn technques by provdng theoretcal performance bounds. By applyng the Lloyd clusterng algorthm to a Lagrangan formulaton of Zador's result t s possble to obtan numercal estmates of Zador's constant. Introducton Wth the growth of the nternet and dgtal meda data compresson s becomng ncreasngly mportant due to the overwhelmng abundance of nformaton that computer users wsh to transmt and store. Quantzaton refers to the methods by whch analog sgnals such as speech musc and mages are converted nto dgtal representatons and compressed thereby mang them sutable for storage. One of the prmary approaches currently employed n analyzng fundamental lmts of quantzer performance s Paul Zador's hgh-rate quantzaton theory whch characterzes the optmal achevable performance of systems wth fxed dmenson and large rate n terms of a constant b. 1 By applyng the Lloyd clusterng algorthm to a Lagrangan formulaton of Zador's result t s possble to obtan numercal estmates of Zador's constant. 3 Knowledge of ths constant may mprove current quantzer desgn technques by provdng theoretcal performance bounds. Ths paper presents the results of numercal smulatons employed to obtan estmates for the value of Zador's constant n the frst through fourth dmensons. In addton the calculated value of Zador's constant for an nfnte dmenson system s gven. Quantzaton Because of the fnte capacty of modern communcaton systems and dgtal storage better methods of encodng data are requred. The goal of quantzaton s to characterze the nput data usng as few bts as possble n such a way that reproducton may be recovered from the bts wth as hgh qualty as possble. One of the earlest examples of quantzaton s "roundng off"; any real number can be rounded off to the nearest nteger and coded as that nteger. Ths example of scalar quantzaton can effectvely tae an nfnte set of real values and approxmate or "compress" them by mappng that set onto a fnte set of dscrete values. Vector quantzaton (VQ) s an extenson of scalar quantzaton to multple dmensons; t s a compresson method that wors by mappng each - dmensonal nput vector onto one of a fnte number of -dmensonal reproducton vectors. A x pxel set or mage would be an example of a 4- dmensonal nput vector. May 00 SURJ 65

2 S U R J A vector quantzer system q s comprsed of two functons an encoder and a decoder whch are desgned usng the Lloyd Algorthm. 4 The encoder α examnes the nput source X blocs t nto vectors of length and maps each vector onto a codeword n the codeboo C. The codeword s then sent through the channel after whch the decoder β maps the codeword onto ts correspondng reproducton vector. The encoder α can be decomposed nto two parts α κ =γοα. In the frst step α maps the sgnal nput onto the reproducton vector n the codeboo that best matches t. In the second step γ converts the reproducton vector nto a codeword. Smlarly the decoder can be broen down nto β κ =βογ 1 where γ 1 assocates the codeword wth ts correspondng vector n the codeboo and β outputs the reproducton vector. The coder (α κ β κ ) can thus be rewrtten as (αγβ). Ths s represented pctorally n Fgure 1. The qualty of a - dmensonal vector quantzer q can be measured n terms of ts dstorton whch quantfes the loss of nformaton resultng from approxmatng X the nput source as Y the code (reproducton) vector. For the sae of smplcty we tae the dstorton to be the meansquared dfference 1 dxy ( ) = X Y = Xl Y l (1) l = 0 between the nput source X and the reproducton vector Y where the subscrpt refers to the th reproducton vector n the codeboo l s the number of bts used to represent each source vector and s the dmenson. Typcally we are nterested n the performance of an algorthm gven a varety of nput sgnals. We thus defne an average dstorton: Df( q) = f( x) X Y dx S () where f(x) s the probablty dstrbuton functon that descrbes the lelhood that the source X wll correspond to a gven reproducton vector x and S s the th codeword (.e. the codeword that corresponds to the reproducton vector x). Whle dstorton measures the fdelty of a quantzer rate measures ts "cost:" the number of bts requred to express the codeword for transmsson to the decoder va the channel. The average rate of a quantzer s gven by R ( q) = p ( S )( l ) (3) f f where p f (S ) s the probablty of usng the th codeword S and l() s the "cost" (.e. length n bts) of the th codeword. The optmalty of a quantzer q s determned by both ts dstorton and rate (.e. ts qualty and cost) whch can be expressed as ρ( f λ q) = Df( q) = λrf( q) (4) where f s the probablty dstrbuton functon (p.d.f) whch characterzes the nput source and λ s a Lagrange multpler whch quantfes the mportance of rate relatve to dstorton for a gven applcaton. If λ s small t allows a large rate (.e. hgh cost or long codeword length) so a larger codeboo can be used and the dstorton ntroduced by compresson wll be mnmal. Conversely f λ s large the rate must be small because cost s crtcal. In ths case a hgh degree of compresson s ey and qualty of the reproducton (.e. nformaton loss or dstorton) s less mportant. For the purposes of ths paper we wll lmt ourselves to consderng systems n the large rate lmt (.e. small values of λ and lttle dstorton). The classc approach to descrbng optmal performance s n terms of the dstorton-rate functon. For rate greater than zero (.e. R > 0) the operatonal dstorton-rate functon s defned as δ f ( R) = nf q : R q R D f ( q) f ( ). Zador proved that under certan condtons on the p.d.f. f ( ) hf lm R δ f( R) = b (5) R where b s Zador's constant whch depends only on and not on f and h( f) f( x)log f( x) dx s the dfferental entropy 3 of f. Whle the exact value of b s nown for = 1 only upper and lower bounds are nown for hgher dmensons (although t s nown that t converges as ). Therefore the goal of ths project was to compute values for Zador's constant (.e. b ) for dmensons greater than one by wrtng a computer program to perform entropyconstraned vector quantzaton (ECVQ) smulatons. A summary of the quantzer terms s presented n Table 1. The ECVQ Algorthm The ECVQ algorthm 5 employed n ths research uses the Lloyd algorthm mentoned earler to desgn vector quantzers wth the least possble dstorton subject to a constrant on rate or entropy. Unle other quantzaton algorthms ECVQ jontly optmzes both the rate R and dstorton D rather than optmzng each quantty separately. ECVQ wors by mnmzng the Lagrangan functonal ρ ( βα α β ) ( ( ( )) = E dx x λ + λr( α( X)) (6) where βα ( ( X )) s the overall 66 SURJ May 00

3 quantzaton operaton on the nput source X to fnd the optmal coder. At the cost of hgher complexty ECVQ generally outperforms other entropy-coded quantzaton schemes ncludng the scalar unform threshold lattce and constraned number-of-ndexes vector quantzaton schemes. The ECVQ algorthm conssts of four man steps. In the frst step the algorthm obtans an ntal reproducton codeboo or nput source. Snce Zador's constant s ndependent of the dstrbuton smulatons were frst performed usng the smplest possble nontrval dstrbuton a unform dstrbuton on the -dmensonal unt cube whch puts equal weght on ntegers between 0 and 1. Later smulatons were also performed usng a Gaussan dstrbuton f = e ( x µ ) / σ where µ s the mean and σ s the standard devaton n order to verfy that the results were n fact ndependent of the dstrbuton used. The second step n the ECVQ algorthm nvolves tranng the codeboo for each value of λ. For decreasng values of λ the algorthm was run untl t met a stoppng crteron ρ old ρ > (7) ρ In the thrd step the encoder maps each nput vector onto the nearest codeword n the codeboo by fndng the codeword that mnmzes ρ (and hence the dstorton): dxβ α( ) arg mn ( ( )) X = I + λ R ( ) (8) Equaton (8) s analogous to nearest neghbor encodng n standard vector quantzaton (VQ). However n ECVQ the rate for the partcular codeword chosen s updated by R ( ) = log ( 1 / p ( )) (9) where p () = P{( α X) = }. The fnal step requres the decoder gven by ( ) β() arg mn E dxy = Y v α( X) = (10) to compute the condtonal expectaton of the dstorton between the output and the nput gven that the encoder produced ndex. It effectvely computes an average or centrod of all the vectors mapped to a partcular cell (.e. codeword) thus far. Ths centrod wll contnue to evolve as more data s mapped to each of the cells. The ECVQ algorthm s summarzed n Table. Computng Zador's Constant We say that a probablty dstrbuton functon f exhbts the Lagrange-Zador property f the followng lmt exsts: ρ lm ( f λ ) + ln λ h( f) = θ λ 0 λ (11) where θ depends only on the dmenson not on the p.d.f. The Zador constant b and θ are related by θ = e ln b (1) Therefore e ρ λ θ λ b ( f q) = ln = + ln h( f) λ Df ( q) = + Rf ( q) h( f) + lnλ λ (13) For the specal case of a unform dstrbuton functon h( f ) = 0. Examnng Equaton (13) t can be seen that t s possble to calculate b f the values of D f and R f are frst obtaned va computer smulatons runnng the ECVQ algorthm. Results Estmates of Zador's constant obtaned from computer smulatons are reported n Table 3 for = 1 through 4. For comparson purposes the upper and lower bounds for dmensons two through four along wth the exact value of Zador's constant for a onedmensonal system are also gven. Because Zador's constant s ndependent of the probablty dstrbuton functon the smulatons were run usng a unform dstrbuton for smplcty. In performng the smulatons both the codeboo sze and the number of tranng vectors were vared as detaled below. For dmensons 1 and 3 the codeboo sze was held constant at 104. For dmenson 1 the test was run fve tmes wth tranng vectors once wth tranng vectors and three tmes wth tranng vectors. The average value obtaned for b 1 from these smulatons was whch devates from the actual value by 0.1%. Because algorthm performance mproves as codeboo sze ncreases we focused on tranng vectors for the second dmenson. The test was run seven tmes wth tranng vectors and once each wth and tranng vectors. The tests gave an average value of for b whch dffers from Zador's constant for fxed rate codng by 1.3%. Although t s not nown whether the values of b for the fxed rate and varable rate cases are dentcal t has been conjectured that the two constants are the same. For = 3 eght smulatons usng tranng vectors and two smulatons usng tranng vectors were run. The results gave an average value of for b 3 whch falls wthn the nown upper and lower bounds. 5 May 00 SURJ 67

4 S U R J For dmenson 4 seven smulatons usng tranng vectors and another three smulatons usng tranng vectors were run wth a codeboo sze of 104. In addton a sngle smulaton was run utlzng tranng vectors and 048 codewords. The average value of b 4 obtaned from these smulatons was whch agan falls wthn the nown upper and lower bounds. 5 As mentoned earler Zador's constant s ndependent of the probablty dstrbuton functon. However to chec that the smulaton results were n fact ndependent of the probablty dstrbuton functon used smulatons were also performed for the frst dmenson usng a Gaussan dstrbuton functon. Ten tests were run usng tranng vectors and 104 codewords. The value obtaned for b 1 from these smulatons was whch agrees well wth both the actual value of and the value of obtaned from the smulatons performed usng a unform dstrbuton functon. Thus t appears that the choce of a unform dstrbuton dd not bas the results. For the nfnte case b can be calculated by usng Zador's result 4 1 β β V b Γ ( 1+ β) V 1+ β (14) where β = r and r = for mean squared error; Γ(x) s the gamma functon and V s the volume of a unt sphere n dmensons. The nfnte case shows b converges upon the value Dscusson As stated above the goal of quantzaton s to convert contnuous sgnals nto bts n a way that optmally trades off dstorton or sgnal to nose rato (SNR) wth bts. Quantzaton theory provdes quanttatve relatons between dstorton and bt rate under certan assumptons. For example the famous "6-dB-per-bt-rule" descrbes how the SNR for a unform scalar quantzer wth hgh rate and low dstorton ncreases 6dB for each one bt ncrease of rate. Ideally we would le to mnmze both dstorton and bt rate but each can be decreased only at the expense of ncreasng the other and hence t s the tradeoff that matters. Zador characterzed the tradeoff under qute general condtons for the "hgh rate" case where the bt rate s hgh and the dstorton small the stuaton arsng n most modern analog-to-dgtal converters. Zador's constant can be vewed as a fundamental constant of nature. It descrbes the relatonshp between dstorton and rate n a manner smlar to the way p descrbes the relatonshp between the crcumference and radus of a crcle. However unle p the exact value of Zador's constant s only nown n the frst dmenson. Estmatng the constant n hgher dmensons s of nterest because t allows the applcaton of theoretcal results to predct the performance of vector quantzers on real world sgnals. Concluson Quantzaton s n essence analog to dgtal converson for the purpose of storage n a dgtal channel. Quantzaton s becomng ncreasngly essental as dgtzaton and the nternet requre mproved methods of conservng memory and storage space. Startng wth the Lagrangan formulaton of Zador's results the generalzed Lloyd ECVQ algorthm has been employed to estmate Zador's constant for = 1 through 4 n hopes that nowledge of Zador's constant may lead to mproved quantzer desgn technques. Wors Cted [1] Zador PL. Topcs n the Asymptotc Quantzaton of Contnuous Random Varables. Bell Lab Tech Memo [] Gray RM Lnder T L J. A Lagrangan Formulaton of Zador's Entropy-Constraned Quantzaton Theorem. IEEE Trans Inform Theory 00; (n press). [3] Shh J Ayer A Gray RM. A Lagrangan Formulaton of Hgh Rate Quantzaton. IEEE Int Conf Acoustcs Speech and Sgnal Processng 001;1-4. [4] Gersho A Gray RM. Vector Quantzaton and Sgnal Compresson. Norwell: Kluwer Academc Publshers 199: [5] Gersho A. Asymptotcally Optmal Bloc Quantzaton. IEEE Trans Inform Theory 1979; 5: [6] Chou PA Looabaugh TD Gray RM. Entropy-Constraned Vector Quantzaton. IEEE Int Conf Acoustcs Speech and Sgnal Processng 1989; 37: SURJ May 00

5 Fgure 1: A Communcaton System Source α γ User β γ -1 C h a n n e l Decoder Table 1. Summary of Quantzer Terms Term Dstorton Rate Optmalty Defnton Quantfes the dfference between the nput source and the reproducton vector. Dstorton measures the fdelty of the quantzer. Rate measures the cost or number of bts requred to express a codeword. The optmalty of a quantzer depends on ts dstorton and rate. Ths value can be expressed as ρ( f λ q) = D () q + λr () q where λ f quantfes the mportance of rate relatve to dstorton. f May 00 SURJ 69

6 S U R J Table. ECVQ Algorthm Steps Descrpton Formula (1) Obtan an ntal reproducton codeboo. () Tran the codeboo for each λ. (3) The encoder maps each nput vector onto the nearest codeword n the codeboo that mnmzes the cost. (4) The decoder computes the centrod for each codeword or cell based upon all the nput vectors mapped onto that codeword or cell. (5) Repeat steps 1-4 untl stoppng crteron has been reached. α( X) = argmn d( X β( )) + λ R( ) I [ ] β() = argmn E d( X Y) α( X) = Y v [ ] ρold ρ > ρ Table 3. Values and Bounds for Quantzaton Coeffcents b K Sphere Lower Bound Actual Value Smulaton Value Cube Upper Bound Zador Upper Bound SURJ May 00