Arithmetic Distribution Matching

Transcription

1 Arithmetic Distributio Matchig Sebastia Baur ad Georg Böcherer Istitute for Commuicatios Egieerig Techische Uiversität Müche, Germay arxiv:48.393v [cs.it] 8 Aug 24 Abstract I this work, arithmetic distributio matchig (ADM) is preseted. ADM ivertibly trasforms a discrete memoryless source (DMS) ito a target DMS. ADM ca be used for probabilistic shapig ad for rate adaptio. Opposed to existig algorithms for distributio matchig, ADM works olie ad ca trasform arbitrarily log iput sequeces. It is show aalytically that as the iput legth teds to ifiity, the ADM output perfectly emulates the target DMS with respect to the ormalized iformatioal divergece ad the etropy rate. Numerical results are preseted that cofirm the aalytical bouds. I. INTRODUCTION Distributio matchig trasforms the output of a discrete memoryless source (DMS) ito a sequece that emulates a target DMS, see Figure for a illustratio. The trasformatio of a distributio matcher is ivertible, i.e., the iput ca be recovered from the output. Distributio matchig is used for example for probabilistic shapig [, Sectio IV.A], [2] ad for rate adaptio [3, Sectio VI.]. Distributio matchers ca be implemeted usig variable legth codig. I [4] ad [5], algorithms for optimal variableto-fixed (v2f) legth ad fixed-to-variable (f2v) legth matchig are preseted, respectively. The drawback of these optimal matchers is that the complete codebook eeds to be calculated offlie, which is ifeasible for large codebook sizes. For data compressio, Huffma [6] ad Tustall [7] codes have a similar problem. Arithmetic codes for data compressio [8], [9] are sub-optimal variable legth codes where ecodig ad decodig ca be doe olie, i.e., o codebook eeds to be stored. The use of arithmetic codig for distributio matchig was proposed i [3, Appedix G]. However, as stated by the authors of [3], their algorithm is icomplete, i particular, it is ot ivertible i the provided descriptio. The authors i [] propose a o-ivertible algorithm for exact radom umber geeratio based o the idea of arithmetic codig. The mai cotributios of this work are the developmet ad the aalysis of a algorithm for arithmetic distributio matchig (ADM). We review f2v legth distributio matchig i Sectio II. I particular, we discuss i Sectio II-E its relatio to data compressio. We the preset i Sectio III our ADM algorithm. We theoretically aalyze the performace of our algorithm i Sectio IV. I particular, we show that as the iput legth teds to ifiity, the output of ADM perfectly This work was supported by the Germa Miistry of Educatio ad Research i the framework of a Alexader vo Humboldt Professorship. DMS P S s, s 2,... c, c 2,... matcher DMS P Z z, z 2,... Fig.. The output s, s 2,... of the DMS P S is trasformed by the distributio matcher. The output sequece c, c 2,... appears similar to the output sequece z, z 2,... of the target DMS P Z. As idicated by the dashed box, the source P S together with the matcher emulates the target DMS P Z. emulates the target DMS with respect to ormalized iformatioal divergece ad etropy rate. We provide umerical results i Sectio V that cofirm our aalytical bouds. Our implemetatio is available at [] ad was used i [2] for coded modulatio with probabilistic shapig. II. FIXED-TO-VARIABLE LENGTH DISTRIBUTION MATCHING The cocept of distributio matchig is illustrated i Figure. We describe i the followig f2v matchig. For clarity of exposure, we cosider biary iput ad biary output. We deote radom variables by capital letters S ad realizatios by small letters s. A biary DMS P S geerates a bit sequece S = S, S 2,..., S of fixed legth. The bits S i are idepedet ad idetically distributed (iid) accordig to the distributio P S () = p src ad P S () = p src. Suppose the source output is s. The matcher trasforms s ito a biary sequece c = c, c 2,..., c l(c) of variable legth l(c), i.e., the matcher outputs codewords of a f2v legth codebook C. The goal of distributio matchig is to emulate a biary DMS with a arbitrary but fixed target distributio P Z () = p code ad P Z () = p code. We explai i the ext paragraphs what we mea by emulatio. A. Iterval represetatio We represet the probabilities of the iput realizatios s ad the target probabilities of the output realizatios c by subitervals of the iterval [; ). We deote the subiterval represetig the probability of s by ad the subiterval represetig the target probability of c by. We idetify the matcher iput s with ad the matcher output c with, i.e., the matcher maps to. We display a example of the iterval represetatio i Figure 2.

2 Fig. 3. A optimal code for iput legth = 2, P S () =.5 ad P Z () =.3. The resultig iformatioal divergece is = Fig. 2. The source is P S () = P S () =.5 ad the target DMS is defied by P Z () = P Z () =.4. The matcher maps to ad to. The sequeces, ad appear at the matcher output with probability.5,, ad.5, respectively. At the output of the target DMS,,, ad would appear with probability.36,.24, ad.4, respectively. B. Iformatioal Divergece The divisio of the iterval [; ) ito subitervals defies the variable legth codebook C. For the example i Figure 2, the codebook is C = {,, }. For the iput legth, the matcher uses 2 output bitsequeces with o-zero probability. I Figure 2, = ad the two possible output sequeces are {, }. The mappig performed by the matcher defies a probability distributio o C. We represet the matcher output takig values i C by the radom variable Y. The codeword c appears at the matcher output with probability P Y (c) = { P X (s), if iput s maps to c, if o s maps to c. The target DMS would have put out the codeword c with probability l(c) PZ(c) C = P Z (c i ). i= We say that P Z iduces the distributio PZ C o the codebook C. I our iterval represetatio, the probability PZ C (c) by which the target DMS P Z would have geerated c is represeted by the iterval size of ad the actual probability is represeted by the iterval size of. The matcher output is a good approximatio of the target DMS output if P Y (c) PZ C(c) ad equivaletly if ad have approximately the same size. This ituitio is formalized by the iformatioal divergece of P Y ad PZ C, which is defied by D(P Y PZ) C = P Y (c) P C Z (c) = (c) () (c) Fig. 4. Example of a code geerated by a arithmetic matcher where = 2, P S () =.5 ad P Z () =.3. The resultig iformatioal divergece is D(P Y PZ C ) =.6346, which is larger tha the divergece of the optimal code i Figure 3. where supp P Y = {c C : P Y (c) > } deotes the support of P Y. We ca see that the iformatioal divergece depeds o the ratio /. It is small if ad have approximately the same size for all codewords that occur with o-zero probability. C. Optimal distributio matchig I [5] a algorithm is described to geerate f2v legth codes for distributio matchig that miimize (). I Figure 3 we show a example for such a optimal code. The iput legth is = 2, the source is uiform ad the target DMS is P Z () = P Z () =.3 The codebook C cosists of all 4 subitervals of [, ) displayed i Figure 3. The resultig iformatioal divergece is.746. The optimal mappig has to be calculated offlie ad stored. The required memory icreases expoetially with the iput legth ad becomes impractical already for reasoably small iput legths. D. Preview: Arithmetic distributio matchig The mai idea of ADM is to require that the code iterval idetifies the source iterval, i.e.,. By this requiremet, we also give up o usig all subitervals of [, ) as codewords. For the same example as i Figure 3, we display i Figure 4 a matcher with the property. The iformatioal divergece of the matcher i Figure 4 is equal to.6346, which is larger tha the divergece of the optimal code i Figure 3, so the code with the property performs worse tha the optimal code. However, as

3 Fig. 5. Example of a arithmetic source compressio code for iput legth = 2, P S () =.3 ad P Z () =.5. we will see i the remaiig sectios, ADM allows us to ecode ad decode olie for arbitrarily log iput sequeces. Furthermore, we will see that for log iput sequeces, ADM results i a smaller divergece tha repeatedly applyig a optimal code. E. Compressio decoder as a matchig ecoder It is claimed i [3] that a ADM for a target DMS P Z ca be realized by applyig the decoder of a arithmetic source compressio code for P Z to the output of a uiform source P S. We illustrate that this is ot possible by a example. We cosider iput legth = 2 ad a source P S () = P S () =.3. We compress P S by emulatig the uiform target DMS P Z () = P Z () =.5. The resultig arithmetic source compressio code is displayed i Figure 5. Suppose ow we wat to apply the correspodig decoder to the output of a uiform source. This meas we apply the iverse mappig from right to left. The output maps to, so this is fie. However, if the output is, this approach fails, sice the ecoder maps othig to, so caot be ecoded. We coclude that i geeral, a arithmetic decoder caot be used as a ecoder. III. ARITHMETIC DISTRIBUTION MATCHING We describe the algorithms for a ADM ecoder ad decoder for biary iput distributios. A. Basic operatios There are two basic operatios, which are used by the ecoder as well as the decoder. ) Read Bits: A bitsequece that was created by a DMS ca be represeted by a iterval by successively readig its bits. We start with the iterval I = [; ). Now we divide the iterval I i two parts accordig to the probability distributio p of the correspodig DMS. The lower subiterval [; p) is assiged to, the upper subiterval [p; ) is assiged to. The we read the first bit of the bitsequece. If there is a, we choose the lower subiterval as the ew iterval I, if there is a, we choose the upper subiterval. This iterval I represets the bitsequece we have read. We cotiue this process recursively by subdividig the curret iterval I accordig to p. Whe all bits of the bitsequece are read, I represets the whole bitsequece. 2) Refie Cadidate List: We subdivide the iterval [; ) accordig to p. I this way we create two subitervals which we call cadidates. The lower subiterval [; p) is assiged to, the upper subiterval [p; ) is assiged to. We call this process refiemet. All itervals created by this process ca also be refied by subdividig them equivaletly accordig to p. So after a secod refiemet we have four cadidates. B. Ecoder Algorithm Ecoder : s iput sequece 2: p src source distributio 3: p code target distributio 4: cadidatelist = refie([; ), p code ) 5: c = empty array 6: for i= to legth(s) do 7: = readbit(, p src, s i ) 8: while j : cadidatelist(j) do 9: apped bits correspodig to j to c : = refie(cadidatelist(j), p code ) : [I u code, Il code ] = refie(, p code ) 2: while k : I u code (k) do 3: I u code = refie(iu code, p code) 4: while l : I l code (l) do 5: I l code = refie(il code, p code) 6: = max(i u code (k), Il code (l)) 7: apped correspodig bits to c The arithmetic ecoder creates a cadidate list by refiig the iterval usig p code. The it reads bits of s util the correspodig iterval idetifies oe of the cadidates. This cadidate is the ew ad. The bit correspodig to this cadidate is writte i the output buffer of c. The the ecoder refies. If idetifies oe of the ew cadidates, the correspodig bit is writte i the output buffer of c too ad this cadidate is the ew agai. The ecoder cotiues the refiemet of ad puts out the correspodig bits if possible, util is ot cotaied i ay of the ew cadidates. The the ecoder starts over agai by readig the ext bit. It repeats this process util all bits are read. To fiish the ecodig, whe all iput bits are read there is a upper cadidate Icode u ad a lower cadidate Il code. The the ecoder refies Icode u util oe cadidate idetifies. It the refies Icode l util a secod cadidate idetifies. The the ecoder chooses the larger oe of the two cadidates as the ew. So holds. Fially it appeds the additioal bits correspodig to this cadidate to c. This fializatio is ecessary to guaratee decodability. Algorithm shows a pseudocode for the ecoder. C. Decoder The arithmetic decoder creates a cadidate list by refiig the iterval usig p src. It the reads bits of c util

4 Algorithm 2 Decoder : c codeword 2: p src source distributio 3: p code target distributio 4: legth of iput sequece s 5: cadidatelist = refie([; ), p src ) 6: s = empty strig 7: i = 8: while legth(s) < do 9: = readbit(, p code, c i ) : i = i + : while j : cadidatelist(j) do 2: apped bits correspodig to j to s 3: = refie(cadidatelist(j), p src ) idetifies oe of the cadidates. The bit correspodig to this cadidate is writte i the output buffer of s. This cadidate is the ew. It is refied, ad if idetifies oe of the ew cadidates, the correspodig bit is writte i the output buffer of s too. Agai, this cadidate is the ew. This is repeated util does ot idetify ay of the cadidates. The the decoder starts readig bits from c agai. It carries o util bits are writte to the output buffer of s. Algorithm shows a pseudocode for the decoder. D. Implemetatio For the floatig poit implemetatio of the algorithms described above we have to prevet the subitervals from becomig too small for a represetatio i floatig poit umbers. That is why they are repeatedly scaled durig the ecodig ad decodig process. Additioally the decoder has to execute the same scaligs as the ecoder, to avoid differet roudig at the ecoder ad the decoder, i.e. the decoder eeds to emulate the ecoder exactly i terms of floatig poit operatios. A. Iformatioal Divergece IV. ANALYSIS Suppose the output of the ecoder is the sequece c. The width of the source iterval is equal to the probability that c is geerated, i.e., (c) = P Y (c). The width of the code iterval is equal to the probability by which the target DMS would geerate c, i.e., (c) = P C Z (c). Propositio. The ratio of the iterval sizes is bouded as (c) (c) = P Y (c) P C Z (c) p code ( p code ). Proof: The left iequality holds sice the algorithm guaratees. The upper boud is proved i the appedix. We boud the iformatioal divergece D(P Y PZ) C = P Y (c) P C Z (c) p code ( p code ) = log 2 p code ( p code ) Thus the u-ormalized iformatioal divergece is bouded from above by a costat that does ot deped o the iput legth. The expected output legth is bouded as (2) (a) H(P Y ) (b) = H(P S ) = H(P S ) (3) where (a) follows by the coverse of the source codig theorem [3, Theorem 5.3.] ad where (b) holds because the mappig of the matcher is oe-to-oe. We ca ow boud the ormalized iformatioal divergece as (a) log 2 p code ( p code ) (b) log 2 p code ( p code ) H(P S ) where (a) follows by Propositio ad where (b) follows by (3). We ca see from (4) that the ormalized iformatioal divergece approaches zero for large iput legths. Our umerical results i Sectio V cofirm this observatio. B. Rate Propositio 2. As the iput legth teds to ifiity, the etropy rate of the matcher output coverges to the etropy of the target distributio, i.e., H(P Y ) H(P Z). (5) Proof: Accordig to (4) which accordig to [4, Propositio 6] implies (5). Sice the mappig is oe-to-oe, we have H(P Y ) = H(P S ). I the average, iput bits are trasformed ito output bits. I terms of the coversio rate /, Propositio 2 states that H(P Z) H(P S ). C. Biary data compressio We ow wat to show how ADM ca be used for data compressio. Suppose P Z () = P Z () = 2. We the have (4) log 2 (p code ( p code )) = 2. (6)

5 With PZ C(c) = 2 l(c) it follows = P Y (c) P C Z (c) P Y (c) = P Y (c)l(c) +. For the first term of this sum we get P Y (c) = H(P Y ). The secod term of the sum is equal to oe, as = P Y (c). Thus = H(P Y ) holds. With H(P Y ) = H(P S ) = H(P S ) we get = H(P S ) + D(P Y P C Z) (a) H(P S ) + 2 (7) where (a) follows by (2) ad (6). The boud (7) recovers the kow boud for arithmetic data compressio, see [5, Exercise 6.]. This shows that ADM ca be used for arithmetic data compressio by usig the target distributio P Z () = P Z () =.5. For Huffma codes H(P S ) + holds [3, Theorem 5.4.]. The additioal bit ecessary for arithmetic codig is the price for calculatig the codewords olie. V. NUMERICAL RESULTS To validate our aalytical results i Sectio IV, we discuss a example applicatio of our ADM implemetatio. We cosider a uiform biary source P S () = P S () =.5 ad the target DMS P Z () = P Z () =.3 ad we evaluate the iformatioal divergece ad the expected output legth. For =, 2,..., 3, we calculate the correct values. For =, 2, 3, 4, we use estimates obtaied from Mote Carlo simulatio. The results for iformatioal divergece are displayed i Figure 6. All obtaied values are below the theoretical boud log 2 (.3.7) = This validates Propositio ad (2). I Figure 7 we plot the rate H(P Y ) = versus the iput legth. As gets large, the rate approaches H(P Z ) =.883 from below. This validates Propositio 2. D(PY P C Z ) ADM (Mote Carlo) ADM (aalytical) optimal (aalytical) boud Fig. 6. Iformatioal divergece D(P Y PZ C ) plotted over iput legth for P S () =.5 ad P Z () =.3. H(P Y ) ADM (Mote Carlo) ADM (aalytical) optimal (aalytical) H(P Z ) Fig. 7. Rate H(P Y ) plotted over iput legth for P S () =.5 ad P Z () =.3. For compariso, we also calculate iformatioal divergece ad rate for the optimal code [5]. As ca be see i Figure 6, the iformatioal divergece is smaller tha for ADM. Suppose ow we would like to ecode 4 iput bits. By (4), the resultig ormalized divergece for ADM would be bouded from above by log 2 p code ( p code ) H(P S ) = Alteratively we could apply the optimal code for = oe thousad times. The resultig ormalized divergece is i this case give by =. = /.884

6 which is higher tha for ADM. This shows that usig suboptimal matchers that ca ecode olie is advatageous for large iput legths. APPENDIX A PROOF OF PROPOSITION To prove the upper boud o the ratio /, we have to take a closer look at the last step of the ecodig algorithm. We cosider the sceario depicted i Figure 8. The last step of the ecodig algorithm ca always be reduced to this sceario. The ecoder refies Icode u ad Il code idepedetly at the ed of the algorithm as described i III-B. To achieve the state i Figure 8, we stop each of these refiemets oe step before a cadidate idetifies. We the drop all cadidates but the two eighborig cadidates, where oe is a subiterval of Icode u ad the other is a subiterval of Il code. The we scale this iterval cosistig of two cadidates to [, ). The last step of the algorithm is ow to refie each of the two subitervals of [, ) ad to choose the largest subiterval idetifyig. The two fial cadidates are cad = p code ( p ratio ) ad cad2 = p ratio ( p code ). We ow show that at least oe of the two fial cadidates is larger tha p code ( p code ). The followig statemets are equivalet: p ratio ( p code ) p code ( p code ) p ratio p code ( p ratio ) ( p code ) p code ( p ratio ) p code ( p code ). This shows i particular that if cad2 is smaller tha p code ( p code ) the cad is larger tha p code ( p code ). Similarly, p code ( p ratio ) p code ( p code ) p ratio p code p ratio p code p ratio ( p code ) p code ( p code ) which shows that if cad is smaller tha p code ( p code ) the cad2 is larger tha p code ( p code ). Sice the algorithm chooses for the fial code iterval Icode fial = max{cad, cad2}, we have Icode fial = max{cad, cad2} p code ( p code ) which is the statemet of the propositio. Fig. 8. p ratio cad cad2 The sceario for the last step of the algorithm. REFERENCES [] J. Forey, G., R. Gallager, G. Lag, F. Logstaff, ad S. Qureshi, Efficiet modulatio for bad-limited chaels, IEEE J. Sel. Areas Commu., vol. 2, o. 5, pp , 984. [2] G. Böcherer, Capacity-achievig probabilistic shapig for oisy ad oiseless chaels, Ph.D. dissertatio, RWTH Aache Uiversity, 22. [Olie]. Available: de/capacityachievigshapig.pdf [3] D. MacKay, Good error-correctig codes based o very sparse matrices, IEEE Tras. If. Theory, vol. 45, o. 2, pp , 999. [4] G. Böcherer ad R. Mathar, Matchig dyadic distributios to chaels, i Proc. Data Compressio Cof., 2, pp [5] R. A. Amjad ad G. Böcherer, Fixed-to-variable legth distributio matchig, i IEEE It. Symp. If. Theory (ISIT), 23, pp [6] D. A. Huffma, A method for the costructio of miimum-redudacy codes, Proc. IRE, vol. 4, o. 9, pp. 98, Sep [7] B. Tustall, Sythesis of oiseless compressio codes, Ph.D. dissertatio, Georgia Istitute of Techology, 967. [8] J. Rissae ad G. G. Lagdo Jr, Arithmetic codig, IBM J. Res. Devel., vol. 23, o. 2, pp , 979. [9] I. H. Witte, R. M. Neal, ad J. G. Cleary, Arithmetic codig for data compressio, Commu. ACM, vol. 3, o. 6, pp , 987. [] T. S. Ha ad M. Hoshi, Iterval algorithm for radom umber geeratio, IEEE Tras. If. Theory, vol. 43, o. 2, pp , 997. [] S. Baur ad G. Böcherer, Arithmetic distributio matchig, Apr. 24. [Olie]. Available: [2] G. Böcherer, Probabilistic sigal shapig for bit-metric decodig, i Proc. IEEE It. Symp. If. Theory (ISIT), 24, pp [3] T. M. Cover ad J. A. Thomas, Elemets of Iformatio Theory, 2d ed. Joh Wiley & Sos, Ic., 26. [4] G. Böcherer ad R. A. Amjad, Iformatioal divergece ad etropy rate o rooted trees with probabilities, Proc. IEEE It. Symp. If. Theory (ISIT), pp. 76 8, 23. [5] D. MacKay, Iformatio Theory, Iferece, ad Learig Algorithms. Cambridge Uiversity Press, 23.