Matching Dyadic Distributions to Channels

Transcription

1 Matchng Dyadc Dstrbutons to Channels G. Böcherer and R. Mathar Insttute for Theoretcal Informaton Technology RWTH Aachen Unversty, 5256 Aachen, Germany Emal: Abstract arxv:9.375v4 [cs.it] 3 Dec 2 Many communcaton channels wth dscrete nut have non-unform caacty achevng robablty mass functons (PMF). By arsng a stream of ndeendent and equrobable bts accordng to a full refx-free code, a modulator can generate dyadc PMFs at the channel nut. In ths work, we show that for dscrete memoryless channels and for memoryless dscrete noseless channels, searchng for good dyadc nut PMFs s equvalent to mnmzng the Kullback-Lebler dstance between a dyadc PMF and a weghted verson of the caacty achevng PMF. We defne a new algorthm called Geometrc Huffman Codng (GHC) and rove that GHC fnds the otmal dyadc PMF n O(m log m) stes where m s the number of nut symbols of the consdered channel. Furthermore, we rove that by generatng dyadc PMFs of blocks of consecutve nut symbols, GHC acheves caacty when the block length goes to nfnty. I. INTRODUCTION For many communcaton channels, the ultmate rate for relable data transmsson s gven by the maxmum nformaton er cost. For dscrete memoryless channels (DMC) and for addtve nose channels wth fnte nut alhabet, the ultmate rate s the maxmum mutual nformaton between nut and outut er channel use. For memoryless dscrete noseless channels (DNC), the ultmate rate s the maxmum entroy of the nut er average weght. In both cases, the maxmum s acheved by an nut that s dstrbuted accordng to a caacty achevng robablty mass functon (PMF). To use non-unform nut PMFs n a dgtal communcaton system, a modulator has to generate ths PMF by mang ndeendent equrobable data bts to the channel nut symbols. One way to do ths s to arse the data bts by a full refx-free code and to ma each codeword to an nut symbol [, Sec. VII]. PMFs that can be generated n ths way are dyadc,.e., the robablty of each ont s of the form 2 l, l N. The caacty achevng PMFs are n general not dyadc, whch rases two questons. Frst, what s an otmal dyadc PMF that maxmzes nformaton er cost, and second, f we ontly generate blocks of consecutve nut symbols by a dyadc PMF, can we asymtotcally acheve caacty by lettng the block length go to nfnty. For noseless channels, an effcent algorthm to fnd the otmal dyadc PMF that maxmzes entroy er average weght was found n [2]. In general, a common aroach n the lterature s to use the dyadc PMF that results from the otmal source code of the caacty achevng PMF. Dyadc PMFs resultng from source codes are n general Ths work has been suorted by the UMIC Research Centre, RWTH Aachen Unversty.

2 not otmal. For the (d, k) constraned noseless channel, t was clamed n [3] that a source code asymtotcally acheves caacty. To the best of our knowledge, for DMCs, there exst no results n the lterature on otmalty and asymtotc behavor of dyadc PMFs. In [], [4], the authors use source codes for addtve nose channels. Whle good numercal results are observed, otmalty and asymtotc behavor are not assessed. In [5], nut entroy er average weght s maxmzed for addtve nose channels. Ths s n general not equvalent to the maxmzaton of mutual nformaton er channel use. Denote the caacty achevng PMF of a channel by. In ths work, we show for DMCs that mnmzng the Kullback-Lebler dstance (KL) D( ) over all dyadc PMFs maxmzes a lower bound on the acheved mutual nformaton er channel use. For DNCs, we show that searchng for the otmal dyadc nut PMF s equvalent to mnmzng the weghted KL-dstance D( R ) log( / R ) over all dyadc PMFs. The value of R s gven by the fracton of the channel caacty that s achevable by dyadc PMFs. We ntroduce an algorthm called Geometrc Huffman Codng (GHC) and rove that GHC mnmzes D( x) over all dyadc PMFs, for any gven vector x wth non-negatve entres. In artcular, for x =, GHC mnmzes D( ) and for x = R, GHC mnmzes D( R ). The comlexty of GHC s O(m log m), where m s the number of nut symbols of the consdered channel. Furthermore, we show that, to asymtotcally acheve caacty for DMCs and DNCs, the normalzed KL-dstance D( (k) (k) )/k has to vansh for block length k. Ths s acheved by GHC. Based on the resent work, we show n [6] that for fnte sgnal constellatons wth average ower constrant, GHC acheves caacty. GHC s as handy as Huffman codng and an mlementaton of GHC n MATLAB s readly avalable at our webste [7]. The remander of ths work s organzed as follows. In Secton II, we defne GHC. In Secton III, we show otmalty and asymtotc otmalty of GHC for DMCs. We show otmalty and asymtotc otmalty of GHC for DNCs n Secton IV. II. GEOMETRIC HUFFMAN CODING For a PMF and a vector x wth non-negatve entres, the KL-dstance s gven by D( x) = log x. () Note that D( x) can be equal to nfnty. The dyadc PMF that mnmzes the KLdstance s drectly gven by the full refx-free code that s constructed by the algorthm of the followng rooston. A refx-free code s full f t fulflls the Kraft nequalty [8, Theorem 5.2.2] wth equalty. Prooston. Wthout loss of generalty, we assume x x 2 x m. The dyadc PMF that mnmzes D( x) s obtaned by constructng a Huffman tree wth the udatng rule { x xm, f x = m 4x m 2 (2) x m x m, f x m < 4x m. Snce t nvolves a geometrc mean, we call ths method Geometrc Huffman Codng. We wrte = GHC(x). Proof: The roof s gven n the aendx.

3 Tree of GHC Tree of HC Tree of HC Fg. : For q = (.328,.32,.22,.,.22) T, the left fgure dslays the code tree of GHC. The fgure n the mddle shows the code tree of Huffman codng. The rght fgure dslays the code tree of Huffman codng aled to (q,..., q 4 ) T. An mlementaton of GHC n MATLAB can be found at our webste [7]. In comarson to GHC, Huffman codng uses the udatng rule x = x m + x m. Furthermore, t can be shown that Huffman codng mnmzes the KL-dstance D(x ) over all dyadc PMFs. Note that ths s not equvalent to mnmzng () because the KL-dstance s not symmetrc n ts arguments. GHC has the same comlexty as Huffman codng, whch s O(m log m) [9, Cha. 6.3]. For llustraton urose, we aly GHC and Huffman codng to the PMF q = (.328,.32,.22,.,.22) T (3) where ( ) T denotes the transose. The resultng code trees are dslayed n Fgure. By readng off the codeword lengths, the corresondng dyadc PMFs are GHC = (2, 2 2, 2 3, 2 3, ) T and HC = (2 2, 2 2, 2 2, 2 3, 2 3 ) T (4) and the KL-dstances to q are D( GHC q) =.369 and D( HC q) =.9548 (5) where we used the dual logarthm. As exected, the KL-dstance resultng from GHC s smaller than the one that results from Huffman codng. Snce GHC assgns zero to q 5, one may want to manually assgn robablty zero to q 5 and then aly Huffman codng to (q,..., q 4 ) T. The corresondng code tree s dslayed n Fgure. The corresondng PMF and the resultng KL-dstance to q are resectvely HC = (2 2, 2 2, 2 2, 2 2, ) T, D( HC q) = (6) Whle HC slghtly mroves uon HC, the KL-dstance s stll larger than the one resultng from GHC. Let q now denote some arbtrary PMF. We consder k subsequent symbols that are ndeendent and dentcally dstrbuted accordng to q. We denote the ont PMF of these symbols by q (k). Our am s to show that for (k) = GHC(q (k) ), the normalzed KLdstance D( (k) q (k) )/k vanshes for k. To show ths, we wll need the followng lemma, whch shows the exstence of dyadc PMFs wth a bounded KL-dstance for any PMF q. Lemma. Wthout loss of generalty, q q 2 q m. Assgn then = 2 log 2 q for k, and = for > k, where k m s chosen such that m = =. Then It can actually be shown that such k always exsts, so GCC s well-defned.

4 s a dyadc PMF and D( q) log 2. We call ths method Greedy Channel Codng (GCC) and wrte = GCC(q). Proof: D( q) = = log q = = = = log 2 log 2 q q (7) log 2 ( log 2 q ) q (8) log 2q q = log 2 (9) where the nequalty n (7) follows from the values that GCC assgns to the. An mlementaton of GCC n MATLAB s avalable at [7]. It s now easy to show the asymtotc behavor of GHC. Prooston 2. For (k) = GHC(q (k) ) t holds that D( (k) q (k) ) k Proof: Defne (k) = GCC(q (k) ). Then k. () D( (k) q (k) ) D( (k) q (k) ) log 2 () k k k where the frst nequalty follows va Prooston from the otmalty of GHC and where the second nequalty follows from Lemma. log 2/k goes to zero for k and the statement of the rooston follows. III. DISCRETE MEMORYLESS CHANNEL We now show how GHC can be used to fnd dyadc PMFs that well aroxmate the caacty of DMCs. A DMC s secfed by a set of m nut symbols, a set of n outut symbols and a matrx of transton robabltes (h ). An nut PMF relates to ts corresondng outut PMF r as r h h m r =. = (2) r n h n h nm m The mutual nformaton between nut and outut s gven by [, Eq. (8.73)] I() = h log h r. (3) The caacty of a DMC s the maxmum mutual nformaton between nut and outut, where the maxmum s taken over all nut PMFs. To fnd the best dyadc nut PMF, we need to solve the otmzaton roblem maxmze subect to I() s a PMF = 2 l, l N, =,..., m. (4)

5 Ths s a nonlnear otmzaton roblem wth nteger constrants and therefore ntractable for ractcal uroses. In order to overcome ths dffculty, we roceed as follows. Frst, we wll dro the restrcton to dyadc PMFs and characterze the caacty achevng PMF. Then, we wll derve the enalty that results from usng a PMF dfferent from. Fnally, we wll mnmze ths enalty over all dyadc PMFs. Caacty and caacty achevng PMF are resectvely defned as C = max I(), = argmax I(). (5) Denote by r and r the outut PMFs that result from usng the nut PMFs and, resectvely. Accordng to [, Eq. (4.5.)], the outut PMF r resultng from the caacty achevng PMF has the mortant roerty that h log h = C, whenever r >. (6) We now use ths roerty to exress the mutual nformaton I() acheved by some PMF n terms of caacty C and caacty achevng PMF. The only assumton that we make about s that =, whenever =. (7) Under ths assumton, we have for I() I() = h log h = h log h r (8) r r r = h log h + r h log r (9) r = C ( ) h log r (2) r = C r log r r (2) = C D(r r ) (22) where equalty n (2) follows from (6) and (7). From the last lne, we see that the enalty of usng nstead of s exactly the KL-dstance between the corresondng outut PMFs r and r. To get a smle exresson that drectly deends on and, we lower bound the last lne. Accordng to [8, Eq. (4.45)] the KL-dstance between the outut PMFs s uer-bounded by the KL-dstance between the nut PMFs,.e, D(r r ) D( ). Thus, I() C D( ). (23) We conclude that for DMCs, the enalty that results from usng nstead of s uer bounded by D( ). Accordng to Prooston, we can now effcently mnmze the enalty bound over all dyadc nut PMFs by usng = GHC( ). Note that GHC guarantees (7): assume s ordered and m =, m >. Then m > 4 m and GHC assgns m =.

6 We now ontly consder the PMF of k consecutve channel nuts. Denote by (k) the caacty achevng ont PMF. Snce the channel s memoryless, (k) s the roduct of k margnal PMFs and we have I( (k) ) = k I( ) = kc. Thus, for a ont PMF (k) we have I( (k) ) kc D( (k) (k) ). (24) The mutual nformaton er channel use Ī((k) ) I( (k) )/k s thus gven by Ī( (k) ) C D((k) (k) ). (25) k By usng (k) = GHC( (k) ), accordng to Prooston 2, Ī((k) ) C for k and we conclude that GHC s asymtotcally caacty achevng. IV. MEMORYLESS DISCRETE NOISELESS CHANNEL Followng [2], a memoryless DNC s gven by a fnte alhabet A = (a,..., a m ) of atomc symbols and an assocated weght functon w : A R >, a w >. The nformaton rate H that s transmtted over the channel s gven by the entroy of the nut PMF dvded by the average weght,.e., H() = H() w, wth H() = log. (26) To fnd the dyadc PMF that maxmzes H(), we need to solve the otmzaton roblem maxmze subect to H() s a PMF = 2 l, l N, =,..., m. (27) As n the case of DMCs, ths s an ntractable nonlnear otmzaton roblem wth nteger constrants. We wll therefore roceed n the same way as we dd for the DMC n Secton III. We start by calculatng the caacty and the caacty achevng PMF. Ths can be done by Lagrange Multlers, see, e.g., [3]. Denote by b the base of the logarthm log. The caacty s acheved by the nut PMF = b Cw, =,..., m (28) where C denotes caacty and s gven by the greatest ostve real soluton of the equaton b sw =. (29) From (28), we have the relaton w = log C. We can thus wrte w = log. (3) C Denote by R the fracton of C that can be acheved by the best dyadc PMF,.e., argmax dyadc H(), R H( ) C. (3)

7 In general, R s not known beforehand, but we wll show n Subsecton IV-A how t can be found. Suose for now that we know R. Assume further that =, whenever =. (32) Furthermore, we use the conventon log =. Wth these assumtons, we can now wrte H() as H() = R log + R log + H() w (33) = RC log R log w (34) log = RC R = RC D( R ) w (35) w where n (34), we used (3) and the defnton of entroy. By (3), for the best dyadc PMF we have H( ) = RC. It follows that for any dyadc PMF, we have D( R ) and for the best dyadc PMF, we have D( R ) =. We conclude that for DNCs, the best dyadc PMF s found by mnmzng D( R ) over all dyadc PMFs and by Prooston, ths PMF s gven by = GHC( R ). Recall that, as we argued n Secton III, GHC guarantees (32). We now consder the PMF of k consecutve symbols. We denote the corresondng weghts by w (k). The caacty achevng ont PMF s the roduct of k coes of and we denote t by (k). Clearly, w (k) kw mn for =,..., m k where w mn = mn{w,..., w m }. Usng ths, we get for H( (k) ) the lower bound H( (k) ) = H((k) ) (k) w (k) = H((k) ) + (k) log (k) C = C D((k) (k) ) (k) C w mn w (k) (k) (k) log (k) log (k) (36) (37) D( (k) (k) ). (38) k For (k) = GHC( (k) ), accordng to Prooston 2, the last term n the last lne vanshes for k and we have H( (k) ) C, thus GHC s asymtotcally caacty achevng. A. Fndng R The exact value of R s n general not known beforehand. However, R and the best dyadc PMF can be found teratvely by the Lemel-Even-Cohn (LEC) algorthm [2]. The dea of the algorthm s to start wth some R, then fnd the best dyadc PMF for ths R, and then udate the value of R. The best PMF for a gven R s n the orgnal formulaton of the LEC algorthm found as follows. A subset of l nonzero entres of s chosen. A Huffman-lke rocedure of comlexty O(m log m) then fnds the best dyadc PMF wth l nonzero entres. There are m values for l that have to be evaluated, the comlexty of the overall rocedure s thus roughly O(m 2 log m).

8 Algorthm Fndng R and the otmal dyadc PMF for DNCs : rocedure LEC( ) 2: R 3: whle D( R ) do 4: GHC( R ) 5: R H()/C 6: end whle 7: end rocedure From (35) and a careful study of the orgnal formulaton n [2, Sec. III,V], t can be shown that the teraton ste s equvalent to mnmzng the weghted KL-dstance D( R ). Ths can be done wth comlexty O(m log m) by GHC. A formulaton of the comlete LEC algorthm wth GHC as the teraton ste s rovded n Algorthm. Besdes mrovng the comlexty of the teraton ste from O(m 2 log m) to O(m log m), our formulaton answers a queston that was rased n [4], namely how the LEC algorthm could be used to fnd the dyadc PMF that mnmzes the KL-dstance D( ). The smle answer s to erform the teraton ste once wth R =. An mlementaton n MATLAB of our formulaton of the LEC algorthm can be found at [7]. APPENDIX Denote by x some non-negatve vector wth m entres. Assume x s ordered,.e., x x 2 x m. We now show that GHC mnmzes D( x) over all dyadc PMFs. The PMF s dyadc f and only f there exst numbers l N, =,..., m, such that = 2 l, and 2 l =. Ths s equvalent to l,..., l m beng the codeword lengths of a full refx-free code [, Sec ]. Usng ths, we can wrte D( x) = log x = log(2) log 2 x (39) = log(2) 2 l ( log 2 x l ). (4) We defne u by u = log 2 x,. Omttng the constant factor log 2, our am s thus to mnmze 2 l (u l ) (4) subect to l,..., l m are the codeword lengths of a full refx-free code. Based on (4), we now rove the otmalty of GHC n a way smlar to the roof gven n [, Sec ] for the otmalty of Huffman codng. Assume for now that an otmal algorthm assgns fnte values to the codeword lengths l m and l m of the two least lkely symbols, whch corresond to the greatest entres u m and u m of u. We now show that n ths case, there s an otmal algorthm for whch l m = l m. Lemma 2. For an otmal algorthm, u > u mles l l.

9 Proof: Assume the contrary,.e., u > u and l < l. Consder the term By nterchangng l and l, the term decreases: 2 l (u l ) + 2 l (u l ). (42) [2 l (u l ) + 2 l (u l )] [2 l (u l ) + 2 l (u l )] (43) so any code wth u > u and l < l s not otmal. = 2 l (u u ) + 2 l (u u ) (44) = (2 l 2 l }{{} )(u u ) < (45) }{{} > < Lemma 3. There s an otmal algorthm for whch the codewords of the two greatest entres u m and u m are sblngs,.e., l m = l m, and n addton, no other codeword s longer than l m and l m. Proof: In a full refx-free code, the sblng of the longest codeword s also a longest codeword. Accordng to Lemma 2, f u m > u m > u m 2, an otmal algorthm assgns the two longest codewords to u m and u m. If only u m u m u m 2, assgnng the two longest codewords to u m and u m does not change otmalty. We can now use l m = l m to rewrte (4): m 2 2 l (u l ) = 2 l (u l ) + 2 l m (u m l m ) + 2 lm (u m l m ) (46) = = m 2 = 2 l (u l ) + 2 lm (u m + u m 2l m ) (47) = m 2 [( = 2 l (u l ) + 2 (lm ) um + u ) m 2 = }{{} u = m 2 = Thus, by combnng u m and u m through ] (l m ) }{{} l (48) 2 l (u l ) + 2 l (u l ). (49) u = u m + u m (5) 2 the sze m roblem s reduced to a sze m roblem. The otmal algorthm may assgn robablty zero to the greatest entry u m, whch corresonds to l m =. We thus have m m 2 l (u l ) = 2 l (u l ) + 2 (u ) = 2 l (u l ) (5) = = where we used the conventon log = and equvalently 2 =. Thus, f l m =, the sze m roblem reduces to a sze m roblem. It remans to check f t s better to assgn robablty zero to u m or to combne u m and u m. Frst, assume the algorthm combnes u m and u m. Then the contrbuton =

10 to the sum (4) s 2 l (u l ). We can now assgn robablty zero to u m and use the codeword of u for u m. The contrbuton of u m to (4) s then zero and the contrbuton of u m s 2 l (u m l ). Thus, snce our am s to mnmze (4), dong the former s better f and only f 2 l (u m l ) > 2 l (u l ) (52) ( (u m l um + u ) m ) > l (53) 2 u m > u m + u m 2 (54) u m > u m 2. (55) Recallng x = 2 u, the udatng rule (5) and the condton (55) can be exressed n terms of x as x = { xm, f x m 4x m 2 x m x m, f x m < 4x m. (56) REFERENCES [] F. R. Kschschang and S. Pasuathy, Otmal nonunform sgnalng for Gaussan channels, IEEE Trans. Inf. Theory, vol. 39, no. 3, , 993. [2] A. Lemel, S. Even, and M. Cohn, An algorthm for otmal refx arsng of a noseless and memoryless channel, IEEE Trans. Inf. Theory, vol. 9, no. 2, , 973. [3] K. J. Kerez, Runlength codes from source codes, IEEE Trans. Inf. Theory, vol. 37, no. 3, , 99. [4] G. Ungerboeck, Huffman shang, n Codes, Grahs, and Systems, R. Blahut and R. Koetter, Eds. Srnger, 22, ch. 7, [5] J. Abrahams, Varable-length unequal cost arsng and codng for shang, IEEE Trans. Inf. Theory, vol. 44, no. 4, , 998. [6] G. Böcherer, F. Altenbach, and R. Mathar, Caacty achevng modulaton for fxed constellatons wth average ower constrant, 2, submtted to ICC 2. [7] G. Böcherer, Geometrc huffman codng, htt:// Dec. 2. [8] T. M. Cover and J. A. Thomas, Elements of Informaton Theory, 2nd ed. John Wley & Sons, Inc., 26. [9] T. H. Cormen, C. E. Leserson, R. L. Rvest, and C. Sten, Introducton to Algorthms, 2nd ed. The MIT Press, 2. [] R. G. Gallager, Prncles of Dgtal Communcaton. Cambrdge Unversty Press, 28. [], Informaton Theory and Relable Communcaton. John Wley & Sons, Inc., 968. [2] R. M. Krause, Channels whch transmt letters of unequal duraton, Inf. Contr., vol. 5,. 3 24, 962. [3] R. S. Marcus, Dscrete noseless codng, Master s thess, MIT, 957. [4] J. Abrahams, Corresondence between varable length arsng and codng, n The mathematcs of nformaton codng, extracton and dstrbuton, G. Cybenko, D. P. O Leary, and J. Rssanen, Eds. Srnger, 999, ch.,. 7.