Converse Bounds for Finite-Length Joint Source-Channel Coding

Transcription

1 Converse Bounds for Fnte-Length Jont Source-Channel Codng Adrà Tauste Campo 1, Gonzalo Vazquez-Vlar 1, Albert Gullén Fàbregas 123, Alfonso Martnez 1 1 Unverstat Pompeu Fabra, 2 ICREA, 3 Unversty of Cambrdge Emal: atauste,gvazquez,gullen,alfonso.martnez}@eee.org Abstract Based on the hypothess-testng method, we derve lower bounds on the average error probablty of fnte-length jont source-channel codng. The extenson of the meta-converse bound of channel codng to jont source-channel codng depends on the codeboo and the decodng rule and thus, t s a pror computatonally challengng. Weaer versons of ths general bound recover nown converses n the lterature and provde computatonally feasble expressons. I. INTRODUCTION Relable communcaton of messages n the fnte bloc length regme can be characterzed by upper and lower bounds on the average error probablty of the best possble code. In order to prove the exstence of a good code, randomcodng technques are often employed to derve upper bounds on the average error probablty. In contrast, the computaton of lower bounds satsfed by every code s n general challengng snce one must optmze the bound over each possble codeboo and decodng rule. For equprobable messages, a number of lower bounds on the average error probablty [1] [5] lead to a proof of the converse part of Shannon s theorem [6] when the bloc length grows to nfnty. More recently, some of these bounds have been generalzed to non-equprobable messages usng nformaton-spectrum measures [7], [8] or the hypothesstestng method [9]. In ths paper, we elaborate the hypothess-testng method n the context of jont source-channel codng to provde lower bounds on the average error probablty. Followng the footsteps of [4], [5], we propose an extenson of the metaconverse by Polyansy et al. [5, Th. 26], whch states that every channel code wth M codewords, bloc length n and average error probablty ɛ satsfes 1 nf sup β 1 ɛ PX P X, P X Q P X Q M, 1 where β α PX P X, P X Q s the mnmum type- II error gven by the Neyman-Pearson lemma [10] for a maxmum type-i error of 1 α when testng between P X P X and P X Q, where P X s the nput dstrbuton Ths wor has been supported by the European Research Councl under ERC grant agreement A. Martnez receved fundng from the Mnstry of Economy and Compettveness Span under grant RC and from the European Unon s 7th Framewor Programme PEOPLE-2011-CIG under grant agreement nduced by the codeboo, P X s the channel law and Q s an arbtrary output dstrbuton. The central dea of our method s to consder an ndependent bnary hypothess test for every source message and obtan a lower bound on the average error probablty by applyng the Neyman-Pearson lemma [10] to each test. Ths approach ntally provdes a converse bound nvolvng a costly optmzaton over all possble codeboos and decodng rules. Moreover, we show that ths bound recovers several nown results, ncludng the nformaton-spectrum bounds [7], [8] and more mportantly, t s proven to attan Csszár s sphere-pacng exponent for jont source-channel codng [11, Th. 3]. Fnally, we weaen the converse result to obtan lower bounds on the average error probablty that can be numercally computed for some source-channel pars of nterest. A. Notaton and System Model We consder the transmsson of a length- dscrete memoryless source over a dscrete memoryless channel usng length-n bloc codes that are nown both at the transmtter and recever. The source s dstrbuted accordng to P V v = P V v, v = v 1,..., v V, where V s a dscrete alphabet wth cardnalty V. The channel law s gven by P X y x = n P Xy x, x = x 1,..., x n X n, y = y 1,..., y n n, where X and are dscrete alphabets wth cardnaltes X and, respectvely. Wthout loss of generalty we assume that the source messages are ndexed as v 1,..., v V. An encoder maps the length- source message v l to a length-n codeword x l, whch s then transmtted over the channel. We refer to the rato t /n as the transmsson rate. Based on the length-n channel output y the decoder guesses whch source message was transmtted. The decodng rule s specfed by the possbly random transformaton P : n V. The decoded message wll be denoted n the followng by z. The average error probablty ɛ s gven by V ɛ = P V v l ɛv l, 2 l=1

2 where ɛv l Pr v l } = 1 y 3 P X y x l P v l y, 4 s the error probablty when message v l s transmtted. In ths paper, we obtan tght lower bounds on the average error probablty of the best code followng a hypothesstestng approach [4], [5]. II. HPOTHESIS-TESTING APPROACH For every par v l, x l we defne a bnary hypothesstestng problem between the channel condtonal dstrbuton P X=xl and an arbtrary output dstrbuton Q l as H 0 : P X=xl, 5 H 1 : Q l. 6 We can construct a sub-optmal test for the above problem from the system descrbed n Secton I-A: for a gven source message v l, upon observaton of the channel output y, we choose H 0 f z = v l, and H 1, otherwse. The performance of ths test can be evaluated accordng to ts type-i and type- II errors. Specfcally, the probablty of choosng Q l when the true dstrbuton s P X=xl type-i error s equal to ɛv l = 1 y P X y x l P v l y. 7 Smlarly, the probablty of choosng P X=xl when the true dstrbuton s gven by Q l type-ii error s gven by Q l v l y Q l yp v l y. 8 The two types of error can be related va the Neyman- Pearson lemma [10]. Ths result states that the optmal type- II error among all possbly randomzed tests P W : n H 0, H 1 } wth a type-i error of at most 1 α s gven by β α P X=xl, Q l mn Q l P W : yp W H 0 y. y P X y x l P W H 0 y α In the rest of the paper, for ease of notaton, we shall use β α x, Q β α P X=x, Q. Consequently, the type-ii error of any test for 5 6 s lower-bounded by β α xl, Q l as long as the type-i error s no greater than 1 α. In partcular, by settng 1 α = ɛv l n 9 and combnng t wth 8 we obtan β 1 ɛvl xl, Q l l Q v l, l = 1,..., V, 10 whch upon recallng 2 gves an mplct lower bound on the average error probablty of our proposed codng scheme. In order to obtan a vald converse bound from 10 one needs to perform a challengng optmzaton over all possble codeboos x 1,..., x V } and decodng transformatons P. In contrast, any choce of Q l, l = 1,..., V, gves y 9 a converse bound as t s ndependent of the codeboo and the decoder. Alternatvely, a converse bound can be derved by defnng P accordng to the MAP decodng rule and optmzng 10 over all possble codeboos. In both cases, the dervaton of a converse bound becomes computatonally unfeasble as the bloc length ncreases. Hence, n the rest of the paper we analyze the performance of lower bounds derved from 10 whch are computable n several cases of nterest. In partcular, n the next secton we weaen 10 to re-derve a generalzed verson of the Verdú- Han lemma for source-channel codng and we show that the lower bound nduced by 2 10 attans Csszár s spherepacng exponent [11, Th. 3]. Then, n Secton IV we further weaen 10 to obtan computable fnte-length lower bounds on the average error probablty of source-channel codng. III. CONNECTION WITH PREVIOUS WORK A. Informaton-Spectrum Bounds In [8, Th. 6], we derved a lower bound on the average error probablty as a generalzaton of the Verdú-Han lemma for channel codng [3], ɛ Pr } P V V P X XV γ γy, 11 where XV denotes the mappng nduced by a specfc codeboo and γ : n R + s an arbtrary non-negatve functon. By choosng γ = γq, wth Q beng an arbtrary output dstrbuton and γ > 0, optmzng the bound over γ, Q and the P X V nduced by each codeboo, one obtans the converse bound ɛ nf sup P X V γ>0 sup Pr Q PV V P X X Q y } } < γ γ, 12 whch has been ndependently gven n [9, Eq. 34]. We next show that 12 can be seen as a consequence of 10. Frst, fx a gven codeboo x 1,..., x V. Then, by combnng the nequalty [2] 1 β 1 ɛ x, Q sup γ>0 γ Pr P X x } < γ ɛ Q 13 where Pr } s computed accordng to P X=x, wth 10 for a message-ndependent dstrbuton Q, one obtans the set of nequaltes Q v l 1 } P X x l Pr < γ l ɛv l, 14 γ l Q for γ l > 0, l = 1,..., V. By choosng γ l = γ P V v l wth γ > 0 for every l = 1,..., V such that P V v l 0, 14 s equvalent to Q v l P V v l γ } P X x l γ Pr < ɛv l, Q P V v l 15

3 for l = 1,..., V. Consder now the set of condtonal dstrbutons P X V nduced by the codeboo,.e, P X V x v j = 1 f = j and P X V x v j = 0, otherwse. By summng both sdes of 15 over l = 1,... V we fnally have } PV V P X X ɛ Pr < γ γ, 16 Q whch upon optmzaton over γ, Q and P X V yelds 12. In partcular, Han s generalzaton of the Verdú-Han lemma derved n [7, Lemma 3.2] can be recovered from 12 by settng Q = P to be the output dstrbuton nduced by a partcular codeboo and by rewrtng 12 wth the defntons of entropy densty hv log P V V, and nformaton densty X, log P X X P [12], as ɛ nf sup Pr X; hv < log γ } 1 P X V γ>0 γ. 17 B. Csszár s Sphere-Pacng Exponent Csszár showed n [11, Th. 3] that the error exponent of every source-channel code s upper-bounded by E sp R J mn te R [thv,t log V ] t, P V +E sp R, P X, 18 where e R, P V s the source relablty functon [13] and E sp R, P X max P X mn DP Q P V 19 Q:HQ R mn P X : IP X,P X R DP X P X P X 20 s the channel-codng sphere-pacng exponent [14]. When the mnmzng R n 18 les above the crtcal rate of the channel [11], [15], the bound 18 s tght and gves the actual error exponent. We next show that usng 10 wth an approprate choce of Q l recovers Csszár s result. We frst decompose the average error probablty usng the set of source-type classes T, = 1,..., N. Rewrtng 2 we have that where N ɛ = Pr T } ɛ, 21 ɛ 1 T ɛv. 22 v T Our re-dervaton reles on the next result. Lemma 1 [4, Thm. 20]: For every v l T consder the bnary hypothess test n 5 between P X=xl and the dstrbuton Q l = QT. Let a decson rule have type-i error equal to ɛv l and type-ii error equal to b. Then, there exsts a dstrbuton Q T such that, f R > 0 satsfes b γe n R+η, η > 0, γ 0, 1, 23 then A R 1 nη 2 γ e n E sp log 2 R n,p X+η ɛv l for all v l T, where A R > 0 s a functon of R ndependent of n. For every source type-class T, = 1,..., N, we defne the probablty dstrbuton Q T Q T v, v T v v T Q T, v 0, otherwse, where Q l must exst v T = QT Q T for all v l T, such that 25. In vew of 25 there T v Q v 1 T 26. Q T Otherwse, v T T v > 1 and Q would not be a probablty dstrbuton. Wthout loss of generalty and for ease of exposton we next assume that the ndexng of the message set s such that v s a source message fulfllng 26 for T, = 1,..., N. Then, we rewrte 26 as Q T v γe n R,n+ηn, 27 for γ 0, 1, ηn = K n, K > 0, and where we defned R, n 1 n log T + 1 n log γ K 28 n such that γe n R,n+ K n = T 1. We now apply Lemma 1 wth P X y x = P X y xv, b = Q T v, and R, n, whch satsfes 27 for γ 0, 1. Then, t follows from 24 that ɛv A R, n K 2 γ e n E sp R,n log 2 n,p X+ K n for all v T. By pluggng 29 nto 22 we have that 29 ɛ = 1 T ɛv 30 v T 1 1 A R, n E 2 K 2 γ e n spr,n,p X+ K n 31 where R, n R, n log 2 n. We now focus on the terms Pr } T, = 1,..., N n 21. Usng [16, Lemma 2.6] we have that PrT } can be lower-bounded as PrT } + 1 V e DP P V, 32

4 for every = 1,..., N, where P s the type assocated to the class T. Hence, combnng 21, 31 and 32, we obtan N ɛ e n tdp tn P V +E spr,n,p X +o 1,n 33 where o 1, n K n V log A R, n + log K 2 γ log 2 34 on account of 28. Fnally, by choosng K > 0 approprately, usng that A s a contnuous functon and E sp R, P X s a non-ncreasng contnuous functon wth respect to R, the proof follows along the same lnes as n [11] to conclude that lm 1 logɛ te n n R t, P V + E sp R, P X, 35 for some R [ thv, t log V ], such that R lm n R, n, whch after mnmzaton over all R [thv, t log V] yelds 18. IV. COMPUTABLE BOUNDS The am of ths secton s to show that 10 can be convenently weaened to obtan practcal converse results n several cases of nterest. Frst, observe from 10 that f β α xl, Q l s nvertble wth respect to α n an approprate range, one may formulate 10 as an explct lower bound on ɛv l for every l = 1,..., V, whch n turn gves a lower bound on ɛ after averagng over all source messages. To ths end, we mae use of the analytcal propertes of β α x, Q as a functon of α. It s nown that β α x l, Q s a pecewse-lnear, convex, and non-decreasng functon n α [0, 1] [17] that taes values n [0, β max ], where β max 1. Then, the fact that β 0 x l, Q = 0 and the convexty n α [0, 1] mples there must exst α mn [0, 1] such that the functon taes the value 0 n [0, α mn and t s strctly ncreasng n [α mn, 1]. As a consequence, the functon β α s nvertble wth respect to α n the range 0, β max ]. The aforementoned arguments can be used to defne the functon α mn, b = 0, α b x, Q a such that β a = b, b 0, β max ], 1, b β max, 1], 36 n the doman [0, 1]. From the above defnton one can chec that for gven a, b [0, 1], β a x, Q b α b x, Q a. 37 Consequently, by applyng 37 to 10 t follows that ɛv l 1 α l xl Q v, Q l l 38 for l = 1,..., V. Averagng 38 over the source messages and upon approprate optmzaton we obtan the next result. Lemma 2: The average error probablty ɛ ncurred by any codeboo s lower-bounded by ɛ 1 sup x 1,...,x V P V l=1 P V v l nf α l Q Q l v l x l, Q l }. 39 In order to provde computatonally feasble bounds, we restrct our attenton to channels for whch, when Q s approprately chosen, the functon β α Q β α x, Q and thus, α b Q α b x, Q, s ndependent of x. Channels of nterest fulfllng ths property are symmetrc channels accordng to [18, p. 94] and Q l y = Q y = n j=1 Q y j wth Q beng the capacty-achevng output dstrbuton [5]. For ths class of channels we can rewrte 39 usng the decomposton of the message set nto N source-type classes as ɛ 1 sup Q N Pr T } 1 T α Q v Q. 40 v T Although α b Q s ndependent of x, the outer sum n 40 stll depends on the codeboo and the decoder through Q. Hence, the optmzaton n 40 can be performed over all possble dstrbutons Q v, v V. Gven that α b, s concave wth respect to b [0, 1] see Appendx A ths s a convex optmzaton problem. However, snce the mnmzaton must be carred out over an exponentally large number of elements, the optmzaton soon becomes computatonally nfeasble as the message length ncreases. A possble approach to smplfy the aforementoned drawbac s to weaen 40 usng Jensen s nequalty. Theorem 1: The average error probablty of every sourcechannel code n a symmetrc channel s lower-bounded as ɛ 1 sup Q T N Pr T } α QT Q, 41 T where Q T v T Q v, = 1,... N, and Q the capacty-achevng output dstrbuton. Proof: Usng Jensen s nequalty, we obtan N ɛ 1 Pr T } 1 T α Q v Q v T 42 N 1 Pr T } α Q v Q 43 v T T N = 1 Pr T } α QT Q. 44 T Theorem 1 depends on the codeboo and the decoder only through the dstrbuton Q T, and therefore t s optmzed over all dstrbutons defned over source-type classes. Snce s

5 10 0 RCU Converse 41 VH 12 Converse RCU Converse 41 VH 12 Converse 45 Error bound, ǫ Error bound, ǫ Bloclength, n Fgure 1. Upper and lower bounds for a BMS-BSC par. Parameters: P V 1 = 0.05, P X 1 0 = P X 0 1 = 0.1, t = Bloclength, n Fgure 2. Upper and lower bounds for a BMS-BEC par. Parameters: P V 1 = 0.001, P X e 0 = P X e 1 = 0.95, t = 1. the dmenson of the doman of Q T grows polynomally wth the bloc length n, ths nvolves an exponentally less complex computaton than that of 40. Remar 1: It can be checed for nstance, by performng the method of the Lagrange multplers that the optmzng Q of 40 s unform over the values of v belongng to the same source-type class. Hence, the optmzng Q T n 41 nduces the optmzng Q of 40 and as a consequence, both bounds concde. Ths s tantamount to state that Theorem 1 also recovers the generalzaton of the Verdú-Han lemma and attans Csszár s sphere-pacng exponent. Theorem 1 can be weaened to obtan a converse bound that does not requre an optmzaton over the dstrbuton Q T. Usng the fact that α b Q s a non-decreasng functon of b [0, 1] and upper-boundng Q T 1, = 1,... N n 41 we obtan the followng result. Corollary 1: The average error probablty of every source-channel code n a symmetrc channel s lowerbounded as N ɛ 1 Pr T } α 1 Q, 45 T where Q s the capacty-achevng output dstrbuton. Equaton 45 does not depend on the decoder nor the codeboo, and thus, t gves drectly a computable converse result. Whle Corollary 1 cannot be used to recover 12 usng 13, t stll can be shown to attan Csszár s exponent by applyng the arguments n [5, Sec. III-F] for each sourcetype class. We next compare the fnte length-bounds gven n Theorem 1 and Corollary 1 for two source-channel pars: a bnary memoryless source BMS transmtted over a bnary symmetrc channel BSC and over a bnary erasure channel BEC respectvely. A. BMS-BSC In ths example we consder a source-channel par gven by a BMS wth P V 1 = 0.05 and a BSC wth crossover probablty P X 1 0 = P X 0 1 = 0.1, t = 1. We now compare the fnte length-bounds gven n 41 from Theorem 1 computed usng [19] and 45 from Corollary 1 wth respect to the Verdú-Han lemma 12, and the RCU upper bound [8] correspondng to a random-codng ensemble generated wth the product unform dstrbuton. For the BSC the capacty-achevng output dstrbuton s the unform dstrbuton. For ths choce of Q and for dstrbuted accordng to P X=x, the random varable Ψ x = P X x Q s ndependent of x, and so t s Pr P V V Ψ x < γ}. Hence, n ths case t s possble to compute a vald lower bound from the Verdú-Han lemma wthout resortng to an optmzaton over all possble codeboos. Fg. 1 shows that n ths scenaro the bound 45 s looser than 12 whle the bound 41 s tghter n the range of bloc lengths shown. Ths agrees wth the fact that whle 12 can be derved by weaenng Thm. 1, ths s not possble from Cor. 1. B. BMS-BEC We now consder an scenaro of a BMS wth P V 1 = whch needs to be transmtted over a BEC wth erasure probablty P X e 0 = P X e 1 = 0.95, t = 1. Fg. 2 shows the same plot as Fg. 1 for ths source-channel par. From the fgure we observe that n ths case the lower bound 45 s tghter than 12, hence none of these two bounds domnates n general. Smlarly to the prevous case, the bound 41 mproves the other two. APPENDIX A CONCAVIT OF THE FUNCTION α b Lemma 3: The functon α b α b, s a concave functon wth respect to b n [0, 1].

6 Proof: Consder α b, α b, where b, b [0, 1] and denote β a β a,. Snce β α s convex, ths mples λ [0, 1] that β λα b + 1 λα b λβ αb + 1 λβ αb 46 λb + 1 λb, 47 where β αb b n 47 follows from 36. If λb+1 λb β max, the monotoncty of β α mples that λα b + 1 λα b α λb+1 λb 48 on account of 36. Otherwse, f λb + 1 λb > β max, we have that λα b + 1 λα b 1 49 = α λb+1 λb, 50 where 49 follows from α b, α b [0, 1] and 50 from λb + 1 λb > β max and 36. REFERENCES [1] S. Armoto, On the converse to the codng theorem for dscrete memoryless channels corresp., IEEE Trans. Inf. Theory, vol. 19, no. 3, pp , [2] J. Wolfowtz, The codng of messages subject to chance errors, Illnos J. Math., vol. 1, pp , [3] S. Verdú and T. S. Han, A general formula for channel capacty, IEEE Trans. Inf. Theory, vol. 40, no. 4, pp , July [4] R. E. Blahut, Hypothess testng and nformaton theory, IEEE Trans. Inf. Theory, vol. IT-20, no. 4, pp , [5]. Polyansy, H. V. Poor, and S. Verdú, Channel codng rate n the fnte bloclength regme, IEEE Trans. Inf. Theory, vol. 56, no. 5, pp , [6] C. Shannon, A mathematcal theory of communcaton, Bell Syst. Tech. J., vol. 27, pp and , July and Oct [7] T. S. Han, Jont source-channel codng revsted: Informatonspectrum approach, arxv preprnt arxv: v1, [8] A. Tauste Campo, G. Vazquez-Vlar, A. Gullén Fàbregas, and A. Martnez, Random-codng jont source-channel codng bounds, n Proc. IEEE Int. Symp. on Inf. Theory, Sant Petersburg, Russa, July-Aug [9] V. Kostna and S. Verdú, Lossy jont source-channel codng n the fnte bloclength regme, arxv preprnt cs/ v1, [10] J. Neyman and E. S. Pearson, On the problem of the most effcent tests of statstcal hypotheses, Phl. Trans. R. Soc. Lond. A, vol. 231, no , p. 289, [11] I. Csszár, Jont source-channel error exponent, Probl. Contr. Inf. Theory, vol. 9, pp , [12] T. S. Han, Informaton-Spectrum Methods n Informaton Theory. Berln, Germany: Sprnger-Verlag, [13] F. Jelne, Probablstc Informaton Theory. New or: McGraw- Hll, [14] C. E. Shannon, R. G. Gallager, and E. R. Berleamp, Lower bounds to error probablty for codng on dscrete memoryless channels. I, Inf. Contr., vol. 10, no. 1, pp , [15]. hong, F. Alajaj, and L. L. Campbell, On the jont source-channel codng error exponent for dscrete memoryless systems, IEEE Trans. Inf. Theory, vol. 52, no. 4, pp , Aprl [16] I. Csszár and J. Körner, Informaton Theory: Codng Theorems for Dscrete Memoryless Systems, 2nd ed. Cambrdge Unversty Press, [17] H. V. Poor, An Introducton to Sgnal Detecton and Estmaton. New or: Sprnger-Verlag, [18] R. G. Gallager, Informaton Theory and Relable Communcaton. New or: John Wley & Sons, Inc., [19] J. Löfberg, ALMIP : A toolbox for modelng and optmzaton n MATLAB, n Proc. of the CACSD Conference, Tape, Tawan, [Onlne]. Avalable: