Phase Transitions for Mutual Information

Transcription

1 Phase Transitions for Mtal Information K. Raj Kmar*, Payam Pakzad t, Amir Hesam Salavati* and Amin Shokrollahi* *Laboratoire d' algorithmiqe Ecole Poly techniqe Federale de Lasanne, 05 Lasanne, Switzerland tqalcomm, Inc., 395 Kifer Road, Santa Clara, CA 9505, USA Abstract-We consider ensembles of binary linear error correcting codes, obtained by sampling each colmn of the generator matrix G or parity check matrix H independently from the set of all binary vectors of weight d (of appropriate dimension). We investigate the circmstances nder which the mtal information between a randomly chosen codeword and the vector obtained after its transmission over a binary inpt memoryless symmetric channel (BIMSC) e is exactly n times the capacity of e, where n is the length of the code. For several channels sch as the binary symmetric channel (BSC) and the binary-inpt additive white Gassian noise (AWGN) channel, we prove that the probability of this event has a threshold behavior, depending on whether n/k is smaller than a certain qantity (that depends on the particlar channel e and d), where k is the nmber of sorce bits. To show this, we prove a generalization of the following wellknown theorem: the expectation of the size of the right kernel of G has a phase transition from to infinity, depending on whether or not n/k is smaller than a certain qantity depending on the chosen ensemble. I. INTRODUCTION The development of the theory of modern codes over the past two decades has reslted in the constrction of practical error correcting codes that operate extremely close to the performance limits dictated by information theory. These modern codes admit low complexity decoding techniqes based on the idea of belief propagation. It has been shown that the ensemble of low density parity check (LDPC) codes and Raptor codes are capacity achieving over the binary erasre channel (BEC). Roghly speaking, the BEC is an idealized channel in which information symbols (bits) are either "lost", or recovered with no error, which may be sed to model for example a packetized commnication system with perfect error detection. However, most practical channels also involve errors, casing information symbols to be confsed with one another. When one moves away from the framework of the BEC to more general memoryless symmetric channels, very few analytical reslts exist regarding the performance of modern coding ensembles. The goal of this paper is to investigate sch reslts from an information theoretic point of view. We consider the transmission of a vector X E lf over a general binary inpt memoryless symmetric channel (BIMSC) e. The vector X is transformed by a linear encoder into a vector Y E lf throgh the mapping Y = XG, with the "generator matrix" G E lf xn (note that we are not assming that n :: k, so G is not a generator matrix in traditional sense). The vector Y is then transmitted over the channel e, to obtain Z. We will consider the case that the encoder G corresponds to a family of "LTlRaptor-like" or "LDPC-like" codes. In order to make this notion more precise, we define Cd(, m) to be the ensemble of all x m matrices whose colmns are sampled independently from the set of vectors v E lf of weight d. For brevity, we will se the notation Cd for the ensemble Cd(, m), the dimensions will be clear from context. Frther, we will call an ensemble of codes to be d-niform if either the generator matrix of the code or the parity check matrix of the code is drawn from Cd. We will be interested in evalating the performance of sch d-niform ensembles of codes. For this setting, the primary qestion that we wold like to answer is the following: nder the assmption of long blocklengths, what is the probability that the mtal information I (X; Z) is close to n times the capacity of the channel e? For a particlar d-niform ensemble of codes, we define the probability IId,e = Pr{I(X; Z) < ncap(e)}. For reasons of analytical tractability, we also define the following probability ITd,e = Pr{I(X; Z) < ncap(e) - o(n)}. We conjectre that the mtal information exhibits the following phase transition. Conjectre : For any BIMSC e and any integer d, there exists a positive real nmber B( d, e) sch that if n / k converges to a vale ] as n -+ 00, { then A 0 if ] < B( d, e) IId,e -+. if ] > B( d, e) In the crrent work, we attempt to prove this conjectre for a few important and practical classes of channels, inclding the binary symmetric channel (BSC) and the additive white Gassian noise (AWGN) channel. In certain cases, we will prove a weaker version of Conjectre, by showing that IId,e converges to 0 if n/k is below a certain threshold. In the most general case of an arbitrary BIMSC, the proof of Conjectre remains an open problem. It is very informative to look first at the case where the channel e = J is the trivial (error-free) channel, sch that Z = Y. In this case, it is easy to see that the mtal information I(X; Z) = rk( G), where rk( G) denotes the rank of the matrix /0/$6.00 0O IEEE 37

2 d Q(d,:!) B(d,:!) TABLE I VALUES OF Q(d,:!) AND B(d,:!) FOR VARIOUS d G. Hence in this case, we have that IId,:/ = Pr{rk(G) < n}. Using the nion bond, one can show that Pr{rk(G) < n} :::; le[llker(g)i] -l, () where lker( G) denotes the left kernel of the matrix G. We have the following well-known reslt on the phase transition behavior of the size of the left-kernel, see [3, Theorem 3.5.]. Theorem : Let the generator matrix G be drawn from the ensemble Cd, with d 3. Frther, let a( d,:) be defined as the first component of the vector (a, x,..\) that is the niqe soltion of the system of eqations e - x cosh(..\) ( ad )a ad _ x ( ad: xr/ d..\ tanh(..\) Sppose that k, n sch that njk -+ a. Then, if a < a( d, :), then E[llker( G) I] -+, and if a > a( d, :), then E[llker(G)I] Notice that this immediately yields that IId,:/ -+ 0 if a < a( d, :), bt only yields a trivial bond on IId,:/ if a > a( d, :). The following theorem from [5] proves Conjectre for the case e =:. Theorem : Let In(d 'Y d - := d(l _ (d) d -l ' where (d is the smallest root of z ( - In z) - ;:t In z - = o for z E [0,]. Then Conjectre is tre for e = : and O(d,:) := 'Y d. In other words, if n, k sch that njk -+ a and a < O( d, :), then ITd,:/ -+ 0, whereas ITd,:/ -+ if a> O(d,:). Table I gives vales of O(d,:) and a(d,:) for varios d. Finally, we wold like to comment on literatre related to the topic of this paper. The reslts that are closest to the spirit of those in this paper are the ones in []-[3], [5]; one can think of these reslts as special cases of or reslts when the channel e is error-free. There is a whole set of other papers that discss nder which conditions I (X; Z) = k, so that ML-decoding is sccessfl'. The most general among sch reslts (bt with a limited range i For G to achieve capacity, we need to have that lex; Z) = k; however, we are interested in this paper in the case when I (X; Z) = ncap( e). These two qantities are eqal only if k = ncap( e), so that the rate of the code is eqal to the capacity.,, x. of applicability) are those of MacMllan and Collins [6] which analyze the inherent gap of certain families of binary linear codes sch as the Hamming and Golay codes to the capacity of the BSC. For ensembles of sparse matrices the qestion of achievability of capacity is not new, of corse. Already in his thesis, Gallager [7, pp ] showed that the rate of a right (or check-) reglar LDPC code that achieves reliable commnication over a BSC sing ML decoding is bonded away from the capacity of the channel by a fnction depending on the right degree of the nderlying graph. In particlar, the right degree has to go to infinity if the code is to approach capacity. Richardson et al. [8] proved that the same conclsion holds for the maximm right degree, if the graph is not rightreglar; this implies that the reslt also applies when taking the average right degree, instead. Brshtein et al. [9] generalized these reslts to general BIMSC. These reslts were themselves generalized and optimized by Sason and Urbanke [0] who gave rather close gaps to capacity for LDPC codes with given average right degree. Thogh it may seem to a reader that this paper is investigating a similar problem as those of the above papers, this is not entirely the case. In all the above cases, either k - I (X; Z) is calclated directly (e.g., in [6]), or an pper bond is obtained on the entropy H (Z) to show that I (X; Z) is bonded away from k (as is the case in [7]-[0]). For s a direct calclation of k - I(X; Z) is very difficlt, so that the reslts of [6] are not directly applicable. Moreover, we are interested in lower bonds for H(Z) (or rather, its expectation), rather than pper bonds, so the mentioned reslts are not applicable either. II. THE CASE e = BSC(p) In this section, we stdy the case of a BSC with crossover probability p, denoted by e = BSC(p). The main theorem of this section is the following. Theorem 3: Let Bw denote the nmber of words of weight w in the right kernel of the matrix G. Then Pr[I(X; Z) < ncap(e)] :::; 0g (t, le[bw](l - P) W). Note that if we assme e = :, so that p = 0, then I(X; Z) is the rank of the matrix G, and Lw Bw is the size of the right kernel of G. Hence, the statement of the theorem says that Pr[rk(G) < n] :::; 0g(lE[llker(G)J) :::; le[llker(g)i] -, and we have retrieved (). To prove Theorem 3, we need an axiliary reslt, which may be interesting in its own right. Theorem 4: Let D be a distribtion on F, with entropy H(D), and let P := PrD[x = ]. For v E F let qv := L, (lv) = l P, where ( v) is the scalar prodct of and v (over F ). Then we have n - H(D) :::; 0g (L ( - Qv) ). vef' 38

3 that Proof" We will first remark some general facts. First, note -qv = -qv-qv = L P- L P = L( -) ( v) p, (lv)=o (lv)=l so that the vector (-qv I v E lf ) is the Hadamard transform of the vector (P I E lf ). Let H be the n x n-hadamard matrix. Since HI y'ri is a nitary matrix, we have ",, P = n ( - qv ). () EIF' v Note that by the concavity of the logarithm fnction, we have for all Xl,..., Xm 0 and all ai,..., am 0 with Li ai = : Specializing to m = k, and a = X = P, we see that L..J - JF ' p > II..,p = -H(D) so that which is the statement of the theorem. We omit the proof of the following corollary for lack of space. Corollary : Sppose that C is an [n, k]-code, and that Y = (Yb..., Yn) is a vector chosen from C niformly at random. Moreover, sppose that for i =,..., n the random variables ei are independent binary Bernolli random variables with Pr[ei = ] = Pi, and sppose that Y, eb.., en are independent. Let Z = (Zl'...' zn) be the vector with Zi = Yi + ei. Then we have Since log(x) is a concave fnction, we have for any random variable U over the positive real nmbers: IE [log (U)] :::; log(ie[u]), hence it sffices to prove that Pr[I(X; Z) < ncap(e)] :::; n - E [H(Z)]. Let be a random variable taking vales in the set {O,,..., t}, and let Pi denote the probability that the vale of is i. Then Pr[ < t] = Po Pt-l = - Pt :::; Pt-l + pt tpo = t - E []. We apply this reslt to = I(X; Z) and t = ncap(e) to obtain that the probability in qestion is pper bonded by ncap(e) -E[I(X; Z)]. Noting that I(X; Z) = H(Z) -H(ZIX) = H(Z) -n h(p), and that Cap(e) = - h(p), the reslt follows. The ineqality of Theorem 4 cannot be improved since it is tight for flat distribtions. In other words, if D is a niform distribtion over a k-dimensional sbspace of lf, where :::; k :::; n, then eqality is achieved. Using the above reslts, we obtain the following theorem whose proof is omitted for lack of space. Theorem 5: Let d 3 be fixed and e = BSC(p). Define f(>..) := ed cosh(>..)d/>..tanh(>..) (tanh(>..)) d (-p), >..tanh(>..) (tanh(>..)) >" tanh( >" ) ( d ) <> g(>.., ) := cosh(>") e d ->..tanh(>..) and ( ) - _ { _ (d ->..tanh(>..) ) >" tanh( >" )/d. >.. tanh(>..) e ( - p), ( <f>d) i)} + e - HI-. where H (z) is the entropy of the probability distribtion of the random variable z. In particlar, if all the Pi are eqal to p, then n -H(z) :::; log (t. Bw(l - P) W), where Bw is the nmber of words of weight w in the right kernel of G. The proof of Theorem 3 follows from the previos corollary. For lack of space, we confine orselves to providing a sketch only. Proof" (Of Theorem 3 - Sketch) Let C be the code generated by the rows of the matrix G. Then, by the previos corollary, we have n - E [H(Z)] :::; IE [IOg (t. Bw(l - P) W) ]. Let /00 be the maximm of f(>..) in the interval (0, (0), and let Ol be the largest positive vale of sch that g( >.., ) :::; for all >.. with >.. tanh( >..) :::; d. Also, let O be the maximm vale of 0 sch that ( ) < p. Set o:(d, e) := min (max(oo, Ol), (). Sppose that n, k go to infinity sch that nlk --+ 0:. Then IId,e if 0: < o:(d, e). The case d = is more involved. In fact, we cannot show that 0:(, BSC(p)) exists. However, it was proved in [4] that in this case the analogos threshold for fi,bsc(p)' viz., 0(, BSC(p)) exists and 0(, BSC(p)) = ( _ p ) Moreover, when d =, one can show that the largest vale of and happens when> O. fnction f(>..)! is eqal to ( p) Table II gives the vale of Cap(BSC(p) )o:( d, BSC(p)) = (-h(p))o:( d, BSC(p)) for varios d and p. One wold expect these vales to converge to as d grows. While this is seen to happen for p «: (see also Table I that corresponds to the limiting case of p = 0), the vales converge to arond / 39

4 d\p ;j A simple maniplation yields TABLE II THE VALUES OF (- h(p))a(d, BSC(p)) FOR VARIOUS VALUES OF d AND p. when p converges to /. This sggests that there is room for improvement in the bonds of Theorem 5. It can be shown that the reslts for the BSC also extend to reslts for the convex combination of BSCs. Details are omitted for brevity. III. THE AWGN CHANNEL We now trn or attention to the case when e = AWGN(p) is a binary inpt (real) AWGN channel, whose otpt Z E IRn may be written as Z=Y+W, where W rv i.i.d. N (0, i I). We assme standard binary phase shift keying (BP K) modlation for transmission over the AWGN channel, i.e., we map component-wise the binary codeword Y t-+ (-) Y prior to transmission over the channel. With a slight abse of notation, we refer to both the binary codeword and the modlated symbols with the same notation Y; the one being referred to will be clear from the context. Hence p denotes the signal to noise ratio (SNR) of the AWGN channel. As before, we first develop a lower bond on the mtal information between the inpt and otpt. I(X; Z) H(Z) -H(ZIX) = H(Z) -! log (7re) n. (3) p n Using Jensen's ineqality, we lower bond the entropy as H(Z) -log [/ p (Z) dz], (4) where p( Z) denotes the pdf of the otpt Z. Define the code Cy = { Y = XG I X E F } { Yb Y,,Yk} (note that if G is not fll-rank, then not all Yi are distinct). Then, we may write k p p (Z) = n/ _ '"' e- IZ-YiI. k ( )n We may hence evalate From (3), (4) and (5), we obtain Since BPSK modlation is sed, IYj -Yi l = 4dH (Yj, Yi), where dh(a, B) denotes the Hamming distance between A and B. Since we employ a linear code, we may frther simplify the above ineqality to obtain I(X; Z) -log " [ ( e) n/ k k exp { - p WH(Yi n, where WH(X) denotes the Hamming weight of X. If we define Cw to be the nmber of codewords with Hamming weight w, we may rewrite the above as: [( e ) n / n ] I(X;Z) -log " k C we- w p. (6) In order to examine Conjectre for e = AWGN(p), we evalate Pr { I(X; Z) < n(cap(e) - n :::; Pr { -log [ G) n / k t Cwe- w p] < n(cap(e) - )} = Pr { ( cap(e)-r ) n t. Cwe- w p > n }, (7) bn b where is a constant independent of n, k. An analysis of the term S n C e-p w L...w=l w, where = IKCap(e)-R Y' is a constant that is independent of n leads to the following theorem. Theorem 6: Let d 3 be a fixed constant, and e = AWGN(p). Define f().., 0) = In [b9 ( edo ) A ta:;h A]_ p).. tanh)" (0 - ).. tanh)" d + In cosh)" + ).. tanh )''In (ta:h).. ) (8) 40

5 and (A ()) d [( tanh I A) A tanh A b g, A tanh A n (()d)b (d76 -AtanhA) -b cosh A] (() - ) 6 (A tanha) e. -po Let ()l «()) denote the maximm vale of () for which the fnction f(a, ()) (respectively, g(a, ()) is less than zero for all non-negative A sch that A tanh A 9 ' Set ()(d, e) = min {, max{()l, ()}}. If n, k sch that n/k --+ 0:, then ITd,e if 0: < ()(d, e). Proof" (Sketch) In order to analyze Cw, we consider two scenarios. When n ;::: k, we se a good channel code to transmit information across the channel. On the other hand, when n < k, we need to compress (qantize) the information to be sent over the channel. We first examine the case when n;::: k. ) Channel coding when n ;::: k: We define or channel coding ensemble in the following manner. Choose the parity check matrix H E lf n-k) x n from the ensemble ed. The qantity Cw is eqal to the nmber of vectors with weight w in the right kernel of H, or in the left kernel of H T. Along the lines of the proof of Theorem 3.5. in Sec. 3.6 of [3], we analyze S by splitting the sm into three regions, Sl = bn L::ow<o Cwe-P w, S = bn n LO n ::ow«l - o ) n Cw e-p w and S3 = (9) bn L ( - o ) n ::ow::o n Cw e-p w, where O. We otline the analysis of these terms in the seqel, omitting details for lack of space. a) Analysis of Sl: From the proof of Lemma 3.5. in [3], we can show that Sl E ', for some 8 > 0, where E ' > 0 is an arbitrary constant. b) Analysis of S: Define () = n/k. Along the lines of the analysis in [3], it can be shown that S vanishes if for all non-negative A sch that A tanh A 9 ' either f(a, ()) < 0 or g(a, ()) < 0, where the fnctions f and g are as defined in (8), (9). c) Analysis of S3: Along the lines of the proof of Lemma 3.5. in [3], we can show that S when we set E ;::: log (JI) bits. ) Compression (Qantization) for n < k: We fix G to be any k x n binary matrix that is of rank n. Hence, as X varies over all k-tples, Y varies over all n-tples, with each n-tple appearing k-n times in the qantizer codebook. This reslts in Cw = k-n ( :). Consider We may now evalate (7) as bnk-n t. (: ) e-p w bnk-n ( + e-pt. Pr{I(X; Z) < n( Cap( C)- E)} Pr {b*- ( + e-p) > }. d\ P(dH) TABLE III THE VALUES OF CAP( AWGN(p» 8(d, AWGN(p» FOR VARIOUS VALUES ofdand p. It can be show that if one sets E ;::: log (JI), the right-hand side converges to zero. Shown in Table III are the vales of Cap(AWGN(p))()(d,AWGN(p)) for several vales of p and d. Notice that the bonds for the case of the A WGN are very tight in terms of the threshold vales for the rate being almost at capacity for even moderate vales of d. However, or main theorem for the AWGN, Theorem 6 is in some sense weaker than the corresponding reslt for the BSC in Theorem 5, since the former proves a statement abot ITd,e involving a linear back-off from capacity, while the latter shows a reslt relating to IId,e with no back-off from capacity. IV. ACKNOWLEDGEMENTS This work was spported by Grant 80-ECCSciEng of the Eropean Research Concil. The athors wold like to thank Mahdi Cheraghchi, Venkat Grswami, Shlomo Shamai, Emre Telatar, and David Tse for helpfl discssions and comments dring the research on this paper. REFERENCES [] J. Blomer, R Karp, and E. Welzl, "The rank of sparse random matrices over finite fields;' Random Strctres and Algorithms, vol. 0, no. 4, pp , 997. [] N. Calkin, "Dependent sets of constant weight binary vectors;' Combinatorics, Probability, and Compting, vol. 6, pp , 003. [3] V.F. Kolchin, Random Graphs, Nmber 53 in Encyclopedia of Mathematics and its Applications, Cambridge University Press, 999. [4] O. Etesami and A. Shokrollahi, "Raptor codes on binary memoryless symmetric channels," IEEE Trans. Inform. Theory, vol. 5, no. 5, pp , 006. [5] P. Pakzad and A. Shokrollahi, "EXIT fnctions for LT and raptor codes, and asymptotic ranks of random matrices," in Proc. of the IEEE International Symposim on Information Theory (ISIT), pp. 4-45, Jne 007. [6] SJ. MacMllan and O.M. Collins, "The capacity of binary channels that se linear codes and decoders;' IEEE Trans. I'!form. Theory, vol. 44, pp. 97-4, 998. [7] RG. Gallager, Low Density Parity-Check Codes, MIT Press, Cambridge, MA,963. [8] T. Richardson, A. Shokrollahi, and R Urbanke, "Finite-length analysis of varios low-density parity-check ensembles for the binary erasre channel;' in Proc. of the IEEE International Symposim on Information Theory (ISIT), pp. I, 00. [9] D. Brshtein, M. Krivelevich, S. Litsyn, and G. Miller, "Upper bonds on the rate of LDPC codes;' IEEE Trans. Inform. Theory, vol. 48, no. 9, pp , 00. [0] I. Sason and R Urbanke, "Parity-check density verss performance of binary linear block codes over memoryless symmetric channels;' IEEE Trans. I'!form. Theory, vol. 49, pp ,003. 4