Lecture 4 : Introduction to Low-density Parity-check Codes

Transcription

1 Lecture 4 : Introduction to Low-density Parity-check Codes LDPC codes are a class of linear block codes with implementable decoders, which provide near-capacity performance. History: 1. LDPC codes were invented by Gallager in his 1960 doctoral dissertation. 2. In 1981, Tanner generalized LDPC codes and introduced a graphical representation of LDPC codes, now called a Tanner graph. 3. The study of LDPC codes was resurrected in the mid 1990s with the work of MacKay, Luby, and others. Advantages over turbo codes: 1. Do not require a long interleaver for near capacity performances. 2. Lower error floor value. 3. More simple decoding architecture. 4. More flexible for code design. 1

2 Matrix Representations of LDPC codes 1. A low-density parity-check (LDPC) code is a linear block code given by the null space of an m n parity-check matrix H that has a low density of 1s. 2. A (g, r) Regular LDPC codes: H has constant column weight g and constant row weight r, where r g = n m and g m. 3. If H is low density, but its row and column weight are not both constant, then the code is an irregular LDPC code. 4. Row-column constraint (RC constraint): No two rows (columns) have more than one position in common that contains a nonzero element. 5. The descriptor low density is unavoidably vague and cannot be precisely quantified, although a density of 0.01 or lower can be called low density. The density need only be sufficiently low to permit effective iterative decoding. 2

3 6. Optimal decoding (e.g. maximum-likelihood) decoding of the general linear block code that is useful for applications is not possible due to the vast complexity involved. 7. The low density aspect of LDPC codes accommodates iterative decoding, which typically has near-ml performance at error rates of interest for many applications. 8. H needs not be full rank. The code rate R is bounded as R (1 m n ) = (1 g r ), with equality when H is full rank. 3

4 The original LDPC codes Gallager codes: 1. Gallager s original LDPC code is a linear code whose m n PCM H has g m ones in each column and r n ones in each row. The matrix H has the form: H = H 1 H 2. H g (1) where the submatrices H a, a = 1,..., g, have the following structure: (a) For any integers µ and r greater than 1, each submatrix H a is µ µr with row weight r and column weight 1. (b) The submatrix H 1 has the following specific form: for i = 0, 1,..., µ 1, the i-th row contains all of its r 1s in columns ir to (i + 1)r 1. 4

5 (c) The other submatrices are obtained by column permutations of H H is regular, has dimension µg µr, and has a column weight of g and a row weight of r. 3. The RC constraint is not guaranteed, but they can be avoided via computer design on H. 4. Gallager showed that the ensemble of such codes has excellent distance properties, provided that g The following H matrix is an example given by Gallager: 5

6 It corresponds to a (20, 5) code with g = 3 and r = 4. Observe that this matrix has the form of (1) and H 1 has the form described above, with µ = 5. 6

7 Graphical Representations of LDPC codes 1. Tanner graph LDPC code Trellis Convolutional code 2. The Tanner graph provides a complete representation of the code and it aids in the description of decoding algorithms. 3. A Tanner graph is a bipartite graph, that is, a graph whose nodes may be separated into two types, with edges conneting only nodes of different types. (a) variable nodes (code-bit nodes) : V N check nodes (constraint nodes) : CN (b) CN i is connected to V N j whenever element h ij in H is A reference for graph theory: N Deo, Graph Theory with Applications to Engineering and Computer Science, Prentice Hall, Englewood Cliffs, N.J.,

8 5. Example: 8

9 6. Each of the nodes acts as a locally operating processor and each edge acts as a bus that conveys information (log-likelihood ratios, LLR s ) from a given node to each of its neighbors. (a) Initialization: n LLR s from the channel n V N processors (b) At the beginning of each half-iteration, each V N processor takes inputs from the channel and from each of its neighboring CNs. Utilizing these input values, a V N processor computes outputs which are forwarded back to each one of its neighboring CN processors. (c) In the next half-iteration, each CN processor takes inputs from each of its neighboring V Ns, and from these computes outputs for each one of its neighboring V N processors. (d) The V N CN iteration continue until a codeword is found or until the preset maximum number of iterations has been reached. 9

10 7. Cycles force the decoder to operate locally in some portions of the graph (e.g., continually around a short cycle) so that a globally optimum solution is impossible. The necessity of a low-density matrix H: at high densities (about half of the entries are 1s), many short cycles will exist, thus precluding the use of an iterative decoder. 8. The length of a cycle is equal to the number of edges which form the cycle. A cycle of length l is called an l-cycle. 9. The minimum cycle length in a given bipartite graph is called the graph s girth. 10. The shortest possible cycle in a bipartite graph is a length-4 cycle and such cycles manifest themselves in the H matrix as four 1s that lie on the four corners of a rectangular submatrix of H. The RC-constraint eliminates length-4 cycles. 10

11 11. In a Tanner graph, the degree of a node is defined as the number of edges that are incident with the node. 12. For irregular LDPC codes, it is usual to specify the V N and CN degree-distribution polynomials, denoted by λ(x) and ρ(x), respectively. (a) In the polynomial, λ(x) = d v d=1 λ dx d 1 [ρ(x) = d v d=1 ρ dx d 1 ] λ d (ρ d ): the fraction of all edges connected to degree-d V Ns. (CNs) d v (d c ): the maximum V N (CN) degree. (b) For (2,4)-regular LDPC code, we have λ(x) = X, ρ(x) = X 3. 11

12 Message passing and the turbo principle The low-density nature of the PCM facilitates iterative decoding. Although the various iterative decoding algorithms are suboptimum, they often provide near-optimum performance, at least for the error rates of interest. Sum-product algorithm (SPA) is a general algorithm that provides near-optimal performance across a broad class of channels. SPA is a kind of message-passing decoding (MPD) algorithm. MPD refers to a collection of low-complexity decoders working in a distributed fashion to decode a received codeword in a concatenated coding scheme. Variants of the SPA and other message-passing algorithms have been invented independently in a number of different contexts, including belief propagation for inference in Bayesian networks and turbo decoding for turbo codes. 12

13 We can consider an LDPC code to be a generalized concatenation of many single parity-check (SPC) codes. A message-passing decoder for an LDPC code employs an individual decoder for each SPC code and these decoders operate cooperatively in a distributed fashion to determine the correct code bit values. 13

14 Example: 14

15 Soldier-counting problem in the Bayesian-inference literature 15

16 1. Case (a): Soldiers in a line. Each soldier receives a number from one side, adds one for himself, and passes the sum to the other side. The soldiers on the ends receive a zero from the side with no neighbor. The sum of the number that a soldier receives from one side, and the number the soldier passes to that side, is equal to the total number of soldiers. 2. Case(b): (Simple tree formation): (a) The message that an arbitrary soldier X passes to arbitrary neighboring soldier Y is equal to the sum of all incoming messages, plus one for soldier X, minus the message that soldier Y had just sent to soldier X. 16

17 (b) Extrinsic information: The idea is that a soldier does not pass to a neighboring soldier any information that the neighboring soldier already has, that is, only extrinsic information is passed. I X Y = I Z X I Y X + I X = Z N(X) Z N(X) {Y } I Z X + I X where N(X) is the set of neighbors of soldier X,I X Y is the extrinsic information sent from soldier X to soldier Y, and I X is sometimes called the intrinsic information. The addition and subtraction operations can be generalized operations, e.g., box-plus operations. 17

18 3. Case (c): A formation containing a cycle (a) No matter what counting rule one may devise, the cycle represents a type of positive feedback, so that the messages passed within the cycle will increase without bound as the trips around the cycle continue. (b) This example demonstrates that message passing on a graph cannot be claimed to be optimal if the graph contains one or more cycles. 4. While most practical codes contain cycles, it is well known that messagepassing decoding performs very well for properly designed codes for most error-rate ranges of interest. It is for this reason that we are interested in MPD. Turbo principle: 1. The notion of extrinsic-information passing described above has been called the turbo principle in the context of iterative decoding of concatenated codes in communication channels. 2. The sequence of the forwarded messages in the Soldiers in a line case can 18

19 be initialized to all zeros and updated as follow: (1, 1, 1, 1, 1) (2, 2, 2, 2, 1) (3, 3, 3, 2, 1) (4, 4, 3, 2, 1) (5, 4, 3, 2, 1). 19

20 The Sum-Product Algorithm Overview: 1. In addition to introducing LDPC codes, Gallager also provided a near-optimal decoding algorithm called sum-product algorithm which is also sometimes called belief-propagation algorithm. 2. The optimality criterion underlying the development of the SPA decoder is symbol-wise maximum a posteriori (MAP). 3. We are interested in computing the a posteriori probability (APP) that a specific bit in the transmitted codeword v = [v 0, v 1,..., v n 1 ] equals 1, given the received word y = [y 0, y 1,..., y n 1 ]. In other words, we are interested in computing the APP, P r(v j = 1 y), the APP ratio (also called the likelihood ratio, LR), 20

21 l(v j y) P r(v j = 0 y) P r(v j = 1 y) or the more numerically stable log-app ratio, also called the log-likelihood ratio (LLR) ( ) P r(vj = 0 y) L(v j y) log P r(v j = 1 y) The natural logarithm is assumed for LLRs. 4. Single-parity-check (SPC) codes: (a) A single-parity-check (SPC) code C spc over GF (2) of length n is a binary (n, n 1) linear code. (b) Let u = (u 0, u 1,..., u n 2 ) be the message to be encoded. Then the codeword for u is v = (c, u 0, u 1,..., u n 2 ) where c = u 0 + u u n 2 (c) Every codeword in C spc has even weight. 21

22 5. The following is a graphical representation of an LDPC code as a concatenation of SPC and REP codes. Π represents an interleaver. 22

23 6. The REP (V N) decoder and SPC (CN) decoder: 23

24 For the computing of the extrinsic information L j i (L i j ), the V N j (CN i ) does not need the L i j (L j i ) from CN i (V N j ). 7. The V N and CN decoders work cooperatively and iteratively to estimate L(v j y) for j = 0, 1,, n According to the flooding schedule, all V Ns (CNs) process their inputs and pass extrinsic information up to their neighboring check nodes (variable nodes). 9. After a preset maximum number of repetitions (or iterations) of this V N/CN decoding round, or after some stopping criterion has been met, the decoder computes (estimate) LLRs L(v j y) from which decisions on the bits v j are made. 10. When the cycles are large, the estimates will be very accurate and the decoder will have near-optimal (MAP) performance. 11. The development of the SPA below relies on the following independence assumption. Clearly, this breaks down when the number of iterations exceeds half of the Tanner graph s girth. 24

25 Repetition code MAP decoder and APP processor 1. Consider a REP code in which the binary code symbol c {0, 1} is transmitted over a memoryless channel d times so that the d-vector r is received. 2. The MAP decoder computes L(c r) = log ( ) P r(c = 0 r) P r(c = 1 r) or, under an equally-likely assumption for the value of c, L(c r) is the LLR ( ) ( d 1 ) P r(r c = 0) l=0 L(c r) = log = log P r(r l c = 0) P r(r c = 1) d 1 l=0 P r(r l c = 1) = d 1 l=0 L(r l c) The MAP decision is ĉ = 0 if L(c r) 0 and ĉ = 1 otherwise. 25

26 3. Following a similar idea, in the context of LDPC decoding, the extrinsic information to be sent from V N j to CN i is L j i = L j + i N(j) {i} L i j where L j is the LLR value computed from the channel symbol y j ( ) P r(cj = 0 y j ) L j = L(c j y j ) = log P r(c j = 1 y j ) we call the V N an APP processor instead of a MAP decoder. At the last iteration, V N j produces a decision based on L total j = L j + i N(j) L i j 26

27 Single parity-check code MAP decoder and APP processor 1. Lemma 5.1: Consider a vector of d independent binary random variables a = [a 0, a 1,, a d 1 ] in which P r(a l = 1) = p (l) 1 and P r(a l = 0) = p (l) 0 Then the probability that a contains an even number of 1s is d 1 (1 2p (l) 1 ) l=0 and the probability that a contains an odd number of 1s is d 1 (1 2p (l) 1 ) l=0 27

28 Proof: 28

29 2. Consider the transmission of a length-d SPC codeword c over a memoryless channel whose output is r. The bits c l in the codeword c have a single constraint: there must be an even number of 1s in c. 3. The MAP rule for c 0 is ĉ 0 = arg max b {0,1} P r{c 0 = b r, SP C} P r{c 0 = 0 r, SP C} = P r{c 1, c 2,, c d 1 have an even no. of 1s r} = d 1 (1 2P r(c l = 1 r l )). l=1 1 2P r(c 0 = 1 r, SP C) = d 1 (1 2P r(c l = 1 r l )). (A) l=1 4. Lemma 5.2: For a generic binary random variable with probabilities p 1 and p 0 ( ( )) ( ) 1 1 2p 1 = tanh 2 log p0 1 = tanh 2 LLR p 1 29

30 Proof: 30

31 5. Using Lemma 5.2, (A) becomes tanh ( ) 1 2 L(c 0 r, SP C) = d 1 l=1 d 1 L(c 0 r, SP C) = 2 tanh 1 [ l=1 tanh( 1 2 L(c l r l )) tanh( 1 2 L(c l r l ))] The MAP decision is ĉ 0 = 0 if L(c 0 r, SP C) 0 and ĉ 0 = 1 otherwise. In the context of LDPC decoding, when the CNs function as APP processors, CN i compute the extrinsic information L i j = 2 tanh 1 ( j N(i) {j} tanh( 1 2 L j i)) and transmits it to V N j. Because the product is over the set N(i) {j}, the message has in effect been subtracted out to obtain the extrinsic information L i j. 31

32 The Gallager SPA decode : Initialization. The decoder is initialized by setting all VN messages L j i equal to L j = L(v j y j ) = log( P r(v j =0 y j ) P r (v j =1 y j )) for all j, i for which h ij = 1. Here, y j represents the channel value that was actually received. 1. BEC: y j {0, 1, e}, p P r (y j = e v j = b) is the erasure probability where b {0, 1}. 1 when y j = b P r (v j = b y j ) = 0 when y j = b c 1 2 when y j = e +, y j = 0 L(v j y j ) =, y j = 1 0, y j = e 32

33 2. BSC: y j {0, 1}. We define ε = P r(y j = b c v j = b) be the error probability L(v j y j ) = ( 1) y j log( 1 ε ). ε 3. BI-AWGNC: Let x j = ( 1) v j be the j-th transmitted binary value. The j-th received sample is y j = x j + n j, where n j N(0, σ 2 ). L(v j y j ) = 2y j σ 2 4. Rayleigh: y i = α j x j + n j, where {α j } are independent Rayleigh random variables with unity variance L(v j y j ) = 2α jy j σ 2 33

34 The Gallager sum-product algorithm: step 1. Initialization : For all j, initialize L j according to L j = L(v j y j ) = log( P r(v j = 0 y j ) P r (v j = 1 y j ) ) for appropriate channel model. Then, for all i, j for which h ij = 1, set L j i = L j. step 2. CN update: Compute outgoing CN message L i j for each CN using L i j = 2 tanh 1 [ j N(i) {j} and then transmit to the V Ns. tanh( 1 2 L j i)] step 3. V N update: Compute outgoing V N message L j i for each V N using L j i = L j + i N(j) {i} L i j and then transmit to the CNs. 34

35 step 4. LLR total: For j = 0, 1,, n 1, compute L total j = L j + i N(j) L i j step 5. Stopping criteria: For j = 0, 1,, n 1, set ˆv j = 1 if L total j < 0 0 else to obtain ˆv. If ˆvH T = 0 or the number of iterations equals the maximum limit, stop; else, go to Step 2. 35

36 The CN update equation can be simplified as follows. 1. L j i = α j i β j i, where α j i = sign(l j i ), β j i = L j i. 2.. tanh( 1 2 L i j) = j N(i) {j} α j i j N(i) {j} tanh( 1 2 β j i) L i j = j α j i 2 tanh 1 [ j tanh( 1 2 β j i)] = j α j i 2 tanh 1 log 1 log[ j tanh( 1 2 β j i)] = j α j i 2 tanh 1 log 1 j log(tanh( 1 2 β j i)) = α j i φ[ φ(β j i)] j N(i) {j} j N(i) {j} 36

37 where φ(x) = log[tanh( x 2 )] = log( ex +1 e x 1 ) Note φ 1 (x) = φ(x) when x > The φ(x) function: The function φ(x) may be implemented by use of a look-up table. However, a table-based decoder can suffer from an error-rate floor. Solution: (a) A piecewise linear approximation (b) Reduced-complexity SPA approximations. 37

38 Example: (3, 2) (3, 2) SPC product code on BEC 38