Codes on graphs and iterative decoding

Transcription

1 Codes on graphs and iterative decoding Bane Vasić Error Correction Coding Laboratory University of Arizona Funded by: National Science Foundation (NSF) Seagate Technology Defense Advanced Research Projects Agency (DARPA) Knowledge Enhanced Compressive Measurement (KECoM) International Disk Drive Equipment and Materials Association (IDEMA) Advanced Storage Technology Consortium (ASTC) European Commission (EC) FET-Open

2 Why you should listen to this lecture? Wireless networks, satellite communications, deep-space communications, power line communications are among applications where the low-density parity check (LDPC) codes are the standardized. Standards include: Digital video broadcast over satellite (DVB-S2 Standard) and over cable (DVB-C2 Standard), terrestrial television broadcasting (DVB-T2, DVB-T2-Lite Standards), GEO-Mobile Radio (GMR) satellite telephony (GMR- Standard), local and metropolitan area networks (LAN/MAN) (IEEE 802. (WiFi)), wireless personal area networks (WPAN) (IEEE c (60 GHz PHY)), wireless local and metropolitan area networks (WLAN/WMAN) (IEEE (Mobile WiMAX), near-earth and deep space communications (CCSDS), wire and power line communications ( ITU-T G.hn (G.9960)), utra-wide band technologies (WiMedia.5 UWB), magnetic hard disk drives, optical communications, flash memories.

3 Basics Outline Error correction codes, linear block codes, parity check matrices, code graphs Decoding using local information, iterative decoders Decoders as finite-state dynamical systems, basins of attraction and decoding failures Failures of iterative decoders Trapping sets Code design Combinatorial designs and codes Quasi-cyclic codes designed from group-theoretic transforms, Latin squares, difference families, finite geometries Many codes in literature are equivalent

4 Information transmission Channel

5 Information transmission signal 0 0 threshold time

6 Noisy memoryless channels x x 2 x 3 x 4 p( Channel y i x i ) y y 2 y 3 y 4 n,..., n,..., n j i P y y x x P y x i

7 Simple memoryless channels Binary symmetric channel (BSC) 0 a -a a 0 -a -e 0 Binary erasure channel (BEC) 0 e e E -e y Binary input additive white Gaussian noise (AWGN) channel, s 2 0 p(y 0) 0 p(y )

8 Channel capacity The channel capacity, C, bounds the information rate R (in bits per channel use) that can be achieved with arbitrarily small error probability. Note that for channels with binary input, C.

9 Channel capacity - examples a -e p(y 0) e a e -a -e p(y ) a y E (db)

10 Channel coding Channel coding idea: in addition to k information bits to be transmitted through the channel, transmit n-k redundant bits. A set of such n-tuples forms a channel code. The code rate is R = k n Noisy-channel coding theorem: It is possible to transmit information over this channel reliably if and only if R<C. Conversely, if R>C, an arbitrarily small probability of error is not achievable. Probability of error achieved by any such code is lower bounded by a strictly positive value. This value is larger for larger rates.

11 Thresholds for binary-input AWGN channel R=0.2 R=0.4 R=0.6 R= P b E b /N 0 (db)

12 Error correction coding m x y Encoder Channel Decoder ˆx ˆm Message Codeword Received word Code rate R = k m ( m,..., m k ) (,..., ) x x x n n y ( y,..., y n ) Decoder tries to find x ( or m ) from y so that the probability of bit/codeword error is minimal. In other words, decoder tries to find a codeword closest to y.

13 Maximum likelihood decoding

14 Bounded distance decoder The decoder circles a sphere of the predetermined radius around the received word and then searches for a codeword within this bounded distance. Two things can happen: Decoder finds a single codeword (success). Note that this codeword may be correct (correct decoding) or wrong (error) Does not find a single (does not find a codeword at all or finds multiple codewords) codeword (failure).

15 Error rate performance FER 0 - SNR 0-6 coded Shannon limit uncoded 0-5

16 Protecting information by coding all words of length n

17 Protecting information by coding all words of length n codewords

18 Minimum distance

19 Protecting information by coding code C

20 Linear block codes

21 Dimension of a linear block code {g,,g k } the basis for code C {h,h 2,,h n-k } the basis of C^ h v H T =0 g 2 g

22 Parity check h g 2 vh T =0 v g

23 Parity check h g 2 vh T =0 v g

24 Parity check h g 2 v vh T =0 g

25 Syndrome h y H T 0 y g 2 g

26 Linear constraints A binary linear (n,k) code C is a k-dimensional subspace of GF(2 n ). A codeword v satisfies n-k equations in n variables Example: H T v H = c : v + v + v + v = c : v + v + v + v = c : v + v + v + v = Binary operations a b a b a b a + b

27 Correcting one error Checks c : v + v + v + v = c : v + v + v + v = c : v + v + v + v = Variables c v c 2 v 4 v 2 v 7 v 6 v 5 v 3 c 3

28 Graphical model for a linear block code Checks c : v + v + v + v = c : v + v + v + v = c : v + v + v + v = v v 2 v 3 v 4 v 5 v 6 v 7 Variables c c 2 c 3

29 Linear block codes achieve capacity Channel coding theorem holds if a code is linear block code. Shannon also proved that if R<C then a typical (random) code has the probability of error decreasing exponentially with the code length. R If R<C, then The complexity of decoding of a random code is O(2 n ), thus e need codes with a nice structure. Algebraic codes such (BCH, Reed-Solomon, Algebraic Geometry) have nice structure, but cannot achieve capacity. 0 C

30 Definitions LDPC codes belong to the class of linear block codes which can be defined by sparse bipartite graphs. The Tanner graph of an LDPC code C is a bipartite graph with two sets of nodes: the set of variable nodes and the set of check nodes The check nodes (variable nodes resp.) connected to a variable node (check node resp.) are referred to as its neighbors. The set of neighbors of a node is denoted by The degree of a node is the number of its neighbors.

31 Definitions A vector is a codeword if and only if for each check node, the modulo two sum of its neighbors is zero. An regular LDPC code has a Tanner graph with variable nodes each of degree and check nodes each of degree. This code has length rate It should be noted that the Tanner graph is not uniquely defined by the code and when we say the Tanner graph of an LDPC code, we only mean one possible graphical representation.

32 An example of a regular n=25 =3, =5 code H

33 Iterative decoding

34 Crossword puzzles Iterate! Across: 4 Animal with long ears and a short tail. 0 Person who is in charge of a country. 2 In no place. 2 3 Down: Pointer, weapon fired from a bow. 6 Accept as true. 7 A place to shoot at; objective. Math. Seminar Tucson

35

36

37 Iterative decoders for BEC

38 Iterative decoding on BEC -e 0 0 e e E erased bit correct bit -e

39 Decoding simulation

40 BEC decoding simulation a check involving a single erased bit other check

41 BEC simulation - a check satisfied after correction

42 BEC simulation - 2

43 BEC simulation - 3

44 BEC simulation - 4

45 BEC simulation - 5

46 BEC simulation - 6 Success!

47 Another example BEC simulation -

48 Another example BEC simulation - 2

49 BEC simulation -final Stuck!

50 Decoding failures A BEC iterative decoder fails to converge to a codeword (correct or wrong) if at any iteration there is no check node connected to less than one erased variable node. A graph induced by such set of check nodes is called a stopping set.

51 Combinatorial definition of a stopping set Consider a set S of variable nodes. Let N(S) be a set of all checks nodes connected to S. If smallest outdegree of nodes in N(S) is two, then S is a stopping set. S N(S) Other channels such as BSC, AWGN do not have such combinatorial definition of a decoding failure.

52 Decoding by belief propagation

53 Decoders for channels with soft outputs In addition to the channel value, a measure of bit reliability is also provided m x y ˆx Encoder Channel Decoder ( x) xm ˆ, ˆ Bit log-likelihood ratio given y i. P( xi 0 yi) ( xi ) log P ( x y ) i i log log p( yi xi 0) P( xi 0) py ( i ) p( yi xi ) P( xi ) py ( ) i log p( yi xi 0) P( xi 0) log p( y x ) P( x ) i i i p( yi xi 0) P( xi 0) p( y x ) P( x ) i i i

54 Log-likelihood ratio Without prior knowledge on x i p( yi xi 0) i ( xi) log p ( y x ) For AWGN (y i = x i + n i, n i ~ N( 0, s 2 ) ) p( y x 0) y log -( y -) ( y ) For BSC with parameter a i i 2 2 i i 2 i i 2 p( yi xi ) 2s 2s i -a log if yi 0 a a log if yi -a i i

55 Soft outputs (x i, i ) Message-passing x i an estimate of the i th bit i - belief, reliability, likelihood, likelihood ratio Example: 5 x i ) x : xˆ : ( xˆ ) : Pr( x i = y i )

56 Soft decoding example x : xˆ : ( xˆ ) : ? -5 M min( -6, 2, -5 ) 2 S sign( -6) sign( 2) sign( -5) A S M A0 A0 A A2 A ? ? -5 A0 '

57 Side remark: some bits voted twice ? ? ? -5

58 Min-sum algorithm For each connected variable-check pair (v,c): Find product of reliability signs of all variables connected to check node c (excluding node v). Find the magnitude of the smallest reliability of variables connected to check node c (excluding node v). Combine sign and minimum reliability to update reliability of the variable node. Repeat until all parity checks are satisfied or maximum number of iterations is reached.

59 The min-sum update rule h x h x ( ) y f y ( ) y f x( x ) h 2 x h x 2 ( ) x x f( x) f y f y ( ) 2 2 y 2 n(x)\{f} n(f)\{x} ( x) ( y) f x yf yn ( f )\{ x} ( x) sgn( ( x)) min ( x) x f hx hx hn ( x)\{ f } hn ( x)\{ f } g ( x ) ( x ) ( x ) i i i h xi i hn ( x ) i

60 Derivation of the check update rule Given the log-likelihoods of (x j ) jm find the loglikelihood of y, L(y). 2 j m- m x x 2 x j x m- x m Pr{ y 0} Pr{# "" in x is even} Ly ( ) log log Pr{ y } Pr{# "" in x is odd} j Ly ( ) 2artanh tanh Ly ( ) jm 2 sgn( j) j jm y jm

61 Derivation of the check update rule Bernoulli trials: Pr{ x 0} q, Pr{ x } p j m m j m- j q p p q 0jm ( ) j m j m j m- j q - p - p q 0jm Pr{# "" in x is even} 2 q p q - p Pr{# "" in x is odd} 2 q p - q - p Generalization : m m Pr{ x 0} q, Pr{ x 0} p, 0 j m j j j j Pr{# "" in x is even} q j p j q j - p j q j - p j 2 2 m m jm jm jm Pr{# "" in x is odd} q j p j - q j - p j - q j - p j 2 2 jm jm jm

62 Derivation of the check update rule Pr{ y 0} Pr{# "" in x is even} Ly ( ) log log Pr{ y } Pr{# "" in x is odd} j e - j jm e e log j e - - j jme e log - jm jm e e e e j j j j - - j j j Ly ( ) 2artanh tanh jm 2 Ly ( ) log - jm jm e e e e /2 - /2 /2 - /2 /2 - /2 /2 - /2 j tanh jm 2 log j - tanh jm 2 j tanh jm 2 2 log 2 j - tanh jm 2 j 2artanh tanh jm 2 j j j j - e e - e e j j j j

63 Sum-product algorithm (Kschischang et. al.) h 2 h x h x 2 x h x ( ) n(x)\{f} ( ) x f x( x ) x f( x) y f y f ( ) n(f)\{x} y f y 2 2 y ( ) y 2 ( x) ( x) xf hx hn ( x)\{ f } ( x) f ( X ) ( y) f x yf ~{ x} hn ( f )\{ x} g ( x ) ( ) i i h x x i i hn ( x ) i

64 Iterative decoders for BSC

65 Decoding on graphs Two basic types of algorithms: Bit flipping If more checks are unsatisfied than satisfied, flip the bit Continue until all checks are satisfied Message passing A variable node sends his value to all neighboring checks A checks computes XOR of all incoming messages and sends this along the edges But it excludes the message on the edge the result is send along! 0 0 Variable takes a majority vote of incoming messages and sends this along If tie, sends its original value

66 Gallager A/B algorithm The Gallager A/B algorithms are hard decision decoding algorithms in which all the messages are binary. With a slight abuse of the notation, let denote the number of incoming messages to which are equal to. Associated with every decoding round and variable degree is a threshold. The Gallager B algorithm is defined as follows.

67 Gallager A/B algorithm The Gallager A algorithm is a special case of the Gallager B algorithm with for all. At the end of each iteration, a decision on the value of each variable node is made based on all the incoming messages and possibly the received value.

68 The update rule The sum-product algorithm The result of decoding after iterations, denoted by is determined by the sign of If otherwise

69 The min-sum algorithm In the limit of high SNR, when the absolute value of the messages is large, the sum-product becomes the minsum algorithm, where the message from the check to the bit looks like:

70 General iterative decoders An iterative decode is defined as a 4-tuple given by is a set the message values are confined to is the set of channel values

71 Finite alphabet iterative decoders Messages belong to a finite alphabet, channel values belong to a finite alphabet. The function used for update at a check node with degree is defined as The function is the update function used at a variable node with degree is defined as is a quantizatation, and is computed using a symmetric function

72 Different thresholds the quantizer and choice of the function lead to different decoders linear-threshold (LT) function decoders or non-linear-threshold (NLT) function decoders

73 Decoders as dynamical systems Let be the vector of messages along all edges in the Tanner graph in the -th iteration, and the received vector, then an iterative decoder on the Tanner graph can be seen as a dynamical system Such dynamical system may have a chaotic behavior When alphabets are finite, a decoder is a finite state machine, with a very large state space. The trajectory converge either to a fixed point or exhibits oscillations around attractor points in the state space. The attractor structure is defined by and.

74 Failures of iterative decoders (intuition)

75 Error floor FER FER 0 - SNR 0 - SNR coded Shannon limit 0-5 uncoded Shannon limit 0-5 Sphere packing bound uncoded

76 Locality of decoding

77 A motivating example Consider a six cycle in a 3-variable regular Tanner Graph. Assume the channel introduces three errors exactly on the variable nodes in the cycle. Also the assume that the neighborhood of the subgraph does not influence the messages propagated within the subgraph (condition to be explained later) Gallager A fails for such error pattern. By adding an extra bit in the message, the decoder can succeed.

78 Gallager A iteration 0 0 Initially wrong variable node Initially correct variable node Odd-degree check node Even-degree check node

79 Gallager A iteration

80 A trapping set illustration Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

81 A trapping set illustration Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

82 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

83 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

84 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

85 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

86 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

87 Concept of a trapping set (3,3) trapping set (5,3) trapping set (8,0) Trapping Set