Codes on graphs and iterative decoding

Transcription

1 Codes on graphs and iterative decoding Bane Vasić Error Correction Coding Laboratory University of Arizona

2 Prelude

3 Information transmission Channel

4 Information transmission signal 0 0 threshold time

5 Noisy memoryless channels x x 2 x 3 x 4 p( Channel y i x i ) y y 2 y 3 y 4 p y,..., y x,..., x p y x n n j i n i

6 Simple memoryless channels Binary symmetric channel (BSC) Binary erasure channel (BEC) E y Binary input additive white Gaussian noise (AWGN) channel, 2 p(y 0) p(y )

7 Channel capacity - BSC C ,

8 Channel capacity - BEC E 0.6 C

9 Channel capacity - BAWGN y p(y 0) p(y ) C EE b /N b =N 0 (db) 0 (db)

10 Error correction coding m x y Encoder Channel Decoder ˆx ˆm Message Codeword Received word m ( m,..., m ) k x ( x,..., x ) n y ( y,..., y ) n k Code rate R n The decoder tries to find x ( or m ) from y so that the probability of bit/codeword error is minimal. In other words, decoder tries to find a codeword closest to y.

11 Error rate performance FER 0 - SNR 0-6 coded Shannon limit 0-5 uncoded

12 Maximum likelihood decoding

13 Protecting information by coding all words of length n

14 Protecting information by coding all words of length n codewords

15 Minimum distance

16 Protecting information by coding code C

17 Linear block codes

18 Dimension of a linear block code {g,,g k } the basis for code C {h,h 2,,h n-k } the basis of C h g 2 g

19 Encoding Generator matrix an k x n matrix of rank k a b a b a b a b

20 Linear block codes as subspaces Given a GF(2) (ground field), we define the vector space - the n-tuple v=( v,v 2, v n ) of elements from the ground filed is a type of vector. Elias and Golay: A binary linear (n,k) code C is a k - dimensional subspace of a vector space Galois Field, GF(2).

21 Parity check h g 2 x h T =0 x g

22 Parity check h g 2 x h T =0 x g

23 Parity check h g 2 x x h T =0 g

24 Syndrome h y h T 0 y g 2 g

25 Let x be a codeword Dual code C parity check matrix A received vector which is not a codeword results in a nonzero syndrome.

26 Linear constraints A codeword x satisfies v H n-k equations in n variables Example: T 0 H c : x x x x c : x x x x c : x x x x =

27 Side observations Since for any codeword x. and since it follows H can be found from G. For any a,b {0,} The parity check matrix can be modified by adding linear combinations of its rows. The ranks of any such new parity matrix is still n-k.

28 LDPC code basics

29 Applications of LDPC codes Wireless networks, satellite communications, deep-space communications, power line communications are among applications where the low-density parity check (LDPC) codes are the standardized. Standards include: Digital video broadcast over satellite (DVB-S2 Standard) and over cable (DVB-C2 Standard), terrestrial television broadcasting (DVB-T2, DVB-T2-Lite Standards), GEO-Mobile Radio (GMR) satellite telephony (GMR- Standard), local and metropolitan area networks (LAN/MAN) (IEEE 802. (WiFi)), wireless personal area networks (WPAN) (IEEE c (60 GHz PHY)), wireless local and metropolitan area networks (WLAN/WMAN) (IEEE (Mobile WiMAX), near-earth and deep space communications (CCSDS), wire and power line communications ( ITU-T G.hn (G.9960)), utra-wide band technologies (WiMedia.5 UWB), magnetic hard disk drives, optical communications, flash memories.

30 Basics Outline Error correction codes, linear block codes, parity check matrices, code graphs Decoding using local information, iterative decoders Decoders as finite-state dynamical systems, basins of attraction and decoding failures Failures of iterative decoders Correcting number of errors linear in code length Finite length analysis Trapping sets Code design Combinatorial designs and codes Quasi-cyclic codes designed from group-theoretic transforms, Latin squares, difference families, finite geometries

31 Graphical model for a linear block code c : v v v v c : v v v v Checks c : v v v v = v v 2 v 3 v 4 v 5 v 6 v 7 Variables c c 2 c 3

32 Definitions LDPC codes belong to the class of linear block codes which can be defined by sparse bipartite graphs. The Tanner graph of an LDPC code is a bipartite graph with two sets of nodes: the set of variable nodes and the set of check nodes V C

33 Definitions The check nodes (variable nodes resp.) connected to a variable node (check node resp.) are referred to as its neighbors. The set of neighbors of a node is denoted by The degree of a node is the number of its neighbors.

34 Definitions A vector is a codeword if and only if for each check node, the modulo two sum of its neighbors is zero. An regular LDPC code has a Tanner graph with variable nodes each of degree and check nodes each of degree. This code has length rate The Tanner graph is not uniquely defined by the code and when we say the Tanner graph of an LDPC code, we only mean one possible graphical representation.

35 An example of a regular n=25 =3, =5 code H

36 Iterative decoding

37

38

39 Message Passing Example:

40 Message Passing Example:

41 2

42 3

43 4

44 5

45 6

46 7

47 8

48 9 Done!

49 An unresolvable configuration Stucked!

50 Iterative decoders for BEC

51 Iterative decoding on BEC E erased bit correct bit

52 Decoding simulation

53 BEC decoding simulation a check involving a single erased bit other check

54 BEC simulation - a check satisfied after correction

55 BEC simulation - 2

56 BEC simulation - 3

57 BEC simulation - 4

58 BEC simulation - 5

59 BEC simulation - 6 Success!

60 Another example BEC simulation -

61 Another example BEC simulation - 2

62 BEC simulation -final Stuck!

63 Decoding failures A BEC iterative decoder fails to converge to a codeword (correct or wrong) if at any iteration there is no check node connected to less than one erased variable node. A graph induced by such set of check nodes is called a stopping set.

64 Combinatorial definition of a stopping set Consider a set S of variable nodes. Let N(S) be a set of all checks nodes connected to S. If smallest outdegree of nodes in N(S) is two, then S is a stopping set. S N(S) Other channels such as BSC, AWGN do not have such combinatorial definition of a decoding failure.

65 Iterative decoders for BSC

66 Decoding on graphs on BSC Two basic types of algorithms: Bit flipping Message passing

67 Bit flipping If more checks are unsatisfied than satisfied, flip the bit. Continue until all checks are satisfied

68 Steps: Message passing A variable node sends his value to all neighboring checks. A check computes XOR of all incoming messages and sends this along the edges, but it excludes the message on the edge the result is send along! Variable takes a majority vote of incoming messages and sends this along, if tie, sends its original value h ( ) x h x ( y ) y f y ( x ) f x h 2 ( ) x h2 x x ( x ) x f f ( y ) y2 f 2 y 2 n(x)\{f} n(f)\{x}

69 Gallager A/B algorithm The Gallager A/B algorithms are hard decision decoding algorithms in which all the messages are binary. With a slight abuse of the notation, let denote the number of incoming messages to which are equal to. Associated with every decoding round and variable degree is a threshold. The Gallager B algorithm is defined as follows.

70 Gallager A/B algorithm The Gallager A algorithm is a special case of the Gallager B algorithm with for all. At the end of each iteration, a decision on the value of each variable node is made based on all the incoming messages and possibly the received value.

71 General iterative decoders An iterative decode is defined as a 4-tuple given by is a set the message values are confined to is the set of channel values The function used for update at a check node with degree. The function is the update function used at a variable node with degree.

72 Decoders as dynamical systems Let be the vector of messages along all edges in the Tanner graph in the -th iteration, and the received vector, then an iterative decoder on the Tanner graph can be seen as a dynamical system Such dynamical system may have a chaotic behavior When alphabets are finite, a decoder is a finite state machine, with a very large state space. The trajectory converge either to a fixed point or exhibits oscillations around attractor points in the state space. The attractor structure is defined by and.

73 Attractors of iterative decoders adapted using and

74 Syndrome Codeword Bit flipping decoder Trajectory examples

75 Trajectory types Fixed point Cyclic Cyclic with a large period

76 An example of a trajectory

77 Failures of iterative decoders

78 Error floor FER FER 0 - SNR 0 - SNR coded Shannon limit 0-5 uncoded Shannon limit 0-5 Sphere packing bound uncoded

79 Locality of decoding

80 A motivating example Consider a six cycle in a 3-variable regular Tanner Graph. Assume the channel introduces three errors exactly on the variable nodes in the cycle. Also the assume that the neighborhood of the subgraph does not influence the messages propagated within the subgraph (condition to be explained later) Gallager A fails for such error pattern. By adding an extra bit in the message, the decoder can succeed.

81 Gallager A iteration 0 0 Initially wrong variable node Initially correct variable node Odd-degree check node Even-degree check node

82 Gallager A iteration

83 A trapping set illustration Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

84 A trapping set illustration Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

85 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

86 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

87 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

88 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

89 Oscillations in the decoder Corrupt variable Correct variable Variable decoded correctly Variable decoded wrongly

90 Concept of a trapping set (3,3) trapping set (5,3) trapping set (8,0) Trapping Set

91 Some ways to construct LDPC codes

92 y LDPC codes - combinatorial designs Affine partial geometry L { ( x, y ) : 0 x k, 0 y m } m - a prime Blocks: the lines starting at points (0,a) with slopes s (0 a,s m-) m each point incident with exactly m blocks m 2 blocks Example: k=3, m= s=0 s= s=2 s=3 s= x k

93 Integer lattice codes s=0 s= s=2 s=3 s= H

94 Affine and projective planes-example Affine Plane Projective Plane 2 3 4

95 Cyclic difference families We can think of the actions of the group V as a partitioning B into classes or orbits. Example: (3,3,) CDF, Z 3 Base blocks B ={0,,4} and B 2 ={0,2,7} B orbits B 2 orbits b +g b 2 +g b 3 +g b 2 +g b 22 +g b 23 +g H

96 Protograph based codes A protograph is a small Tanner graph. Example (Thorpe): V = 4 variable nodes and C = 3 check nodes, connected by E = 8 edges. 2 3 In this case Tanner graph of an (n = 4, k = ) LDPC code (in this case, a repetition code). Double edges are allowed

97 Protograph codes 0 0 H 0 0 A B C A A A B A A 2 A 3 B B 2 B 3 C C 2 C 3 H B B C C C A A 2 A 3 B B 2 B 3 C C 2 C 3

98 Parity check masking Start from a quasi-cyclic code and force some blocks to be zeros (in the Tanner graph, disconnect groups of checks and variables) H H

99 Decoding by belief propagation

100 Crossword puzzles Iterate! Across: 4 Animal with long ears and a short tail. 0 Person who is in charge of a country. 2 In no place. 2 3 Down: Pointer, weapon fired from a bow. 6 Accept as true. 7 A place to shoot at; objective. Math. Seminar Tucson

101 Decoders for channels with soft outputs In addition to the channel value, a measure of bit reliability is also provided ˆx m xˆ, mˆ x y Encoder Channel Decoder Bit log-likelihood ratio given y i. ( x ) ( x ) lo g i P ( x 0 y ) i P ( x y ) i i i lo g lo g p ( y x 0 ) P ( x 0 ) i i i p( y ) p ( y x ) P ( x ) i i i i p( y ) i lo g p ( y i x i 0 ) lo g P ( x i 0 ) p ( y x ) P ( x ) i i i p ( y x 0 ) P ( x 0 ) i i i p ( y x ) P ( x ) i i i

102 Log-likelihood ratio Without prior knowledge on x i i ( x ) lo g i p ( y x 0 ) i p ( y x ) i i i For AWGN (y i = x i + n i, n i ~ N( 0, 2 ) ) p ( y x 0 ) y lo g ( y ) ( y ) p ( y x ) 2 2 i i 2 2 i i 2 i i 2 i i For BSC with parameter lo g if y 0 i i lo g if y i

103 Soft outputs (x i, i ) Message-passing x i an estimate of the i th bit i- belief, reliability, likelihood, likelihood ratio Example: 5 x i ) x : xˆ : ( xˆ ) : Pr( x i = y i )

104 Soft decoding example x : xˆ : ( xˆ ) : ? -5 M m in ( 6, 2, 5 ) 2 S sig n ( 6 ) sig n ( 2 ) sig n ( 5 ) A S M ? A A A A A ? -5 A 0 '

105 Side remark: some bits voted twice ? ? ? -5

106 The min-sum update rule h ( ) x h x ( y ) y f y ( x ) f x h 2 ( ) x h2 x x ( x ) x f f ( y ) y2 f 2 y 2 n(x)\{f} n(f)\{x} x f x h f h n ( x ) \{ f } ( x ) sg n ( ) m in f x y f y f y n ( f ) \{ x } y n ( f ) \{ x } g ( x ) ( x ) i i i h x h n ( x ) i i

107 Derivation of the check update rule Given the log-likelihoods of (x j ) j m find the loglikelihood of y, L(y). 2 j m- m x x 2 x j x m- x m y L( y) P r{ y lo g 0} P r{ # " " in x is e v e n} lo g P r{ y } P r{ # " " in x is o d d } L( y) 2 a rta n h ta n L h ( y) sg n ( ) 2 j j j m j j m j m

108 Derivation of the check update rule B e rn o u lli tria ls: P r{ x 0} q, P r{ x } p m m j m j q p p q 0 j m q p ( ) p q 0 j m j m j m j m j j P r{ # " " in x is e v e n} q p m q p P r{ # " " in x is o d d } 2 q p m q p 2 m m G e n e r a liz a tio n : P r{ x 0} q, P r{ x 0} p, 0 j m j j j j P r{ # " " in x is e v e n} q p q p q p 2 2 j j j j j j j m j m j m P r{ # " " in x is o d d } q p q p q p 2 2 j j j j j j j m j m j m

109 Derivation of the check update rule P r{ 0} P r{ # " " } ( ) lo g lo g P r{ } P r{ # " " } lo g lo g j j j j j j j j j j j m j m j m j m y in x is e v e n L y y in x is o d d e e e e e e e e e e / 2 / 2 / 2 / 2 / 2 / 2 / 2 / 2 ( ) lo g ta n h 2 lo g ta n h 2 ta n h 2 2 lo g 2 ta n h 2 2 a rta n h ta n h 2 j j j j j j j j j m j m j j m j j m j j m j j m j j m e e e e L y e e e e ( ) 2 a rta n h ta n h 2 j j m L y

110 Min-sum approximation (x) = -log tanh(x/2) = log( (e x +)/(e x -) )= - (x) i m in m in i f i f i f i i'

111 Sum-product algorithm (Kschischang et. al.) h ( ) x h x ( y ) y f y ( x) ( x) x f h x h n ( x ) \{ f } h 2 ( ) x h2 x x ( x ) f x ( x ) x f f ( y ) y2 f 2 y 2 ( x ) f ( X ) ( y ) f x y f ~ { x } h n ( f ) \{ x } n(x)\{f} n(f)\{x} g ( x ) ( x ) i i h x i h n ( x ) i i

112 The update rule The sum-product algorithm The result of decoding after iterations, denoted by is determined by the sign of If otherwise

113 The min-sum algorithm In the limit of high SNR, when the absolute value of the messages is large, the sum-product becomes the minsum algorithm, where the message from the check to the bit looks like: