Low-density parity-check codes

Transcription

1 Low-density parity-check codes From principles to practice Dr. Steve Weller School of Electrical Engineering and Computer Science The University of Newcastle, Callaghan, NSW 2308 Australia Work supported by the Australian Research Council (ARC) under Discovery Project Grant DP Page 1

2 The binary symmetric channel 1 ε 0 0 ε ε ε Until the late 1940 s communication engineers believed that reliable use of channel required either: increase in signal-to-noise ratio repeated transmission of message Binary symmetric channel (BSC) with crossover probability ε Like shouting for a beer in a crowded pub Page 2

3 Channel coding for reliable communication Shannon s 1948 paper A mathematical theory of communication revolutionised this view: either communication can be made reliable, or it cannot k message Channel code defines the mapping from messages to codewords use channel n times but use these n transmissions to send just k n bits of information n code rate R k n (n,k) code codeword Page 3

4 Channel coding theorem for BSC Define BSC capacity CBSC ( ε ) = 1 h( ε ) ( ) log ( 1 ) log ( 1 ) h x x x x x 2 2 With code rate R, arbitrarily reliable use of channel possible if and only if R < C BSC Example: with ε = 0.11 have C BSC = 0.5 for rate-1/2 code, expect to tolerate at most 11% channel errors Page 4

5 In this overview Shannon s 1948 Noisy Channel Coding theorem is nonbut indicates that capacity-approaching codes: Use long codewords Use random codewords Exhibit threshold behaviour w.r.t noise In this overview we look at low-density parity-check (LDPC) codes capable of getting extremely close to capacity on BSC, AWGN (+others) LDPC codes: Use long codewords Use codewords specified by random-ish parity-check matrix Exhibit threshold behaviour w.r.t. noise Page 5

6 LDPC codes for BSC Start by designing an LDPC code for the BSC Will introduce most of the key ingredients: Two representations of the code: parity-check matrix Tanner graph Parity-check matrix randomly selected from ensemble Message passing decoder on Tanner graph Noise thresholds Won t produce a code close to capacity limited to crossover probabilities ε Won t use soft information from channel for BSC, each received bit is equally unreliable Message-passing decoder limited to discrete alphabet {0,1} Page 6

7 Low-density parity-check codes c.1962 LDPC codes introduced by PhD student Robert Gallager in 1960 thesis (MIT) then almost totally neglected for 35 years Total of 14 citations of paper or monograph to /year 380 citations /week Page 7

8 Codes from parity-check equations Start with a small example: (10,5) linear block code with codewords x 1 x 2 x 3 x 10 Each length n = 10 codeword satisfies all five of the following constraints: x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 1 x 2 x 3 x 4 x 5 x 9 = 0 x 1 x 2 x 4 x 7 x 9 x 10 = 0 x 3 x 4 x 6 x 7 x 8 x 10 = 0 = addition modulo-2 = 2-input XOR gate Easy to confirm there are 32 = 2 5 length-10 words satisfying all constraints, so this is indeed a (10,5) code Page 8

9 Parity-check matrices Parity-check matrices represent parity-check equations in the obvious way: Each row of matrix represents one equation H = x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 1 x 2 x 3 x 4 x 5 x 9 = 0 x 1 x 2 x 4 x 7 x 9 x 10 = 0 x 3 x 4 x 6 x 7 x 8 x 10 = 0 Page 9

10 Parity-check matrices Parity-check matrices represent parity-check equations in the obvious way: Each row of matrix represents one equation Each column of matrix indicates the parity-check equations involving the corresponding codeword bit H = x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 1 x 2 x 3 x 4 x 5 x 9 = 0 x 1 x 2 x 4 x 7 x 9 x 10 = 0 x 3 x 4 x 6 x 7 x 8 x 10 = 0 Page 10

11 (d v,d c )-regular LDPC codes The LDPC codes considered by Gallager were (d v,d c )-regular codes. Parity-check matrix H has: constant column weight d v H = d v =3 Page 11

12 (d v,d c )-regular LDPC codes The LDPC codes considered by Gallager were (d v,d c )-regular codes. Parity-check matrix H has: constant column weight d v constant row weight d c H = d c =6 (3,6)-regular code d v and d c are typically fixed, independent of dimensions of H Thus for example a (3,6)-regular (1000,500) code has just 0.6% of entries of H being non-zero Sparsity of H gives rise to low-density in LDPC name Page 12

13 Tanner graphs variable nodes x 1 x 2 x 3 check nodes x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 4 x 5 x 6 x 7 x 8 x 9 x 10 Page 13

14 Tanner graphs variable nodes x 1 x 2 x 3 x 4 check nodes x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 5 x 6 x 7 x 8 x 9 x 10 Page 14

15 Tanner graphs variable nodes x 1 check nodes x 2 x 3 x 4 x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 5 x 6 x 1 x 2 x 3 x 4 x 5 x 9 = 0 x 7 x 8 x 9 x 10 Page 15

16 Tanner graphs variable nodes x 1 check nodes x 2 x 3 x 4 x 1 x 5 x 6 x 8 x 9 x 10 = 0 x 2 x 3 x 5 x 6 x 7 x 8 = 0 x 5 x 6 x 7 x 8 x 9 x 1 x 2 x 3 x 4 x 5 x 9 = 0 x 1 x 2 x 4 x 7 x 9 x 10 = 0 x 3 x 4 x 6 x 7 x 8 x 10 = 0 x 10 Page 16

17 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 17

18 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 18

19 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 19

20 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 20

21 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 21

22 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 22

23 LDPC codes from ensembles LDPC codes are typically constructed by selecting a parity-check matrix at random from a suitably defined ensemble e.g. ensemble of (3,6)-regular codes But how to select a bi-regular graph uniformly at random? Following construction is easy to implement extends readily to irregular codes (to follow) Page 23

24 LDPC codes from ensembles In practice, purely random graph realisation often modified slightly: forbid repeated edges between nodes avoid short cycles in Tanner graph, especially cycles of length-4 Page 24

25 Gallager s hard-decision decoding algorithm initialisation received bit is broadcast 1 check-to-variable x 1 =x 5 x 6 x 8 x 9 x 10 variable-to-check 0 If incoming messages on other edges are unanimous, send this message, otherwise send received bit Page 25

26 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.02 after 1 round of bit-flipping Page 26

27 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.02 after 2 rounds of bit-flipping Page 27

28 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.02 after 3 rounds of bit-flipping Page 28

29 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.02 after 4 rounds of bit-flipping Page 29

30 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 1 round of bit-flipping Page 30

31 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 2 rounds of bit-flipping Page 31

32 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 3 rounds of bit-flipping Page 32

33 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 4 rounds of bit-flipping Page 33

34 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 5 rounds of bit-flipping Page 34

35 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 6 rounds of bit-flipping Page 35

36 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 7 rounds of bit-flipping Page 36

37 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 8 rounds of bit-flipping Page 37

38 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = 0.03 after 9 rounds of bit-flipping Page 38

39 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 1 round of bit-flipping Page 39

40 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 2 rounds of bit-flipping Page 40

41 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 3 rounds of bit-flipping Page 41

42 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 4 rounds of bit-flipping Page 42

43 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 5 rounds of bit-flipping Page 43

44 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 6 rounds of bit-flipping Page 44

45 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 7 rounds of bit-flipping Page 45

46 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 8 rounds of bit-flipping Page 46

47 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 9 rounds of bit-flipping Page 47

48 (3,6)-regular code, rate-1/2, length = 1000 hard-decision decoding received bits ε = after 10 rounds of bit-flipping Page 48

49 Preparing to compute the noise threshold Simulation clearly suggests existence of noise threshold Simplicity of hard-decision decoding algorithm makes it possible * to track expected fraction of errors p i+1 remaining after k+1 rounds p 0 = ε * conditions apply: read the fine print shortly! ( p ) 1 dv 1 d 1 dv 1 ( p ) dc c k k pk + 1 = p0 p0 + ( 1 p0) 2 2 Page 49

50 (3,6)-regular code threshold on BSC Page 50

51 (3,6)-regular code threshold on BSC Page 51

52 (3,6)-regular code threshold on BSC Page 52

53 (3,6)-regular code threshold on BSC Page 53

54 Definition of noise threshold * ε = sup p0 such that lim pk = 0 p [0,1] k 0 For BSC, threshold for (3,6)-regular codes is * ε = Threshold upper bounds tolerable noise level for harddecision decoder on BSC Randomly chosen member of ensemble will achieve arbitrarily small target error probability, with probability that approaches 1 exponentially fast in length of code Page 54

55 The fine print Ability to readily track average fraction of decoded bits in error assumes neighbourhood of each message node is a tree for arbitrarily many rounds which it isn t Page 55

56 What was known in 1963? From Gallager s 1962 paper + monograph: Hard-decision, iterative message-passing decoder although Tanner graphs came much later in 1981 Message-passing decoder exchanging soft (=probabilistic) messages, now known as the sum-product algorithm, or belief propagation decoder Calculation of thresholds for regular codes on BSC Specific constructions of regular LDPC codes Why didn t Gallager + others discover just how good regular LDPC codes are? Page 56

57 Reflections on Gallager s work In 1965, IBM 7090 on which Gallager performed his numerical experiments was typically rented from IBM at about $A600,000 per month in today s money, allowing for inflation! Page 57

58 LDPC codes for AWGN channels LDPC codes described thus far are good, but not exceptional Additional ingredients are needed: Soft information from channel Sum-product, or belief propagation, decoder Irregular codes: degrees of variable- and check-nodes no longer constant Key to irregular codes is density evolution 2002 IEEE Information Theory Society paper award jointly to two groups of authors for work on density evolution and irregular LDPC codes: Richardson & Urbanke Luby, Mitzenmacher, Shokrollahi, and Spielman Independent rediscovery (re-invention!) of Gallager s work by MacKay (& Neal) in 1996 Page 58

59 Soft messages On additive white Gaussian noise (AWGN) channel, transmit x {-1,+1}, and receive y = x+n, n ~ N(0,σ 2 ) Soft value read from channel contains important information about x condense pair of conditional probabilities into a single log-likelihood ratio (LLR) Px ( = + 1 y) 2 log = Px ( = 1 y ) σ 2 y node v u node c node v v node c u P( x=+ 1 t) = log P ( x= 1 t ) v P( x= + 1 s) = log P ( x= 1 s ) t denotes LLR messages from channel, and from all edges into v, other than from node c s denotes LLR messages sent from all edges into node node c, other than from node v Page 59

60 Sum-product algorithm initialisation u 0 u 0 u 0 u Px ( = + 1 y) 2 = log = Px ( = 1 y ) σ 0 2 y u 0 check-to-variable u v 1 v d c-1 u d c 1 v 1 j = 2tanh tanh j= 1 2 variable-to-check 1 u 0 v u d v-1 v d v = u i= 0 i u 1 Page 60

61 Density evolution For BSC: Computed threshold by tracking expected number of bits in error after each round of hard-decision decoding This worked because messages were discrete For AWGN: Messages are LLRs, and hence are continuous To track expected behaviour of iterative decoder, need to track evolution of LLR probability densities through sum-product algorithm This is density evolution Page 61

62 Density evolution: (3,6)-regular code on AWGN Page 62

63 Density evolution: (3,6)-regular code on AWGN Page 63

64 Density evolution: (3,6)-regular code on AWGN Page 64

65 Density evolution: (3,6)-regular code on AWGN Page 65

66 Density evolution: (3,6)-regular code on AWGN Page 66

67 Density evolution: (3,6)-regular code on AWGN Page 67

68 Density evolution: (3,6)-regular code on AWGN Page 68

69 Density evolution: (3,6)-regular code on AWGN Page 69

70 Density evolution: (3,6)-regular code on AWGN Page 70

71 Density evolution: (3,6)-regular code on AWGN Page 71

72 Threshold of (3,6)-regular code on AWGN channel Page 72

73 Threshold of (3,6)-regular code on AWGN channel estimate threshold as E b /N db Page 73

74 Irregular weight distributions Have restricted attention to parity-check matrices from ensembles with column weight = d v and row weight = d c Much to be gained from allowing irregular distributions of row and column weights d v i 1 ( x) ix i 1 ( x) ix λ = λ i= 2 d c ρ = ρ i= 2 λ i = fraction of edges in Tanner graph connected to variable nodes of degree i ρ i = fraction of edges in Tanner graph connected to check nodes of degree i Page 74

75 Irregular weight distributions Example ( x) x x x ( x) x x λ = + + ρ = + Variables nodes of degrees 2, 3 and % edges connected to variable nodes of degree-2 Check nodes of degrees 5 and % of edges connected to check nodes of degree-6 Density evolution used to compute thresholds of irregular code ensembles Linear programming and (confusingly named!) differential evolution used to obtain distributions with large thresholds Page 75

76 Threshold of irregular code on AWGN channel Page 76

77 Threshold of irregular code on AWGN channel estimate threshold as E b /N db compare with threshold of 1.12 db for (3,6)-regular code Page 77

78 World champion code Weight distribution computed by Chung et al. in 2001: ( x) x x x x x λ = x x x x x x x x x x x ( x) 0.5x 0.5x ρ = + Shannon limit E b /N 0 = db at rate-1/2 With density evolution, compute threshold as E b /N 0 * = db Threshold is just db from Shannon limit! Page 78

79 BER performance results Compare bit-error rate (BER) performance of several codes All codes are rate-1/2 Comparisons intended to be illustrative rather than fair Block lengths and decoder complexities differ markedly Page 79

80 BER performance results Page 80

81 BER performance results constraint length 7 convolutional code soft-decision Viterbi decoding Page 81

82 BER performance results (3,6)-regular LDPC code block length 10 6?? iterations of sum-product decoder Page 82

83 BER performance results original turbo code of Berrou-Glavieux- Thitimajshima (ICC 93) block length 65, iterations Page 83

84 BER performance results irregular LDPC (Chung) block length 10 7 max iterations of sum-product decoder Page 84

85 Challenges to LDPC design & implementation Irregular codes designed for very long block lengths Better design methods needed for codes of length ~1000 Density evolution algorithm is numerically intensive and acutely sensitive to selection of tuning parameters and uses impractically large number of decoder iterations Gaussian approximations trade off accuracy for speed Trade-off between steepness of waterfall and depth of error floor How to design powerful irregular codes with low error floors? Efficient encoding of irregular codes is feasible, but awkward Applications typically use regular codes Silicon for highly irregular codes is problematic ASIC designers want regular structures to ease automatic placement & routing Page 85

86 Practical implementations Agere Blanksby & Howland s first proof-of-concept LDPC ASIC (2001) Flarion Technologies (Richardson) Integrated into Flash-OFDM broadband wireless (WiMAX competitor) Digital Fountain (Luby & Shokrollahi) Application of LDPC codes to packet erasure channel = Internet Lucent Technologies LDPC for optical networking;10 Gb/s, rate R = 0.93, target BER = Consultative C tee for Space Data Systems (CCSDS) Application to deep-space comms (JPL and NASA) DVB satellite communications LDPC code recently adopted as part of standard Several IEEE standards working groups: IEEE WirelessMAN (WiMAX) IEEE n WLAN (optional in both WWiSE and TGnSync proposals) IEEE 802.3an 10GBT Page 86

87 Closing remarks Titles of three AusCTW talks: MIMO systems: From theory to practice Today - LDPC codes: From principles to practice Friday - Wireless communications: Taking theory to practice The difference between theory and practice in theory is much less than the difference between theory and practice in practice. But at least in communications the difference is getting smaller! Page 87