5. Density evolution. Density evolution 5-1

Size: px

Start display at page:

Download "5. Density evolution. Density evolution 5-1"

Corey Bailey
5 years ago
Views:

1 5. Density evolution Density evolution 5-1

2 Probabilistic analysis of message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ; F ; E) and (x ) = 1 Z Y a2f Y i2v i(x i) sum-product algorithm and max-product algorithms are instances of message-passing algorithms discrete xi 2 X two sets of messages fi!a (x i )g and f a!i (x i )g update: (t+1) i!a (t) = F i!a (f (t) b!i : b n ag) a!i = G a!i(f (t) j!a : j n ig) Density evolution 5-2

3 Density evolution 5-3 assumptions for probabilistic analysis a random graph is a graph G = (V ; F ; E) where E is drawn randomly from a set of possible graphs e.g., Erdös-Renyi graph, random regular graph asymptotic analysis: in the limit n! 1 density evolution is used in analyzing channel codes analyzing solution space of XORSAT analyzing a message-passing algorithm for crowdsourcing analyzing belief propagation for community detection etc.

4 Density evolution 5-4 Example: channel coding sending messages through a noisy channel x Channel y channel is defined by P Y jx (yjx ) Binary Erasure Channel (BEC) input x i 2 f; 1g, output y i 2 f; 1; g goal: estimate bx 1 : : : : ; bx n given y 1 ; : : : ; y n performance metric: average bit error probability P error 1 n nx i=1 P(x i 6= bx i )

5 Message length vs. block length no coding: (11) ) (1 ) message k = 5 bits, block length n = 5 ) rate of this code r k =n = 1, delay is one Perror = =2 repetition code: (111111) ) ( ) k = 5, n = 15 rate r = 1=3 and P error = 3 =2, delay is 3 in general, Perror = 1=r =2 > (unless rate is zero) information theory capacity of a BEC is 1 there exists a code such that lim n!1 P error = with rate r < 1 using the BEC n times, one can reliably send k = (1 )n bits of messages Density evolution 5-5

6 Modern coding theory modern codes = iterative decoding (belief propagation) Turbo code Low-Density Parity Check (LDPC) code Polar code etc. LDPC code is defined by a factor graph model variable nodes factor nodes x i 2 f; 1g x1 a x2 a(x i ; x j ; x k ) = I(x i x j x k = ) x3 b x4 block length n = 4 number of factors m = 2 allowed messages = f; 111; 11; 111g message size k log2 (# of allowed messages) = 2 (k = n m) rate r k =n = 1=2 received y = ( 1 ), then bx = (111) received y = ( ), then? Density evolution 5-6

7 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

8 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

9 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

10 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

11 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

12 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

13 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

14 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

15 Density evolution 5-7 Peeling decoder is equivalent to sum-product for BEC without loss of generality, suppose all s sent

16 Density evolution 5-8 Modern coding theory decoding using belief propagation y(x ) = 1 Z Y Y P Y jx (y i jx i ) I( = ) i2v a2f use (parallel) sum-product algorithm to find (x i ) and let bx i = arg max (x i ) minimizes bit error rate

17 Density evolution 5-9 Decoding by sum-product algorithm Directly applying parallel sum-product algorithm (t+1) i!a (x i) = P(y i jx i ) Q b2@infag (t) b!i (x i) (t+1) a!i = P Qj 2@anfig j!a(x j )I( = ) Notice that all ; s can take only one of the following three values: (" # " # " #) 1 1=2 i!a(x i ) 2 ; ; 1 1=2 hence, we will map these vectors to symbols f; 1; g because (proof by induction) initially, () a!i (x i) = 1=2 ; (1) 1=2 i!a (x i) = 8 >< >: 1 1 1=2 1=2 if y i = if y i = 1 if y i =

18 Density evolution 5-1 recursively, assuming the input messages up to t are one of the three types, 8 1 if all other bits are determined and add up to (t+1) a!i (x i ) = (t+1) i!a (x i) = >< >: 8 >< >: 1 1=2 1= =2 1=2 if all other bits are determined and add up to 1 if there is at least one bit that is not determined 1 if at least one of the input message is if at least one of the input message is 1=2 if all input messages are 1=2 1 consequently, the messages only take those three values we will denote those three types of messages as ; 1, and, meaning determined to be or 1, or not determined.

19 Density evolution 5-11 (simplified) Parallel sum-product for BEC (t) i!a 2 f; 1; g our belief about x i (t) a!i 2 f; 1; g our belief about x i () at iteration : i!a = y i at iteration t: (t) a!i = if any of the incoming messages is a (t) i!a = x b!i otherwise this is equivalent to the peeling decoder if all of the incoming messages are otherwise

20 Density evolution 5-12 Probabilistic analysis: density evolution an LDPC code is defined by a graph G probabilistic analysis: we want to predict the performance of a given LDPC code G to this end, we use density evolution on the computation tree if G is locally tree like up to depth k, and if we run sum-product algorithm for k iterations, then the resulting message (k) i!a described by the computation tree for the message (k) i!a : is fully () `!c (2) i!a (1) j!b

21 Density evolution 5-13 however, it is not always possible to apply density evolution a few assumptions sparse random graph construction (e.g. random (`; r)-regular graph from the configuration model) asymptotic analysis: in the limit n! 1 but finite number of iterations t why do we need these assumptions? it is difficult to analyze one particular graph, so we resort to the expected performance where the expectation also take into account the randomness in the graph generation random sparse graphs are locally tree-like if we consider random (d ; d)-regular graphs, the expected number of 2-cycles is ( 1 + n + d 1 ) n, which is small compared to the n number of edges

22 Density evolution 5-14 Probabilistic analysis: density evolution locally-tree like structure ensures that the incoming messages are independent formally, as n! 1 local neighborhood of a node converges in probability to a random tree P( lim depth k neighborhood of a random i is a tree) = 1 n!1 density evolution for (`; r)-regular graph zt 2 [; 1] be the probability a randomly chosen message from f (t) i!a g is an erasure w t 2 [; 1] be the probability a randomly chosen message from f (t) a!i g is an erasure in the limit n! 1, they satisfy the density evolution equations w t = 1 (1 z t 1 ) r 1 z t = w ` 1 t

23 z t = (1 (1 z t 1 ) r 1 )` 1 with initial condition z = density evolution for (3,6) code with = :4(left) and :45(right) rate of this code =.5, threshold '.4xxx, this simple code achieves rate less than the capacity = 1 P error (t) = lim n!1 P error (n ; t) analyze lim t!1 lim n!1 P error (n ; t), is this what we want? Density evolution 5-15

24 Density evolution 5-16 for a given value of, we can numerically run the density evolution, since it is an evolution of a scalar value, which gives bit error rate of (3; 6)-codes how do we find?

25 let s change the equation to z t = (1 (1 z t 1) r 1 )` 1 zt 1=(` 1) = 1 (1 z t 1) r 1 = :4 = : ` r `((1 ` ) (1 )) r for a given, if there is no overlap, then achieve zero error probability Z 1 rate of the code = 1 y ` 1 dy = ` ; ` r Z 1 vs. capacity = 1 (1 (1 x ) r 1 )dx = 1 extend this analysis to construct capacity achieving tornado codes Density evolution r

26 density evolution for general message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ; F ; E) and update: (x ) = 1 Z (t+1) i!a density evolution equation Y a2f Y i2v i(x i) = F i!a (f (t) b!i : b n ag) (t) a!i = G a!i(f (t) j!a : j n ig) z (t+1) = F (w (t) 1 ; : : : ; w (t) ` 1 ) w (t) = G(z (t) 1 ; : : : ; z (t) k 1 ) Density evolution 5-18

27 Density evolution 5-19 formally, as n! 1 a randomly chosen message from f (t) i!a g converge in probability to z (t) who cares about random graphs? who cares about asymptotics? alphabet x i 2 X messages i!a 2 Y density Z discrete f; 1g discrete f; 1; g continuous R discrete continuous R jx j 1 distribution over R jx j 1 continuous R distribution over R dist. over dist. over R how do we compute evolution of distributions? quantization Gaussian approximation population dynamics: represent the density using samples

EE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes

EE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check