Growth Rate of Spatially Coupled LDPC codes

Transcription

1 Growth Rate of Spatially Coupled LDPC codes Workshop on Spatially Coupled Codes and Related Topics at Tokyo Institute of Technology 2011/2/19

2 Contents 1. Factor graph, Bethe approximation and belief propagation 2. Relation between annealed free energy and belief propagation 3. Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon Here, growth rate is G(ω) = lim N 1 N loge[z(ω)] Z(ω): the number of codewords of relative weight ω [0,1]. 2 / 34

3 Factor graph, Bethe approximation and belief propagation 3 / 34

4 Factor graph Factor graph: bipartite graph which defines probability measure p(x) = 1 f a (x a ) Z a Z := f a (x a ), (partition function) f a (x a ) : X r a R 0 x X n a 4 / 34

5 Gibbs free energy p(x) = 1 Z f a (x a ) Approximation by simple distribution q of p which is defined by factor graph a D(q p) = x q(x)log q(x) p(x) ( ) = logz x q(x) log f a (x a ) + x q(x) log q(x) =: logz +U(q) H(q) =: logz +F Gibbs (q) a U(p): internal energy H(p): entropy F Gibbs (p): Gibbs free energy 5 / 34

6 Mean field approximation and Bethe approximation Mean field approximation q(x) = i b i (x i ) Degree of freedom is reduced from q n to nq Bethe approximation d i : degree of variable node i q(x) = ab a(x a ) i b i(x i ) d i 1 When factor graph is tree, Bethe approximation can be exact 6 / 34

7 Bethe free energy U(q) = x a ( ) q(x) log f a (x a ) a b a (x a )logf a (x a ) =: U Bethe ({b a }) x a b(x) ab a(x a ) i b i(x) d i 1 H(b) = x x = a b(x) log b(x) b(x) log ab a(x a ) i b i(x) d i 1 x a b a (x a )logb a (x a )+ =: H Bethe ({b i },{b a }) (d i 1) i i b i (x i )logb i (x i ) 7 / 34

8 Minimization of Bethe free energy F Bethe ({b i },{b a }) := U Bethe ({b a }) H Bethe ({b i },{b a }) minimize : F Bethe ({b i },{b a }) subject to : b i (x i ) 0, i b a (x a ) 0, a b i (x i ) = 1 i b a (x a ) = 1 a x a \x i b a (x a ) = b i (x i ), a, i a 8 / 34

9 Stationary point of Lagrangian of Bethe free energy [Yedidia, Freeman, and Weiss 2005] L := F Bethe ({b i },{b a })+ [ ] γ a b a (x a ) 1 + [ ] γ i b i (x) 1 a x a i x + λ ai (x i ) b i (x i ) b a (x a ) a i a x i x a \x i Stationary points of Lagrangian is fixed points of BP b a (x a ) f a (x a ) m i a (x i ) b i (x i ) i a i a m a i (x i ) where m i a (x i ) m c i (x i ) c i\a m a i (x i ) x a \x i f a (x a ) m j a (x j ) j a\i 9 / 34

10 Relation between annealed free energy and belief propagation 10 / 34

11 Random regular factor graph ensemble Factor graph: bipartite graph which defines probability measure µ(x) = 1 f a (x a ) Z a Z := f a (x a ), x X n a (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs 11 / 34

12 Annealed free energy Factor graph: bipartite graph which defines probability measure µ(x) = 1 f a (x a ) Z a Z := f a (x a ), x X n a (partition function) Random (l, r)-regular factor graph ensemble: l: degree of variable nodes, r: degree of factor nodes Random ensemble of factor graphs (Quenched) free energy: lim N 1 N E[logZ] Annealed free energy: lim N 1 N loge[z] 12 / 34

13 Contribution to partition function of particular types {v x } x X : the number of variable nodes of value x X is v x {u x } r: x X the number of factor nodes of value x Xr is u x Z = f(x a ) x X N = E[N({v},{u})] = {v},{u} a N({v},{u}) x X r f(x) ux. ( )( N l N ) r x X (v xl)!. {v x } x X {u x } x X r (Nl)! lim N 1 N loge[z({ν},{µ})] = l r H({µ}) (l 1)H({ν})+ l r x X r µ(x)logf(x). 13 / 34

14 Annealed free energy of fixed type and Bethe free energy F Bethe ({b i },{b a }) = b a (x a )logf a (x a ) a x a + b a (x a )logb a (x a ) (d i 1) a x a i i b i (x i )logb i (x i ) lim N 1 N loge[z({ν},{µ})] = l µ(x)logf(x)+ l r r H({µ}) (l 1)H({ν}). x X r 14 / 34

15 Maximization of the exponents of contributions maximize : l r H({µ}) (l 1)H({ν})+ l r subject to : ν(x) 0, x X µ(x) 0, x X r ν(x) = 1 x X µ(x) = 1 x X r 1 r µ(x) = ν(z), z X r k=1 x\x k x k =z x X r µ(x)logf(x) 15 / 34

16 The stationary condition The stationary condition is where ν(x) m f v (x) l µ(x) f(x) r m v f (x i ) i=1 m v f (x) m f v (x) l 1 r m f v (x) f(x) m v f (x j ). j k k=1 x\x k x k =x If f(x) is invariant under any permutation of x X r m f v (x) x\x 1 x 1 =x 16 / 34 f(x) j 1m v f (x j ).

17 Annealed free energy Theorem 1. lim N 1 N loge[z] = max (m f v,m v f ) S where S denotes the set of saddle points, and where { } l r logz f +logz v l logz fv. Z v := x Z f := x Z fv := x m f v (x) l r f(x) m v f (x i ) i=1 m f v (x)m v f (x). 17 / 34

18 Number of solutions If r k=1 x\x k x k =x f(x) is constant among all x X, the uniform message m f v (x) m v f (x) is a saddle point. The contribution of the uniform message is lim N 1 N loge[z(ν,µ)] = logq + l r log ( Nf q r ) (design rate) where q := X N f := x f(x). For CSP i.e., f(x) {0,1}, the expected number of solutions is about q N ( Nf q r ) l r N. This intuitively means all constraints are independent. 18 / 34

19 Contribution to partition function of fixed variable type Z({ν}) := {µ} Z({ν},{µ}) lim N 1 N loge[z({ν})] { = sup {µ} l r H({µ}) (l 1)H({ν})+ l r x X r µ(x)logf(x) } where {µ} satisfies 1 r µ(x) 0, x X r µ(x) = 1 x X r r µ(x) = ν(z), z X k=1 x\x k x k =z Convex optimization problem with linear constraints. 19 / 34

20 The stationary condition The stationary condition is µ(x) f(x) r i=1 m v f (x i ) where ν(x) h(x)m f v (x) l m v f (x) h(x)m f v (x) l 1 r m f v (x) f(x) m v f (x j ). j k k=1 x\x k x k =x If f(x) is invariant under any permutation of x X r m f v (x) x\x 1 x 1 =x 20 / 34 f(x) j 1m v f (x j ).

21 Growth rate of contribution to partition function of fixed variable type Theorem 2. lim N 1 N loge[z({ν})] { = max (m f v,m v f ) S l r logz f +logz v l logz fv x ν(x) log h(x) } where S denotes the set of saddle points, and where Z v := x Z f := x Z fv := x h(x)m f v (x) l r f(x) m v f (x i ) i=1 m f v (x)m v f (x). 21 / 34

22 Growth rate of regular LDPC codes G(ω) = l r log 1+zr 2 +log [ e h ( 1+y 2 where ω := 1 2ω and ω = tanh(h+l tanh 1 (y)) y = z r 1 ) l +e h ( 1 y z = tanh(h+(l 1)tanh 1 (y)). 2 ) l ] l log 1+yz 2 ω h This result can be easily understood from correspondings ω = ν(0) ν(1) y = m f v (0) m f v (1) z = m v f (0) m v f (1) h(x) = e ( 1)x h 22 / 34

23 Growth rate of regular LDPC codes (5,10) (6,10) (7,10) (8,10) (9,10) (10,10) (11,10) G w 23 / 34

24 Growth rate of binary CSP k=1 k=2 k= G w 24 / 34

25 Growth rate of (3,2)-regular-3-coloring / 34

26 Replica theory This story continues into the replica theory (see the paper in arxiv). But, we don t deal with it here. E[logZ] = loge[zn ] n n=0 lim N 1 E[logZ] = lim N N 1 N lim n 0? 1 = lim n 0 n lim N loge[z n ] n 1 N loge[zn ] The replica method is methematically not rigorous e.g., exchange of limits, analytic continuation of n. 26 / 34

27 Growth rate of spatially coupled LDPC codes and threshold saturation phenomenon 27 / 34

28 Protograph ensemble The similar results also holds for protograph ensemble [Vontobel 2010] In this morning, Kenta has explained Definition of protograph ensemble Definition of spatially coupled LDPC codes Threshold saturation phenomenon of EXIT curve 28 / 34

29 Growth rate of spatially coupled LDPC codes [ G(ω) = 1 l 2L+1 r + L i= L log L+l 1 j= L [ log e h l 1 ω = 1 2L+1 k=0 ( log 1+ l 1 k=0 z j,k 2 ( 1+yi,k L i= L z j,k = tanh h+ y i,k = z i+k,k r l 1 2 tanh l 1 L r l ) )+e h l 1 l 1 i= Lk=0 ( k =0,k k l 1 k =0,k k h+ k=0 ( 1 yi,k 2 ) ] ( ) ] 1+yi,k z i+k,k log ω h. 2 l 1 k=0 tanh 1 (y i,k ) tanh 1( y j k,k z i+k,k r l ) ) 29 / 34

30 ω versus h L=2,4,8,16,32,64,128,256 (5,10) h ω 30 / 34

31 ω versus h: Derivative of growth rate 1 L=2,4,8,16,32,64,128,256 (5,10) 0.5 h ω 31 / 34

32 Growth rate L=2,4,8,16,2048 (5,10) G(ω) ω 32 / 34

33 Conclusion Contribution to annealed free energy of particular type has similar form of Bethe free energy. The stationary condition of maximization problem for annealed free energy is similar to equation of belief propagation. There exists threshold saturation phenomenon in the calculation of growth rate of spatially coupled LDPC codes. We now can calculate annealed free energy of any coupled factor graphs. Effect of boundary condition is not obvious. BP iterations does not necessarily converge (even for uncoupled cases). 33 / 34

34 Acknowledgment The basic idea that the growth rate approaches to the concave hull is given by Nicolas Macris. I acknowledges Hamed Hassani and Toshiyuki Tanaka for encouragement and discussion. arxiv: , paper about connection between Bethe and annealed free energies (submitted to ISIT 2011). Joint paper with Hamed and Nicolas about growth rate of coupled LDPC codes was submitted to ISIT / 34