Sparse Superposition Codes for the Gaussian Channel

Transcription

1 Sparse Superposition Codes for the Gaussian Channel Florent Krzakala (LPS, Ecole Normale Supérieure, France) J. Barbier (ENS) arxiv: presented at ISIT 14 Long version in preparation

2 Communication through the Gaussian Channel N (0, ) X coding Ỹ = F X noisy output Y = Ỹ + decoding ˆX F, Y ˆX Message (K bits) Codeword, size M M N matrix Forney & Ungerboeck 98 Review (modulation, shaping and coding) Characteristics: Power constraint: hỹ 2 i =1 Richardson & Urbanke 08 (LDPC and turbo codes) Arikan 09, Abbe & Barron 11 (Polar codes) Signal to noise ratio: SNR = 1 Shannon Capacity: C = 1 2 log 2 (1 + SNR)

3 Sparse Superposition codes Joseph, Barron ISIT 10 N (0, ) X coding Ỹ = F X noisy output Y = Ỹ + decoding ˆX F, Y ˆX Message (K bits) M N matrix with i.i.d random elements Codeword, size M L sections vector X= { B components Only one 0 Dimension N=LB K=L log 2 (B) bits of information Rate: R = N log 2(B) BM

4 Sparse Superposition codes Joseph, Barron ISIT 10 For large B, the maximum likelihood solution is capacity achieving minˆx Y F ˆx `2 such that ˆx has a single 1 per section Hard computational problem

5 Sparse Superposition codes Joseph, Barron ISIT 10 For large B, the maximum likelihood solution is capacity achieving With proper power allocation, the adaptive successive decoder of Joseph and Barron achieve capacity when B vector X=

6 Sparse Superposition codes Joseph, Barron ISIT 10 For large B, the maximum likelihood solution is capacity achieving With proper power allocation, the adaptive successive decoder of Joseph and Barron achieve capacity when B vector X= Power constraint: 0 c1 0 0 c c3 hc 2 i i =1 c 2 i / e 2Ci L c

7 Sparse Superposition codes Joseph, Barron ISIT 10 For large B, the maximum likelihood solution is capacity achieving With proper power allocation, the adaptive successive decoder of Joseph and Barron achieve capacity when B vector X= 0 c1 0 0 c Power constraint: hc 2 i i = c3 c 2 i / e 2Ci L c In practice, however, results are FAR from capacity when B is finite

8 Sparse Superposition codes Joseph, Barron ISIT 10 SNR=15 SNR=7

9 1 The approximate message-passing algorithm

10 Sparse Superposition codes = Multi-dimensional estimation problem L components x 1 B-dim variables Coding matrix F random Mₒ N matrix x 2 Noise M components = 1 x Noisy output Y M components Sparse signal X L B-dimensional components

11 Graphical model Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) P( x l )= prior «only one non zero component» / e snr 2 (y µ likelihood P (y µ F, X) / P N i=1 F µix i ) 2

12 Graphical model Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) Belief Propagation? Not really an option i) Densely connected model ii) O(L 2 ) messages iii) Combinatoric factor O(B L ) iv) Way too slow! m i!µ (x i ) / P (x i ) Y 6=µ m!i (x i )

13 From belief propagation to AMP Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) (i) A Gaussian relaxation (aka Gaussian-BP) (ii) Write the recursion in terms of single marginals (equivalent to TAP in Stat. Phys.) m i!µ (x i ) / P (x i )e x 2 i 2 P 6=µ A!i+ P 6=µ B µ!ix i A Gaussian-like relaxation m µ!i (x i ) / e x 2 i 2 A µ!i+b µ!i x i

14 From belief propagation to AMP Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) (i) A Gaussian relaxation (aka Gaussian-BP) (ii) Write the recursion in terms of single marginals (equivalent to TAP in Stat. Phys.) m i!µ (x i ) m i (x i ) + correction Closing the equations on two moments for each variables: 2N variables (AMP) vs N 2 variables (BP)

15 From belief propagation to AMP Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) Matlab implementation + demo:

16 From belief propagation to AMP Compute the marginal of P (X F, Y ) / P (X)P (Y F, X) A (perhaps) more familiar form from monday s talk (Donoho, Montanari, Maleki 09) a t+1 = f t a(f 0 z t + a t ) z t = y Fa t + 1 < f t+1 a (F 0 z t 1 + a t 1 ) >z t 1 Complexity is N 2 for iid matrix Nlog(N) for Hadamard/Fourier Matlab implementation + demo:

17 2 Asymptotic analysis

18 A heuristic computation of the optimal SER Bayes-optimal estimate: Marginalize w.r.t. P (X Y,F) = P (X) Z MY µ=0 e SNR 2 (y µ P N i=1 F µi ˆx i) 2 The replica method : Asymptotic estimates when N (statistical physics, information theory, optimization, etc..) In CDMA Tanaka 02 Guo, Verdu 06 In Compressed sensing Rangan et al. 09 Kabashima 09 Guo Baron Shamai 09 Krzakala et al. 11

19 A heuristic computation of the optimal SER Bayes-optimal estimate: Marginalize w.r.t. P (X Y,F) = P (X) Z MY µ=0 e SNR 2 (y µ P N i=1 F µi ˆx i) 2 The replica method : Asymptotic estimates when N (statistical physics, information theory, optimization, etc..) 1) Assume that log(z) is concentrated NON RIGOROUS 2) Use the following identity: 3) After (a bit) of work: log Z / log Z =lim n!0 Z n 1 n Z NON RIGOROUS dee N (E) MMSE = max E (E) 4) From the MSE E, one can compute the average SER

20 Single letter characterization of the SER B-dimensional integral with and The SER can be computed from E as with

21 Single letter characterization of the SER Here B=2, SNR=15, C= Φ R=1.5 below the BP threshold R=1.5 at R=1.68 the BP threshold R=1.68 at R=1.75 the optimal threshold R=1.775 maxima SER

22 Gap to capacity closing polynomially

23 Error floor decay polynomially with B

24 State Evolution Analysis of AMP (As in Bayati-Montanari 10 in the scalar case) The MSE obey the following recursion: with: Define: with The Section Error Rate can be deduced from the MSE at all times by

25 State Evolution vs AMP for finite sizes B= 4, SNR=15, N=32768, RBP=1.55

26 State Evolution Analysis and the replica potential with: Define: Replica Potential: The fixed point of the state evolution corresponds = extrema of the potential Potential = Bethe free energy

27 Phase diagram SNR=15 (C=2) high error «low» error Optimal threshold R C when B 1.6 rate BP threshold Optimal threshold exp 1: AMP, L=100 exp 1: SC AMP + Hadamard, L=100 exp 1: it. successive, exp. dist., L=100 exp 2: SC AMP + Hadamard, L=2 13 exp 2: AMP + Hadamard, L= section size B

28 Phase diagram SNR=15 (C=2) high error «low» error Optimal threshold R C when B 1.6 high error for AMP rate AMP+Hadamard, «low» error for L=2^13 AMP R AMP = [2 log(2) (1 + 1/SNR)] BP threshold Optimal threshold exp 1: AMP, L=100 exp 1: SC AMP + Hadamard, L=100 exp 1: it. successive, exp. dist., L=100 exp 2: SC AMP + Hadamard, L=2 13 exp 2: AMP + Hadamard, L= section size B AMP gets worst as B increases :-(

29 Phase diagram SNR=15 (C=2) high error «low» error Optimal threshold R C when B 1.6 high error for AMP rate AMP+Hadamard, «low» error for L=2^13 AMP R AMP = [2 log(2) (1 + 1/SNR)] BP threshold Optimal threshold exp 1: AMP, L=100 exp 1: SC AMP + Hadamard, L=100 exp 1: it. successive, exp. dist., L=100 exp 2: SC AMP + Hadamard, L=2 13 exp 2: AMP + Hadamard, L= section size B but is still way better than the alternative :-)

30 3 Two strategies to reach capacity

31 Two ways to make AMP capacity achieving: Spatial coupling Power Allocation

32 Two ways to make AMP capacity achieving: Spatial coupling Power Allocation vector X= vector X= 0 c1 0 0 c c3 c Generalization the State evolution for structured matrices shows that bamp can reach capacity as B c 2 i / 2 2Ci/L Power constraint: hc 2 i i =1 In practice, however, this turns out to be not so efficient

33 Two ways to make AMP capacity achieving: Spatial coupling 0 h 1 0 F 1 0 e 1 Seed block 1 J J 0 = w J B C J C J J A J : unit coupling : coupling J small w : interaction range Propagation : no coupling (null elements) Use of a structured matrix (Similar constructions in compressed sensing): Pfister-Kudekar 09 Krzakala et al 11 Donoho et al 11 C A

34 Spatial coupling Lena: L=256^2 B=256 levels of grey Matlab implementation + demo:

35 4 Practical tests!

36 Phase diagram SNR=15 (C=2) high error «low» error Optimal threshold R C when B 1.6 high error for AMP rate AMP+Hadamard, «low» error for L=2^13 AMP R AMP = [2 log(2) (1 + 1/SNR)] BP threshold Optimal threshold exp 1: AMP, L=100 exp 1: SC AMP + Hadamard, L=100 exp 1: it. successive, exp. dist., L=100 exp 2: SC AMP + Hadamard, L=2 13 exp 2: AMP + Hadamard, L= section size B but is still way better than the alternative :-)

37 Comparing everything snr = 15, B = 512, L = 1024 Blocklenght ~ average SER Opt. pow. alloc. + random op. + off-line AMP Opt. pow. alloc. + Had. op. + on-line AMP Fixed sp.-coupling + Had. op. + on-line AMP Full Had. op. + on-line AMP Fixed sp.-coupling + opt. pow. alloc. + Had. op. + on-line AMP BP threshold Capacity R

38 . Conclusion Simple coding, capacity achieving with spatial coupling/power allocation Similar phenomenology as LDPC codes Efficient when used with structured operator such as Hadamard Can be analyzed by state evolution/replica method Interesting finite size performances?