Optimum Rate Communication by Fast Sparse Superposition Codes

Transcription

1 Optimum Rate Communication by Fast Sparse Superposition Codes Andrew Barron Department of Statistics Yale University Joint work with Antony Joseph and Sanghee Cho Algebra, Codes and Networks Conference Universite de Bordeaux June 18, 2014

2 Quest for Provably Practical and Reliable High Rate Communication The Channel Communication Problem Gaussian Channel History of Methods Communication by Regression Sparse Superposition Coding Adaptive Successive Decoding Rate, Reliability, and Computational Complexity Distributional Analysis Simulations

3 Shannon Formulation Input bits: U = (U 1, U 2,......, U K ) Encoded: x = (x 1, x 2,..., x n ) Channel: p(y x) Received: Y = (Y 1, Y 2,..., Y n ) Decoded: Û = (Û1, Û2,......, ÛK ) indep Bern(1/2) Rate: R = K n Capacity C = max PX I(X; Y ) Reliability: Want small Prob{Û U} or reliably small fraction of errors

4 Gaussian Noise Channel Input bits: U = (U 1, U 2,......, U K ) 1 Encoded: c = (c 1, c 2,..., c n ) n n i=1 c2 i = P Channel: p(y c) Y = c + ε, ε N(0, σ 2 I) snr = P/σ 2 Received: Y = (Y 1, Y 2,..., Y n ) Decoded: Û = (Û1, Û2,......, ÛK ) Rate: R = K n Capacity C = 1 2 log(1 + snr) Reliability: Want small Prob{Û U} or reliably small fraction of errors

5 The Gaussian Noise Model The Gaussian noise channel is the basic model for wireless communication radio, cell phones, television, satellite, space wired communication internet, telephone, cable

6 Shannon Theory meets Coding Practice Forney and Ungerboeck 1998 review: modulation, shaping and coding for the Gaussian channel Richardson & Urbanke 2008, state of the art: Empirically good LDPC and turbo codes with fast encoding and decoding based on Bayesian belief networks New spatial coupling techniques, Urbanke 2013 Proof of rates up to capacity in some cases Arikan 2009, Arikan and Teletar 2009 polar codes: Adapted to Gaussian channel (Abbe and Barron 2011) Tropp 2006,2008 codes from compressed sensing and related sparse signal recovery work: Wainwright; Fletcher, Rangan, Goyal; Zhang; others Donoho,Montenari, et al 2012, role of spatial coupling l 1 -constrained least squares practical, has positive rate but not capacity achieving Knowledge of above not necessary to follow presentation

7 Sparse Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size M, a power of 2. One non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: X, n by ML dictionary; Normal(0, 1) entries X β, superposition of selected columns

8 Sparse Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size M, a power of 2. One non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by ML dictionary; Normal(0, 1) entries X β, superposition of selected columns Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log M

9 Sparse Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size M, a power of 2. One non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by ML dictionary; Normal(0, 1) entries X β, superposition of selected columns Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log M Reliability: small Prob{Fraction ˆβ mistakes δ}, small δ Is it reliable with rate up to capacity?

10 Sparse Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size M, a power of 2. One non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by ML dictionary; Normal(0, 1) entries X β, superposition of selected columns Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log M Ultra-sparse case: Impractical M = 2 nr/l with L constant (reliable at all rates R < C: Cover 1972,1980) Moderately-sparse: Practical M = n with L = nr/ log n (Still reliable at all R < C)

11 Sparse Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size M, a power of 2. One non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: X, n by ML dictionary; Normal(0, 1) entries Codeword: X β, superposition of selected columns Receive: Y = Xβ + ε Decode: ˆβ and Û Rate: R = K n from K = L log M Reliability: small Prob{Fraction mistakes δ} small δ Outer RS code: rate 1 2δ, corrects remaining mistakes Explanation: Outer code selects (j 1, j 2,..., j L ) where any two differ in more than fraction 2δ sections Overall rate: R tot = (1 2δ)R with small δ Overall rate: up to capacity

12 Power Allocation Coefficients: β =( , ,..., ) Indices of nonzeros: sent = (j 1, j 2,..., j L ) Coeff. values: β jl = P l Power control: L l=1 P l = P Codewords: Power Allocations for l = 1, 2,..., L X β, have average power P Constant power: Variable power: P l = P/L 2C l/l P l proportional to e

13 Variable Power Allocation Power control: L l=1 P l = P β 2 = P Variable power: P l proportional to e 2Cl/L for l = 1,..., L Power allocation with snr=7, L=512 Power Allocation section index

14 Variable Power Allocation Power control: L l=1 P l = P β 2 = P Variable power: P l proportional to e 2Cl/L for l = 1,..., L Successive decoding motivation Incremental capacity ( ) 1 2 log P l 1 + σ 2 = C + P l P L L matching the section rate R L = log M n

15 Adaptive Successive Decoder [Joseph & B ISIT, 2013 IT] Decoding Steps (with thresholding) Start: [Step 1] Compute the inner product of Y with each column of X See which are above a threshold Form initial fit as weighted sum of columns above threshold Iterate: [Step k 2] Compute the inner product of residuals Y Fit k 1 with each remaining column of X Standardize by dividing by Y Fit k 1 See which are above the threshold 2 log M + a Add these columns to the fit Stop: At Step k = 1 + snr log M, or if there are no additional inner products above threshold

16 Complexity of Adaptive Successive Decoder Complexity in parallel pipelined implementation Space: (use k = snr log M copies of the n by N dictionary) knn = snr C M n 2 memory positions kn multiplier/accumulators and comparators Time: O(1) per received Y symbol

17 Adaptive Successive Decoder: soft decision [Cho & B 2012 ISIT, 2013 WITMSE] Decoding Steps (with iteratively optimal statistics) Start: [Step 1] Compute the inner product of Y with each column of X Form initial fit Iterate: [Step k 2] Compute inner product of residuals Y Fit k 1 with each X j. Adjusted form: stat k,j equals (Y X ˆβ k 1, j ) T X j Standardize by dividing it by Y X ˆβ k 1. Form the new fit ˆβ k,j = P l ŵ j (α) = P l e α stat k,j j sec l e α stat k,j Stop: When estimated success stops increasing At Step k = O(log M)

18 Iteratively Bayes optimal ˆβ k With prior j l Unif on sec l, the Bayes estimate based on stat k ˆβ k = E[β stat k ] has representation ˆβ k,j = P l ŵ k,j with ŵ k,j = Prob{j l = j stat k }.

19 Iteratively Bayes optimal ˆβ k With prior j l Unif on sec l, the Bayes estimate based on stat k ˆβ k = E[β stat k ] has representation ˆβ k,j = P l ŵ k,j with ŵ k,j = Prob{j l = j stat k }. Here, when the stat k,j are independent N(α l,k 1 {j=jl }, 1), we have the logit representation ŵ k,j = ŵ(α l,k ) where ŵ k,j (α) = e α stat k,j j sec l e α stat. k,j

20 Iteratively Bayes optimal ˆβ k With prior j l Unif on sec l, the Bayes estimate based on stat k ˆβ k = E[β stat k ] has representation ˆβ k,j = P l ŵ k,j with ŵ k,j = Prob{j l = j stat k }. Here, when the stat k,j are independent N(α l,k 1 {j=jl }, 1), we have the logit representation ŵ k,j = ŵ(α l,k ) where ŵ k,j (α) = e α stat k,j j sec l e α stat k,j. Recall stat k,j is standardized inner product of residuals with X j stat k,j = (Y X ˆβ k 1, j ) T X j Y X ˆβ k 1

21 Distributional Analysis Approximate distribution of these statistics: independent standard normal, shifted for terms sent where Here stat k,j = α l,xk 1 1 {j sent} + Z k,j np l α = α l,x = σ 2 + P(1 x) E ˆβ k 1 β 2 = P (1 x k 1 ) Update rule x k = g(x k 1 ) where g(x) = L (P l /P)E[w jl (α l,x )]. l=1

22 Distributional Analysis Approximate distribution of these statistics: independent standard normal, shifted for terms sent where Here stat k,j = α l,xk 1 1 {j sent} + Z k,j np l α = α l,x = σ 2 + P(1 x) E ˆβ k 1 β 2 = P (1 x k 1 ) Update rule x k = g(x k 1 ) where g(x) = L (P l /P)E[w jl (α l,x )]. l=1

23 Success Rate Update Rule Update rule x k = g(x k 1 ) where g(x) = L (P l /P)E[w jl (α l,x )] l=1 this is the success rate update function expressed as a weighted sum of the posterior prob of the term sent

24 Success Rate Update Rule Update rule x k = g(x k 1 ) where g(x) = L (P l /P)E[w jl (α l,x )] l=1 this is the success rate update function expressed as a weighted sum of the posterior prob of the term sent Empirical success rate x k = L (P l /P)w jl (α l,xk ) l=1

25 Success Rate Update Rule Update rule x k = g(x k 1 ) where g(x) = L (P l /P)E[w jl (α l,x )] l=1 this is the success rate update function expressed as a weighted sum of the posterior prob of the term sent Empirical success rate x k = L (P l /P)w jl (α l,xk ) l=1 Empirical estimated success rate ˆx k = L (P l /P) l=1 j sec l [w j (α l,xk )] 2

26 Decoding progression g(x) x M = 2 9, L = M snr=7 C=1.5 bits R=1.2 bits(0.8c) Figure : Plot of g(x) and the sequence x k. x

27 Update functions g(x) Lower bound a=0 a=0.5 M = 2 9, L = M snr=7 C=1.5 bits R=1.2 bits(0.8c) Figure : Comparison of update functions. Blue and light blue lines indicates {0, 1} decision using the threshold τ = 2logM + a with respect to the value a as indicated. x

28 Success Progression plots Expected weight of the terms sent x=0 soft decision hard decision with a=1/2 x=0.6 x=0.2 x=0.8 x=0.4 x= u(l) u(l) u(l) Figure : Progression plots : M = 2 9, L = M, C = 1.5 bits and R = 0.8C. We used Monte Carlo simulation with replicate size The horizontal axis depicts u(l) = 1 e 2Cl/L which is an increasing function of l. Area under curve equals g(x).

29 Rate and Reliability Result for Optimal ML Decoder [Joseph and B IT], with outer RS decoder, and with equal power in the sections Prob error exponentially small in n for all R < C Prob{Error} e n (C R)2 /2V In agreement with the Shannon-Gallager exponent of optimal code, though with a suboptimal constant V depending on the snr

30 Rate and Reliability of Fast Superposition Decoder Practical: Adaptive Successive Decoder, with outer RS code. prob error exponentially small in n/(log M) for R < C [Joseph and B ISIT, 2013 IT] for thresholding [Cho and B ISIT, 2013 WITMSE] for soft-decision C M is identified approaching capacity at rate 1/(log M) 0.5, or, with a variant of the power allocation, at rate 1/ log M. Probability error exponentially small in L for R < C M Prob { Error } e L(C M R) 2 c 1 Improves to e c 2L(C M R) 2 (log M) 0.5 using a Bernstein bound.

31 Simulation Figure : L = 512, M = 64, snr = 15, C = 2 bits, R = 1 bits, n = Ran 20 trials for each residual-based method. Green lines for hard thresholding; blue and red for soft decision.

32 Framework: For k 1 Improved iterative statistics Codeword fits: F k = X ˆβ k statistic vector stat k constructed from inner products of (Y, F 1,..., F k ) with columns of X Want stat k,j better than X T j (Y F k ) Update estimate ˆβ k+1 is a function of stat k as before Thresholding: Adaptive Successive Decoder ˆβ k+1,j = P l 1 {statk,j >thres} Soft decision: ˆβ k+1,j = E[β j stat k ] = P l ŵ k,j with thresholding on the last step

33 Orthogonal Components Codeword fits: F k = X ˆβ k Orthogonalization : Let G 0 = Y and for k 1 G k = part of F k orthogonal to G 0, G 1,..., G k 1 Components of statistics Z k,j = X j T G k G k Statistics such as stat k,j built from Xj T (Y F k, j ) are linear combinations of these Z k,j

34 Distribution Evolution Lemma 1: shifted normal conditional distribution Given F k 1 = ( G 0,..., G k 1, Z 0, Z 1,..., Z k 1 ), the Z k has the distributional representation Z k = G k b k + Z k σ k G k 2 /σ 2 k Chi-square(n k) b 0, b 1,..., b k the successive orthonormal components of [ ] [ ] [ ] β ˆβ, 1 ˆβ,..., k ( ) σ 0 0 Z k N(0, Σ k ) indep of G k Σ k = I b 0 b T 0 b 1b T 1... b kb T k = projection onto space orthogonal to ( ) σ 2 k = ˆβ T k Σ k 1 ˆβ k

35 Combining Components Class of statistics stat k formed by combining Z 0,..., Z k n stat k,j = Zk,j comb + ˆβk,j ck with Z comb k = λ 0,k Z 0 + λ 1,k Z λ k,k Z k, k λ2 k,k = 1

36 Combining Components Class of statistics stat k formed by combining Z 0,..., Z k n stat k,j = Zk,j comb + ˆβk,j ck with Z comb k = λ 0,k Z 0 + λ 1,k Z λ k,k Z k, Ideal Distribution of the combined statistics n statk ideal = β + Zk comb ck for c k = σ 2 + (1 x k )P with Z comb k Normal(0, I). For the terms sent the shift α l,k has an effective snr interpretation α l,k = n P l c k = P l n σ 2 + P remaining,k k λ2 k,k = 1

37 Oracle statistics Weights of combination: based on λ k proportional to ( (σ Y b0 T ˆβ k ), (b1 T ˆβ k ),..., (bk T ˆβ ) k ) Combining Z k with these weights, replacing χ n k with n, it produces the desired distributional representation n stat k = σ 2 + β ˆβ β + Z comb k 2 k with Z comb k N(0, I) and σ 2 Y = σ2 + P.

38 Oracle statistics Weights of combination: based on λ k proportional to ( (σ Y b0 T ˆβ k ), (b1 T ˆβ k ),..., (bk T ˆβ ) k ) Combining Z k with these weights, replacing χ n k with n, it produces the desired distributional representation n stat k = σ 2 + β ˆβ β + Z comb k 2 k with Z comb k N(0, I) and σ 2 Y = σ2 + P. Can t calculate the weights not knowing β, β ˆβ k 2 is close to its known expectation e.g. b 0 = β/σ Provides the desired distribution of the stat k,j

39 Statistics based on weights of combination Oracle weights of combination: λ k proportional to ( (σ Y b0 T ˆβ k ), (b1 T ˆβ k ),..., (bk T ˆβ ) k ) Estimated weights of combination: λ k proportional to ( ( Y Z0 T ˆβ k ), (Z1 T ˆβ k ),..., (Zk T ˆβ ) k ) These estimated weights produce the residual based statistics previously discussed

40 Orthogonalization Interpretation of Weights Estimation of ( (σ Y b0 T ˆβ k ), (b1 T ˆβ k ),..., (bk T ˆβ ) k ) These b T k ˆβ k arise in the QR-decomposition for B = [β, ˆβ 1,..., ˆβ k ]

41 Cholesky Decomposition for B T B The b T k ˆβ k arise in the Cholesky decomposition B T B =R T R,

42 Cholesky Decomposition for B T B Replace entries with deterministic estimates based on x k P, with c k = σ 2 + (1 x k )P and with ω k = 1/c k 1/c k 1 for k 1 and ω 0 = 1/c 0.

43 Cholesky Decomposition for B T B Replace entries with deterministic estimates based on x k P, with c k = σ 2 + (1 x k )P and with ω k = 1/c k 1/c k 1 for k 1 and ω 0 = 1/c 0. The rightmost column yields suitable deterministic weights of combination

44 Deterministic weights of combinations For given ˆβ 1,..., ˆβ k and c 0 >... > c k, c k = σ 2 + (1 x k )P Combine Z k = n b k + Z k with λ k = c k ( ω 0, ω 1,..., ω k ) yielding approximately optimal statistics stat k = k k =0 λ k,k Z k + n ck ˆβ k

45 Update Plots for Deterministic and Oracle Weights Figure : L = 512, M = 64, snr = 15, C = 2 bits, R = 0.7C, blocklength = Ran 10 experiment for each method. We see that they follow the expected update function.

46 Improving the End Game Variable power: P l proportional to e 2Cl/L for l = 1,..., L We use alternative power allocation: constant leveling the power allocation for the last portion of the sections Figure : L = 512, snr = 7, C = 1.5 bits

47 Progression Plot using Alternative Power Allocation Figure : L = 512, M = 64, snr = 7, C = 1.5 bits, R = 0.7C, blocklength = Progression plot of the final step. The area under the curve might be the same, the expected weights for the last sections are higher when we level the power at the end L=512, M=64, n=2926, C=1.5 bits, rate=1.05 bits, snr.ref=7, snr=7 weights for the true term in section l with leveling without leveling P_l P

48 Bit Error Rate Figure : L = 512, M = 64, R = 1.05 bits, blocklength n = 2926, snr.ref = 7, snr = (4, 5, 7, 10, 15). Ran 10,000 trials. Average of error count out of 512 sections.

49 Block Error Rate Figure : L = 512, M = 64, R = 1.05 bits, blocklength n = 2926, snr.ref = 7, snr = (4, 5, 7, 10, 15). Ran 10,000 trials.

50 Block Error Rate Large L regime, M not yet large. Identify rate limits. Tradeoff in R inner and in x where g(x) = x. Let R (snr) be the maximized value of the total rate x R inner Figure : M = 64, 128, R target = 1.05 bits log(prob of Error) Shannon limit M=64 (0.72) M=128 (0.7) log(snr) Snr snr(db) Vertical bars at the snr at which R target equals capacity and at the snr at which R (snr) matches the target.

51 Summary Sparse superposition codes with adaptive successive decoding Simplicity of the code permits: distributional analysis of the decoding progression low complexity decoder exponentially small error probability for any fixed R < C

52 Improving the adaptive successive decoder Introduce successively decode chapters of the dictionary Update coefficient estimates across all sections Use a Reed-Solomon code for each chapter to clean away section errors once estimates are high Reduced interference yields improved success progression

53 Chaptering Rule to compute number of chapters and sections in each Set total number of sections L and the rate R = 0.8C. Precompute progressions g l (x) at current success rate x k. Identify section index l k at which area to the left is x k. For specified rate, it s equivalent to l where Ew jl 0.78 Use these indices as the chapter boundaries. Reed-Solomon code decodes the most recent chapter. Remove the associated interferrence when updating g l (x) where g l (x) = Ew jl Chapter creation ends when can nolonger increase x k.

54 Example use of Chapters Parameters: L = 512, M = 64, n = 2926, R = 1.05bits = 0.7C Number of sections in each chapter: L 1 = 120, L 2 = 117, L 3 = 109, L 4 = 88, L 5 = 78

55 Progression of success with chaptering L=512, M=64, n=2926, snr=7, Rate=1.05 bits (0.7C) Expected weight of the terms sent without chapter with chapter step=1 step=2 step=3 step=4 step=5 step=6 step=7 step=8 step= u(l) u(l) u(l)

56 update function x(k+1) L=512, M=64 n=2926, snr=7 Rate=1.05 bits (0.7C) x(k) Figure : Total success rate is improved from to