Communication by Regression: Achieving Shannon Capacity

Transcription

1 Communication by Regression: Practical Achievement of Shannon Capacity Department of Statistics Yale University Workshop Infusing Statistics and Engineering Harvard University, June 5-6, 2011

2 Practical Communication by Regression Achieving Shannon Capacity Gaussian Channel with Power Constraints History of Methods Communication by Regression Sparse Superposition Coding Adaptive Successive Decoding Distributional Analysis Rate, Reliability, and Computational Complexity

3 Shannon Formulation Input bits: U = (U 1, U 2,......, U K ) Encoded: x = (x 1, x 2,..., x n ) = x(u) Channel: p(y x) Received: Y = (Y 1, Y 2,..., Y n ) Decoded: Û = (Û1, Û2,......, ÛK ) indep Bern(1/2) Rate: R = K n Capacity C = max PX I(X; Y ) Reliability: Want small Prob{Û U} and small Prob{Fraction mistakes α}

4 Gaussian Noise Channel Input bits: U = (U 1, U 2,......, U K ) Encoded: x = (x 1, x 2,..., x n ) 1 n n i=1 x 2 i P Channel: p(y x) Y = x(u) + ε, ε N(0, σ 2 I) snr = P/σ 2 Received: Y = (Y 1, Y 2,..., Y n ) Decoded: Û = (Û1, Û2,......, ÛK ) Rate: R = K n Capacity C = 1 2 log(1 + snr) Reliability: Want small Prob{Û U} and small Prob{Fraction mistakes α}

5 Gaussian Noise Channel Input bits: U = (U 1, U 2,......, U K ) [ Encoded: x = (x 1, x 2,..., x n ) E 1 n U n i=1 x i 2 ] P Channel: p(y x) Y = x(u) + ε, ε N(0, σ 2 I) Received: Y = (Y 1, Y 2,..., Y n ) Decoded: Û = (Û1, Û2,......, ÛK ) snr = P/σ 2 Rate: R = K n Capacity C = 1 2 log(1 + snr) Reliability: Want small Prob{Û U} and small Prob{Fraction mistakes α}

6 Shannon Theory Channel Capacity: Supremum of rates R such that reliable communication is possible, with arbitrarily small error probability Information Capacity: C = max PX I(X; Y ) Where I(X; Y ) is the Shannon information, also known as the Kullback divergence between P X,Y and P X P Y Shannon Channel Capacity Theorem: The supremum of achievable communication rates R equals the information capacity C Books: Shannon (49), Gallager (68), Cover & Thomas (06)

7 The Gaussian Noise Model The Gaussian noise channel is the basic model for wireless communication radio, cell phones, television, satellite, space wired communication internet, telephone, cable

8 Shannon Theory meets Coding Practice Forney and Ungerboeck 1998 review: modulation, shaping and coding for the Gaussian channel Richardson & Urbanke 2008 cover coding state of the art: Empirically good LDPC and turbo codes with fast encoding and decoding based on Bayesian belief networks Some tools for theoretical analysis, yet obstacles remain for proof of rates up to capacity for the Gaussian channel Arikan 2009, Arikan and Teletar 2009 polar codes: Adapted to Gaussian channel (Abbe and Barron 2011) Tropp 2006,2008 codes from compressed sensing and related sparse signal recovery work: Wainwright; Fletcher, Rangan, Goyal; Zhang; others l 1 -constrained least squares practical, has positive rate but not capacity achieving Method here different from those above. Prior knowledge of such is not necessary to follow what we present.

9 Sparse Superposition Code Input bits: U = (U U K ) Coefficients: β = ( ) T Sparsity: L entries non-zero out of N Matrix: X, n by N, all entries indep Normal(0, 1) Codeword: X β, superposition of a subset of columns Receive: Y = Xβ + ε, a statistical linear model Decode: ˆβ and Û from X,Y

10 Sparse Superposition Code Input bits: U = (U U K ) Coefficients: β = ( ) T Sparsity: L entries non-zero out of N Matrix: X, n by N, all entries indep Normal(0, 1) Codeword: X β, superposition of a subset of columns Receive: Y = Xβ + ε Decode: ˆβ and Û from X,Y Rate: R = K n from K = log ( ) ( N L, near L log N L e)

11 Sparse Superposition Code Input bits: U = (U U K ) Coefficients: β = ( ) T Sparsity: L entries non-zero out of N Matrix: X, n by N, all entries indep Normal(0, 1) Codeword: X β, superposition of a subset of columns Receive: Y = Xβ + ε Decode: ˆβ and Û from X,Y Rate: R = K n from K = log ( ) N L Reliability: small Prob{Fraction ˆβ mistakes α}, small α

12 Sparse Superposition Code Input bits: U = (U U K ) Coefficients: β = ( ) T Sparsity: L entries non-zero out of N Matrix: X, n by N, all entries indep Normal(0, 1) Codeword: X β, superposition of a subset of columns Receive: Y = Xβ + ε Decode: ˆβ and Û from X,Y Rate: R = K n from K = log ( ) N L Reliability: small Prob{Fraction ˆβ mistakes α}, small α Is it reliable with rate up to capacity?

13 Partitioned Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size B =N/L, a power of 2. 1 non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by N, splits into L sections X β, superposition of columns, one from each Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log N L = L log B

14 Partitioned Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size B =N/L, a power of 2. 1 non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by N, splits into L sections X β, superposition of columns, one from each Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log N L = L log B Ultra-sparse case: Impractical B = 2 nr/l with L constant (reliable at all rates R < C: Cover 1972,1980) Moderately-sparse: Practical B = n with L = nr/ log n Is it reliable with rate up to capacity?

15 Partitioned Superposition Code Input bits: U = (U 1...,...,...,... U K ) Coefficients: β =( , ,..., ) Sparsity: L sections, each of size B =N/L, a power of 2. 1 non-zero entry in each section Indices of nonzeros: (j 1, j 2,..., j L ) specified by U segments Matrix: Codeword: Receive: Decode: X, n by N, splits into L sections X β, superposition of columns, one from each Y = Xβ + ε ˆβ and Û Rate: R = K n from K = L log N L = L log B Reliability: small Prob{Fraction mistakes α} small α Outer RS code: rate 1 α, corrects remaining mistakes Overall rate: R tot = (1 α)r Overall rate: up to capacity?

16 Power Allocation Coefficients: β =( , ,..., ) Indices of nonzeros: sent = (j 1, j 2,..., j L ) Coeff. values: β jl = P l Power control: L l=1 P l = P Codewords: Power Allocations Constant power: Variable power: for l = 1, 2,..., L X β, have average power P P l = P/L 2C l/l P l proportional to e Variable with leveling: P l proportional to max{e 2C l/l, cut}

17 Power Allocation power allocation Power = 7 L = section index

18 Adaptive Successive Decoder Decoding Steps Start: [Step 1] Compute the inner product of Y with each column of X See which are above a threshold Form initial fit as weighted sum of columns above threshold Iterate: [Step k 2] Compute the inner product of residuals Y Fit k 1 with each remaining column of X See which are above threshold Add these columns to the fit Stop: At Step k = log B, or if there are no inner products above threshold

19 Contrast Two Decoders Decoders using received y = Xβ + ε Optimal: l 0 -constrained least squares decoder using equal power ˆβ = argmin Y Xβ 2 maximizes posterior probability of codeword given the data minimizes probability of error with uniform input distribution reliable for all R < C, with best form of error exponent Practical: Adaptive Successive Decoder fast decoder reliable using variable power allocation for all R < C

20 Rate and Reliability Optimal Decoder: l 0 -constrained least squares Prob error exponentially small in n for all R < C Prob{Error} e n (C R)2 /2V In agreement with the Shannon-Gallager exponent of optimal code, though with a suboptimal constant V depending on the snr

21 Rate and Reliability Practical: Adaptive Successive Decoder, with outer RS code. prob error exponentially small in n/(log B) 1/2 for R < C Value C B approaching capacity C B = C 1 + c 1 / log B Probability error exponentially small in L for R < C B Prob { Error } e L(C B R) 2 c 2 Improves to e c 3L(C B R) 2 (log B) 0.5 using a Bernstein bound. Nearly optimal when C B R is at least C C B. Our c 1 is near ( /snr) log log B + 4C

22 Ingredients in the Practical Decoder Adaptive Successive Decoder Ingredients Intialization: res 1 = Y and J 1 = {1, 2,..., N}, with N = LB Loop: Residual: Test Stat: res k = Y Fit k 1 Z comb k,j = Xj T res k / res k Threshold: τ = 2 log B + a Detections: 1 Hk,j = 1 {Z comb k,j τ} Fit Increment: F k = j J k Pj X j 1 Hk,j Fit Update: Fit k = Fit k 1 + F k Remaining: J k+1 = { j J k : Z k,j < τ}

23 Tracking Progress Message sent = (j 1, j 2,..., j L ) False Alarms Increment: ˆf k = j J k (not sent) π j 1 Hk,j Total: ˆf1,k = ˆf 1 + ˆf ˆf k Correct Detections Increment: ˆq k = j J k sent π j 1 Hk,j Total: ˆq 1,k =ˆq 1 + ˆq ˆq k Weights π j = P j /P where P j is the power allocated to the section containing j

24 Decoding Progression g L(x) x B = 2 16, L = B snr = 1 C = 0.5 bits R = 0.31 bits (0.62C) No. of steps = Figure: Plot of of likely progression of weighted fraction of correct detections ˆq 1,k, for snr = 1. x

25 Distributional Analysis Test statistic ingredients: Z k,j = X j T G k / G k The Y, F 1,..., F k 1 have orthogonalized components G 1, G 2,..., G k Approximate distrib: Z k,j is shift of indep N(0, 1) r.v.s (w k u j C/R) 2 log B 1 {j sent} + Z k,j with u jl = u(l/l) = e 2Cl/L where w k has sum s k = w 1 + w w k equaling s(x) = 1/(1 νx) ν = snr/(1 + snr) evaluated at x =x k 1 based on wght. frac. prev. detections x k is at least q adj 1,k q 1,k f 1,k initialized with w 1 = 1 and x 0 = 0.

26 Distributional Analysis Combined test statistic: Z comb k,j = k k =1 λk Z k,j Weights of combination λ k sum to 1 Best weights λ k = w k /s k proportional to w k Approximate distrib: Z comb N(0, 1) k,j (u j C/R) 2 log B 1 νx, maximized shift of indep 1 {j sent} + Z comb k,j evaluated at x = x k 1 and u jl = u(l/l) = e 2C(l 1)/L Expected weighted fraction of terms sent above threshold g(x) = u π j Φ τ j C/R 1 νx 1 j sent called the performance update function

27 Lower Bounding Correct Detections Correct Detections Increment: ˆq k = j J k sent π j 1 Hk,j Total: ˆq 1,k = ˆq 1 + ˆq ˆq k Equivalent: j sent π j 1 H1,j H 2,j... H k,j Lower Bound: j sent π j 1 Hk,j LB Expectation: q1,k Reliability: ˆq 1,k > q 1,k with high prob for q 1,k < q1,k

28 Lower Bounding Correct Detections Correct Detections Increment: ˆq k = j J k sent π j 1 Hk,j Total: ˆq 1,k = ˆq 1 + ˆq ˆq k Equivalent: j sent π j 1 H1,j H 2,j... H k,j Lower Bound: j sent π j 1 Hk,j Expectation: q 1,k Reliability: ˆq 1,k > q 1,k with high prob for q 1,k = q1,k η Recursive Evaluation: q1,k = g( q adj ) 1,k 1 g(x) shown to exceed x by at least const/ log B for R R B g(x) evaluated at x k 1 = q adj 1,k 1 yields x k likely lower bound on correct detections reaches x k 1 const/ log B in order log B steps

29 Upper Bounding False Alarms False Alarms Increment: ˆfk = j J k (not sent) π j 1 Hk,j Upper bound: j not sent π j 1 Hk,j Expectation: f Target level: f less than const/(log B) 2 Total: ˆf1,k = ˆf 1 + ˆf ˆf k UB expectation: kf Reliability: ˆf1,k less than kf with high prob for f > f Total bound: const/ log B with #steps k of order log B

30 Decoding Progression g L(x) x B = 2 16, L = B snr = 15 C = 2 bits R = 1.04 bits (0.52C) No. of steps = Figure: Plot of likely progression of weighted fraction of correct detections ˆq 1,k, for snr = 15. x

31 Decoding progression, example bounds g L(x) x B = 2 16, L = B snr = 15 C = 2 bits R = 1.04 bits (0.52C) No. of steps = Figure: Plot of g(x) and the sequence x k for snr = 15, with variable power allocation. The threshold uses a = The final false alarm and failed detection rates are less than and respectively, with probability of at least that fraction of mistakes less than x

32 Decoding Progression g L(x) x B = 2 16, L = B snr = 1 C = 0.5 bits R = 0.31 bits (0.62C) No. of steps = Figure: Plot of g(x) and the sequence x k for snr = 1, with constant power allocation. The threshold uses a = The final false alarm and failed detection rates are and respectively, with probability bound x

33 Summary Sparse superposition coding is fast and reliable at rates up to channel capacity Formulation and analysis blends modern statistical regression and information theory

34 Rate versus Section Size R (bits/ch. use) detailed envelope curve when L = B curve using simulation runs capacity snr = Figure: Rate as a function of B for snr =15 and error probability B

35 Rate versus Section Size R (bits/ch. use) detailed envelope curve when L = B curve using simulation runs capacity snr = Figure: Rate as a function of B for snr = 1 and error probability B