Sparse Regression Codes for Multi-terminal Source and Channel Coding

Size: px

Start display at page:

Download "Sparse Regression Codes for Multi-terminal Source and Channel Coding"

Georgia Newman
5 years ago
Views:

1 Sparse Regression Codes for Multi-terminal Source and Channel Coding Ramji Venkataramanan Yale University Sekhar Tatikonda Allerton / 20

2 Compression with Side-Information X Encoder Rate R Decoder ˆX Y The Wyner-Ziv Problem Compress X with side-info Y known only to decoder Applications in video coding & peer-to-peer file sharing 2 / 20

3 Communication with Channel State S M X Z Encoder + Decoder ˆM Noise Z = X + S + Noise The Gelfand-Pinsker problem Channel state S known only to the encoder - Gaussian case [Costa 83]: Writing on Dirty Paper Applications in - Wireless communication (MIMO broadcast) - Digital watermarking 3 / 20

4 Other multi-terminal networks Multiple-access Broadcast Noise M 1 X 1 Noise + Z 1 ˆM 1 M2 X 2 + Z ˆM 1 ˆM2 X + Noise Z 2 ˆM2 ˆM 3 M 3 X 3 + Z 3 ˆM 3 Noise 4 / 20

5 Achieving the Optimal Rates Textbook Code Constructions Random codes for point-to-point source & channel coding Random Binning Superposition 5 / 20

6 Achieving the Optimal Rates Textbook Code Constructions Random codes for point-to-point source & channel coding Random Binning Superposition Rate-optimal but... High Storage and Coding Complexity - exponential in block length n Lattice constructions: compact representation - Eyboglu-Forney, Zamir-Shamai-Erez,... 5 / 20

7 Achieving the Optimal Rates Textbook Code Constructions Random codes for point-to-point source & channel coding Random Binning Superposition Rate-optimal but... High Storage and Coding Complexity - exponential in block length n Lattice constructions: compact representation - Eyboglu-Forney, Zamir-Shamai-Erez,... GOAL: Compact representation + fast encoding/decoding Fast polynomial in n 5 / 20

8 In this talk... Ensemble of codes based on sparse linear regression - For point-to-point & multi-terminal problems Provably achieve rates close to info-theoretic limits - with fast encoding + decoding Based on construction of Barron & Joseph for AWGN channel - Achieve capacity with fast decoding [ISIT 10, Arxiv 12] 6 / 20

9 Sparse Regression Codes for point-to-point Source and Channel coding 7 / 20

10 Source Coding R bits/sample Codebook 2 nr S = S 1,..., S n Ŝ = Ŝ1,..., Ŝn 1 Distortion criterion: n S Ŝ 2 = 1 n k (S k Ŝk) 2 For i.i.d N (0, σ 2 ) source, min distortion = σ 2 e 2R 8 / 20

11 Source Coding R bits/sample Codebook 2 nr S = S 1,..., S n Ŝ = Ŝ1,..., Ŝn 1 Distortion criterion: n S Ŝ 2 = 1 n k (S k Ŝk) 2 For i.i.d N (0, σ 2 ) source, min distortion = σ 2 e 2R Can we achieve this with low-complexity algorithms? - Storage & Computation 8 / 20

12 Sparse Regression Codes (SPARC) A: n rows β: 0, 0, c 1, 0, c 2, 0, c L, 0,, 0 T A: design matrix or dictionary with ind. N (0, 1) entries Codewords Aβ - sparse linear combinations of columns of A 9 / 20

13 SPARC Construction Section 1 Section 2 Section L A: n rows β: 0, 0, c 1, 0, c 2, 0, c L, 0,, 0 T n rows, ML columns 10 / 20

14 SPARC Construction Section 1 Section 2 Section L A: n rows β: 0, 0, c 1, 0, c 2, 0, c L, 0,, 0 T Choosing M and L: n rows, ML columns For rate R codebook, need M L = 2 nr 10 / 20

15 SPARC Construction Section 1 Section 2 Section L A: n rows β: 0, 0, c 1, 0, c 2, 0, c L, 0,, 0 T n rows, ML columns Choosing M and L: For rate R codebook, need M L = 2 nr Choose M polynomial of n L n/log n Storage Complexity Size of A: polynomial in n 10 / 20

16 A Simple Encoding Algorithm A: Section 1 n rows β: 0, 0, c 1, T Step 1: Choose column in Sec.1 that minimizes X c 1 A j 2 - Max among inner products X, A j 11 / 20

17 A Simple Encoding Algorithm A: Section 1 n rows β: 0, 0, c 1, T Step 1: Choose column in Sec.1 that minimizes X c 1 A j 2 - Max among inner products X, A j - Residue R 1 = X c 1 Â 1 11 / 20

18 A Simple Encoding Algorithm A: Section 2 n rows β: 0, c 2, 0, T Step 2: Choose column in Sec.2 that minimizes R 1 c 2 A j 2 - Max among inner products R 1, A j - Residue R 2 = R 1 c 2 Â 2 11 / 20

19 A Simple Encoding Algorithm A: Section L n rows β: c L, 0,, 0 T Step L: Choose column in Sec.L that minimizes R L 1 c L A j 2 - Max among inner products R L 1, A j - Final residue R L = R L 1 c L Â L 11 / 20

20 Performance Theorem (RV, Sarkar, Tatikonda 12) The proposed encoding algorithm approaches the rate-distortion function with exponentially small probability of error. In particular, ( ) P Distortion > σ 2 e 2R + e L for 1 log M. Computation Complexity ML inner products and comparisons polynomial in n Storage Complexity Design matrix A: n ML polynomial in n 12 / 20

21 Performance Theorem (RV, Sarkar, Tatikonda 12) The proposed encoding algorithm approaches the rate-distortion function with exponentially small probability of error. In particular, ( ) P Distortion > σ 2 e 2R + e L for 1 log M. Theorem [RV, Joseph, Tatikonda, ISIT 12] With ML encoding, SPARCs attain the rate-distortion function with the optimal error-exponent for all D < D / 20

22 Point-to-point Communication Noise M X + Z ˆM Z = X + Noise, X 2 n P, Noise Normal(0, N) SPARCs Provably good with low-complexity decoding - [Barron-Joseph, ISIT 10, 11, Arxiv 12] 13 / 20

23 SPARC Construction Section 1 Section 2 Section L A: n rows β: 0, 0, c 1, 0, c 2, 0, c L, 0,, 0 T n rows, ML columns β message, Codeword Aβ For rate R codebook, need M L = 2 nr - choose M polynomial of n L n/log n Adaptive successive decoding achieves R < Capacity 14 / 20

24 Sparse Regression Codes for multi-terminal networks 15 / 20

25 Wyner-Ziv coding X Encoder R Decoder ˆX Y Side-info Y = X + Z X N (0, σ 2 ), Z N (0, N) 16 / 20

26 Wyner-Ziv coding X Encoder R Decoder ˆX U Y Side-info Y = X + Z 2 nr1 cwds X N (0, σ 2 ), Z N (0, N) Encoder Quantize X to U U = X + V, V N (0, Q) - Find U that minimizes X au 2, a = σ2 σ 2 +Q 16 / 20

27 Wyner-Ziv coding X Encoder R Decoder ˆX U 2 nr bins Y Side-info Y = X + Z 2 nr1 cwds X N (0, σ 2 ), Z N (0, N) Encoder Quantize X to U U = X + V, V N (0, Q) - Find U that minimizes X au 2, a = σ2 σ 2 +Q 16 / 20

28 Wyner-Ziv coding X Encoder R Decoder ˆX U 2 nr bins Y Side-info Y = X + Z 2 nr1 cwds X N (0, σ 2 ), Z N (0, N) Decoder Y = X + Z Y = au + Z Find U within bin that minimizes Y au 2 - Reconstruct ˆX = E [ X UY] 16 / 20

29 Binning with SPARCs A: Section 1 Section 2 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Quantize X to au using n ML SPARC (rate R 1 ) 17 / 20

30 Binning with SPARCs M A: Section 1 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Quantize X to au using n ML SPARC (rate R 1 ) (M/M ) L = 2 nr 17 / 20

31 Binning with SPARCs M A: Section 1 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Quantize X to au using n ML SPARC (rate R 1 ) (M/M ) L = 2 nr Bin: defined by 1 subsection from each section - Encoder only sends indices of non-zero subsections Decodes Y to U within smaller n M L SPARC 17 / 20

32 Writing on Dirty Paper S M X Z Encoder + Decoder ˆM Noise Z = X + S + N, X 2 n P 18 / 20

33 Writing on Dirty Paper Z = X + S + N, X 2 n P A: Section 1 Section 2 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Encoder n ML SPARC of rate R 1 Divide each section into M subsections - Defines (M/M ) L = 2 nr bins 18 / 20

34 Writing on Dirty Paper Z = X + S + N, X 2 n P M A: Section 1 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Encoder n ML SPARC of rate R 1 Divide each section into M subsections - Defines (M/M ) L = 2 nr bins 18 / 20

35 Writing on Dirty Paper Z = X + S + N, X 2 n P M A: Section 1 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Encoder Within message bin quantize S to U U = X + αs, U N (0, P + α 2 σ 2 s ) 18 / 20

36 Writing on Dirty Paper Z = X + S + N, X 2 n P M A: Section 1 Section L β: 0, 0, c 1,, c 2, 0, c L, 0,, 0 T Decoder Z = X + S + N Z = (1 + κ)u + N Decode U from Z the big (rate R 1 ) codebook 18 / 20

37 Main Result X Encoder Rate R Decoder ˆX Y S M X Y Encoder + Decoder ˆM Noise Theorem SPARCs attain the optimal information-theoretic limits for the Gaussian Wyner-Ziv and Gelfand-Pinsker problems with exponentially decaying probability of error. 19 / 20

38 Summary Sparse Regression Codes Rate-optimal codes for compression and communication Low-complexity coding algorithms Nice structure that enables - Binning (Wyner-Ziv, Gelfand-Pinsker) - Superposition (Multiple-access, Broadcast) Future Directions Interference channels, Multiple descriptions,... Improved coding algorithms - l 1 minimization etc.? General design matrices Finite-field analogs? 20 / 20

39 21 / 20

40 Simulation Gaussian source: Mean 0, Variance SPARC 0.7 Distortion Scalar Q Shannon Limit Rate (bits) 22 / 20

Generalized Writing on Dirty Paper

Generalized Writing on Dirty Paper Aaron S. Cohen acohen@mit.edu MIT, 36-689 77 Massachusetts Ave. Cambridge, MA 02139-4307 Amos Lapidoth lapidoth@isi.ee.ethz.ch ETF E107 ETH-Zentrum CH-8092 Zürich, Switzerland