Error-Correcting Codes: Progress & Challenges Madhu Sudan Microsoft/MIT
Communication in presence of noise We are not ready Sender Noisy Channel We are now ready Receiver If information is digital, reliability is critical
Shannon s Model: Probabilistic Noise Sender Receiver Encode (expand) Noisy Channel Decode (compress?) E:Σ k Σ n D:Σ n Σ k Probabilistic Noise: E.g., every letter flipped to random other letter of Σ w.p. p Focus: Design good Encode/Decode algorithms.
Hamming Model: Worst-case error Errors: Upto t worst-case errors Focus: Code: C = Image(E) = {E(x) x Є Σ k } (Note: Not encoding/decoding) Goal: Design code to correct every possible pattern of t errors.
Problems in Coding Theory, Broadly Combinatorics: Design best possible errorcorrecting codes. Probability/Algorithms: Design algorithms correcting random/worst-case errors.
Part I (of III): Combinatorial Results
Hamming Notions Hamming Distance: (x,y) = {i x i y i } Distance of Code: (C) = min x,y 2 C { (x,y)} Code of distance 2t+1 corrects t errors. Main question: Four parameters: Length n, message length k, distance d, alphabet q = Σ. - How do they relate? - Want + n, " k, " d,? q Let: R = k/n; δ= d/n; How do R, δ, q relate?
Simple results Ball(x,r) = {y \in Σ n Δ(x,y) r} Volume of Ball:Vol(q,n,r) = Ball(x,r) Entropy function: H q (δ) = c s.t. Vol(q,n, δn) ¼ q cn Hamming (Packing) Bound: Balls of radius δn/2 around codewords are disjoint. q k q H q(δ/2)n q n R + H q (δ/2) 1
Simple results (contd.) Gilbert-Varshamov (Greedy) Bound: Let C:Σ k Σ n be maximal code of distance d. Then balls of radius d-1 around codewords cover Σ n So q k q H q(δn) q n Or R 1 Hq(δ)
Simple results (Summary) For the best code: 1 H q (δ) R 1 H q (δ/2) Which is right? After fifty years of research We still don t know.
Binary case (q =2): Case of large distance: δ = ½ - ², ² 0. Ω(² 2 ) R O * (² 2 ) GV/Cherno LP Bound Case of small (relative) distance: No bound better than R 1 (1-o(1)) H(δ/2) Hamming Case of constant distance d: (d/2) log n n-k (1-o(1)). (d/2) \log n BCH Hamming
Binary case (Closer look): For general n,d: # Codewords 2 n / Vol (2,n, d-1) Can we do better? Twice as many codewords? (won t change asymptotics of R, δ ) Recent progress [Jiang-Vardy]: # Codewords d 2 n / Vol(2,n,d-1)
Major questions in binary codes: Give explicit construction meeting GV bound. Specifically: Codes with δ = ½ - ² & R = Ω(² 2 ) Is Hamming tight when δ 0? Do there exist codes of distance δ with R = 1 [ c (1 o(1)) δ log 2 (1/δ) ] for c < 1? [Hamming: c > ½ ] Is LP Bound tight?
Combinatorics (contd.): q-ary case Fix δ and let q 1 (then fix q and let n 1) 1 δ O(1/log q) R 1 δ 1/q GV bound Plotkin Surprising result ( 80s): Algebraic Geometry yields: R 1 δ 1/( q 1) (Also a negative surprise: BCH codes only yield 1 R (q-1)/q log q n) Not Hamming
Major questions: q-ary case Suppose R = 1 δ f(q) What is the fastest decaying function f(.)? (somewhere between 1/ q and 1/q). Give a simple explanation for why f(q) 1/ q Fix d, and let q 1 How does (n-k)/(d log q n) grow in the limit? Is it 1 or ½? Or somewhere in between?
Part II (of III): Correcting Random Errors
Recall Shannon 1948 Σ-symmetric channel w. error prob. p: Transmits σ 2 Σ as σ w.p. 1-p; Shannon s Coding Theorem: and as 2 Σ- {σ} w.p. p/(q-1). Can transmit at rate R = 1 H q (p) - ², 8 ² > 0 If R = 1 H q (p) - ², then for every n and k = Rn, there exist E:Σ k Σ n and D:Σ n Σ k s.t. Pr Channel,x [D(Channel(E(x)) x] exp(-n). Converse Coding Theorem: Can not transmit at rate R = 1 H q (p) + ² So: No mysteries?
Constructive versions Shannon s functions: E random, D brute force search. Can we get poly time E, D? [Forney 66]: Yes! (Using Reed-Solomon codes correcting ²-fraction error + composition.) [Sipser-Spielman 92, Spielman 94, Barg- Zemor 97]: Even in linear time! Still didn t satisfy practical needs. Why? [Berrou et al. 92] Turbo codes + belief propagation: No theorems; Much excitement
What is satisfaction? Articulated by [Luby,Mitzenmacher,Shokrollahi,Spielman 96] Practically interesting question: n = 10000; q = 2, p =.1; Desired error prob. = 10-6 ; k =? [Forney 66]: Decoding time: exp(1/(1 H(p) (k/n))); Rate = 90% ) decoding time 2 100; Right question: reduce decoding time to poly(n,1/ ²); where ² = 1 H(p) (k/n)
Current state of the art Luby et al.: Propose study of codes based on irregular graphs ( Irregular LDPC Codes ). No theorems so far for erroneous channels. Strong analysis for (much) simpler case of erasure channels (symbols are erased); decoding time = O(n log (1/²)) (Easy to get composition based algorithms with decoding time = O(n poly(1/²)) Do have some proposals for errors as well (with analysis by Luby et al., Richardson & Urbanke), but none known to converge to Shannon limit.
Part III: Correcting Adversarial Errors
Motivation: As notions of communication/storage get more complex, modeling error as oblivious (to message/encoding/decoding) may be too simplistic. Need more general models of error + encoding/decoding for such models. Most pessimistic model: errors are worst-case.
Gap between worst-case & random errors In Shannon model, with binary channel: Can correct upto 50% (random) errors. ( 1-1/q fraction errors, if channel q-ary.) In Hamming model, for binary channel: Code with more than n codewords has distance at most 50%. So it corrects at most 25% worst-case errors. ( ½(1 1/q) errors in q-ary case.) Shannon model corrects twice as many errors: Need new approaches to bridge gap.
Approach: List-decoding Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. List-decoding: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. Code is (p,l) list-decodable if it corrects p fraction error with lists of size L.
List-decoding Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. List-decoding [Elias 57, Wozencraft 58]: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. Code is (p,l) list-decodable if it corrects p fraction error with lists of size L.
What to do with list? Probabilistic error: List has size one w.p. nearly 1 General channel: Need side information of only O(log n) bits to disambiguate [Guruswami 03] (Alt ly if sender and receiver share O(log n) bits, then they can disambiguate [Langberg 04]). Computationally bounded error: Model introduced by [Lipton, Ding Gopalan L.] List-decoding results can be extended (assuming PKI and some memory at sender) [Micali et al.]
List-decoding: State of the art [Zyablov-Pinsker/Blinovskii late 80s] There exist codes of rate 1 H q (p) - \epsilon that are (p,o(1))-list-decodable. Matches Shannon s converse perfectly! (So can t do better even for random error!) But [ZP/B] non-constructive!
Algorithms for List-decoding Not examined till 88. First results: [Goldreich-Levin] for Hadamard codes (non-trivial in their setting). More recent work: [S. 96, Shokrollahi-Wasserman 98, Guruswami-S. 99, Parvaresh-Vardy 05, Guruswami-Rudra 06] Decode algebraic codes. [Guruswami-Indyk 00-02] Decode graphtheoretic codes. 02/17/2010 ECC: Progress/Challenges (@CMU)
Results in List-decoding q-ary case: [Guruswami-Rudra 06] Codes of rate R correcting 1 R - ² fraction errors with q = q(²) Matches Shannon bound (except for q(²) ) 9 Codes of rate ²c correcting 1 ² fraction errors. 2 c = 4: Guruswami et al. 2000 c! 3: Implied by Parvaresh-Vardy 05 c = 3: Guruswami Rudra
Major open question ² Construct (p; O(1)) list-decodable binary code of rate 1 H(p) ² with polytime list decoding.. ² Note: If running time is poly(1=²) then this implies a solution to the random error problem as well.
Conclusions Coding theory: Very practically motivated problems; solutions influence (if not directly alter) practice. Many mysteries remain in combinatorial setting. Significant progress in algorithmic setting, but many more questions to resolve.