An Introduction to Algorithmic Coding Theory
|
|
- Clifford Kelly
- 6 years ago
- Views:
Transcription
1 An Introduction to Algorithmic Coding Theory M. Amin Shokrollahi Bell Laboratories
2 Part : Codes -
3 A puzzle What do the following problems have in common? 2
4 Problem : Information Transmission MESSAGE G W D A S U N C L R V J K F T X M I H Z B Y E MASSAGE?? 3
5 Problem 2: Football Pool Problem Bayern München :. FC Kaiserslautern Homburg : Hannover München : FC Karlsruhe 2. FC Köln : Hertha BSC Wolfsburg : Stuttgart Bremen : Unterhaching 2 Frankfurt : Rostock Bielefeld : Duisburg Feiburg : Schalke 4 2 4
6 Problem 3: In-Memory Database Systems Database Corrupted! User memory malloc malloc malloc malloc Code memory 5
7 Problem 4: Bulk Data Distribution 6
8 What do they have in common? They all can be solved using (algorithmic) coding theory 7
9 Basic Idea of Coding Adding redundancy to be able to correct! Objectives: MESSAGE Add as little redundancy as possible; correct as many errors as possible. MESSAGE MESSAGE MASSAGE 8
10 Codes A code of block-length n over the alphabet GF(q) is a set of vectors in GF(q) n. If the set forms a vector space over GF(q), then the code is called linear. [n,k] q -code 9
11 ??? Encoding Problem source??? Codewords Efficiency!
12 Linear Codes: Generator Matrix Generator matrix source codeword k n k x n O(n 2 )
13 Linear Codes: Parity Check Matrix Parity check matrix codeword O(n 2 ) after O(n 3 ) preprocessing. 2
14 The Decoding Problem: Maximum Likelihood Decoding a b c d a b c d a b c d a b c d Received: a x c d a b z d a b c d. a x c d a b z d a b c d a x c d a x c d a x c d 3 a b z d a b z d a b z d 3 a b c d a b c d a b c d 2 3 3
15 Hamming Distance Hamming distance between two vectors of equal dimension is number of positions at which they differ. [n,k,d] q -code 4
16 Maximum Likelihood Decoding Given received word, find a codeword that has least Hamming distance to it. Intractable in general. 5
17 Worst Case Error Correction [n,k,d] q -code is capable of correcting up to e = (d )/2 errors in whatever locations! Hamming balls of radius e d e 6
18 Errors in Known Locations: Erasures [n,k,d] q -code is capable of correcting up to d errors in whatever locations if the locations are known. 7
19 Projective plane over GF(2) Minimum distance = 4 8
20 Hamming Code ,4,3] [ -code 2 9
21 A Solution to the Football Match Problem [4,2,3] 3 -Hamming Code with generator matrix: 2 Codewords:
22 A Solution to the Football Match Problem This Hamming code is perfect: Hamming balls of radius fill the space GF(3) 4 : ( ) i = 9 (+4 2) = 8. i i= Any vector in GF(3) 4 has Hamming distance at most to a codeword. Hamming code Hamming code Fixed game 2
23 Bounds How can we prove optimality of codes? (Fix any two of the three parameters n, k, d and maximize the third.). Hamming bound: e i=( n i) (q ) i q n k. Equality: perfect codes 2. Plotkin bound: d+k n+. Equality: MDS codes. 3. Other more refined bounds... 22
24 Perfect Codes Have been completely classified by van Lint and Tieätväinen. Essentially: Hamming codes, Golay codes. Not desirable in communication scenarios. 23
25 MDS Codes Not classified completely. Open problem: Given dimension k and field size q, determine maximum block length of an MDS-code. (Conjecturally q + or q +2.) MDS codes are desirable in practice in worst case scenarios if efficient encoding and decoding available. Prototype: Reed-Solomon codes. 24
26 Reed-Solomon Codes: Applications. Satellite Communication, 2. Hard disks, 3. Compact Discs, Digital Versatile Disks, Digital Audio Tapes, 4. Wireless Communication, 5. Secret sharing,
27 Reed-Solomon Codes: Definitions Choose n different elements x,...,x n in GF(q). Reed-Solomon code is image of the morphism GF(q)[x] <k GF(q) n f (f(x ),...,f(x n )) Block length = n. Dimension? Minimum distance? 26
28 Reed-Solomon Codes: Parameters Theorem. Nonzero polynomial of degree m over a field can have at most n zeros over that field. 27
29 Reed-Solomon Codes: Dimension GF(q)[x] <k GF(q) n f (f(x ),...,f(x n )) Kernel: if k n since nonzero polynomial of degree k has at most k zeros. Dimension: k (if k n). 28
30 Reed-Solomon Codes: Minimum Distance GF(q)[x] <k GF(q) n f (f(x ),...,f(x n )) Minimum distance: Maximal number of zeros in a nonzero codeword is k, since evaluating polynomial of degree k. Minimum distance is n k +, hence equal, hence MDS code!. 29
31 Reed-Solomon Codes: Encoding f (f(x ),...,f(x n )) is easy to compute! O(n 2 ) using naive algorithm. O(nlog 2 (n)loglog(n) using fast algorithms. 3
32 Reed-Solomon Codes: Decoding No efficient maximum likelihood decoding known. Concentrate on bounded distance decoding. Transmitted word Received word Number of agreements (n+k)/2. Number of disagreements (n k)/2. 3
33 Welch-Berlekamp Algorithm Transmitted word: (f(x ),...,f(x n )). Received word: (y,...,y n ). Number of agreements (n+k)/2. Find f! 32
34 Welch-Berlekamp Algorithm Step : Find g(x) GF(q)[x] <(n+k)/2 and h(x) GF(q)[x] (n k)/2, not both zero, such that i =,...,n: g(x i )+y i h(x i ) =. (Solving a system of equations!) Step 2: Then f = g/h! 33
35 Welch-Berlekamp Algorithm: Proof H(x) := g(x) f(x)h(x). Degree of H(x) < (n+k)/2. If y i = f(x i ), then H(x i ) =. H(x) has at least (n+k)/2 zeros. H(x) is zero. f(x) = g(x)/h(x). 34
36 Welch-Berlekamp Algorithm: Running time Step : Solving a homogeneous n (n+) system of equations; O(n 3 ). Can be reduced to O(n 2 ) (Welch-Berlekamp, 983; displacement method (Olshevsky-Shokrollahi, 999)). Step 2: Polynomial division; O(n 2 ). 35
37 Welch-Berlekamp Algorithm: Generalization Has been generalized to more than (n k)/2 errors (list-decoding, Sudan, 997, Guruswami-Sudan, 999). Step 2 requires factorization of bivariate polynomials. Can be done more efficiently (Gao-Shokrollahi 999, Olshevsky-Shokrollahi 999). 36
38 A Solution to the In-Memory Database Problem Database Redundant User memory malloc malloc malloc malloc Code memory 37
39 Reed-Solomon Codes: Generalization Disadvantage of RS-codes: GF(q) must be large to accommodate many points, so long codes impossible. Interpret GF(q) as affine line over itself, and generalize to more complicated algebraic curves. Lead to best known codes in terms of minimum distance, dimension, block-length. Above algorithms can be generalized to these Algebraic-geometric codes. 38
40 Probabilistic Methods 38-
41 Input alphabet Channels Transition probabilities 39 Output alphabet
42 Entropy and Mutual Information X and Y discrete random variables on alphabets X and Y and distributions p(x) and q(x). p(x, y) their joint distribution. Entropy H(X) of X H(X) = x X p(x)log p(x). Mutual information I(X; Y) I(X;Y) = y Y x X p(x,y)log p(x,y) p(x)p(y). 4
43 Entropy and Mutual Information H(X) is the amount of uncertainty of random variable X. I(X;Y) is the reduction in the uncertainty of X due to the knowledge of Y. 4
44 Capacity Capacity of a channel with input alphabet X and output alphabet Y and probability transition matrix p(y x) is C = max p(x) I(X;Y), where maximum is over all possible input distributions p(x). 42
45 Binary Erasure Channel: Examples of Capacity: BEC -p p p E -p Capacity = p 43
46 Binary Symmetric Channel: Examples of Capacity: BSC -p p p -p Capacity = +plog 2 (p)+( p)log 2 ( p) 44
47 Capacity and Communication Shannon s Coding Theorem, 948: C channel with capacity C. For any rate R C there exists a sequence of codes of rate R such that the probability of error of the Maximum Likelihood Decoding for these codes approaches zero as the block-length approaches infinity. The condition R C is necessary and sufficient. 45
48 Problems How to find the sequences of codes? (Random codes, Concatenated codes, ) How to decode efficiently? Has been open for almost 5 years. Low-Density Parity-Check Codes 46
49 46- Part 2: Low-Density Parity-Check Codes
50 Low-Density Parity Check Codes Gallager 963 Zyablov 97 Zyablov-Pinsker 976 Tanner 98 Turbo Codes 993 Berroux-Glavieux-Thitimajshima 47
51 Sipser-Spielman, Spielman 995 MacKay-Neal, MacKay 995 Luby-Mitzenmacher-S-Spielman-Stemann 997 Luby-Mitzenmacher-S-Spielman 998 Richardson-Urbanke 999 Richardson-Shokrollahi-Urbanke
52 Code Construction Codes are constructed from sparse bipartite graphs. 49
53 Code Construction Any binary linear code has a graphical representation. a b a c f! = c b c! d e= d e! a c e= f Not any code can be represented by a sparse graph. 5
54 Parameters n r Rate Rate n-r n - average left degree average right degree 5
55 Dual Construction a b c a b f Source bits d e a b c g c e g h Redundant bits f g b d e f g h h Encoding time is proportional to number of edges. 52
56 Encoding? Algorithmic Issues Is linear time for the dual construction Is quadratic time (after preprocessing) for the Gallager construction. More later! Decoding? Depends on the channel, Depends on the fraction of errors. 53
57 Decoding on a BSC: Flipping satisfied check unsatisfied check 54
58 Decoding on a BSC: Gallager Algorithm A (Message passing) b u x y z m x y m z u m = x b if x=y=z=u else m=x y z u MESSAGE CHECK 55
59 Decoding on a BSC: Belief Propagation b u x y z m hyperbolic transform x y m z u m = x+y+z+u+b m=x * y z u * * ( a,b) * (c,d):=(a+c, b+d mod 2) MESSAGE CHECK Messages in log-likelihood ratios. 56
60 Optimality of Belief Propagation Belief propagation is bit-optimal if graph has no loops. Maximizes the probability P(c m = b y) = c CP(c y). 57
61 Performance on a (3,6)-graph Shannon limit: % Flipping algorithm: %? Gallager A: 4% Gallager B: 4% (6.27%) Erasure decoder: 7% Belief propagation: 8.7% (.8%) 58
62 The Binary Erasure Channel (BEC) -p p p E -p 59
63 Decoding on a BEC: Luby-Mitzenmacher-Shokrollahi-Spielman- Stemann x y z u m x y m z u m = if x y z u = else m = if x = y = z = u = else MESSAGE CHECK 6
64 Decoding on a BEC Phase : Direct recovery a? c?? f g h b b e b d d e 6
65 Decoding on a BEC Phase 2: Substitution a b c?? f g h b d e d e 62
66 Example (a) (b) (c) Complete Recovery (d) (e) (f) 63
67 Have: fast decoding algorithms. The (inverse) problem Want: design codes that can correct many errors using these algorithms. Focus on the BEC in the following. 64
68 Choose regular graphs. Experiments An (d,k)-regular graph has rate at least d/k. Can correct at most an d/k-fraction of erasures. Choose a random (d, k)-graph. p := maximum fraction of erasures the algorithm can correct. What are these numbers? d k d/k p
69 A Theorem Luby, Mitzenmacher, Shokrollahi, Spielman, Stemann, 997: A randomly chosen (d,k)-graph can correct a p -fraction of erasures with high probability if and only if p ( ( x) k ) d < x for x (,p ). 66
70 67
71 Analysis: (3,6)-graphs Expand neighborhoods of message nodes. 68
72 Analysis: (3,6)-graphs p i probability that message node is still erased after ith iteration. p i+ Message Check p i Message p i+ = p ( ( p i ) 5 ) 2. 69
73 Successful Decoding Condition: p ( ( p i ) 5 ) 2 <p i 7
74 Making arguments exact: Analysis: (3,6)-graphs Neighborhood is tree-like: high probability, standard argument. Above argument works for expected fraction of erasures at lth round. Real value is sharply concentrated around expected value p l : Edge exposure martingale, Azuma s inequality. 7
75 The General Case Let λ i and ρ i be the fraction of edges of degree i on the left and the right hand side, respectively. Let λ(x) := i λ ix i and ρ(x) := i ρ ix i. Condition for successful decoding for erasure probability p is then for all x (,p ). p λ( ρ( x)) < x 72
76 Richardson-Urbanke, 999: Belief propagation f l : density of the probability distribution of the messages passed from the check nodes to the message nodes at round l of the algorithm. P : density of the error distribution (in log-likelihood representation). Consider (d, k) regular graph. Γ(f l+ ) = ( ( )) (d ) (k ) Γ P f l, where Γ is a hyperbolic change of measure function, and denotes convolution. Γ(f)(y) := f(lncothy/2)/sinh(y), 73
77 We want f l to converge to a Delta function at. Gives rise to high-dimensional optimization algorithms. 74
78 Achieving capacity Want to design codes that can recover from a fraction of R of erasures (asymptotically). Want to have λ and ρ so that p λ( ρ( x)) < x for all x (,p ), and p arbitrarily close to R = ρ(x)dx. λ(x)dx 75
79 Tornado codes Extremely irregular graphs provide for any rate R sequences of codes which come arbitrarily close to the capacity of the erasure channel! Degree structure? Choose design parameter D. λ(x) := H(D) (x+ x xd D ) ρ(x) := exp(µ(x )), where H(D) = +/2+ +/D and µ = H(D)/( /(D +)). 76
80 Tornado Codes: Left Degree Distribution 77
81 Right regular codes Shokrollahi, 999: Graphs that are regular on the right. Degrees on the left are related to the Taylor expansion of ( x) /m. These are the only known examples of LDPC codes that achieve capacity on a nontrivial channel using a linear time decoding algorithm. 78
82 Other channels? f density function. λ(f) := i λ if (i ). ρ(f) := i ρ if (i ). Want P such that f l. Γ(f l+ ) = ρ(γ(p λ(f l ))). 79
83 Conditions on the density functions Richardson-Shokrollahi-Urbanke, 999: Consistency: if the channel is symmetric, then the density functions f l satisfy f(x) = f( x)e x. Fixed point theorem: If P err (f i ) = P err (f j ) for i < j, then f i = f j is a fixed point of the iteration. 8
84 Conditions on the density functions Stability: let r := lim n n logp err(p n ). Then for λ 2 ρ () > e r we have P err (f l ) > ǫ for some fixed ǫ and all l. If λ 2 ρ () < e r, then the fixed point is stable. is the error probability. P err (f) := f(x)dx 8
85 Stability Erasure channel with erasure probability p : λ 2 ρ () p. BSC channel: with probability p: λ 2 ρ () AWGN channel: with variance σ 2 : 2 p( p). λ 2 ρ () e 2σ 2. 82
86 Shokrollahi, 999: Stability for the Erasure Channel stable not stable p λ( ρ( x ))- x p λ( ρ( x)) - x 83
87 Flatness: Higher Stability Conditions Shokrollahi, 2: (λ m (x),ρ m (x)) capacity achieving sequence of degree distributions. Then: ( R)λ m ( ρ m ( x)) x converges uniformly to the zero-function on the interval [, R]. No equivalent known for other channels. 84
88 Flatness: Higher Stability Conditions p λ( ρ( x)) x 85
89 Capacity achieving No sequences of c.a. degree distributions for channels other than the erasure channel known. Conjecture: They exist! irregular LDPCC; n= 6 Turbo Code; n= 6-5 Shannon Limit Threshold Threshold (3, 6)-regular LDPCC; n= E b /N [db] σ P b 86
90 Applications to computer networks Distribution of bulk data to a large number of clients. Want fully reliable, low network overhead, support vast number of receivers with heterogeneous characteristics users want to access data at times of their choosing and these access times overlap. 87
91 A Solution Broadcast Server 88
92 A Solution Client joins multicast group until enough of the encoding has been received, and then decodes to obtain original data. Amount of encoding received Time Digital Fountain, 89
93 Open problems Asymptotic theory. Classification of capacity achieving sequences for the erasure channel. 2. Capacity achieving sequences for other channels. 3. Exponentially small error probabilities for the decoder (instead of polynomially small). Explicit constructions. Constructions using finite geometries. 2. Construction using Reed-Solomon-Codes. 3. Algebraic constructions. Graphs with loops. Short codes 9
94 Algorithmic issues. Design and analysis of new decoding algorithms. 2. Design of new encoders. Packet based wireless networks. Applications Randomness Use of randomness in other areas: random convolutional codes?. 9
EE229B - Final Project. Capacity-Approaching Low-Density Parity-Check Codes
EE229B - Final Project Capacity-Approaching Low-Density Parity-Check Codes Pierre Garrigues EECS department, UC Berkeley garrigue@eecs.berkeley.edu May 13, 2005 Abstract The class of low-density parity-check
More informationECEN 655: Advanced Channel Coding
ECEN 655: Advanced Channel Coding Course Introduction Henry D. Pfister Department of Electrical and Computer Engineering Texas A&M University ECEN 655: Advanced Channel Coding 1 / 19 Outline 1 History
More informationLow-density parity-check codes
Low-density parity-check codes From principles to practice Dr. Steve Weller steven.weller@newcastle.edu.au School of Electrical Engineering and Computer Science The University of Newcastle, Callaghan,
More informationIntroduction to Low-Density Parity Check Codes. Brian Kurkoski
Introduction to Low-Density Parity Check Codes Brian Kurkoski kurkoski@ice.uec.ac.jp Outline: Low Density Parity Check Codes Review block codes History Low Density Parity Check Codes Gallager s LDPC code
More informationAn Introduction to Low Density Parity Check (LDPC) Codes
An Introduction to Low Density Parity Check (LDPC) Codes Jian Sun jian@csee.wvu.edu Wireless Communication Research Laboratory Lane Dept. of Comp. Sci. and Elec. Engr. West Virginia University June 3,
More informationBelief-Propagation Decoding of LDPC Codes
LDPC Codes: Motivation Belief-Propagation Decoding of LDPC Codes Amir Bennatan, Princeton University Revolution in coding theory Reliable transmission, rates approaching capacity. BIAWGN, Rate =.5, Threshold.45
More informationPerformance Analysis and Code Optimization of Low Density Parity-Check Codes on Rayleigh Fading Channels
Performance Analysis and Code Optimization of Low Density Parity-Check Codes on Rayleigh Fading Channels Jilei Hou, Paul H. Siegel and Laurence B. Milstein Department of Electrical and Computer Engineering
More informationCodes on graphs and iterative decoding
Codes on graphs and iterative decoding Bane Vasić Error Correction Coding Laboratory University of Arizona Funded by: National Science Foundation (NSF) Seagate Technology Defense Advanced Research Projects
More informationCoding Techniques for Data Storage Systems
Coding Techniques for Data Storage Systems Thomas Mittelholzer IBM Zurich Research Laboratory /8 Göttingen Agenda. Channel Coding and Practical Coding Constraints. Linear Codes 3. Weight Enumerators and
More informationIterative Encoding of Low-Density Parity-Check Codes
Iterative Encoding of Low-Density Parity-Check Codes David Haley, Alex Grant and John Buetefuer Institute for Telecommunications Research University of South Australia Mawson Lakes Blvd Mawson Lakes SA
More informationOn Generalized EXIT Charts of LDPC Code Ensembles over Binary-Input Output-Symmetric Memoryless Channels
2012 IEEE International Symposium on Information Theory Proceedings On Generalied EXIT Charts of LDPC Code Ensembles over Binary-Input Output-Symmetric Memoryless Channels H Mamani 1, H Saeedi 1, A Eslami
More informationCodes on graphs and iterative decoding
Codes on graphs and iterative decoding Bane Vasić Error Correction Coding Laboratory University of Arizona Prelude Information transmission 0 0 0 0 0 0 Channel Information transmission signal 0 0 threshold
More informationError-Correcting Codes:
Error-Correcting Codes: Progress & Challenges Madhu Sudan Microsoft/MIT Communication in presence of noise We are not ready Sender Noisy Channel We are now ready Receiver If information is digital, reliability
More informationCS6304 / Analog and Digital Communication UNIT IV - SOURCE AND ERROR CONTROL CODING PART A 1. What is the use of error control coding? The main use of error control coding is to reduce the overall probability
More informationBounds on Achievable Rates of LDPC Codes Used Over the Binary Erasure Channel
Bounds on Achievable Rates of LDPC Codes Used Over the Binary Erasure Channel Ohad Barak, David Burshtein and Meir Feder School of Electrical Engineering Tel-Aviv University Tel-Aviv 69978, Israel Abstract
More informationSlepian-Wolf Code Design via Source-Channel Correspondence
Slepian-Wolf Code Design via Source-Channel Correspondence Jun Chen University of Illinois at Urbana-Champaign Urbana, IL 61801, USA Email: junchen@ifpuiucedu Dake He IBM T J Watson Research Center Yorktown
More information5. Density evolution. Density evolution 5-1
5. Density evolution Density evolution 5-1 Probabilistic analysis of message passing algorithms variable nodes factor nodes x1 a x i x2 a(x i ; x j ; x k ) x3 b x4 consider factor graph model G = (V ;
More informationLow-Density Parity-Check Codes
Department of Computer Sciences Applied Algorithms Lab. July 24, 2011 Outline 1 Introduction 2 Algorithms for LDPC 3 Properties 4 Iterative Learning in Crowds 5 Algorithm 6 Results 7 Conclusion PART I
More informationBifurcations in iterative decoding and root locus plots
Published in IET Control Theory and Applications Received on 12th March 2008 Revised on 26th August 2008 ISSN 1751-8644 Bifurcations in iterative decoding and root locus plots C.M. Kellett S.R. Weller
More informationLower Bounds on the Graphical Complexity of Finite-Length LDPC Codes
Lower Bounds on the Graphical Complexity of Finite-Length LDPC Codes Igal Sason Department of Electrical Engineering Technion - Israel Institute of Technology Haifa 32000, Israel 2009 IEEE International
More informationLecture 4 : Introduction to Low-density Parity-check Codes
Lecture 4 : Introduction to Low-density Parity-check Codes LDPC codes are a class of linear block codes with implementable decoders, which provide near-capacity performance. History: 1. LDPC codes were
More informationIntroducing Low-Density Parity-Check Codes
Introducing Low-Density Parity-Check Codes Sarah J. Johnson School of Electrical Engineering and Computer Science The University of Newcastle Australia email: sarah.johnson@newcastle.edu.au Topic 1: Low-Density
More informationAn Introduction to Low-Density Parity-Check Codes
An Introduction to Low-Density Parity-Check Codes Paul H. Siegel Electrical and Computer Engineering University of California, San Diego 5/ 3/ 7 Copyright 27 by Paul H. Siegel Outline Shannon s Channel
More informationJoint Iterative Decoding of LDPC Codes and Channels with Memory
Joint Iterative Decoding of LDPC Codes and Channels with Memory Henry D. Pfister and Paul H. Siegel University of California, San Diego 3 rd Int l Symposium on Turbo Codes September 1, 2003 Outline Channels
More informationGraph-based codes for flash memory
1/28 Graph-based codes for flash memory Discrete Mathematics Seminar September 3, 2013 Katie Haymaker Joint work with Professor Christine Kelley University of Nebraska-Lincoln 2/28 Outline 1 Background
More informationOptimal Rate and Maximum Erasure Probability LDPC Codes in Binary Erasure Channel
Optimal Rate and Maximum Erasure Probability LDPC Codes in Binary Erasure Channel H. Tavakoli Electrical Engineering Department K.N. Toosi University of Technology, Tehran, Iran tavakoli@ee.kntu.ac.ir
More informationBounds on the Performance of Belief Propagation Decoding
Bounds on the Performance of Belief Propagation Decoding David Burshtein and Gadi Miller Dept. of Electrical Engineering Systems Tel-Aviv University Tel-Aviv 69978, Israel Email: burstyn@eng.tau.ac.il,
More informationOn the Typicality of the Linear Code Among the LDPC Coset Code Ensemble
5 Conference on Information Sciences and Systems The Johns Hopkins University March 16 18 5 On the Typicality of the Linear Code Among the LDPC Coset Code Ensemble C.-C. Wang S.R. Kulkarni and H.V. Poor
More informationIterative Decoding for Wireless Networks
Iterative Decoding for Wireless Networks Thesis by Ravi Palanki In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy California Institute of Technology Pasadena, California
More informationLDPC Codes. Intracom Telecom, Peania
LDPC Codes Alexios Balatsoukas-Stimming and Athanasios P. Liavas Technical University of Crete Dept. of Electronic and Computer Engineering Telecommunications Laboratory December 16, 2011 Intracom Telecom,
More informationTHE seminal paper of Gallager [1, p. 48] suggested to evaluate
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 50, NO. 11, NOVEMBER 2004 2657 Extrinsic Information Transfer Functions: Model and Erasure Channel Properties Alexei Ashikhmin, Member, IEEE, Gerhard Kramer,
More informationBelief propagation decoding of quantum channels by passing quantum messages
Belief propagation decoding of quantum channels by passing quantum messages arxiv:67.4833 QIP 27 Joseph M. Renes lempelziv@flickr To do research in quantum information theory, pick a favorite text on classical
More informationAn Efficient Maximum Likelihood Decoding of LDPC Codes Over the Binary Erasure Channel
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 5, NO. 11, NOVEMBER 24 1 An Efficient Maximum Likelihood Decoding of LDPC Codes Over the Binary Erasure Channel David Burshtein and Gadi Miller School of Electrical
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationCapacity-Achieving Ensembles for the Binary Erasure Channel With Bounded Complexity
Capacity-Achieving Ensembles for the Binary Erasure Channel With Bounded Complexity Henry D. Pfister, Member, Igal Sason, Member, and Rüdiger Urbanke Abstract We present two sequences of ensembles of non-systematic
More informationCapacity-approaching codes
Chapter 13 Capacity-approaching codes We have previously discussed codes on graphs and the sum-product decoding algorithm in general terms. In this chapter we will give a brief overview of some particular
More informationLecture 12. Block Diagram
Lecture 12 Goals Be able to encode using a linear block code Be able to decode a linear block code received over a binary symmetric channel or an additive white Gaussian channel XII-1 Block Diagram Data
More informationLow Density Parity Check (LDPC) Codes and the Need for Stronger ECC. August 2011 Ravi Motwani, Zion Kwok, Scott Nelson
Low Density Parity Check (LDPC) Codes and the Need for Stronger ECC August 2011 Ravi Motwani, Zion Kwok, Scott Nelson Agenda NAND ECC History Soft Information What is soft information How do we obtain
More informationLecture 8: Shannon s Noise Models
Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have
More informationConvergence analysis for a class of LDPC convolutional codes on the erasure channel
Convergence analysis for a class of LDPC convolutional codes on the erasure channel Sridharan, Arvind; Lentmaier, Michael; Costello Jr., Daniel J.; Zigangirov, Kamil Published in: [Host publication title
More informationStructured Low-Density Parity-Check Codes: Algebraic Constructions
Structured Low-Density Parity-Check Codes: Algebraic Constructions Shu Lin Department of Electrical and Computer Engineering University of California, Davis Davis, California 95616 Email:shulin@ece.ucdavis.edu
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationAnalysis of Sum-Product Decoding of Low-Density Parity-Check Codes Using a Gaussian Approximation
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 47, NO. 2, FEBRUARY 2001 657 Analysis of Sum-Product Decoding of Low-Density Parity-Check Codes Using a Gaussian Approximation Sae-Young Chung, Member, IEEE,
More informationPhysical Layer and Coding
Physical Layer and Coding Muriel Médard Professor EECS Overview A variety of physical media: copper, free space, optical fiber Unified way of addressing signals at the input and the output of these media:
More informationError Correcting Codes Questions Pool
Error Correcting Codes Questions Pool Amnon Ta-Shma and Dean Doron January 3, 018 General guidelines The questions fall into several categories: (Know). (Mandatory). (Bonus). Make sure you know how to
More informationA Non-Asymptotic Approach to the Analysis of Communication Networks: From Error Correcting Codes to Network Properties
University of Massachusetts Amherst ScholarWorks@UMass Amherst Open Access Dissertations 5-2013 A Non-Asymptotic Approach to the Analysis of Communication Networks: From Error Correcting Codes to Network
More informationThe BCH Bound. Background. Parity Check Matrix for BCH Code. Minimum Distance of Cyclic Codes
S-723410 BCH and Reed-Solomon Codes 1 S-723410 BCH and Reed-Solomon Codes 3 Background The algebraic structure of linear codes and, in particular, cyclic linear codes, enables efficient encoding and decoding
More informationError Correcting Codes: Combinatorics, Algorithms and Applications Spring Homework Due Monday March 23, 2009 in class
Error Correcting Codes: Combinatorics, Algorithms and Applications Spring 2009 Homework Due Monday March 23, 2009 in class You can collaborate in groups of up to 3. However, the write-ups must be done
More informationLow-complexity error correction in LDPC codes with constituent RS codes 1
Eleventh International Workshop on Algebraic and Combinatorial Coding Theory June 16-22, 2008, Pamporovo, Bulgaria pp. 348-353 Low-complexity error correction in LDPC codes with constituent RS codes 1
More informationDecoding Codes on Graphs
Decoding Codes on Graphs 2. Probabilistic Decoding A S Madhu and Aditya Nori 1.Int roduct ion A S Madhu Aditya Nori A S Madhu and Aditya Nori are graduate students with the Department of Computer Science
More informationLDPC Code Ensembles that Universally Achieve Capacity under BP Decoding: A Simple Derivation
LDPC Code Ensembles that Universally Achieve Capacity under BP Decoding: A Simple Derivation Anatoly Khina EE Systems Dept., TAU Tel Aviv, Israel Email: anatolyk@eng.tau.ac.il Yair Yona Dept. of EE, UCLA
More informationOn the Block Error Probability of LP Decoding of LDPC Codes
On the Block Error Probability of LP Decoding of LDPC Codes Ralf Koetter CSL and Dept. of ECE University of Illinois at Urbana-Champaign Urbana, IL 680, USA koetter@uiuc.edu Pascal O. Vontobel Dept. of
More informationFactor Graphs and Message Passing Algorithms Part 1: Introduction
Factor Graphs and Message Passing Algorithms Part 1: Introduction Hans-Andrea Loeliger December 2007 1 The Two Basic Problems 1. Marginalization: Compute f k (x k ) f(x 1,..., x n ) x 1,..., x n except
More informationFountain Uncorrectable Sets and Finite-Length Analysis
Fountain Uncorrectable Sets and Finite-Length Analysis Wen Ji 1, Bo-Wei Chen 2, and Yiqiang Chen 1 1 Beijing Key Laboratory of Mobile Computing and Pervasive Device Institute of Computing Technology, Chinese
More informationOne Lesson of Information Theory
Institut für One Lesson of Information Theory Prof. Dr.-Ing. Volker Kühn Institute of Communications Engineering University of Rostock, Germany Email: volker.kuehn@uni-rostock.de http://www.int.uni-rostock.de/
More informationList Decoding of Reed Solomon Codes
List Decoding of Reed Solomon Codes p. 1/30 List Decoding of Reed Solomon Codes Madhu Sudan MIT CSAIL Background: Reliable Transmission of Information List Decoding of Reed Solomon Codes p. 2/30 List Decoding
More informationMaking Error Correcting Codes Work for Flash Memory
Making Error Correcting Codes Work for Flash Memory Part I: Primer on ECC, basics of BCH and LDPC codes Lara Dolecek Laboratory for Robust Information Systems (LORIS) Center on Development of Emerging
More informationLecture 2 Linear Codes
Lecture 2 Linear Codes 2.1. Linear Codes From now on we want to identify the alphabet Σ with a finite field F q. For general codes, introduced in the last section, the description is hard. For a code of
More informationLecture 12: November 6, 2017
Information and Coding Theory Autumn 017 Lecturer: Madhur Tulsiani Lecture 1: November 6, 017 Recall: We were looking at codes of the form C : F k p F n p, where p is prime, k is the message length, and
More informationBridging Shannon and Hamming: List Error-Correction with Optimal Rate
Proceedings of the International Congress of Mathematicians Hyderabad, India, 2010 Bridging Shannon and Hamming: List Error-Correction with Optimal Rate Venkatesan Guruswami Abstract. Error-correcting
More informationBlock Codes :Algorithms in the Real World
Block Codes 5-853:Algorithms in the Real World Error Correcting Codes II Reed-Solomon Codes Concatenated Codes Overview of some topics in coding Low Density Parity Check Codes (aka Expander Codes) -Network
More informationLDPC Codes. Slides originally from I. Land p.1
Slides originally from I. Land p.1 LDPC Codes Definition of LDPC Codes Factor Graphs to use in decoding Decoding for binary erasure channels EXIT charts Soft-Output Decoding Turbo principle applied to
More informationLOW-density parity-check (LDPC) codes were invented
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL 54, NO 1, JANUARY 2008 51 Extremal Problems of Information Combining Yibo Jiang, Alexei Ashikhmin, Member, IEEE, Ralf Koetter, Senior Member, IEEE, and Andrew
More informationB I N A R Y E R A S U R E C H A N N E L
Chapter 3 B I N A R Y E R A S U R E C H A N N E L The binary erasure channel (BEC) is perhaps the simplest non-trivial channel model imaginable. It was introduced by Elias as a toy example in 954. The
More informationBOUNDS ON THE MAP THRESHOLD OF ITERATIVE DECODING SYSTEMS WITH ERASURE NOISE. A Thesis CHIA-WEN WANG
BOUNDS ON THE MAP THRESHOLD OF ITERATIVE DECODING SYSTEMS WITH ERASURE NOISE A Thesis by CHIA-WEN WANG Submitted to the Office of Graduate Studies of Texas A&M University in partial fulfillment of the
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More informationLP Decoding Corrects a Constant Fraction of Errors
LP Decoding Corrects a Constant Fraction of Errors Jon Feldman Columbia University (CORC Technical Report TR-2003-08) Cliff Stein Columbia University Tal Malkin Columbia University Rocco A. Servedio Columbia
More informationOn Encoding Symbol Degrees of Array BP-XOR Codes
On Encoding Symbol Degrees of Array BP-XOR Codes Maura B. Paterson Dept. Economics, Math. & Statistics Birkbeck University of London Email: m.paterson@bbk.ac.uk Douglas R. Stinson David R. Cheriton School
More information6.1.1 What is channel coding and why do we use it?
Chapter 6 Channel Coding 6.1 Introduction 6.1.1 What is channel coding and why do we use it? Channel coding is the art of adding redundancy to a message in order to make it more robust against noise. It
More informationTime-invariant LDPC convolutional codes
Time-invariant LDPC convolutional codes Dimitris Achlioptas, Hamed Hassani, Wei Liu, and Rüdiger Urbanke Department of Computer Science, UC Santa Cruz, USA Email: achlioptas@csucscedu Department of Computer
More informationConstructions of Nonbinary Quasi-Cyclic LDPC Codes: A Finite Field Approach
Constructions of Nonbinary Quasi-Cyclic LDPC Codes: A Finite Field Approach Shu Lin, Shumei Song, Lan Lan, Lingqi Zeng and Ying Y Tai Department of Electrical & Computer Engineering University of California,
More informationSpatially Coupled LDPC Codes
Spatially Coupled LDPC Codes Kenta Kasai Tokyo Institute of Technology 30 Aug, 2013 We already have very good codes. Efficiently-decodable asymptotically capacity-approaching codes Irregular LDPC Codes
More informationCodes designed via algebraic lifts of graphs
p./40 Codes designed via algebraic lifts of graphs Clemson Mini-Conference on Discrete Mathematics Oct. 3, 2008. Christine A. Kelley Department of Mathematics University of Nebraska-Lincoln email: ckelley2@math.unl.edu
More informationOn Achievable Rates and Complexity of LDPC Codes over Parallel Channels: Bounds and Applications
On Achievable Rates and Complexity of LDPC Codes over Parallel Channels: Bounds and Applications Igal Sason, Member and Gil Wiechman, Graduate Student Member Abstract A variety of communication scenarios
More informationRevision of Lecture 5
Revision of Lecture 5 Information transferring across channels Channel characteristics and binary symmetric channel Average mutual information Average mutual information tells us what happens to information
More informationON THE MINIMUM DISTANCE OF NON-BINARY LDPC CODES. Advisor: Iryna Andriyanova Professor: R.. udiger Urbanke
ON THE MINIMUM DISTANCE OF NON-BINARY LDPC CODES RETHNAKARAN PULIKKOONATTU ABSTRACT. Minimum distance is an important parameter of a linear error correcting code. For improved performance of binary Low
More informationLecture 4: Codes based on Concatenation
Lecture 4: Codes based on Concatenation Error-Correcting Codes (Spring 206) Rutgers University Swastik Kopparty Scribe: Aditya Potukuchi and Meng-Tsung Tsai Overview In the last lecture, we studied codes
More informationGraph-based Codes and Iterative Decoding
Graph-based Codes and Iterative Decoding Thesis by Aamod Khandekar In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy California Institute of Technology Pasadena, California
More informationCodes on Graphs. Telecommunications Laboratory. Alex Balatsoukas-Stimming. Technical University of Crete. November 27th, 2008
Codes on Graphs Telecommunications Laboratory Alex Balatsoukas-Stimming Technical University of Crete November 27th, 2008 Telecommunications Laboratory (TUC) Codes on Graphs November 27th, 2008 1 / 31
More informationCoding for loss tolerant systems
Coding for loss tolerant systems Workshop APRETAF, 22 janvier 2009 Mathieu Cunche, Vincent Roca INRIA, équipe Planète INRIA Rhône-Alpes Mathieu Cunche, Vincent Roca The erasure channel Erasure codes Reed-Solomon
More information2 Information transmission is typically corrupted by noise during transmission. Various strategies have been adopted for reducing or eliminating the n
Finite size eects and error-free communication in Gaussian channels Ido Kanter and David Saad # Minerva Center and Department of Physics, Bar-Ilan University, Ramat-Gan 52900, Israel. # The Neural Computing
More informationGLDPC-Staircase AL-FEC codes: A Fundamental study and New results
GLDPC-Staircase AL-FEC codes: A Fundamental study and New results Ferdaouss Mattoussi, Vincent Roca, Bessam Sayadi To cite this version: Ferdaouss Mattoussi, Vincent Roca, Bessam Sayadi. GLDPC-Staircase
More informationNotes 10: List Decoding Reed-Solomon Codes and Concatenated codes
Introduction to Coding Theory CMU: Spring 010 Notes 10: List Decoding Reed-Solomon Codes and Concatenated codes April 010 Lecturer: Venkatesan Guruswami Scribe: Venkat Guruswami & Ali Kemal Sinop DRAFT
More informationPart III Advanced Coding Techniques
Part III Advanced Coding Techniques José Vieira SPL Signal Processing Laboratory Departamento de Electrónica, Telecomunicações e Informática / IEETA Universidade de Aveiro, Portugal 2010 José Vieira (IEETA,
More informationRaptor Codes: From a Math Idea to LTE embms. BIRS, October 2015
Raptor Codes: From a Math Idea to LTE embms BIRS, October 2015 The plan is to... 1 introduce LT codes and Raptor codes 2 provide insights into their design 3 address some common misconceptions 2 / 31 The
More informationSIDDHARTH GROUP OF INSTITUTIONS :: PUTTUR Siddharth Nagar, Narayanavanam Road UNIT I
SIDDHARTH GROUP OF INSTITUTIONS :: PUTTUR Siddharth Nagar, Narayanavanam Road 517583 QUESTION BANK (DESCRIPTIVE) Subject with Code : CODING THEORY & TECHNIQUES(16EC3810) Course & Branch: M.Tech - DECS
More informationCSCI 2570 Introduction to Nanocomputing
CSCI 2570 Introduction to Nanocomputing Information Theory John E Savage What is Information Theory Introduced by Claude Shannon. See Wikipedia Two foci: a) data compression and b) reliable communication
More informationCOMPSCI 650 Applied Information Theory Apr 5, Lecture 18. Instructor: Arya Mazumdar Scribe: Hamed Zamani, Hadi Zolfaghari, Fatemeh Rezaei
COMPSCI 650 Applied Information Theory Apr 5, 2016 Lecture 18 Instructor: Arya Mazumdar Scribe: Hamed Zamani, Hadi Zolfaghari, Fatemeh Rezaei 1 Correcting Errors in Linear Codes Suppose someone is to send
More informationLow-Density Parity-Check codes An introduction
Low-Density Parity-Check codes An introduction c Tilo Strutz, 2010-2014,2016 June 9, 2016 Abstract Low-density parity-check codes (LDPC codes) are efficient channel coding codes that allow transmission
More informationPractical Polar Code Construction Using Generalised Generator Matrices
Practical Polar Code Construction Using Generalised Generator Matrices Berksan Serbetci and Ali E. Pusane Department of Electrical and Electronics Engineering Bogazici University Istanbul, Turkey E-mail:
More informationTurbo Compression. Andrej Rikovsky, Advisor: Pavol Hanus
Turbo Compression Andrej Rikovsky, Advisor: Pavol Hanus Abstract Turbo codes which performs very close to channel capacity in channel coding can be also used to obtain very efficient source coding schemes.
More informationConstruction and Performance Evaluation of QC-LDPC Codes over Finite Fields
MEE10:83 Construction and Performance Evaluation of QC-LDPC Codes over Finite Fields Ihsan Ullah Sohail Noor This thesis is presented as part of the Degree of Master of Sciences in Electrical Engineering
More informationPart III. Cyclic codes
Part III Cyclic codes CHAPTER 3: CYCLIC CODES, CHANNEL CODING, LIST DECODING Cyclic codes are very special linear codes. They are of large interest and importance for several reasons: They posses a rich
More informationRecent Results on Capacity-Achieving Codes for the Erasure Channel with Bounded Complexity
26 IEEE 24th Convention of Electrical and Electronics Engineers in Israel Recent Results on Capacity-Achieving Codes for the Erasure Channel with Bounded Complexity Igal Sason Technion Israel Institute
More informationSIPCom8-1: Information Theory and Coding Linear Binary Codes Ingmar Land
SIPCom8-1: Information Theory and Coding Linear Binary Codes Ingmar Land Ingmar Land, SIPCom8-1: Information Theory and Coding (2005 Spring) p.1 Overview Basic Concepts of Channel Coding Block Codes I:
More informationLecture 14 February 28
EE/Stats 376A: Information Theory Winter 07 Lecture 4 February 8 Lecturer: David Tse Scribe: Sagnik M, Vivek B 4 Outline Gaussian channel and capacity Information measures for continuous random variables
More informationLecture 4 Noisy Channel Coding
Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem
More informationNoisy channel communication
Information Theory http://www.inf.ed.ac.uk/teaching/courses/it/ Week 6 Communication channels and Information Some notes on the noisy channel setup: Iain Murray, 2012 School of Informatics, University
More informationLecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
More information