Lecture 4: Linear Codes Copyright G. Caire 88
Linear codes over F q We let X = F q for some prime power q. Most important case: q =2(binary codes). Without loss of generality, we may represent the information message as a sequence of k symbols from F q. We have C = q k, and R = k n log 2 q bits/symbol. Definition 22. A (q k,n) block code over X = F q is called a linear (n, k) code if its codewords form a k-dimensional vector subspace of the vector space F n q. Copyright G. Caire 89
Elementary properties of linear codes The code C is an additive group, in particular, if c, c 0 2 C then c + c 0 2 C and c 2 C. The all-zero vector is a codeword: 0 2 C. Linear combination of codewords are codewords: c 1,...,c` 2 C and a 1,...,a` 2 F q, then a 1 c 1 + a`c` 2 C There exist (non-unique) sets of k linearly independent codewords that generate the whole code, i.e., ( k 1 ) X C = u`g` : u 0,...,u k 1 2 F q `=0 where g 0,...,g k 1 are codewords that form a basis for the code C. Copyright G. Caire 90
Generator matrix We can arrange the basis as rows of a k n matrix G = 2 6 4 g 0,0 g 0,1 g 0,2 g 0,n 1 g 1,0. g 1,1 g 1,2 g 1,n. 1 g k 1,0 g k 1,1 g k 1,2 g k 1,n 1 3 7 5 This is called a generator matrix for the code. Letting u =(u 0,...,u k can write the encoding function as 1 ) we c = ug Copyright G. Caire 91
(7, 4) binary Hamming code G = 2 6 4 1 1 0 1 0 0 0 0 1 1 0 1 0 0 1 1 1 0 0 1 0 1 0 1 0 0 0 1 3 7 5 Copyright G. Caire 92
Repetition and single-parity-check codes G = apple 1 1 0 1 0 1 G = 1 1 1 Copyright G. Caire 93
Systematic encoding Sometimes, a desirable property of the encoding function is that the vector of information symbols appears explicitly as part of the codeword. For linear codes, a generator matrix in systematic form is given by G =[P I k ] k (n k) where P 2 Fq and I k denotes the k k identity. In this way, we have c = ug =[up u]. Copyright G. Caire 94
Parity-check matrix Since C is a vector subspace of F q of dimension k, then it can be seen as the Kernel of some linear transformation F n q! F n k q. The matrix of such linear transformation is called parity-check matrix and it is denoted by H T, where H 2 F (n k) n q : C = c 2 F n q : ch T = 0 In particular, the rows of H are n-vectors in F n q that are orthogonal to all codewords. Indeed, they generate the orthogonal subspace C?, also known as the dual code of C. If G =[P I k ], then H =[I n k P T ], in fact, we have GH T =[P I k ] apple In k P = P P = 0 Copyright G. Caire 95
q-ary symmetric channels A q-ary symmetric channel can always be represented as an additive noise channel over F q, such that y = c + z where z 2 F n q. The noise pmf is given by P Z (z) = 1 for z =0 /(q 1) for z 6= 0 The syndrome of the error vector is given by s = zh T Copyright G. Caire 96
Syndrome decoding Notice that the decoder can compute the syndrome even though it does not know z, in fact, yh T =(c + z)h T = ch T + zh T = zh T = s Therefore, the syndrome of the error vector is an index that can be used by the decoder to undo the bit-flips. If s = 0, then y 2 C. In this case, we let bc = y. If s 6= 0, then y /2 C. In this case the decoder knows that an error has occurred (detectable error). Copyright G. Caire 97
In order to correct the error, the decoder needs to find bz, an estimate of z, and correct the errors as bc = y bz. Unfortunately, the system of equations s = zh T where s is known (the syndrome) and z is unknown, is underdetermined (n unknowns and n k equations). For every syndrome s 2 F n q k, we have q k possible error vectors. We have to solve this problem in a probabilistic sense... for each set of q k possible error vectors corresponding to a given syndrome, we shall pick the most likely. Copyright G. Caire 98
Standard array The linear map F n q! F n q k such that y 7! s = yh T has Kernel Ker(H T )=C Linear maps are group homomorphisms (they preserve the group operation, that in this case is componentwise addition in F q ). A coset of C in F n q is given by the translate v + C for some v 2 F n q. The factor group (group of cosets), with respect to the coset addition, satisfies (canonical isomorphism): F n q /C F n k q The standard array is the correspondence between the syndromes s 2 F n q and the cosets of C in F n q, induced by this isomorphism. k Copyright G. Caire 99
Building the standard array s 0 = 0 c 0 = 0 c 1 c q k 1 s 1 v 1 v 1 + c 1 v 1 + c q k 1 s 2 v 2 c 2 + c 1 v 2 + c q k 1... s q n k 1 v q n k 1 v q n k 1 + c 1 v q n k 1 + c q k 1 For every new row, find the vector in F n q with minimum Hamming weight that not yet appeared in the array. The corresponding row is obtained by adding this vector to all codewords. All row are distinct, and yield the same syndrome. These vectors of minimum weight are called coset leaders. Copyright G. Caire 100
Example Consider the SPC (3, 2), with parity-check matrix H =[ 1 1 1 ] Exercise: build the standard array for the Hamming (7, 4) code. Copyright G. Caire 101
Error correction algorithm 1. Compute the syndrome s = yh T. 2. Use the standard array and find the most likely error vector bz compatible with s (coset leader). 3. The minimum Hamming distance decision rule is given by bc = y bz 4. A (n, k) linear code is able to correct q n k error patterns (error vectors) Copyright G. Caire 102
The Tanner Graph of a linear code Consider a linear (n, k) block code C = {c : ch T = 0}. It is defined by the set of parity-check equations h 1,1 x 1 + h 1,2 x 2 + + h 1,n x n = 0 h 2,1 x 1 + h 2,2 x 2 + + h 2,n x n = 0. h n k,1 x 1 + h n k,2 x 2 + + h n k,n x n = 0 The Tanner graph of the code is a bipartite graph with n bit-nodes and n k check-nodes, such that an edge (i, j) exists if h i,j = 1, that is, if bit x j participate in the parity-check equation i. Copyright G. Caire 103
Hamming (7, 4) code Parity-check equations of the Hamming (7, 4) code: x 1 + x 2 + x 4 + x 5 = 0 x 1 + x 3 + x 4 + x 6 = 0 x 2 + x 3 + x 4 + x 7 = 0 + + + Copyright G. Caire 104
LDPC codes Low-Density Parity-Check (LDPC) codes are linear binary codes with the characteristic that their parity-check matrix is sparse: the number of ones in the matrix is proportional to the block length n. Notice that a randomly generated binary matrix H with dimensions n(1 R) n has an average number of ones equal to n 2 (1 R)/2, i.e., quadratic with n. A regular (d`,d r ) LDPC code has Tanner graph with constant left and right degrees d` and d r, respectively. Example: a (3, 6) regular LDPC code of length n = 10, given by the paritycheck matrix 2 3 0 1 1 1 1 1 0 0 0 1 1 0 0 1 0 1 0 1 1 1 H = 6 0 0 1 0 1 1 1 0 1 1 7 4 1 1 1 0 1 0 1 1 0 0 5 1 1 0 1 0 0 1 1 1 0 Copyright G. Caire 105
(3, 6) LDPC code with n = 10 Copyright G. Caire 106
Introducing memory A binary linear block encoder is a linear transformation F k 2! F n 2. What about using a Linear time-invariant linear system for encoding? Convolutional codes consider small k and n, but introduce memory into the encoding process of a sequence of consecutive blocks. We may see this a the convolution of a sequence of information blocks {u i } with a matrix G of impulse responses, in order to generate a sequence of coded blocks {c i }. Copyright G. Caire 107
MA k n linear systems A k n Moving Average (MA) system is defined by: c (1) i = c (2) i =. c (n) i = m 1,1 X `=0 m 1,2 X `=0 m X 1,n `=0 g (1,1) ` u (1) i g (1,2) ` u (1) i g (1,n) ` u (1) i mk,1 ` + + X `=0 mk,2 ` + + X `=0 mk,n ` + + X `=0 g (k,1) ` g (k,2) ` g (k,n) ` u (k) i ` u (k) i ` u (k) i ` Copyright G. Caire 108
Defining a vector output sequence c 0, c 1, c 2,...such that c i =(c (1) i and a vector input sequence u 0, u 1, u 2,...such that u i =(u (1) i can write mx where we let m = max{m (i,j) }. c i = `=0 u i `G`,...,c (n),...,u (k) i i ) ), we We obtain a block-toeplitz notation 2 (c 0, c 1, c 2, c 3,...)=(u 0, u 1, u 2, u 3,...) 6 4 G 0 G 1 G m 0 0 G 0 G m 1 G m. 0.... G m 1... G 0. 0 G 0... 3 7 5 The impulse responses g (i,j) are called the code generators. Copyright G. Caire 109
D-transform D-transform domain u i! u(d) = X i u i D i (Laurent series) Convolutional encoding in the D-transform domain: c(d) =u(d)g(d) or, equivalently, (c 1 (D),...,c n (D)) = (u 1 (D),...,u k (D)) 2 6 4 g 1,1 (D) g 1,2 (D) g 1,n (D) g 2,1 (D). g 2,2 (D) g 2,n (D). g k,1 (D) g k,2 (D) g k,n (D) 3 7 5 Copyright G. Caire 110
Example: a (2, 1) convolutional code + c (1) i = u i + u i 2 u i + c (2) i = u i + u i 1 + u i 2 Copyright G. Caire 111
Encoder canonical forms A code C is defined as the set of all output sequences (code sequences). As for block codes, a convolutional code C may have several input-ouput encoder implementations. We seek encoders in canonical form: a general problem in system theory is for a given system, defined as the ensemble of all its output sequences, what is the minimal canonical realization? State-space representation (ABCD): s i+1 = s i A + u i B, c i = s i C + u i D a minimal representation is a representation with the minimum number of state variables. Copyright G. Caire 112
More on the (2, 1) example In the (2, 1) example of before, the state is defined as the content of the memory elements, s i =(u i 1,u i 2 ) therefore we have and s i+1 = s i apple 0 1 0 0 c i = s i apple 0 1 1 1 + u i 1 0 + u i 1 1 Copyright G. Caire 113
Generalization... A convolutional code can be seen as a block code defined on the field F q (D) of rational functions over F q. Roughly speaking: rational functions are to polynomials as rationals Q to the integers Z. Generalizing what seen before, we can consider G(D) with rational elements g i,j (D). In system theory, this corresponds to AR-MA linear systems. The code is preserved by elementary row operations. It follows that for any G(D), we can find a systematic generator matrix in the form G(D) =[I P(D)] where I is the k k identity, and P(D) is a k (n k) matrix of rational functions. Copyright G. Caire 114
State diagram A state-space realization with m binary state variables is a finite-state machine (FSM) with a state space = F m 2. In general, a FSM is described by its state transition diagram, i.e., by a graph with vertices, corresponding to all possible state configurations, and edges connecting those states for which a transition is possible. Each edge (s, s 0 ) 2 is labeled by input and output vectors b 2 F k 2 and c 2 F n 2, corresponding to the state transition between s and s 0. Copyright G. Caire 115
Example: the (2, 1) 4-state code Faculty of Electrical Engineering and Computer Systems Department of Telecommunication Systems Information and Communication Theory Prof. Dr. Giuseppe Caire Einsteinufer 25 10587 Berlin Telefon +49 (0)30 314-29668 Telefax +49 (0)30 314-28320 caire@tu-berlin.de Sekretariat HFT6 Patrycja Chudzik Telefon +49 (0)30 314-28459 Telefax +49 (0)30 314-28320 sekretariat@mk.tu-berlin.de 1/11 (1, 0) 1/10 0/01 0/00 (0, 0) 1/00 (1, 1) 1/01 0/10 0/11 (0, 1) Copyright G. Caire 116
Trellis diagram An alternative representation consists of the trellis section, i.e., by a bipartite graph with state vertices on the left and state vertices on the right. Left vertices represent the possible states at time i, and right vertices represent the possible states at time i +1. Edges represent the possible state transitions corresponding to input u i and output c i. The trellis representation follows from the state transition diagram by introducing the time axis. A trellis diagram for a convolutional code consists of the concatenation of an infinite number of trellis sections. Given an initial state at time i =0, an input sequence u(d) determines an output sequence c(d) and a state sequence s(d) that correspond to a path in the trellis. Copyright G. Caire 117
Example: the (2, 1) 4-state code Faculty of Electrical Engineering and Computer Systems Department of Telecommunication Systems Information and Communication Theory Prof. Dr. Giuseppe Caire Einsteinufer 25 10587 Berlin Telefon +49 (0)30 314-29668 Telefax +49 (0)30 314-28320 caire@tu-berlin.de Sekretariat HFT6 Patrycja Chudzik Telefon +49 (0)30 314-28459 Telefax +49 (0)30 314-28320 sekretariat@mk.tu-berlin.de (0, 0) (1, 0) 0/00 1/11 (0, 0) 0/11 (1, 0) 1/00 (0, 1) 0/01 1/10 (0, 1) 0/10 (1, 1) Copyright G. Caire 1/01 (1, 1) 118
The Factor Graph for a convolutional code Three types of variable nodes: information bitnodes u i, coded bitnodes c i and states nodes s i. The function nodes correspond to the state and output mappings s i+1 = s i A + u i B, c i = s i C + u i D Copyright G. Caire 119
End of Lecture 4 Copyright G. Caire 120