The Channel Capacity of Constrained Codes: Theory and Applications

Transcription

1 The Channel Capacity of Constrained Codes: Theory and Applications Xuerong Yong 1 The Problem and Motivation The primary purpose of coding theory channel capacity; Shannon capacity. Only when we transmit information at a rate below its capacity can we make reliable transmission. Good code re- The capacity is usually unknown. quires matching capacity.

2 2 Outline The Problem and Motivation Constrained Codes Encoding and Decoding Perron-Frobenius Theory Characterizations of Capacity Representative Research and Open Problems 2

3 3 Constrained Code as a Language Encoding of random data as a constrained code is accomplished by means of a finite state machine. A constrained code can be thought of as a regular language, so its encoder can be chosen as a finite state deterministic automaton. 3

4 4 Constrained Code for Recording Codes in magnetic, digital and optical recordings: (d,k)- RLL codes are the codes over binary alphabet {, 1} where d (k) is the minimum (maximum) permitted number of s separating consecutive 1 s in a legal binary sequence S = s 1 s 2. The k is imposed to guarantee sufficient sign-changes in the recording waveform to prevent clock drifting in the clock synchronization; d is used to prevent intersymbol interference. Multiple-spaced (d, k, s)-rll codes, where s indicates that the runlengths of s must be given in the form d + is, where i is an integer. Others: 4

5 5 Data Storage Devices Appear in computer center at business location and in desk-top workstation in office, etc. Variety of such devices includes: Conventional diskette and hard diskette drivers; optical read-only drivers such as CD, CD-ROM drivers. Magnetic tape drivers, digital audio tape systems and digital compact cassette audio tape systems. The data recording and retrieval process are illustrated in Figure 1. 5

6 data data Compression Encoder Compression Decoder Error Correction Encoder Modulation Encoder Error Correction Decoder Modulation Decoder Signal Generator Detector Write Equalizer Read Equalizer Figure 1: Data recording schematic. 6

7 6 Encoding and Decoding Given a code, we can construct an encoder, which accepts an input block of p user bits and generates a length-q codeword. The sequences obtained by concatenating the lengthq codewords satisfy the constraint. The encoder should automatically be decodable. The state dependent decoder accepts a length-q codeword and produces a length-p block of user bits. It is desirable that the code have the highest rate possible. Shannon proved that the rate p/q can not exceed the capacity. 7

8 7 Representations of Constrained Codes The sequences permitted to appear in a given channel can be represented by a graph (encoder). Conversely, an encoder for the channel may give words only from this system. The graph that characterises the configurations is called a labeled graph, a labeled directed multi-graph G = (V, E, L). The labeled graph is conveniently expressed by a matrix, called its adjacency matrix A, where the entry (A) u,v is the number of edges from vertex u to vertex v in the graph G. 8

9 8 Some Definitions Let A be an n n matrix. Its eigenvalues are n roots of the characteristic polynomial det(λi A). Let λ(a) be the largest absolute value of the eigenvalues of A. Right, left eigenvectors: For nonzero vectors x, y, Ax = λx, y t A = λy t, where t stands for the transpose. period of a graph G is the greatest common divisor of the lengths of all cycles in G. A graph is called primitive if its period is 1. A constrained system S is called irreducible or primitive if its graph is strongly connected or primitive. 9

10 9 Perron-Frobenius Theorem Theorem. Let A be a nonnegative irreducible matrix. Then λ(a) is an eigenvalue of A and A has positive right and left eigenvectors associated with the eigenvalue λ(a). λ(a) is a simple eigenvalue of A, i.e. λ(a) appears as a root of the characteristic polynomial of A with multiplicity 1. min u v (A) u,v λ(a) max u v (A) u,v. 1

11 1 Characterization of Capacity Viewing from combinatorics, algebra and probability. Combinatorial description: The capacity cap(s) measures the growth rate of the number N(n; S) of sequences of length n in S. Precisely, Example: log cap α (S) = n lim α N(n; S). n Algebraic description: Let S be an irreducible constrained system and let G be an irreducible lossless presentation of S. Then cap(s) = log λ(a G ), where A G is the adjacency matrix of G. 11

12 Probabilistic Description: An n n nonnegative matrix Q = (q ij ) is stochastic if all its row sums are 1. Let y = (y 1, y 2,..., y n ) t, n i=1 y i = 1 be the left eigenvector corresponding to 1. The entropy of Q is H(Q) = n n q ij log q ij. y i i=1 j=1 Let p ij be the probability that there is an edge from vertex u i to vertex u j in the labeled graph G of a constrained system S. Then Q = (p ij ) is stochastic. Let S be a primitive constrained system and let G be a primitive presentation of S. Then sup Q H(Q) = cap(s) = log λ(a G ). 12

13 11 Capacity of 1-D Constrained Codes The capacity of 1-D codes is paid much attention. The following are the representatives. Norris and Bloomberg considered (1981) the cap(s d,k) (1) of 1-D (d, k)-rll constraints for a large range of parameters and proved cap(s d,k) (1) = log 2 λ 1 where λ 1 is the largest root of the polynomial f d,k (x) = x k+1 x k d... x 1, if k <, f d, (x) = x d+1 x d 1, if k =. Ashley and Siegel proved (1987) that for all d 1 we have cap(s (1) d, ) = cap(s (1) d 1,2d 1). Their approaches are combinatorics and algebra. 13

14 12 Capacity of 2-D Constrained codes 2-D system S has constraints both horizontally and vertically. 2-D (d, k)-rll system S (2) d,k satisfies the RLL constraint both two directions. The capacity cap(s) measures the growth rate of the number N(m, n; S) of m n arrays in S: cap(s) = lim n, m 1 nm log 2 N(m, n; S), where N(m, n; S) equals the number of m n (, 1) matrices that satisfy S. cap(s d,k) (2) cap(s d,k). (1) However, the 1-D and 2-D capacities are different. Ashley and Marcus (1996) proved that cap(s 1,2) (1) = , whereas cap(s 1,2) (2) = (Kato and Zeger, 1999). 14

15 13 2-D RLL Constrained code S (2) 1, Called hard-square, or hard-core lattice gas system (Forchhammer and Justesen, 1999), Engel (1982) called Fibonacci number of a lattice, Calkin and Wilf (1998) called independent sets in grid graph. Denote it simply by S HS. It contains all matrices that do not have adjacent horizontal or vertical 1 s. For example, the first one of the following two matrices satisfies the constraint but the second does not ,

16 14 Work on Capacity Weber seems to be the first to consider this problem. Weber (1988) obtained cap(s HS) 1.554, Engel (199) cap(s HS) Calkin and Wilf (1998) cap(s HS) , i.e cap(s HS ) Later, Weeks and Blahut (1998) improved the these bounds, and recently further refined by Nagy and Zeger (21) and become cap(s HS )

17 15 The Tools Used in S HS The basic tool used is the transfer matrix technique. Consider the set V m of possible 1 m strings that are allowed to appear as rows in the constrained matrix. We say that the pair (v i, v j ) is valid if v i, v i V m can be put together without violating the constraint. The transfer matrix T m is defined as (T m ) ij = 1 if (v i, v j ) is valid and otherwise. It is symmetric and primitive. 17

18 16 Lowerbounding the Capacity N(m, n; S) = 1 t Tm n 1 1 where 1 is the appropriate size vector of all 1s, and m 1,n 1. Let λ m be the largest eigenvalue of T m. Then using Perron-Frobenius Theorem, log cap(s) = lim 2 λ m m m. By maximum principle, for an n n real symmetric matrix A, the largest eigenvalue λ n satisfies λ n max x x t Ax x t x. p, q, cap(s) log 2 ( λ p+2q ) /p (1) λ 2q bounding cap(s) from below by computing the first few λ. 18

19 17 Upperbounding the Capacity Their upper bound is obtained by counting the number of cylinders; the arrays that the leftmost and rightmost columns can be put next to each other without violating the constraint. This is equivalent to constructing a related transfer matrix B 2p for an arbitrary integer p such that trace(a 2p ) = 1 t B2p m 1 1, and then using the fact λ m (trace(a 2p )) 1 2p = (1 t B m 1 2p 1 ) 1 2p, and computing the 2pth positive root of the largest eigenvalue of B 2p. 19

20 18 The Difficulties The largest eigenvalue λ m of T m is computed by power method (although there are several methods, this method has been considered to be most applicable). [The method has n square complexity where n is the size of the matrix.] The crucial fact is that the size of the matrices grows exponentially (T m is an F m+3 F m+3 matrix and F m ( 1+ )m 5 2 ) so exactly calculating any more than the first few λ m is very difficult. Better approaches involve eigen-spaces of the first two largest eigenvalues. 2

21 19 General Technique In the case that the transfer matrix is symmetric, almost all of the research into considering the capacity has been using transfer matrix technique maximum principle Perron-Frobenius theory power method BUT since the size usually grows exponentially, the research concentrated on getting better bounds using loworder λ m would be desirable. 21

22 2 Read/Write isolated memory A serial, binary (, 1) memory is said to be read isolated if no consecutive positions can store 1 s in the memory. Freiman and Wyner (1964, 1965) first considered this problem, and then Kautz (1965) explored a subcase. A serial, binary (, 1) memory is said to be write isolated if no two consecutive positions can be changed during rewriting (Cohen, 1993). These two memories have same capacity A read/write isolated memory is a binary linearly ordered, rewritable storage medium obeying both the two restrictions, which considered by Cohn in

23 21 The Boundings It is equivalent to finding the capacity of a constrained system S RW in which no matrix can contain a summatrix like (11), ( ) 1, or ( ) The transfer matrix is symmetric and primitive. He proved that for m, cap(s RW ) log 2 λ m m. (2) His lower bound came from a combinatorial argument. 23

24 Diamond Hexagonal Square Figure 2: Three kinds of checkerboard constraints 22 Checkerboard Constraints A 2-D arrangement of zeros and ones, where every 1 is surrounded by a specific pattern of s, considered by Weeks and Blahut (1998). Three kinds: Diamond, Hexagonal and Square. If every one 1 is surrounded by l rings of zeros, the constraint is lth order. First two orders are shown in Figure 2. 24

25 23 The Techniques in Checkerboard All transfer matrices (for the first order) are symmetric. So Weeks and Blahut used the same approaches as the Calkin and Wilf did. When the order exceeds 1, the transfer matrix is not symmetric but irreducible. Applying Perron-Frobenius Theorem they obtained some rough lower and upper bounds on capacity. And then using these bounds, a numerical convergencespeeding technique called Richardson extrapolation was developed tighter results. 25

26 24 2-D RLL Constrained Systems S (2) d,k Kato and Zeger (1999) studied 2-D RLL S (2) (d,k) and proved that ( ) for d >, cap(s (2) d,k) = if and only if k = d + 1. ( ) Inequalities between cap(s,k) (2) and cap(s 1, ) (2) and between cap(s d,k) (2) and cap(s 1, ). (2) ( ) As d grows, cap(s d, ) (2) decays to zero exactly at the cap(s (2) d, rate (log 2 d)/d, i.e. lim ) d (log 2 d)/d = 1. They do not use the transfer matrix approach but rather a combinatorial one in their proofs. 26

27 25 Zero Capacity In a recent paper (1999) by Ito, Kato, Nagy and Zeger it is proved for the n-d constraint S (n) d,k n 2 that ( ) for d >, and k d cap(s (n) d,k ) = if and only if k = d + 1. and ( ) for d >, and k d, if and only if k 2d. lim n cap(s(n) d,k ) = 27

28 K K K K K K K K K K K K K K K Figure 3: An example on 6 1 board 26 Nonattacking Kings Probem Nonattacking Kings: the kings are not allowed to be placed consecutively both horizontally and vertically in a lattice, and one in every 2 2 chessboard. Wilf (1995) considered this problem from combinatorics. Let N(m, n) stand for the number of ways that mn nonattacking kings can be placed on a 2m 2n chessboard. He obtained N(m, n) = (c m n + d m )(m + 1) n + O(θ n m), n +. 28

29 27 Its Transfer Matrix This is a constrained 2-D (1, )-RLL constrained system. The transfer matrix is neither symmetric nor irreducible ,

30 28 Two Recent Approaches Transfer matrix is symmetric: a work by Nagy and Zeger (21) provides a technique for upper bounding the 3-D RLL constraint S,1. (3) they extended the ideas of Calkin and Wilf (1998) to prove that cap(s (3),1) They used cylinders, and introduced toroidal constraint. Thansfer matrix T is non-symmetric: Forchhammer (2) provides a new technique based on transfer matrix to upperbound the capacity of a 3-D constraint. 3

31 29 Construction of Constrained Codes Siegel and Wolf (1998) gave lower bounds on the capacities of two 2-D constrained systems S (2) d, and S (2),k called bit-stuffing encoder. Bit-stuffing Encoder They describe a mapping of 1-D constrained system S (1) 2d, to 2-D constrained system S d,, (2) then represent the 2-D constrained system S (2) d, as the lower right quadrant of a rectangular grid. Then cap(s (2) d, ) cap(s (1) 2d, ). 31

32 3 Approach by Analyzing a Bit-stuffing Encoder A binary data sequence is first converted to a sequence of statistically independent binary digits with the probability of a 1 equal to p and the probability of a equal to 1 p. This conversion occurs at a rate penalty of entropy function H 2 (p) = p log 2 p (1 p) log 2 (1 p). Using the Bit-stuffing map 1-D S (1) 2d, to 2-D S (2) d,. Under certain assumptions they obtained lower bounds on cap(s d, ) (2) and cap(s,k). (2) 32

33 31 Results based on Bit-stuffing Technique Two recent papers both by Roth, Siegel, Wolf (1999, 21), who describe efficient schemes, based on the bit stuffing technique, for constructing codes satisfying the S d,, (2) S,k, (2) and S (2) 1, (Hard-Square). Their average code rate is within 1% of the capacity cap ( S (2) 1, ). 33

34 32 Possible Work - Developing a Software 1. Generate the constrained sequences recursively 2. Generate the transfer matrices recursively. 3. Find the corresponding compressed matrices. 4. Compute the largest eigenvalues of the compressed matrix. 5. Estimate the bounds on capacity. 6. Check the accuracy: if the error bound is bigger than a designed tolerance, then go back to (1). 34

35 32.1 Conjecture It is known that log 2 ( λ 2m+1 λ 2m ) cap(s). (3) There is a conjecture (Engel, 199) for hard square system: log 2 ( λ 2m λ 2m 1 ) cap(s). (4) Our calculations indicate that the conjecture seems also true for other constraints. If it is true, it will refine many obtained results. 35