ARITHMETIC coding (AC) [1] is an effective method for

Transcription

1 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 1 Hamming Distance Spectrum of DAC Coes for Equiprobable Binary Sources Yong Fang, Member, IEEE, Vlaimir Stankovic, Senior Member, IEEE, Samuel Cheng, Member, IEEE, an En-hui Yang, Fellow, IEEE Abstract Distribute Arithmetic Coing DAC) is an effective technique for implementing Slepian-Wolf coing SWC). It has been shown that a DAC coe partitions source space into unequalsize coebooks, so that the overall performance of DAC coes epens on the carinality an structure of these coebooks. The problem of DAC coebook carinality has been solve by the so-calle Coebook Carinality Spectrum CCS). This paper extens the previous work on CCS by stuying the problem of DAC coebook structure. We efine Hamming Distance Spectrum HDS) to escribe DAC coebook structure an propose a mathematical metho to calculate the HDS of DAC coes. The theoretical analyses are verifie by experimental results. Inex Terms Distribute source coing, Slepian-Wolf coing, istribute arithmetic coing, Hamming istance spectrum, coebook carinality spectrum. I. INTRODUCTION ARITHMETIC coing AC) [1] is an effective metho for ata compression that works by mapping each source sequence onto a half-open interval [l, h), where 0 l < h < 1. Though the principle of AC coes is rather simple, a major technical problem when putting AC coec into practice is that one has to use infinite-precision real numbers to represent l an h, which is impossible for a igital circuit. Fortunately, there is a canonical implementation in [2] that represents l an h with finite-precision integers an utilizes some scaling rules to solve the problems of renormalization an unerflow that are cause by finite-precision operations. An alternative solution to the complexity an precision problems in the AC coec is to use quasi-ac QAC) coes, which can be seen as a reuce-precision version of AC coes [3]. As other variable-length coes, AC coes suffer from error propagation when the bitstream is conveye over noisy channels. This problem can be solve by reserving a forbien interval in [0, 1) for error etection [4] an running maximum a posteriori MAP) ecoing for error correction [5]. For This work was supporte by the National Science Founation of China grant nos an ), the Program for New Century Excellent Talents in University of China grant no. NCET ), Provincial Founation for Youth Nova of Science an Technology of Shaanxi, China grant no. 2014KJXX-41), an the Funamental Research Fun for the Central Universities of China grant nos. 2014YQ001 an QN ). Y. Fang is with the College of Information Engineering, Northwest A&F University, Yangling, Shaanxi , China yfang79@gmail.com). V. Stankovic is with the Department of Electronic an Electrical Engineering, University of Strathclye, Glasgow, UK vlaimir.stankovic@strath.ac.uk). S. Cheng is with the School of Electrical an Computer Engineering, University of Oklahoma, Tulsa, OK samuel.cheng@ou.eu). E.-H. Yang is with the Department of Electrical an Computer Engineering, University of Waterloo, Waterloo, ON N2L 3G1, Canaa ehyang@uwaterloo.ca). The corresponing author is Y. Fang. QAC coes, state moels can be efine an use in a straightforwar manner for MAP or soft ecoing [6]. Such solutions are known as joint source-channel AC JSCAC). Forbien interval reservation is not the only solution to this problem, e.g., [7] achieves the same goal by inserting segment markers at fixe positions of bitstreams. Besies har markers, the soft synchronization mechanism is also a powerful option for the JSCAC which allows controlling the trae-off between reunancy an resilience [8]. To preict an evaluate the effectiveness of the JSCAC, [9] provies an analytical tool to erive the istance spectrum of the JSCAC an proposes an algorithm to compute the free istance of the JSCAC. Recently, AC coes also fin their application to lossless istribute source coing DSC), or Slepian-Wolf coing SWC) [10], which has traitionally been implemente with channel coes, e.g., turbo coes [11] an low-ensity paritycheck LDPC) coes [12], [13], [14]. Such solutions are known as istribute AC DAC) coes. In fact, DAC coes are ual coes of JSCAC coes, so they can be realize by either interval overlapping [15], [16], [17] or bitstream puncturing [18], [19], [20]. Naturally, DAC coes can be combine with JSCAC coes to obtain the so-calle istribute JSCAC DJSCAC) coes, which allow the coexistence of overlappe an forbien intervals to realize ata compression an error correction simultaneously [21]. Since the emergence of DAC coes, a lot of work has been one to verify the coing efficiency of DAC coes [16]. An important fining is that the resiual errors of DAC coes cannot be remove by increasing coe rate an/or length [16]. Thus, it is better to quantitatively measure the coing efficiency of DAC coes in terms of frame-error-rate FER) or symbolerror-rate SER) at a given coe rate. Moreover, it is shown that at least for short coe length, DAC coes outperform LDPC-base SWC coes with acceptable ecoer complexity [16]. However, the above results are heuristic an lack strict theoretical analyses. To obtain an illuminating insight into the coing efficiency an ecoer complexity of DAC coes, the concept of spectrum was introuce, an the following finings were reporte in [22], [23], [24]: A DAC coe partitions source space into unequal-size coebooks whose carinalities are proportional to the socalle initial spectrum [23]. Accoring to this fining, we can raw the following conclusion: For a DAC coe with initial spectrum f 0 u) see Sect. III for a formal efinition of the initial spectrum), its total rate loss to SWC limit [10] will ten to a constant 1 0 f 0u) log 2 f 0 u)u as coe

2 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 2 length goes to infinity, an hence the per-symbol rate loss will vanish as coe length increases [24]. DAC spectrum will become uniformly istribute as the ecoing procees, which implies that 1-away in Hamming istance) coewors in each coebook cannot be remove by increasing coe length [24]. Further, a loose lower boun of ecoing error probability is given as ɛ2 2 R ), where ɛ is the crossover probability between source an sie information SI), an R is coe rate [24]. Two techniques can be use to improve the coing efficiency of DAC coes [24]. First, the permutation technique can remove those closely-packe in Hamming istance) coewors in each coebook. Secon, the weighte branching technique can reuce the mis-pruning risk of proper paths uring the ecoing. Besies the above avances, the authors of [25] also notice the existence of 1-away in Hamming istance) coewors in each DAC coebook an propose the istribute block arithmetic coing DBAC) to solve this problem. In summary, the problem of eucing DAC coebook carinality has been solve, but we still know very little about DAC coebook structure. The only thing we know about the latter problem is that 1-away in Hamming istance) coewors in each DAC coebook almost always exist [24]. Obviously, to analyze the coing efficiency of DAC coes, more knowlege about DAC coebook structure is necessary. Motivate by this problem, this paper introuces the concept of Hamming istance spectrum HDS), which is essentially proportional to the average number of -away in Hamming istance) coeworpairs insie each DAC coebook. We enote the HDS by ψ n,r ), a function with respect to w.r.t.) inter-coewor Hamming istance {0,, n} that is parameterize by coe length n an rate R, an propose a mathematical metho to calculate ψ n,r ). Equippe with the HDS, it may be possible to calculate the FER an SER of DAC coes. Notice that to istinguish from the HDS, the spectrum efine in [22], [23], [24] will be formally referre to as coebook carinality spectrum CCS). The rest of this paper is arrange as follows. Section II escribes the encoing proceure of DAC coes. Section III briefly reviews the previous work on the CCS. Section IV efines the HDS for DAC coes an gives an example to illustrate how to calculate it by exhaustive enumeration. Section V evelops a mathematical metho to calculate ψ n,r 1), which is then generalize to ψ n,r ) for 2 in Sect. VI. Two implementation issues uring calculating ψ n,r ), i.e., complexity an convergency, are iscusse in Sects. VII an VIII, respectively. Experimental results are presente in Sect. IX to verify the correctness of the propose metho. Finally, Sect. X conclues this paper. Source Moel Following [22], [23], [24], this paper restricts the research scope to equiprobable binary sources. The reason is that the tackle issue is ifficult, thus we have to begin with the simplest but non-trivial source moel to simplify the analysis an make many har problems tractable. Note that, the concepts propose in this paper an previous work [22], [23], [24]) cannot easily be extene to nonuniform sources, because in contrast to uniform sources, for nonuniform sources, the DAC behaves like a source coe rather than a channel coe, making it very ifficult to buil the concepts of coebook an space partitioning. Notation This paper will aopt the notations efine in [26], which are also use in [24]. We use X to enote a ranom variable an fx) to enote a function of X. Corresponingly, we use x X to enote a realization of X, where X is the alphabet of X, an fx) to enote a function of x. We use X n X 0,, X n 1 ) to enote the tuple of n ranom variables an x n x 0,, x n 1 ) to enote a realization of X n. We use 0 n to enote the tuple of n consecutive 0s, while the meaning of 1 n is similar. We efine [i : j] {i,, j} an i : j) {i + 1),, j 1)}, while the meanings of [i : j) an i : j] are similar. Further, we efine q [i:j) q i,, q j 1 ) an the meanings of q [i:j], q i:j), an q i:j] are similar. For brevity, the crossover probability between source an SI is abbreviate to source-si crossover probability an enote by ɛ. Moreover, we use q to enote the length of enlarge intervals the same as [22] an [23], while ifferent from [24]). The operation of may enote the absolute value of a number, the carinality of a set, or the length of an interval, epening on the operan. The ot prouct of x n an y n is enote by x n, y n, an the Hamming istance between x n an y n is enote by H x n, y n ). We use {l, h]+ } an {ξl, h]} to enote the interval shifting an interval scaling operations, respectively, i.e., { {l, h] + } l +, h + ]. 1) {ξl, h]} ξl, ξh] The clip function max0, ) is abbreviate to ) +, i.e., ) + max0, ). II. REVIEW OF DAC ENCODING Let Y n be a tuple of n inepenent an uniformlyistribute i.u..) binary ranom variables with Y i py) = 0.5, where y B {0, 1} an i [0 : n). Let X n be another tuple of n i.u.. binary ranom variables with X i {Y i = y} px y) for x B. The correlation between X n an Y n is moele as a virtual binary symmetric channel BSC) with crossover probability p0 1) = p1 0) = ɛ. Accoring to the Slepian-Wolf theorem [10], if only Y n is available at the ecoer, lossless recovery of X n will be possible at rates R HX Y ) = H b ɛ) bits per symbol bps), where H b ) enotes the binary entropy function BEF), no matter whether Y n is available at the encoer or not. To compress X n, the rate-r, where 0 < R < 1, DAC encoer iteratively maps source symbols onto partiallyoverlappe intervals [0, q) an [1 q), 1), where q 2 R [15], [16]. Let [L i, H i ) be the interval after coing X i. It is easy to show that [L 0, H 0 ) = [0, 1) an H i L i ) = q i = 2 ir [24]. Therefore, we only nee to trace either L i or H i. It is usually more convenient to trace L i. As shown in [24], L i = lx i ), where lx i ) 1 q) q [0:i), X i. 2)

3 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 3 Since the length of the final interval after coing X n is always q n, it can be uniquely ientifie by log 2 q n = nr bits. To obtain the bitstream of X n, we scale L n to get S 2 nr L n. It is easy to see S = sx n ) 2 nr lx n ). The final interval [L n, L n + q n ) is now mappe onto [S, S + 2 nr nr ), which will be referre to as scale final interval. An important problem is: What is the range of S? This problem can be solve by consiering the following two extreme cases: If X n = 0 n, then [L n, H n ) = [0, q n ); an if X n = 1 n, then [L n, H n ) = [1 q n ), 1). Hence, L n [0, 1 q n )] an further S [0, 2 nr 2 nr nr )]. Then we calculate S. Because nr nr) [0, 1), we have 2 nr nr [1, 2) an further 2 nr 2 nr nr = 2 nr 1). Therefore, S [0 : 2 nr ), implying that S can be binarize into a string of nr bits, which is just the DAC bitstream of X n. Length of Scale Final Interval For simplicity, we will no longer consier the case nr > nr in the following. The reason is: If nr > nr, we can always re-encoe X n at rate R = nr /n, which will prouce a bitstream with exactly the same length. Therefore, in the rest of this paper, S = q n L n an sx n ) = q n lx n ) = 1 q) q [0:n) n, X n. 3) It is easy to know S [0, 2 nr 1)] an S [0 : 2 nr ). The scale final interval after coing X n is always [S, S + 1), i.e., the length of the scale final interval is always 1. Illustration of DAC Encoing An illustration to better unerstan DAC encoing is shown in Fig. 1. As shown in Fig. 1, if we take each coewor x n B n as a ball, DAC encoing is equivalent to putting 2 n balls into 2 nr bins accoring to sx n ). The rule is: If sx n ) = 0, x n is put into the 0-th bin; otherwise if sx n ) m 1), m], where m [1 : 2 nr ), x n is put into the m-th bin cf. Fig. 1). III. REVIEW ON CODEBOOK CARDINALITY SPECTRUM In this section, we briefly review the main results on CCS [22], [23], [24]. As shown in [24], a 2 nr, n) binary DAC coe is efine as an encoer m : B n [0 : 2 nr ) that assigns inex m [0 : 2 nr ) to each source sequence x n B n, an a ecoer ˆx n : [0 : 2 nr ) B n {e} that assigns an estimate ˆx n B n or an error message e to each inex m [0 : 2 nr ). The DAC encoing is in fact a many-to-one nonlinear mapping B n [0 : 2 nr ), which unequally partitions source space B n into 2 nr coebooks. Let C m, where m [0 : 2 nr ), be the m-th coebook. If x n C m, then sx n ) = m an sx n ) m 1), m] [0, 2 nr 1)]. 4) Especially, if x n C 0, sx n ) 0; otherwise, sx n ) m 1), m]. Since lx n ) = q n sx n ), lx n ) m 1)q n, mq n ] [0, 1 q n )]. 5) Especially, if x n C 0, lx n ) 0; otherwise, lx n ) m 1)q n, mq n ]. An important property of DAC coebooks is carinality, which is etermine in [22], [23], [24] by efining the socalle initial spectrum. Initial Spectrum Let X X 0, X 1, ) an q [0: ) q 0, q 1, ). Both L an H will converge to the following continuous ranom variable U 0 1 q) q [0: ), X, 6) whose probability ensity function pf) f 0 u) is calle the initial spectrum [22], [23], [24]. Accoring to the efinition of f 0 u), for m [1 : 2 nr ), the carinality of C m is proportional to the integral of f 0 u) over m 1)q n, mq n ] in the asymptotic sense, i.e., as n, mq n C m 2 n m 1)q n f 0 u)u. 7) For n sufficiently large, the interval m 1)q n, mq n ] will be so short that f 0 u) almost hols constant in m 1)q n, mq n ]. Thus C m f 0 mq n )2 n1 R) as n, i.e., C m is proportional to f 0 mq n ) in the asymptotic sense. For this reason, we will call f 0 u) the coebook carinality spectrum CCS) from now on. For equiprobable binary sources, Pr{X n = x n } 2 n for all x n B n. Thus, for all x n C m, Pr{X n = x n X n C m } 1/ C m, an for all m [0 : 2 nr ), Pr{ sx n ) = m} = C m 2 n. The 0-th Coebook Because sx n ) = 0 only if x n = 0 n, C 0 has one an only one coewor 0 n in any case, an its carinality is always 1, i.e., C 0 1. Conitional CCS The pf of U 0 given X j = b B is calle the conitional CCS given X j = b, an enote by f 0,j u b) [27]. Though DAC coebook carinality has been well stuie, very little about DAC coebook structure is known up to now. The only thing we know is that for a 2 nr, n) binary DAC coe, as n, the proportion of twin leaf noes in the ecoing tree will ten to 2 2 R ). Thus, the ecoing error probability is lower boune by ɛ2 2 R ), where ɛ is the source-si crossover probability [24]. IV. HAMMING DISTANCE SPECTRUM Just as channel coes, it is rather intuitive that another very important property of DAC coes is how far in Hamming istance) the coewors in each coebook keep away from each other. Therefore, we will below efine the Hamming istance spectrum HDS) to measure quantitatively the istribution of inter-coewor Hamming istances within each DAC coebook, which will be helpful for unerstaning DAC coebook structure. A. Definition of Hamming Distance Spectrum Coewor HDS The HDS w.r.t. coewor X n is efine as k X n ) { Xn : sx n ) = s X n ) an H X n, X n ) = }. 8)

4 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 4 bin 1 bin 2 bin 2 nr 2) bin 2 nr 1) nr 3 2 nr 2 2 nr 1 Fig. 1. Explanation of DAC encoing with ball binning. The horizontal axis is sx n ). In plain wors, k X n ) is the number of coewors X n in coebook sx n ) = m that are -away in Hamming istance) from X n. It is easy to see [0 : n] an 0 k X n ) n ). If we efine k0 X n ) = 1, then n =0 k X n ) = C m. Coebook HDS The HDS of the m-th coebook is efine as φ m ) E[k X n ) X n C m ]. 9) Coe HDS The HDS of the 2 nr, n) DAC coe is efine as ψ n,r ) E[k X n )]. 10) It is easy to see that φ m ) is proportional to the number of -away coewor-pairs within the m-th coebook an similarly, ψ n,r ) is proportional to the average number of -away coewor-pairs within all coebooks of the 2 nr, n) DAC coe. Asymptotic Coe HDS The asymptotic HDS of the rate-r DAC coe is efine as λ R ) lim n ψ n,r). 11) B. Calculating HDS by Exhaustive Enumeration In practice, ψ n,r ) can be calculate by exhaustive enumeration. Let us first consier φ m ). For equiprobable binary sources, φ m ) = Pr{X n = x n X n C m }k x n ) x n C m = 1/ C m ) k x n ), 12) x n C m where k x n ) is a realization of k X n ) [23]. Further, we can obtain ψ n,r ) = 2 nr 1 m=0 2 nr 1 = 2 n Pr{ sx n ) = m}φ m ) m=0 x n C m k x n ). 13) Convexity of Sum-of-HDS Since n =0 k x n ) C m for all x n C m, we have [24] n =0 2 nr 1 ψ n,r ) = 2 n C m 2 m=0 1 2 n1 R) f0 2 u)u, 14) 0 as n. Hence, n =0 Γ n ψ n,r) 2 n1 R) 1 0 f 2 0 u)u 1. 15) After expansion, we have Γ n = n 1 i=0 γ i, where γ i is the leveli expansion factor that is efine as the ratio of the number of level-i+1) noes to that of level-i noes in the DAC ecoing tree [23]. Apparently, Γ is a nonnegative an convex function in f 0 u), which takes the minumum value 1 only when f 0 u) is uniform over [0, 1). Similarly, we have n ψ n,r ) 2 n1 R) 16) =0 an the equality hols only if C m 2 n1 R), i.e., source space B n is equally partitione into 2 nr coebooks of carinality 2 n1 R). C. Example of Hamming Distance Spectrum To illustrate the concept of HDS, we give an example to show how to calculate ψ n,r ) for n = 4 an R = 0.5. The source space B n contains 2 n = 16 coewors an is partitione into 2 nr = 4 coebooks. We list all coewors of the source space in Tab. I. For each coewor x n, sx n ) the lower boun of the scale final interval) an m the corresponing coebook inex) are inclue in Tab. I, where ifferent coebooks are marke with ifferent colors for clarity. We also plot the positions of sx n ) for all coewors x n in Fig. 2. It can be seen that C 0 = 1, C 1 = 4, C 2 = 7, an C 3 = 4. We list the HDS of each coewor in Tab. I. After a simple calculation, we obtain the HDS of each coebook an the coe HDS, as shown in Tab. II. It is easy to verify Γ n = 5.125/4 > 1. V. MATHEMATICAL CALCULATION OF ψ n,r 1) For a large n, it is ifficult to calculate coe HDS ψ n,r ) through exhaustive enumeration in Subsect. IV-C because it nees the HDS k x n ) of all 2 n coewors. To get aroun it, we propose below a mathematical metho that is able to obtain ψ n,r ) irectly in the absence of k x n ). The proceure of the propose metho is still very time-consuming for large. Nevertheless, this is usually enough in practice because the ecoing failure of DAC coes is cause mainly by closely-packe in Hamming istance) coewors within each coebook. For clarity, we first use the simplest case = 1 to illustrate the principle of the evelope metho in

5 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) n s x ) Fig. 2. Example for illustrating the mapping of x n an sx n ), where n = 4 an R = 0.5. Each noe at the horizontal axis enotes the position of sx n ) corresponing to coewor x n. Different coebooks are marke with ifferent colors. TABLE I EXAMPLE OF CODEWORD HDS x n sx n ) m k 0 x n ) k 1 x n ) k 2 x n ) k 3 x n ) k 4 x n ) Sum TABLE II EXAMPLE OF CODEBOOK HDS AND CODE HDS Term = 0 = 1 = 2 = 3 = 4 Sum φ 0 ) φ 1 ) 1 4/4 6/4 2/4 0 4 φ 2 ) 1 10/7 16/7 10/7 6/7 7 φ 3 ) 1 6/4 6/ ψ n,r ) 1 20/16 28/16 12/16 6/ this section an then exten it to the general case 2 in the next section. The core iea of our propose metho is to expan ψ n,r ) as the sum of multiple tractable terms calle atoms below). To achieve this goal, we efine the following important concept. XOR Pattern We refer to Z n = X n X n ) as the XOR pattern between X n an X n, where X n an X n are two binary vectors. A. Expansion of ψ n,r 1) as Sum-of-Atoms Given H X n, X n ) = 1, there are n 1) = n ifferent XOR patterns between X n an X n, which must take the form of z n j) 0 j, 1, 0 n j 1 ), where j [0 : n). Let z i j) be the i-th element of z n j), then z j j) = 1 an z i j) = 0 for all other i j. We efine k j) 1 Xn ) { Xn : s X n ) = sx n ) an X n X n ) = z j)} n. 17) In plain wors, k j) 1 Xn ) is the number of coewors X n in coebook sx n ) = m that satisfy X n X n ) = z n j). It is easy to see k j) 1 Xn ) = 0 or 1, an n 1 j=0 kj) 1 Xn ) = k 1 X n ), where k X n ) is the coewor HDS of X n see Subsect. IV-A). With the help of XOR patterns, we can expan ψ n,r 1) as ψ n,r 1) = n 1 j=0 ω j, where ω j E[k j) 1 Xn )]. We refer to ω j as molecule, which can be further expane as ω j = Pr{X j = 0}β j 0) + Pr{X j = 1}β j 1) = 1/2) β j 0) + β j 1)), 18) where β j b) E[k j) 1 Xn ) X j = b] for b B. Similarly, we refer to β j b) as atom. In this way, we expan ψ n,r 1) as the sum of n molecules, each of which is the average of two atoms. The problem finally boils own to calculating atoms β j b) for all j [0 : n) an b B.

6 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 6 B. Definition of Risky Interval Before calculating β j b), we nee to introuce the concept of risky interval. From 3), it is easy to see that given X n X n ) = z n j), { s X sx n n ) + 1 q)q j n, if X j = 0 ) = sx n ) 1 q)q j n, if X j = 1, 19) which can be abbreviate to s X n ) = sx n ) + τ j b), where b B is the value of X j an τ j b) 1 q) 1) b q j n. For m [1 : 2 nr ), notice the following two points: if sx n ) = m, then sx n ) m 1), m]; if s X n ) = m, then s X n ) m 1), m] an sx n ) {m 1), m] τ j b)}, where {m 1), m] τ j b)} enotes a shifte version of m 1), m] refer to the Notation part of Sect. I for the efinition of interval shifting operation). Clearly, given X j = b an sx n ) = m [1 : 2 nr ), the necessary an sufficient conition for the existence of a binary vector Xn in the m-th coebook satisfying X n X n ) = z n j) is sx n ) I b), where I b) {m 1), m] τ jb)} m 1), m]. 20) Conversely, once sx n ) falls into I b), there must exist a binary vector X n in the m-th coebook that satisfies X n X n ) = z n j). Let { δ j b) min 1, τ jb) τ j b)) /2) δ + j b) min 1, τ, 21) jb) + τ j b)) /2) where τ j b) is the absolute value of τ j b). Then 20) can be rewritten as I b) = m 1) + δ j b), m δ+ j b)]. 22) where m [1 : 2 nr ), j [0 : n), an b B. We refer to I b) as a risky interval. It is easy to know I b) = if τ jb) 1. The 0-th Risky Interval Because C 0 contains only one coewor 0 n, I b) is meaningless for m = 0. Thus, we will ignore in the following iscussion. I b) 0,j Length of Risky Interval Let I b) be the length of Ib) an ) + max0, ), then I b) = 1 τ jb) ) + = 1 1 q)q j n ) +. 23) Obviously, I b) [0, 1), I0) = I1), an Ib) I b) 2 nr 1,j. In aition, Ib) 1,j = = is a nonecreasing function w.r.t. j, i.e., 0 I b) m,0 Ib) m,n 1 < 1. Example of Risky Interval Let n = 4 an R = 0.5, then m [0 : 2 nr ) = {0, 1, 2, 3}, j [0 : n) = {0, 1, 2, 3}, an q = 1/ 2. It is easy to obtain τ 0 b), τ 1 b), τ 2 b), τ 3 b) ) = , , , ). 24) The risky intervals I b) for all m {1, 2, 3}, j {0, 1, 2, 3}, an b B are liste in Tab. III. For clarity, the relative TABLE III EXAMPLE OF RISKY INTERVALS Term j = 0 j = 1 j = 2 j = 3 I 0) 1,j 0, ] 0, ] 0, ] I 0) 2,j 1, ] 1, ] 1, ] I 0) 3,j 2, ] 2, ] 2, ] I 1) 1,j , 1] , 1] , 1] I 1) 2,j , 2] , 2] , 2] I 1) 3,j , 3] , 3] , 3] I b) relationship of I b) for ifferent j an b is illustrate by Fig. 3. It is easy to see that I b) Ib) for j < j. It can be seen that because τ 0 b) > 1, I b) m,0 In aition, we can fin that I b) w.r.t. j. C. Link between Atom an Risky Interval = for all m {1, 2, 3}. is inee nonecreasing Accoring to the efinitions of β j b) an I b), we can easily link them as β j b) = where 2 nr 1 m=0 Pr{ sx n ) = m X j = b}pi b) m, b), 25) pi b) m, b) Pr{sXn ) I b) sxn ) = m, X j = b}. 26) It is easy to see that in the asymptotic sense, i.e., as n, pi b) m, b) {q n I b) } f 0,ju b)u mq n m 1)q n f 0,j u b)u, 27) where f 0,j u b) is the conitional CCS given X j Sect. III) an {q n I b) = b see } enotes a scale version of Ib) see the Notation part of Sect. I for the efinition of interval scaling operation). As n increases, m 1)q n, mq n ] will converge to a real number. Hence, for n sufficiently large, f 0,j u b) will be approximately uniform over m 1)q n, mq n ] an m, b) q n I b) m 1)q n, mq n ] = Ib) pi b) where j [0 : n). Equivalently, = 1 1 q)q j n ) +, 28) pi b) m,n j m, b) 1 1 q)q j ) +, 29) where j [1 : n]. It can be seen that pi b) m,n j m, b) keeps the same for all m [1 : 2 nr ), thus for n sufficiently large, β n j b) 1 1 q)q j ) +. It is easy to know that β n j b) [0, 1), β n j 0) = β n j 1), an β n j b) is a nonincreasing function w.r.t. j.

7 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 7 m 1, m] 0) I m, n 1 1) I m, n 1 0) I m, n 2 1) I m, n 2 m 1 Fig. 3. 0) I m, n Illustration of risky interval I b) for q = 1/ 2. The horizontal axis is sx n ). 1) I m, n 3 m D. Calculation of Coe HDS After knowing atoms, we can obtain molecules ω n j 1 1 q)q j ) +, where j [1 : n]. In turn, we can obtain the coe HDS as below n ψ n,r 1) 1 1 q)q j ) +. 30) j=1 Finally, we can obtain the asymptotic coe HDS as below λ R 1) = 1 1 q)q j ) +. 31) j=1 VI. MATHEMATICAL CALCULATION OF ψ n,r ) FOR 2 One can easily exten the metho evelope in Sect. V to the general case 2. This section will first expan ψ n,r ) as the sum of atoms, then efine the risky interval to calculate atoms, an finally give the expression for ψ n,r ). A. Expansion of ψ n,r ) as Sum-of-Atoms Given H X n, X n ) =, there are n ) ifferent XOR patterns between X n an X n. Let j j 1,, j ), where 0 j 1 < < j < n. The XOR pattern between X n an X n must take the form of z n j) 0 j1, 1, 0 j2 j1 1,, 0 j j 1 1, 1, 0 n j 1 ). 32) In other wors, z j1 j) = = z j j) = 1 an z i j) = 0 for other i / j, where z i j) enotes the i-th element of z n j). Beginning with j 1 [0 : n )], we can obtain j j 1 : n + ) for [2 : ] by recursion. Let us efine k j) Xn ) { Xn : s X n ) = sx n ) an X n X n ) = z j)} n an ω j E[k j) Xn )]. Then we can expan ψ n,r ) as ψ n,r ) = n j 1=0 n 1 j =j ) ω j. 34) Let X j X j1,, X j ) an b b 1,, b ) B, then we can further expan ω j as ω j = 2 1 b=0 β jb), where β j b) E[k j) Xn ) X j = b]. B. Length of Risky Interval Accoring to 3), given X n X n ) = z n j), we have s X n ) = sx n ) + τ j b), where b B is the value of X j an τ j b) 1 q) 1) b q j n. 35) Given X j = b an sx n ) = m [1 : 2 nr ), the necessary an sufficient conition for the existence of a binary vector X n in the m-th coebook satisfying X n X n ) = z n j) is sx n ) I b), where I b) {m 1), m] τ jb)} m 1), m]. 36) Let us efine { δ j b) min 1, τ jb) τ j b)) /2) δ + j b) min 1, τ jb) + τ j b)) /2) It is easy to obtain the risky interval Obviously, I b). 37) I b) = m 1) + δ j b), m δ+ j b) ]. 38) I b) = 1 τ jb) ) +. = if τ jb) 1. The length of I b) C. Link between Atom an Risky Interval Accoring to the efinitions of β j b) an I b), we have β j b) = where 2 nr 1 m=0 Pr{ sx n ) = m X j = b}pi b) m, b), is 39) pi b) m, b) Pr{sXn ) I b) sxn ) = m, X j = b}. 40) In the asymptotic sense, pi b) m, b) Ib) = 1 τ jb) ) +. 41) Let n j) n j 1,, n j ) for 1 j 1 < < j n, then 41) is equivalent to pi b) m,n j m, b) 1 1 q) ρ jb) ) +, 42)

8 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 8 where ρ j b) 1) b q j. 43) Therefore, for n sufficiently large, β n j b) 1 1 q) ρ j b) ) +. 44) After a simple euction, we obtain the tight ranges of j in 43) as follows: j 1 [1 : n + 1)] an j j 1 : n + )] for [2 : ]. D. Calculation of Coe HDS After knowing atoms, we can obtain molecules as below 1 ω n j 2 b=0 1 1 q) ρ j b) ) +, 45) where 1 j 1 < < j n. In turn, we can obtain the coe HDS as below n +1 ψ n,r ) 2 n 1 1 q) ρ j b) ) +. j 1=1 j =j 1 +1 b=0 46) Finally, we can obtain the asymptotic coe HDS as below λ R ) = 2 j 1=1 j =j b=0 1 1 q) ρ j b) ) +. 47) VII. COMPLEXITY OF CALCULATING DAC HDS The complexity of 46) is O n ) 2 ). To reuce the complexity, we exploit the fact ρ j b) = ρ j 1 b) to obtain n +1 ψ n,r ) = 2 1 n 0,1 1 ) 1 1 q) ρ j b) ) +. θ0 2 ) = j 1=1 j =j 1 +1 b=0 48) Therefore, in the following, the leaing bit of b will always be 0 without explicit eclaration. Though the complexity of 46) is now reuce to O n ) 2 1 ), it is still unacceptable for large n an. Thus the propose metho is feasible only for small n an. The complexity of 48) can further be reuce by swapping the orer of summations 0,1 1 ) ψ n,r ) = 2 1 where θb) n j = ηb, j ) an further ηb, j ) 1 j 1 = 1 b=0 θb), 49) 1 1 q) ρ j b) ) +. 50) j 1=1 The complexity of θb) is O n ) ), still high for large n an. However, we fin that in some special cases, ηb, j ) 0 for all j > Jb), where Jb) is an integer totally epening on q while unrelate to n. Thus, we can obtain θb) = Jb) j = ηb, j ), whose complexity is O ) Jb) ). For n Jb), the complexity of θb) will be significantly reuce. The trick for fining Jb) is to make ρ j b) 1 q) 1 for all j > Jb). However, up to now, this problem is solve only for the special case that there is no more than one 1 in b, while still remains open for the general case. To facilitate the escription, we ivie the case that there is no more than one 1 in b into three subcases: b is an all-0 vector, i.e., b = 0 ; there is only one 0 before the 1 in b, i.e., b = 0, 1, 0 2 ); there are two or more 0s before the 1 in b, i.e., b = 0 a, 1, 0 a 1 ), where a 2. We list the expressions of Jb) an θb) in the above three subcases in Tab. IV, while the etaile euctions are place in the Appenix. Below we will give some examples of Jb) an θb) in special cases by looking up Tab. IV an iscuss the existence of Jb) in general cases. Afterwars, we will propose an approximation of ψ n,r ) for large n an, an finally justify the practical values of 46). A. Examples in Special Cases 1) Examples when b = 0 : For = 1, by looking up Tab. IV, we can obtain J0) = log q 1 q) J0) θ0) = 1 1 q)q j ). 51) j=1 For = 2, by looking up Tab. IV, we can obtain J0 ) = log q 1 q) log q 2 q 1 ) J0 2 ) 1 1 q)q j 2 + q j1 ) ). 52) j 2=2 j 1=1 2) Examples when b = 0, 1, 0 2 ): For = 2, by looking up Tab. IV, we can obtain J01) = 2 log q 1 q) J01) θ01) = 1 1 q)q j 2 q j1 ) ). 53) j 2=2 j 1=1 For = 3, by looking up Tab. IV, we can obtain J010) = 2 log q 1 q) log q 2 q 1 ) J010) 3 1 θ010) = 1 1 q)q j 3 q j2 + q j1 ) ). j 3=3 j 2=2 j 1=1 54) 3) Examples when b = 0 a, 1, 0 a 1 ) an a 2: For = 3 an a = 2, by looking up Tab. IV, we can obtain J001) = log q 1 q) log q 2q 1 q 2 ) J001) 3 1 θ001) = 1 1 q)q j 3 + q j2 q j1 ) ). j 3=3 j 2=2 j 1=1 55)

9 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 9 TABLE IV EXAMPLE OF Jb) AND θb) Term J0 ) J010 2 ) J0 a 10 a 1 ) θ0 ) θ010 2 ) θ0 a 10 a 1 ) J010 2 ) j = J0 a 10 a 1 ) j = J0 ) Expression log q 1 q) log q 2 q 1 ) 2 log q 1 q) log q 2 q 2 ) log q 2 q 1 )1 q) 1 + 2q a ) j 1 j = j 1 = 1 j 1 j 1 = 1 j 1 j 1 = j 1 =1 2 1 j 1 =1 2 1 j 1 =1 1 1 q) ) q j )) 1 1 q) q j 2q j 1 )) 1 1 q) q j 2q j a+1 B. Discussions in General Cases As shown above, if the leaing bit of b is 0 an b contains no more than one 1, there will exist an integer Jb) unrelate to n such that ηb, j ) 0 for all j > Jb). The secret hien behin it is that ρ j b) is always positive in this case. On the contrary, if there are more than one 1s in b, the positiveness of ρ j b) cannot be guarantee so that it is unknown whether there still exists an integer Jb) unrelate to n such that ηb, j ) 0 for all j > Jb). Through many experiments, we fin that ηb, j ) usually tens to zero as j increases. Thus, in most cases, there exists an integer Jb) unrelate to n such that ηb, j ) 0 for all j > Jb). However, we are not able to prove this conjecture. Note that, if Jb) exists, the two coewors X n an X n belonging to the same coebook iffer from each other only in the last Jb) symbols. C. Approximation of ψ n,r ) for large n an The complexity of calculating ψ n,r ) by 46) is unacceptable for large n an, so we give below a simple metho to calculate the approximation of ψ n,r ) for large n an. Let X n an X n be two binary sequences belonging to the same coebook. As H X n, X n ) = increases, X n an X n will become less correlate. Thus, for a large, X n an X n can be taken as two binary sequences that are inepenently rawn from B n. This means: For sufficiently large, ψ n,r ) can be well approximate by the scale combination formula ) n 1 ) ψ n,r ) f0 2 u)u 2 n1 R), 56) 0 ) 1 where 0 f 0 2 u)u 2 n1 R) is in fact the average coebook carinality [24]. D. Practical Values of the Propose Metho To obtain a complete HDS by 46), one must try all possible source sequences x n B n an all possible XOR patterns z n B n. Actually, by the Monte-Carlo metho, one may obtain an approximate HDS much faster. Naturally, one may ask: Does the propose metho make sense in practice? Our answer is YES. There are two reasons for this answer. First, the ecoing failures of DAC coes come mainly from closely-packe in Hamming istance) coewors in each coebook cf. Fig. 7a) in Subsect. IX-D), so it is unnecessary to calculate the exact value of ψ n,r ) for large by 46). Though the complexity of computing ψ n,r ) by 46) is rather high for large, it is very low for small much faster than the Monte-Carlo metho). Hence, the propose metho is useful in practice. Secon, the propose metho may be use to compute the FER of DAC coes. Let y n be the SI available only at the ecoer an ˆx n be the recovere version of x n. If we assume that x n 1 is known at the ecoer, then = x n 1. Let Pr{e} be the FER an Pr{e x n 1 } be the conitional FER given x n 1 known at the ecoer, then Pr{e x n 1 } = Pr{ˆx n 1 x n 1 } an Pr{e x n 1 } < Pr{e}. Given x n C m, c n = x n 0 n 1, 1) may or may not belong ˆx n 1 to C m. If c n / C m, the ecoing will always be correct, regarless of y n 1 ; otherwise, i.e., if c n C m, the correctness of the ecoing purely epens on y n 1. Therefore, Pr{e x n 1 } = Pr{c n C m } Pr{y n 1 x n 1 }. 57) It is easy to obtain Pr{c n C m } = Pr{sx n ) I b) m,n 1 }. 58) Let Pr{X Y } = ɛ. By 29), we can obtain Pr{e x n 1 } 2 2 R )ɛ as n, which is just the lower boun given in [24]. The above metho can be extene to more complex cases, i.e., all but the last n > 1 symbols of each sequence are known at the ecoer. When n = n, Pr{e x n n } = Pr{e}. Actually, the resiual errors of DAC coes happen mainly at sequence tails cf. Fig. 7b) in Subsect. IX-D), so Pr{e x n n } may be very close to Pr{e} for n n. Therefore, Pr{e x n n }, where n n, may be taken as an approximation of Pr{e} an the complexity of computing the FER of DAC coes is significantly reuce. VIII. CONVERGENCY OF DAC HDS Another interesting question regaring the DAC HDS is: Will ψ n,r ) finally converge to a finite value for any finite as n goes to infinity, i.e., λ R ) < for <? The answer

10 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 10 in general is NO. However, we will show below that for = 1 an 2, the answer is YES. In the case of = 1, accoring to the analysis in Subsect. VII-A, we can obtain J0) λ R 1) = 1 1 q)q j ), 59) j=1 where J0) is given by 51). Apparently, λ R 1) <, i.e., λ R ) converges for = 1. In the case of = 2, accoring to the analysis in Subsect. VII-A, we can obtain λ R 2) = 1 2 J00) j 2=2 j 1=1 J01) j 2=2 j 1=1 1 1 q)q j 2 + q j1 ) ) q)q j 2 q j1 ) ), 60) where J00) an J01) are given by 52) an 53), respectively. Apparently, λ R 2) <, i.e., λ R ) converges for = 2. The convergency of ψ n,r ) for 3 is unknown. However, we have foun that ψ n,r ) may not converge in some cases. For example, if = 3 an q = 5 1)/2, it is easy to verify q j3 q j3 1) q j3 2) ) 0 an thus 1 1 q) q j3 q j3 1) q j3 2) ) ) TABLE V COMPARISON OF THEORETICAL AND EMPIRICAL VALUES OF Γ n R 2/6 3/6 4/6 5/6 Theoretical Γ n Empirical Γ n f 2 u)u A. Correctness Verification of 46) The aim of the first experiment is to verify the correctness of 46), which was achieve by comparing the theoretical values of ψ n,r ) with its empirical values. Some results for coe length n = 12 are presente in Fig. 4. Notice that since ψ n,r 0) 1, it is not plotte in Fig. 4. We teste four ifferent coe rates: R = 2/6, 3/6, 4/6, an 5/6, but only the results for R = 2/6 an 5/6 are inclue in Fig. 4 for conciseness. It can be seen that the theoretical values of ψ n,r ) coincie with its empirical values perfectly, which confirms the correctness of 46). Similar results are obtaine for ifferent values of n an R. In aition, we calculate the theoretical values of Γ n using 15) an its empirical values by experiments. The results are liste in Tab. V, where the numerical values of 1 0 f 2 u)u, which were obtaine through the numerical algorithm in [23], are also inclue. It can be seen that the theoretical values of Γ n, the empirical values of Γ n, an the numerical values of 1 0 f 2 u)u are very close to each other. These finings also confirm the correctness of 46). Therefore, θ011) = > n 3 1 j 3=3 j 2=2 j 1=1 1 1 q) q j3 q j2 q j1 ) + n 1 1 q) q j3 q j3 1) q j3 2) ) + j 3=3 = n 2. 62) As n goes to infinity, θ011) will ten to infinity, i.e., ψ n,r ) oes not converge for = 3 if q = 5 1)/2. IX. EXPERIMENTAL RESULTS We implemente 46) in MATLAB to calculate the theoretical values of ψ n,r ) an the DAC coec in C language to obtain the empirical values of ψ n,r ). Since DAC HDS oes not epen on SI, we ignore SI in implementation for simplicity. We first generate a length-n equiprobable binary sequence as the source an compresse it by the DAC encoer. Then, the DAC ecoer parse the bitstream through a epthfirst full search that was implemente by a recursive function. For each [0 : n], the ecoer counte the number of paths that were -away in Hamming istance) from the source. For fairness, 10 3 trials were run an the average number of paths that were -away from the source was output as the empirical value of ψ n,r ). The precision of the use DAC coec was 32-bit. Four experiments were conucte to stuy the properties of DAC coes from ifferent aspects. B. Convergency of DAC HDS The aim of the secon experiment is to stuy the convergency of DAC HDS, which was achieve by trying ifferent coe lengths n range from 12 to 28. Both theoretical an empirical values of ψ n,r ) are plotte in Fig. 5. Consiering the computational complexity, only the results of = 1, 2, an 3 are inclue in Fig. 5. To show how coe rate R impacts the convergency of DAC HDS, two special coe rates R = 0.5 an log 2 [ 5 1)/2] were trie. From Fig. 5, it can be seen that the theoretical values of ψ n,r ) coincie with its empirical values perfectly. For = 1 an 2, ψ n,r ) remains constant as coe length n increases, i.e., ψ n,r ) converges as n goes to infinity. For = 3, ψ n,r ) remains constant when coe rate R = 0.5 while grows continuously when R = log 2 [ 5 1)/2] as n increases. Therefore, the convergency of ψ n,r ) for 3 epens on coe rate: At some coe rates, ψ n,r ) may ten to infinity as coe length increases, i.e., oes not converge. This property of DAC coes is very ifferent from that of ranom coes because accoring to the law of large numbers LLN), for any <, ψ n,r ) of ranom coes will ten to 0 as coe length goes to infinity. C. Comparison of DAC Coes with Other Coes The aim of the thir experiment is to compare DAC coes with ranom coes an some practical channel coes. For coe length n = 24, the empirical HDS of DAC coes an the theoretical HDS of ranom coes are given in Fig. 6. For ranom coes, the inter-coewor Hamming istances within

11 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 11 n = 12, R = 2/6 n = 12, R = 5/6 80 Theoretical Empirical 0.7 Theoretical Empirical ψn,r) ψn,r) a) b) Fig. 4. Correctness verification of 46) for n = 12. a) R = 2/6. b) R = 5/ R = 1/2 Theoretical, = 1 Theoretical, = 2 Theoretical, = 3 Empirical, = 1 Empirical, = 2 Empirical, = R = log 2 [ 5 1)/2] Theoretical, = 1 Theoretical, = 2 Theoretical, = 3 Empirical, = 1 Empirical, = 2 Empirical, = 3 ψn,r) ψn,r) n a) n b) Fig. 5. Convergency of DAC HDS. a) R = 0.5. b) R = log 2 [ 5 1)/2]. each coebook obey the binomial istribution, so ψ n,r ) can be calculate by 56). It can be seen that at low rates, the HDS of DAC coes is similar to that of ranom coes, while at high rates, the HDS of DAC coes is ifferent from that of ranom coes. Similar results are also obtaine for ifferent values of n an R. For turbo coes, the Fano algorithm was moifie to compute the HDS in [28], where some examples of selecte turbo coes with short interleaving were given. For a rate-0.5 turbo coe base on the 7, 5) recursive systematic convolutional RSC) coe with 8 8 nonuniform interleaving, the HDS is w8) = 0.34, w9) = 1.5, w10) = 0.63, w11) = 0.12, an w12) = The minimum Hamming istance MHD) is 8. For LDPC coes, the nearest nonzero coewor search NNCS) algorithm was propose to fin the MHD an multiplicity in [29], where the results for some well-known LDPC coes were reporte. For the 3, 6)-regular 504, 252) an 1008, 504) MacKay coes [30], the MHDs are 20 an 34, an the multiplicities are 2 an 1, respectively. For the p-11 Margulis coe [31], the MHD is 40 an the multiplicity is 66. For the 13, 5) an 17, 5) Ramanujan-Margulis coes [32], the MHDs are 14 an 24, an the multiplicities are 2184 an 204, respectively. From the above results, we can fin that compare to ranom coes, turbo coes, an LDPC coes, the main rawback of DAC coes is that the MHD is almost always 1, regarless of coe length n an rate R. This is because for <, ψ n,r ) of DAC coes oes not converge to 0 as coe length n goes to infinity refer to Sect. VIII for the examples of = 1 an 2). On the contrary, the MHDs of turbo coes an LDPC coes are far larger than 1. Even for ranom coes, as coe length n goes to infinity, the MHD will graually ten

12 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 12 n = 24, R = 2/6 n = 24, R = 5/ Ranom coes DAC coes Ranom coes DAC coes ψn,r) ψn,r) a) b) Fig. 6. Comparison of the HDS of DAC coes with that of ranom coes for n = 24. a) R = 2/6. b) R = 5/6. to infinity because ψ n,r ) will ten to zero for any < accoring to the LLN. In aition, it can be seen that ψ n,r ) of DAC coes is greater than that of ranom coes for small while smaller than that of ranom coes for large, implying that the HDS of DAC coes is inferior to that of ranom coes especially at high rates). Base on these results, it can be conclue that the pure DAC coes mapping all symbols of each sequence onto overlappe intervals) shoul be worse than ranom coes, turbo coes, an LDPC coes. D. Properties of Decoing Errors Let us first stuy the property of frame errors. In Fig. 7a), we plot the occurrence rate of -errors frames, where n = 64 an R = 0.5. For clarity, the occurrence rate of error-free frames is not inclue in Fig. 7a). It can be seen that most of erroneous frames inclue only very few erroneous symbols, implying that DAC ecoing errors are cause mainly by closely-packe coewors within the same coebook. Next we stuy the property of symbol errors. It is pointe out in Subsect. VII-B that the two coewors belonging to the same coebook iffer mainly in the last few symbols, so symbol errors shoul happen mainly at the tail of each frame. To verify this point, we investigate how the error probability of each symbol is impacte by its position in a frame. For n = 64 an R = 0.5, the iniviual SER of each symbol versus its inex i is plotte in Fig. 7b). It can be seen that symbol errors are not uniformly istribute over i [0 : n) an most of symbol errors o happen at the last few symbols of each frame, as reporte in [15], [16]. The above phenomena inspire some improvements of DAC coes, e.g., permutating coewors in each coebook [24], mapping the last few symbols of each sequence onto nonoverlappe intervals [15], [16], [27], etc. After these improvements, DAC coes may outperform LDPC coes an turbo coes, especially for short sequences [15], [16], [24], [27]. These improvements are all base on the principle of refining the HDS of DAC coes by sparsifying the last few levels of DAC ecoing trees. In such a way, closely-packe coewors in each coebook can be remove, i.e., ψ n,r ) 0 for small. X. CONCLUSION The analysis of DAC coes is an interesting an challenging task. By extening our previous work, this paper makes another step forwar. We efine Hamming istance spectrum to escribe how DAC coebooks are constructe an propose a metho to calculate DAC HDS by the mathematical means. The correctness of the propose metho is verifie by experiments. We also show how DAC coes can be improve from the viewpoint of HDS. Despite the above avances, many important questions regaring DAC HDS, e.g., complexity, convergency, etc., still remain open. First, uner which conitions will the HDS converge as coe length increases? Secon, if the HDS oes converge as coe length increases, can we fin some way to calculate the HDS with linear or near-linear complexity? Thir, if the HDS oes not converge, how fast will it grow as coe length increases? These ifficult problems will be tackle in the future. Finally, another big challenge is the generalization of DAC HDS to nonuniform binary sources an even non-binary sources. A. Deuction of Jb) APPENDIX 1) Special Case of b = 0 : In this case, since j for all [1 : ], we have ρ j b) = q j q j + 1 q ) > 0. 63)

13 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) 13 n =64 an R =0.5 n =64 an R = ǫ=0.04 ǫ=0.05 ǫ= ǫ=0.04 ǫ=0.05 ǫ= Occurrence Rate Iniviual SER Number of Erroneous Symbols a) i b) Fig. 7. Properties of DAC ecoing errors, where n = 64 an R = 0.5. a) Occurrence rate of -errors frames, where means the number of erroneous symbols in a recovere frame. b) Iniviual SER versus symbol position. A necessary conition for ρ j b) < 1 q) 1 is 1 ) q j < 1 q) 1 q = 2 q 1 )1 q) 1, 64) which is followe by j < log q 1 q) log q 2 q 1 ). 2) Special Case of b = 0, 1, 0 2 ): In this case, since j for all [1 : ], we have ρ j b) = q j ) 2q j 1 q j q j 1) + 2 q ) > 0. 65) A necessary conition for ρ j b) < 1 q) 1 is 2 ) q j 1 q) < 1 q) 1 q = 2 q 2 )1 q) 1, 66) which is followe by j < 2 log q 1 q) log q 2 q 2 ). 3) Special Case of b = 0 a, 1, 0 a 1 ) for a 2 an 3: In this case, since j for all [1 : ], we have ρ j b) = q j + q j ) 2q j a 1 q ) 2q a > 0. 67) A necessary conition for ρ j b) < 1 q) 1 is 1 ) q j < 1 q) 1 q + 2q a = 2 q 1 )1 q) 1 + 2q a, 68) which is followe by j < log q 2 q 1 )1 q) 1 + 2q a ). REFERENCES [1] J. Rissanen, Generalize Kraft inequality an arithmetic coing, IBM J. Research & Development, vol. 20, no. 3, pp , May [2] I. Witten, R. Neal, an J. Cleary, Arithmetic coing for ata compression, Commun. of the ACM, vol. 30, no. 6, pp , Jun [3] P. Howar an J. Vitter, Practical implementations of arithmetic coing, in: Image an Text Compression, J. A. Storer, e., pp , Kluwer Acaemic Publishers, Boston, Mass, USA, [4] C. Boy, J. Cleary, S. Irvine, I. Rinsma-Melchert, an I. Witten, Integrating error etection into arithmetic coing, IEEE Trans. Commun., vol. 45, no. 1, pp. 1 3, Jan [5] B. Pettijohn, M. Hoffman, an K. Sayoo, Joint source/channel coing using arithmetic coes, IEEE Trans. Commun., vol. 49, no. 5, pp , May [6] T. Guionnet an C. Guillemot, Soft an joint source-channel ecoing of quasi-arithmetic coes, EURASIP J. Applie Signal Process., no. 3, pp , Mar [7] I. Soagar, B. Chai, an J. Wus, A new error resilience technique for image compression using arithmetic coing, in: Proc. IEEE ICASSP, pp , Istanbul, Turkey, Jun [8] T. Guionnet an C. Guillemot, Soft ecoing an synchronization of arithmetic coes: Application to image transmission over noisy channels, IEEE Trans. Image Process., vol. 12, no. 12, pp , Dec [9] S. Ben-Jamaa, C. Weimann, an M. Kieffer, Analytical tools for optimizing the error correction performance of arithmetic coes, IEEE Trans. Commun., vol. 56, no. 9, pp , Sep [10] D. Slepian an J. Wolf, Noiseless coing of correlate information sources, IEEE Trans. Inform. Theory, vol. 19, no. 4, pp , Jul [11] J. Garcia-Frias an Y. Zhao, Compression of correlate binary sources using turbo coes, IEEE Commun. Lett., vol. 5, no. 10, pp , Oct [12] A. Liveris, Z. Xiong, an C. Georghiaes, Compression of binary sources with sie information at the ecoer using LDPC coes, IEEE Commun. Lett., vol. 6, no. 10, pp , Oct [13] Y. Fang, LDPC-base lossless compression of nonstationary binary sources using sliing-winow belief propagation, IEEE Trans. Commun., vol. 60, no. 11, pp , Nov [14] Y. Fang, Asymmetric Slepian-Wolf coing of nonstationarily-correlate M-ary sources with sliing-winow belief propagation, IEEE Trans. Commun., vol. 61, no. 12, pp , Dec

14 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) [15] M. Grangetto, E. Magli, an G. Olmo, Distribute arithmetic coing, IEEE Commun. Lett., vol. 11, no. 11, pp , Nov [16] M. Grangetto, E. Magli, an G. Olmo, Distribute arithmetic coing for the Slepian-Wolf problem, IEEE Trans. Signal Process., vol. 57, no. 6, pp , Jun [17] X. Artigas, S. Malinowski, C. Guillemot, an L. Torres, Overlappe quasi-arithmetic coes for istribute vieo coing, in: Proc. IEEE ICIP, 2007, vol. II, pp [18] S. Malinowski, X. Artigas, C. Guillemot, an L. Torres, Distribute coing using puncture quasi-arithmetic coes for memory an memoryless sources, IEEE Trans. Signal Process., vol. 57, no. 10, pp , Oct [19] X. Chen an D. Taubman, Distribute source coing base on puncture conitional arithmetic coes, in: Proc. IEEE ICIP, pp , Sep [20] X. Chen an D. Taubman, Couple istribute arithmetic coing, in: Proc. IEEE ICIP, pp , Sep [21] M. Grangetto, E. Magli, an G. Olmo, Distribute joint source-channel arithmetic coing, in: Proc. IEEE ICIP, 2010, pp [22] Y. Fang, Distribution of istribute arithmetic coewors for equiprobable binary sources, IEEE Signal Process. Lett., vol. 16, no. 12, pp , Dec [23] Y. Fang, DAC spectrum of binary sources with equally-likely symbols, IEEE Trans. Commun., vol. 61, no. 4, pp , Apr [24] Y. Fang an L. Chen, Improve binary DAC coec with spectrum for equiprobable sources, IEEE Trans. Commun., vol. 62, no. 1, pp , Jan [25] J. Zhou, K. Wong, an J. Chen, Distribute block arithmetic coing for equiprobable sources, IEEE Sensors Journal, vol. 13, no. 7, pp , Jul [26] A. Gamal an Y. Kim, Network information theory, Cambrige University Press, [27] Y. Fang, V. Stankovic, S. Cheng, an E.-H. Yang, Depth-first ecoing of istribute arithmetic coes for uniform binary sources, submitte to IEEE Trans. Commun. [28] R. Poemski, W. Holubowicz, C. Berrou, an G Battail, Hamming istance spectra of turbo-coes, Annals of Telecommunications, vol. 50, no. 9 10, pp , Sep.-Oct., [29] X. Hu, M. Fossorier, an E. Eleftheriou, On the computation of the minimum istance of low-ensity parity-check coes, in: Proc. IEEE Int l Conf. Commun., vol. 2, pp , Jun [30] D. MacKay, Goo error-correcting coes base on very sparse matrices, IEEE Trans. Inform. Theory, vol. 45, no. 3, pp , Mar [31] G. Margulis, Explicit constructions of graphs without short cycles an low ensity coes, Combinatorica, vol. 2, no. 1, pp , Jan [32] J. Rosenthal an P. Vontobel, Construction of LDPC coes base on Ramanujan graphs an ieas from Margulis, in: Proc. 38th Annual Allerton Conf. Commun., Computing, an Control, Monticello, IL, Oct Yong Fang receive his BEng, MEng, an PhD from Xiian University, Xi an China, in 2000, 2003, an 2005, respectively, all in signal processing. Then, he was a post-octoral fellow for one year with Northwestern Polytechnical University, Xi an China. From 2007 to 2008, he joine Hanyang University, Seoul Korea, as a research professor. He is currently a full professor with Northwest A&F University, Shaanxi Yangling, China. He ha long experiences in harware system evelopment, e.g., FPGA-base Xilinx Vertex series) vieo coec esign, DSPbase TI C64 series) vieo surveilliance system, etc.. His research interests inclue istribute source coing, joint source-channel coing, network information theory, an image/vieo coing, processing, an transmission. 14 Vlaimir Stankovic M03-SM10) receive the Dr.Ing. Ph.D.) egree from the University of Leipzig, Leipzig, Germany in From 2003 to 2006, he was with Texas A&M University, College Station, first as Research Associate an then as a Research Assistant Professor. From 2006 to 2007 he was with Lancaster University. Since 2007, he has been with the Dept. Electronic an Electrical Engineering at University of Strathclye, Glasgow, where he is currently a Reaer. He has co-authore 4 book chapters an over 160 peer-reviewe research papers. He was an IET TPN Vision an Imaging Executive Team member, Associate Eitor of IEEE Communications Letters, member of IEEE Communications Review Boar, an Technical Program Committee co-chair of Eusipco Currently, he is Associate Eitor of IEEE Transactions on Image Processing, IEEE Transactions on Communications, an Elsevier Signal Processing: Image Communication. His research interests inclue source/channel/network coing, user-experience riven image processing an communications an energy isaggregation. Samuel Cheng receive the B.S. egree in Electrical an Electronic Engineering from the University of Hong Kong, an the M.Phil. egree in Physics an the M.S. egree in Electrical Engineering from Hong Kong University of Science an Technology an the University of Hawaii, Honolulu, respectively. He receive the Ph.D. egree in Electrical Engineering from Texas A&M University in He worke in Microsoft Asia, China, an Panasonic Technologies Company, New Jersey, in the areas of texture compression an igital watermarking uring the summers of 2000 an In 2004, he joine Avance Digital Imaging Research, a research company base near Houston, Texas, as a Research Engineer to perform biomeical imaging research an was promote to Senior Research Engineer the next year. Since 2006, he joine the School of Electrical an Computer Engineering at the University of Oklahoma an is currently an associate professor. He has been aware six US patents in miscellaneous areas of signal processing. He is a senior member of IEEE an a member of ACM. His research interests inclue information theory, image/signal processing, an pattern recognition.

15 IEEE TRANSACTIONS ON COMMUNICATIONS SUBMISSION) En-hui Yang M 97-SM 00-F 08) receive the B.S. egree in applie mathematics from Huaqiao University, Quanzhou, China, an Ph.D. egree in mathematics from Nankai University, Tianjin, China, in 1986 an 1991, respectively. Since June 1997, he has been with the Department of Electrical an Computer Engineering, University of Waterloo, ON, Canaa, where he is currently a Professor an Canaa Research Chair in information theory an multimeia compression, an the founing Director of the Leitch-University of Waterloo multimeia communications lab. He hel a Visiting Professor position at the Chinese University of Hong Kong, Hong Kong, from September 2003 to June 2004; positions of Research Associate an Visiting Scientist at the University of Minnesota, Minneapolis-St. Paul, the University of Bielefel, Bielefel, Germany, an the University of Southern California, Los Angeles, from January 1993 to May 1997; an a faculty position first as an Assistant Professor an then an Associate Professor) at Nankai University, Tianjin, China, from 1991 to A Co-Founer of SlipStream Data Inc. now a subsiiary of BlackBerry), he currently also serves as an Executive Council Member of China Overseas Exchange Association, an Overseas Avisor for the Overseas Chinese Affairs Office of the City of Shanghai, an a irector of Boar of Trustees of Huaqiao University, China, an serves on the Overseas Expert Avisory Committee for the Overseas Chinese Affairs Office of the State Council of China. His current research interests are: multimeia compression, multimeia transmission, igital communications, information theory, source an channel coing, image an vieo coing, image an vieo unerstaning an management, big ata analytics, an information security. Dr. Yang is a recipient of several awars an honors, a partial list of which inclues the prestigious Inaugural Premier s Catalyst Awar in 2007 for the Innovator of the Year; the 2007 Ernest C. Manning Awar of Distinction, one of the Canaa s most prestigious innovation prizes; the 2013 CPAC Professional Achievement Awar; the 2014 IEEE Information Theory Society Paovani Lecture; an the 2014 FCCP Eucation Founation Awar of Merit. He has exemplifie research excellence in both theory an practice. Proucts base on his early inventions an commercialize by his previous company, SlipStream, receive the 2006 Ontario Global Traers Provincial Awar. With over 210 papers an more than 200 patents/patent applications worlwie, his research work has benefite people over 170 countries through commercialize proucts, vieo coing open sources, an vieo coing stanars. He is a Fellow of the Canaian Acaemy of Engineering an a Fellow of the Royal Society of Canaa: the Acaemies of Arts, Humanities an Sciences of Canaa. He serve, among many other roles, as a review panel member for the International Council for Science; a General Co-Chair of the 2008 IEEE International Symposium on Information Theory; an Associate Eitor for IEEE Transactions on Information Theory; a Technical Program Vice-Chair of the 2006 IEEE International Conference on Multimeia & Expo ICME); the Chair of the awar committee for the 2004 Canaian Awar in Telecommunications; a Co-Eitor of the 2004 Special Issue of the IEEE Transactions on Information Theory; a Co-Chair of the 2003 US National Science Founation NSF) workshop on the interface of Information Theory an Computer Science; an a Co-Chair of the 2003 Canaian Workshop on Information Theory. 15