Leaving Ranomness to Nature: -Dimensional Prouct Coes through the lens of Generalize-LDPC coes Tavor Baharav, Kannan Ramchanran Dept. of Electrical Engineering an Computer Sciences, U.C. Berkeley {tavorb, kannanr} @berkeley.eu Abstract Prouct coes (that are eterministic constructions) an Low-Density-Parity-Check (LDPC) coes (that come from a ranom ensemble of coes on graphs) have ha little overlap in esign an analysis methoology. We show that when nature offers suitable ranomness in the form of the unerlying statistical communications channel (e.g. iscrete symmetric erasure an error channels), these two families of coes can be unifie via their prune resiual Tanner graph post channel-corruption, allowing prouct coes to cross-leverage the esign an analysis literature of LDPC coes. Further, for a subclass of these prouct coes, we can leverage the power of ensity evolution methos use successfully for LDPC coes. In this work, focusing on symmetric erasure channels, we show that -imensional prouct coes are isomorphic to a class of -left regular Generalize Low-Density-Parity-Check (GLDPC) coes. This insight allows us to characterize the performance of -imensional prouct coes with component coes having constant erasure correcting capability, with relative simplicity using ensity evolution methos. The isomorphism also reveals that the choice of 3D prouct coes with a single parity-check per component offers superior performance at asymptotically high rates for the erasure channel to the preominantly-stuie 2D prouct coes. Inee, 3-imensional prouct coes having a single parity check per component perform ientically to 3-left-regular LDPC coes, which for the erasure channel (using ensity evolution methos) are about 22% away from Shannon capacity. This, combine with the simplicity an linear-time encoing an ecoing of prouct coes, makes them attractive to a host of applications, with one particularly exciting one being their key role in performing coe istribute matrix multiplication [] for straggler mitigation in moern large-scale istribute computing platforms. I. INTRODUCTION Prouct Coes an Low-Density-Parity-Check (LDPC) coes have co-existe separately for ecaes with little effort to unify them in esign an analysis. This may have been in part ue to prouct coes being eterministic constructions whereas LDPC coes come from a ranom ensemble of coes on graphs. However, when nature offers suitable ranomness in the form of the unerlying statistical communications channel, these two families of coes can be unifie via their prune tanner graph post channel-corruption. Important examples inclue iscrete symmetric erasure an error channels, an more generally, iscrete channels that rener every subset of encoe symbols equally likely to be corrupte. We show how for these settings it is possible to unify these two isparate families of coes, allowing prouct coes to cross-leverage the esign an analysis literature of LDPC coes. We further show that for a subclass of these prouct coes, we can leverage the power of ensity evolution methos use successfully for LDPC coes. Specifically, focusing on symmetric erasure channels in this work, we show that -imensional prouct coes are isomorphic to a class of -left regular Generalize Low- Density-Parity-Check (GLDPC) coes. This insight allows us to characterize the performance of -imensional prouct coes with component coes having constant erasure correcting capability, with relative simplicity using ensity evolution methos in contrast to the more cumbersome analytical tools typically eploye in this literature. The isomorphism also reveals that while the bulk of the prouct coe literature has focuse on 2D coes with heavy component coes, the choice of 3D coes with a single parity-check per component offers superior performance at asymptotically high rates for the erasure channel. While the rigorous proof is more etaile, the unerlying iea is quite simple; the way we are able to raw such an isomorphism between eterministic an ranom constructions is by utilizing the ranom appearance of the prune resiual graph. Before being sent through the channel, a -imensional prouct coe has a eterministic tanner graph; there is no ensemble of graphs to analyze. However, the channel ranomly corrupting a subset of symbols allows us to analyze the resiual graph with only those corrupte symbols as variable noes. Since a ranom subset is corrupte, the resiual graph appears sufficiently ranom, an so we are able to analyze the ensemble of prune resiual graphs. A. Prouct Coes Prouct coes have a very intuitive an regular structure that makes them easy to conceptualize, implement, an unerstan. In stanar prouct coes, = 2, one selects two systematic linear coes, C = (n, k, ) an C 2 = (n 2, k 2, 2 ). The While -imensional prouct coes are asymptotically rate, the term is misleaing; while using the traitional efinition of rate, a coe of length N with k = N N δ is rate, these constructions aren t simply a point on the spectrum. For any fixe values of N, K, one can fin a δ such that K = N N δ. Practically, these asymptotically rate schemes translate to aroun 0% reunancy, which is very practical in many applications
resulting coe is (n n 2, k k 2, 2 ), which is achieve by arranging the ata bits in a k 2 k rectangle, an encoing to a n 2 n rectangle. This is one by encoing the k 2 ata rows with C, etermining the last n k bits of each row, an then encoing all n columns with C 2 to etermine the last n 2 k 2 bits of each column. Since C 2 is linear, the last n 2 k 2 rows are also coewors of C. These prouct coes are very appealing ue to their eterministic construction an linear encoing time. This construction also allows for the use of a peeling ecoer, which yiels simple, linear time ecoing. -Dimensional prouct coes are a natural extension of this 2-imensional framework. For the sake of clarity, we focus on the case where the component coes in each imension are the same, C = (n, k, min ), but our analysis can be extene to the case where ifferent component coes are use in each imension. We aitionally assume that C is a linear coe an has been put in systematic form. To obtain our prouct coe, we form the ata into a -imensional hypercube with sie length k, an inex each ata location with its Cartesian coorinates, (l, l 2,..., l ), with l i = 0,..., k. We encoe this to a -imensional hypercube with sie length n in a similar manner to the 2D case, such that i [(l, l 2,..., l ) : l i = 0,..., n ] C. This means that every "row" is a coewor, with a -imensional row being a set of n elements obtaine by fixing all but inex, an varying that inex 0,..., n. B. LDPC Backgroun Low Density Parity Check (LDPC) coes were first introuce by Gallager in 962 [2]. As they ve evolve, more complex variants have been propose, like Generalize LDPC coes (GLDPC). A GLDPC coe is a low-ensity parity check coe in which the constraint noes of the coe graph are arbitrary linear block coes, rather than just single parity checks [3]. LDPCs are very appealing in their ease of analysis, particularly using ensity evolution. II. RELATED WORK High-imensional prouct coes have been stuie in the literature, e.g. a scheme similar to our Pseuo Prouct Coe one in [4] focusing on the aitive white gaussian noise channel setting using soft-ecision ecoing. However, no isomorphism is mae between these coes an GLDPC coes. 3D prouct coes have been shown to perform empirically better than 2D prouct coes in [5]. However, this is justifie base on an increase minimum-istance of the 3D coe over the 2D coe for finite lengths, rather than a characterization of threshols from Shannon capacity, as we target here using an asymptotic analysis. EXIT chart analysis has been propose for 3D coes in [6], but to the best of our knowlege, ensity evolution has only been applie to the 2D coe setting in [7]. Our paper is motivate by the esire to characterize the funamental performance of eterministic coes in terms of the gol stanar, namely linear time encoing, gap to capacity, an linear time ecoing. We argue that eterministic construction is a highly esirable property in practice ue to its harware-frienliness, an its preictability in esign an performance in contrast with ranom coe constructions like LDPC coes an their variants like the Irregular-Repeat- Accumulate (IRA) coes [8] which can achieve the gol stanar but using a ranom esign. Our construction, at high rates over the erasure channel, asymptotically sacrifices a 22% gap from capacity in orer to gain this ae benefit of eterministic construction, while maintaining linear time encoing an ecoing. A. Ranomness through nature The iea of using a eterministic esign an letting nature supply the ranomness is not new. Work on this ates back to at least [9], with the observation that by erasing ranom variable noes on an erasure channel, a eterministic coe coul be inuce to have its resiual graph appear ranom. A similar observation is mae in [7]. Following in these footsteps, we avoi introucing ranomness in the construction by allowing nature to a ranomness via its ranom erasure of variable noes. III. MAIN RESULT We provie a new lens through which to view -imensional prouct coes, an provie a rigorous asymptotic analysis of their performance over the erasure channel using ensity evolution. We brige the gap between the ranom construction of LDPC coes an the eterministic construction of prouct coes by using the ranomness of the erasure channel in generating the resiual graph. A similar analysis can be one for other iscrete symmetric channels, like the q-ary symmetric channel; the isomorphism will still hol, as there exists a ranom subset of corrupte variable noes, an so the resiual appears to have been generate ranomly. Instea of the simple ensity evolution equation, however, a more complicate message passing algorithm will nee to be analyze. For reaability, we restrict our focus to the erasure channel for the rest of the paper. A. Coe Construction In this paper we present two constructions for -imensional prouct coes. The first we ub a "Pseuo Prouct Coe", which entails a single parity check coe applie to every imensional slice (this can be generalize to a systematic, linear (n, k, ) coe). An example for a 3D Pseuo prouct coe can be foun in Fig. (a), with the colore lines representing the parities for each plane (2 imensional slice). This construction is similar to [4], but where our - imensional pseuo prouct coe has n( ) + parity bits (as one is a global parity, common to all stages), their alternate construction has n parities ae to the outsie of the hypercube of ata, instea of woven into it. A stanar Prouct Coe, can be viewe as having a parity check (or arbitrary systematic linear (n, k, ) component coe) on every imensional slice (row). This translates to Fig. (b), where each colore plane represents the generalize parity checks in a given imension.
(a) 3D Pseuo Prouct Coe (b) 3D Prouct Coe Fig.. Visualization of parities of 3D prouct an pseuo prouct coes Lemma : A -imensional prouct coe with a systematic linear C = (n, k, min ) component coe is equivalent to the Tanner graph unerlying a eterministically constructe -left regular GLDPC coe with C at each check noe. Proof: We note that for a prouct coe, in each imension there are n rows, each representing a constraint on the length N coewor. In orer to construct an equivalent GLDPC coe, we simply nee to have n check noes arrange in stages of n check noes each (one representing each "face" of the -imensional hypercube). We take the N coe inices, an for each stage map them to the check noe corresponing to their row in that particular stage. The check noes (colore planes) will apply the constraint implie by the C to the mappe inices. This construction has equivalent checks to the initial imensional prouct coe. We note that an ientical result hols for pseuo prouct coes. A -imensional prouct coe with a linear component coe is linear as well, an can thus be characterize by its generator matrix. Fig. 2 shows such an example generator matrix. n n {}} {{ }} { G = } {{ } Fig. 2. Generator Matrix of -imensional prouct coe with C = (4, 3, 2) For any linear component coe C, the generator matrix of the -imensional prouct coe corresponing to C can be expresse as the tensor prouct of copies of the generator matrix of C. IV. PSEUDO PRODUCT CODE We now show that a class of eterministically constructe GLDPC coes have resiual graphs equivalent to those of a traitional balls an bins moel, after transmission through the BEC. A. Ensemble C l (n, ) of bi-partite graphs constructe using a balls an bins moel Bipartite graphs in the ensemble C l (n, ) have l variable noes on the left, an n check noes on the right. We group these check noes into stages, each containing n check noes. Each variable (left) noe is connecte to check noes, which are selecte uniformly at ranom, from each stage of check noes. B. Ensemble C l 2(n,, Φ) of bipartite graphs constructe using the mapping Φ For a given mapping Φ : Z N Z n... Z n, Φ(v) = (Φ (v),..., Φ (v)), the ensemble C2(n, l, Φ) of left regular bipartite graphs with l variable noes, an n check noes, is efine as follows. As before, we partition the check noes into sets of n check noes each. Now, we consier a set I of l integers, where each element of the set I is between 0 an N. Assign the l integers from the set I to the l variable noes in arbitrary orer. Label the check noes in each set from 0 to n, for all i =,...,. A -left regular bipartite graph with l variable noes an n check noes is then obtaine by connecting a variable noe with an associate integer v to check noe Φ i (v) in stage i, for i =,...,. The ensemble C2(n, l, Φ) is the collection of -left regular bipartite graphs inuce by all possible sets I. Lemma 2: For any isomorphism Φ, the ensembles C(n, l ) an C2(n, l, Φ) are ientical. Proof: It is trivial to see that C2(n, l, Φ) C(n, l ). Now, we show the reverse. Consier a graph G C(n, l ). Suppose a variable noe v G is connecte to the check noes numbere {f i } i=. Since Φ is an isomorphism, we know there exists Φ : Z n Z n, an thus we can fin an integer q = Φ (f,..., f ) between 0 an n such that Φ i (q) = f i i =,.... Thus, for every graph G C(n, l ), there exists a set I of k integers that will result in an ientical graph using the Φ base construction, an so G C2(n, l, Φ). Hence, C(n, l ) = C2(n, l, Φ) Using the cartesian mapping for Φ, we are able to obtain a pseuo prouct coe by mapping to Φ as in C2(n, l, Φ) for Φ(l) = (l//n,..., (l//n i )%n,..., l%n). V. STANDARD PRODUCT CODE Now that we have establishe the above, we continue to a traitional -imensional prouct coe. We can see an example 3D prouct coe in Fig. 3. We can analyze the performance of this scheme in a similar manner to the Pseuo Prouct Coe. Fig. 3. Construction for 3D Prouct Coe from [6] A. Balls-an-Bins Construction We construct a bipartite graph with l variable noes on the left, an n check noes on the right as follows. We partition the check noes into stages of n check noes each, an arrange them in a imensional hypercube with
sie length n. We inex the check noes for a given stage by a length tuple, with inices ranging from 0 n, representing its coorinates (like the cartesian coorinates above). Each variable noe selects a tuple uniformly at ranom from Z n = (v, v 2,..., v ). These tuples are chosen uniformly at ranom an with replacement for all l variable noes. For each variable noe (v, v 2,..., v ), we connect it to the check noe inexe by (v,..., v i, v i+,..., v ) in stage i for i =,...,. We refer to this ensemble as C l (n, ). B. Φ Guie Construction For any Φ : Z N Z n... Z n with Φ(v) = (Φ (v),..., Φ (v)), we generate graphs in the following manner: choose l integers at ranom, with replacement, from 0,..., N. Call this set I. Partition the n check noes into stages as before, arrange an inexe in a imensional hypercube with sie length n. For convenience, we efine Φ j (v) (Φ (v),..., Φ i (v), Φ i+ (v),..., Φ (v)) For each inex v I, we attach noe v to check noe Φ j (v) in stage j, for j =,...,. We refer to this ensemble as C l 2(n,, Φ) Lemma 3: For any isomorphism Φ, the ensembles C l (n, ) an C l 2(n,, Φ) are equivalent. Proof: The proof is ientical to Lemma 2. C. Implication Theorem : For any isomorphism Φ : Z N Z n... Z n, the eterministically constructe GLDPC coe constructe by mapping each variable noe v = 0,..., n to check noes Φ (v) in stages,..., as in C l 2(n,, Φ) can be analyze using ensity evolution. This also hols using the mapping Φ as in C l 2(n,, Φ) Proof: For any isomorphism Φ : Z N Z n... Z n, picking a resiual graph at ranom from C l 2(n,, Φ) is equivalent to picking a graph at ranom from the ensemble C l (n, ) generate using the balls an bins moel, as the two ensembles are equivalent by Lemma 3. Similarly, by Lemma 2, C l (n, ) = C l 2(n,, Φ). This means we can use ensity evolution to analyze graphs eterministically generate by Φ an ranomly erase by nature. Again, we use the cartesian mapping for Φ, an are able to obtain a stanar prouct coe by mapping to Φ, as in Cl 2 (n,, Φ), using same Φ(l) = (l//n,..., (l//n i )%n,..., l%n). VI. PERFORMANCE ANALYSIS We procee to analyze the asymptotic performance of such a -imensional prouct coe. We restrict our performance analysis to (n, k, r + ) component coes with constant r. Theorem 2: -imensional prouct coes with (n, k, r + ) component coes with linear encoing an ecoing time have the following properties: ) Can asymptotically correct for n k erasures 2) Can be encoe an ecoe in linear time For constants.22 < η(, r) < 2 for most useful regimes, Table I. Proof: The ensemble of resiual graphs of these - imensional prouct coes is equivalent to C l 2(n,, Φ) by Lemma 2, which by Theorem is equivalent to the ensemble of resiual graphs generate with the balls-an-bins moel. Thus, we raw conclusions about the performance of prouct coes (or pseuo prouct coes) by analyzing the ensemble of balls an bins resiual graphs. There are three main steps to analyzing the performance of this ensemble of balls-anbins graphs, which we will briefly outline in the following subsections. More etaile proofs can be foun in [9]. A. Density Evolution We analyze a message-passing algorithm s performance over a typical graph from the ensemble. We assume the graph has local neighborhoos that are cycle free up to epth 2l. Thus, for the first l rouns of message passing, messages are inepenent. B. Convergence to Cycle-free Case We can show that the expecte behavior of graphs in the ensemble C l (n, ) converges to cycle-free. Furthermore, the proportion of eges left unecoe after l iterations is tightly concentrate aroun p l, the proportion ensity evolution tells us shoul be unecoe. C. Expaner Graph We show that graphs in the ensemble C l (n, ) are expaners with high probability. This means that if our peeling ecoer ecoes all but a small fraction of variable noes, then it will ecoe all the variable noes. For =2, we cannot prove this last portion. In fact, we can show that the ensemble of graphs C l (n, 2) are not expaner graphs with probability boune away from zero. We can see that our peeling ecoer will not be able to resolve a graph G C l (n, ) if there are two variable noes in G that share all the same check noes. We examine this scenario by relating it to the birthay paraox. If l people have birthays ranomly selecte between 0,..., n, then the probability that at least two people have the same birthay is approximately e l2 /2n. In our scenario, the number of erase variable noes is l = O(n ), an so for 2, the probability that two people have the same birthay, i.e. share all the same check noes, is boune away from zero. This means that the expaner graph conition is not met, yieling an error floor. A. Threshols VII. SIMULATION RESULTS Now that we have establishe an isomorphism between - imensional prouct coes an -left regular GLDPC coes, we can analyze the asymptotic performance of -imensional prouct coes through ensity evolution analysis on -left regular GLDPC coes. 2 We note that there is a ifference between rawing the set I with an without replacement, but this ifference oes not affect the analysis as in Proposition 5 in [0].
We use the accepte notation for the ege-egree istribution polynomials of the bipartite graphs in the ensemble as λ(α) i= λ iα i an ρ(α) i= ρ iα i where λ i (resp. ρ i ) enotes the probability that an ege of the graph is connecte to a left (resp. right) noe of egree i. Thus for the resiual graphs of a -imensional prouct coe, equivalently the ensemble C (n, l ) constructe using the balls-an-bins proceure, λ(α) = α, as the graph is -left regular. The ege egree istribution ρ(α) can be erive as in [9] to be ρ(α) = exp( ( α)/η(, r)). Uner the tree-like (cyclefree) assumption that we make, we can write an equation for p j, the probability that an ege in the tanner graph is left unecoe after j iterations, starting with p 0 =. ( ) r p j+ = λ ρ( p j ) i=0 We can specialize this ensity evolution equation to the case of C l (n, ), obtaining the upate equation. r p j+ = l=0 ( ) l pj e p j l! This ensity evolution process () will converge to 0 in l iterations for sufficiently large η(, r). We list minimum values of η(, r) for various (, r) in Table I. r 2 3 4 5 6 7 3.228.2880.3797.4564.5202.574.622 4.2949.4998.6568.778.8760.9575 2.0275 5.4250.7275.9409 2.03 2.2327 2.3406 2.4329 6.5697.9577 2.2244 2.4256 2.5864 2.799 2.834 7.789 2.869 2.505 2.7446 2.936 3.0953 3.2309 TABLE I THRESHOLDS: η(, r) Asymptotically, a sharp phase transition can be observe in η(, r), an thus the coe will be able to correct for n k erasures, for component coe (n, k, r + ). We can see from ensity evolution fixe point analysis that out of all 3, r, it is optimal for a given rate to choose =3, r=. We ignore = 2 ue to the inherent error floor of C (n, l ), as explaine with the birthay paraox. 3 4 5 6 3.25 3.00 2.75 2.50 2.25 2.00.75 () VIII. CONCLUSIONS AND FUTURE WORK We have shown that when nature supplies suitable ranomness in the form of the communication channel, prouct coes an LDPC coes can be unifie in their esign an analysis via their prune resiual Tanner graph, allowing prouct coes to leverage the esign an analysis tools of LDPC coes. Concretely, we have shown that uner a iscrete symmetric channel, the ensemble of resiual graphs generate by transmitting eterministically constructe prouct coes is isomorphic to the ensemble of resiual graphs generate by transmitting ranomly constructe GLDPC coes using a balls an bins moel. We have further shown that the subset of these with component coes that correct for a constant number of erasures can be analyze using the powerful an elegant metho of Density Evolution. Our isomorphism provies new insights into the esign an performance analysis of -imensional prouct coes, an unveils the superiority of 3D prouct coes over the well-stuie class of 2D prouct coes for the erasure channel at high rates. While we have limite our isomorphism to the equivalence of the ensemble of prune Tanner graphs in this work, we conjecture that this isomorphism can be extene to more general settings as well, which will be part of our future work. Further, while we have limite our ensity evolution analysis of - imensional prouct coes at high rates, we believe that this can be extene to more general rate settings as well. We are excite about the potential for further insights, an will explore this as part of future work. REFERENCES [] T. Baharav, K. Lee, O. Ocal, an K. Ramchanran. (208, Jan.) Straggler-proofing massive-scale istribute matrix multiplication with -imensional prouct coes. [Online]. Available: https://people.eecs. berkeley.eu/~kannanr/assets/pro-coes-ist-comp.pf [2] R. Gallager, Low-ensity parity-check coes, IRE Trans. on Inf. Theory, vol. 8, no., pp. 2 28, 962. [3] R. M. Tanner, A Recursive Approach to Low Complexity Coes, IEEE Trans. Inf. Theory, vol. 27, no. 5, pp. 533 547, 98. [4] J. M. Shea an T. F. Wong, Multiimensional coes, Encyclopeia of Telecommunications, 2003. [5] X. Wu, Y. He, an G. Zhu, Performance of improve three-imensional turbo prouct coe ecoer, in Integration Technology, 2007. ICIT 07. IEEE International Conference on. IEEE, 2007, pp. 563 567. [6] Y. He, G. Sun, J. Yang, an F. C. Lau, D-GLDPC coes with 3-D single parity-check prouct coes as super check noes, International Conference on Wireless Communications an Signal Processing, 204. [7] C. Häger, H. D. Pfister, A. G. i Amat, an F. Brännström, Density evolution for eterministic generalize prouct coes on the binary erasure channel at high rates, IEEE Trans. Inf. Theory, vol. 63, no. 7, p. 4357, 207. [8] H. Jin, A. Khanekar, an R. McEliece, Irregular repeat-accumulate coes, in Proc. 2n Int. Symp. Turbo coes an relate topics, 2000. [9] S. Pawar an K. Ramchanran, Computing a k-sparse n-length Discrete Fourier Transform using at most 4k samples an O(k log k) complexity, in IEEE Int. Symp. on Inf. Theory, may 203, pp. 464 468. [0] R. Scheibler, S. Haghighatshoar, an M. Vetterli, A fast haamar transform for signals with sublinear sparsity in the transform omain, IEEE Trans. Inf. Theory, vol. 6, no. 4, pp. 25 232, 205. 7.50.25 2 4 6 8 Fig. 4. Heatmap Representationr of Table I: η(, r) for imension, erasure correcting capability r