Endcoding Complexity Versus Minimum Distance

Transcription

1 Edcodig Complexity Versus Miimum Distace Louay M.J. Bazzi, Sajoy K. Mitter Abstract We establish a boud o the miimum distace of a biary error correctig code give costraits o the computatioal time-space complexity of its ecoder where the ecoder is modeled as a brachig program. The boud we obtai asserts that if the ecoder uses liear time ad subliear memory i the most geeral sese, the the miimum distace of the code caot grow liearly with the block legth whe the rate is ovaishig, i.e. miimum relative distace of the code teds to zero i such a settig. The settig is geeral eough to iclude oserially cocateated turbo like codes ad various geeralizatios. Our argumet is based o brachig program techiques itroduced by Ajtai [1]. We cosider also the case of costatdepth AND-OR circuits ecoders with ubouded fais. 1 Itroductio A biary error correctig code is specified by a ijective map E : {0, 1} {0, 1} m from the biary strigs of legth to biary strigs of legth m. The map E is called the ecodig map or the ecoder, the image C of E i {0, 1} m is called the code, is called the message legth, m is called the block legth, the elemets of C are called the codewords, ad the ratio /m is called the rate of the code. A fudametal parameter that characterizes the worst case error correctio capabilities of the code is its miimum distace which is defied as the miimum Hammig distace betwee two distict codewords. Oe of the mai goals i combiatorial codig theory is to fid codes with a good tradeoff betwee rate ad miimum distace, as the message legth teds to ifiity. A code (meaig a family of codes idexed by the block legth) is called a asymptotically good code if the block legth grows liearly with the message legth ad the miimum distace grows liearly with the block legth. Otherwise, the code is called asymptotically bad. I this paper, we cosider the followig questio: What ca we say about the growth of the miimum distace of a biary code give costraits o the computatioal complexity of its ecoder? We cocetrate maily o the time-space complexity of the ecoder. I this settig, the above questio is a atural tradeoff questio betwee the parameters: miimum distace, rate, ecodig time, ad ecodig space. From a practical perspective, this questio is importat sice there are popular error correctig codes that have low time-space ecodig complexity. We are referrig here to turbo codes, or more precisely to parallel cocateated turbo codes itroduced by Berrou, Glavieux, ad Thitimajshima i [4], ad repeat-covolute codes itroduced by Divsalar, Ji, ad McEliece i [6]. This low time-space ecodig complexity is crucial for the correspodig iterative decodig This research was support by ARO grat DAA L03-92-G-0115, ad MURI grat DAAD Louay.Bazzi@aub.edu.lb, Departmet of Electrical ad Computer Egieerig, AUB, Beirut, Lebao. mitter@mit.edu, Departmet of Electrical Egieerig ad Computer Sciece, MIT, Cambridge, MA

2 algorithms. These decodig algorithms use the state space represetatio of the ecoder ad their ruig time is proportioal to the cardiality of the state space. Sharp bouds o the miimum distace of turbo codes were first obtaied by Kahale ad Urbake [8] for radom iterleavers ad costat memory covolutioal codes. I a recet joit paper with Mahdia ad Spielma [3], we derived strog bouds o the miimum distace of turbo like codes i a variety of cases. Oe of these cases is the well structured settig of geeralized repeat-covolute codes where the covolutioal code is replaced by a arbitrary automato. We argued that such codes are asymptotically bad whe the memory of the automato is subliear ad the umber of repetitios is costat. I this paper, we exted this particular result to the much more geeral settig where the ecoder is a biary brachig program, or equivaletly a ouiform radom-access machie with biary iput registers. We establish a geeral theorem that asserts that if the ecoder is a biary brachig program that uses liear time ad subliear space, the the miimum distace of the code caot grow liearly with the block legth whe the rate is ovaishig, which is a rather surprisig result. I geeral we derive a boud relatig the ivolved parameters. Our proof is based o the brachig programs techiques itroduced i the recet paper of Ajtai [1]. We cosider also the case of costat-depth AND-OR circuit ecoders with ubouded fais. We coclude with a cojecture about the strogest possible time-space tradeoffs oe ca obtai for ecodig asymptotically good codes. 2 Brachig program ecoders By a brachig program (biary by default, i.e., 2-way) B we mea a coected directed acyclic graph with a sigle source ad a sigle sik, together with a set of biary iput variables ad a set of m biary output variables satisfyig the followig. There are exactly two edges leavig each o-sik ode, the first labeled with a oe iput-label ad the secod with a zero iput-label. Every o-sik ode is associated with a iput variable. Some of the edges of the graph are associated with output variables (possibly more tha oe output variable per edge), i which case the edges are labeled by biary output-labels (a separate 0 or 1 for each output variable associated with the edge). The odes of the graph are called states, its source is called the start state, ad its sik is called the ed state. The brachig program B computes a ecodig map E : {0, 1} {0, 1} m as follows. The computatio starts by readig the value of the variable associated with the start state ad movig accordig to its value to the ext state ad so o by readig more bits util the ed state is reached. At each trasitio alog a edge, the output variables associated with the edge (if ay) are set accordig to the edge output labels. We may wat to assume that o ay iput each output variable will be set at least oce, or we ca assume that the output variables are arbitrarily preset. The computatio of the brachig program o a iput is the correspodig sequece of states startig with the start state ad edig with the ed state. More formally, by a a brachig program B we mea a 5-tuple B = (, m, G, i-var, out-val), where: ad m are positive itegers. G = (V, E) is a coected directed acyclic graph. V is the set of vertices of the graph ad E is its set of edges. The graph G has a sigle source ad a sigle sik. The elemets of V are called states. We call the source ode of G the start state ad we deote it by start V. We call the sik ode of G the ed state ad we deote it by sik V. The graph G has exactly two edges e s,0 ad e s,1 leavig each o-sik ode s V \{sik}. 2

3 Thus E is i oe-to-oe correspodece with (V \{sik}) {0, 1}. 1 We associate with G the state trasitio map tras : (V \{sik}) {0, 1} V give by e s,b = (s, tras(s, b)) for each s V ad for each b {0, 1}. i-var is the the iput-variables map i-var : V \{sik} []. (If i is a iteger, we mea by [i] the set {1,..., i}.) out-val is the output-values map out-val : E A [] {0, 1} A. We associate with the output-values map, the output-variables map out-var : E 2 [m] which is specified by out-val(e) {0, 1} out-var(e) for all e E. The brachig program B computes a ecodig map E : {0, 1} {0, 1} m as follows. The computatio of the brachig program B o a iput x {0, 1} is the sequece of states s 0, s 1,..., s l 1, where s 0 = start, s l 1 = sik, ad s i = tras(s i 1, x i-var(s i 1)) for i = 1,...,l 2. Note that the ed state must be reached after a fiite umber of steps sice the graph G is acyclic. The output y = E(x) of the brachig program B o the iput x is a strig i {0, 1} m defied by y = y (l 1), where y (0),..., y (l 1) are defied iteratively as: y (0) is the all zeros strigs i {0, 1} m, ad y (i) is derived from y (i 1) via { y (i) (i 1) y j = j if j out-var(s i 1, s i ) out-val(s i 1, s i ) j if j out-var(s i 1, s i ) for j = 1,...,m ad i = 1,...,l 1. The legth of the computatio of B o x is defied as its umber of states, i.e., l. The time of the computatio of B o x is defied as its umber of states plus the total umber of times each output variable is set durig the computatio of B o x, i.e., l + l 1 i=1 out-var(s i 1, s i ). We say also that the brachig program B ecodes the code C, which is defied as the image of E i {0, 1} m. We call also B a brachig program ecoder for the code C. Fially, we associate three complexity measures with a brachig program: legth, time, size, ad memory (or space). The legth of the brachig program B is the maximum legth of a computatio of B o a iput x where the maximizatio is over all iput strigs x {0, 1}. Similarly, the time of the brachig program is the maximum time of a computatio. The size S of the brachig program is its umber of states, i.e., S = V. The memory or the space M of a brachig program is defied as M = log 2 S. 2.1 Radom access machies ecoders The codes ecodable by such geeral brachig programs correspod to those ecodable by ouiform radom access machies with biary iput registers which we defie below. The brachig program model was itroduced by Borodi ad Cook [5] as a model of radom access machies for studyig time-space tradeoffs for sortig algorithms. See also Ajtai [1]. We explai i this sectio the relatio betwee the two models i the cotext of ecoders. A o-uiform radom access machie ecoder is as a 8-tuple where: ad m are positive itegers. V is a fiite set of states M = (, m, V, start, sik, tras, i-var, out-val), 1 Sice the edges are labeled, we allow multiple edges, i.e., we allow the possibility that e s,0 ad e s,1 have the same destiatio. 3

4 start ad sik are special states i V called the start ad the ed states respectively. tras is the state trasitio map tras : (V \{sik}) {0, 1} V. We associate with tras a graph G = (V, E) whose vertex-set is the set of states V ad whose set E of edges is costructed as follows. For each o-sik state s V \{sik}, we have two edges (s, tras(s, 0)) ad (s, tras(s, 1)) i E labeled with 0 ad 1 respectively. We assume that the state trasitio map satisfies the coditio that G is a acyclic graph with start as its sigle source vertex ad sik as its sigle sik vertex. i-var is the the iput-variables map i-var : V \{sik} []. out-val is the output-values map out-val : (V \{sik}) {0, 1} A [] {0, 1} A. We associate with the output-values map, the output-variables map out-var : (V \{sik}) {0, 1} 2 [m] which is specified by out-val(s, b) {0, 1} out-var(s,b) for all s V ad b {0, 1}. The equivalece of the radom access machie model ad the brachig program model of a ecoder is obvious from the defiitio. The machie M fuctios i exactly the same way as the uderlyig brachig program B, ad its time ad memory are defied as those of B. Note that this model, beig ouiform, does ot restrict the size of the code descriptio. This cotributes to the geerality of the results i this paper sice we are derivig lower bouds. Oe cosequece of this urestricted code descriptio size is that ay code ca be ecoded i liear time ad liear space. This ca be doe usig a expoetial-size code descriptio as explaied i Example 2.4. I the rest of the paper, we will be workig with the brachig program model. 2.2 Some special types of brachig programs The brachig program is called leveled if the states are divided ito a ordered collectio of sets each called a level where edges are betwee cosecutive levels oly. I such a case, the width of the brachig program is the maximum umber of states per level. The brachig program is called oblivious if the iput variables (ad the output variables) are read (respectively, set) i the same order regardless of the iput uder cosideratio. Thus a oblivious brachig program is aturally leveled i such a way that all the odes i the same level read the same iput variables, ad set the same output variables whe movig to the ext level. The brachig program is called a read-k-times brachig program if each iput variable is read at most k times o ay iput. The brachig program is called a write-w-times if at most w output variables are set per trasitio. 2.3 Examples Example 2.1 The trellis of a rate-1/w covolutioal code is a oblivious, read-oce, write-wtimes brachig program. Sice we decided for simplicity to allow brachig program ecoders to have oly oe ed state, we are assumig here that the states i the last level of the trellis are grouped ito a sigle state. The width of the brachig program is 2 M0, where M 0 is the umber of memory registers of the covolutioal code. The size of the brachig program is 2 M0 + 1, where is the block legth. The legth of the brachig program is + 1 ad its time is (w + 1) + 1. The additioal 1 is eed to accout for the ed state. Note that ot ay oblivious read-oce brachig program ecoder is a trellis of some covolutioal code sice such a brachig program ca ecode a oliear code. Moreover, the graph structure of such a brachig program eed ot be uiform i the sese that the trasitios structure ca chage from oe level to the other. 4

5 Example 2.2 Parallel cocateated turbo codes are ecodable by low complexity oblivious brachig programs as follows. A parallel cocateated turbo ecoder [4] E with a costat umber k of braches, message legth, ad memory M 0 is specified by k permutatios π 1,...,π k each o bits ad a rate-1 covolutioal code Q (the compoet code) with M 0 memory registers. For x i {0, 1}, E(x) is ecoded as E(x) = (x, Q(π 1 (x)),..., Q(π k (x))), where π i (x) is the strig obtaied by permutig the bits of x accordig to the permutatio π i, ad Q(y) is the output of the covolutioal ecoder Q o the iput strig y. E is aturally ecodable by a oblivious read-k-times write-2-times brachig program B as follows. Let B 1 be the read-oce write-2-times brachig program correspodig to the trellis of the rate-1/2 systematic covolutioal code E 1 (x) = (x, Q(π 1 (x)) (See Example 2.1). For i = 2,...,k, let B i be the read-oce write-oce brachig program correspodig to the trellis of the covolutioal code E i (x) = Q(π 1 (x)). B is the cocateatig of B 1,...,B k resultig from idetifyig the ed state of B i with the start state of B i+1 for i = 1,...,k 1. The start state of B is that of B 1 ad its ed state is that of B k. Thus B has legth k + 1 ad time (2k + 1) + 1. The width of B is 2 M0, ad its size is at most k2 M Example 2.3 Repeat-covolute codes fit i the same picture. A repeat-covolute code [6] cosists of a repeat-k-times code, a covolutioal code, ad a permutatio. More precisely, a repeat-covolute ecoder E of message legth ad memory M 0 is specified by a costat iteger k, a permutatio π o k bits, ad a rate-1 covolutioal ecoder Q with M 0 memory registers. For x i {0, 1}, E(x) is ecoded as E(x) = (x, Q(π(r(x)))), where r is the repeat-ktimes map, i.e., r(x) is the cocateatio of k copies of x. E is aturally ecodable by a leveled, oblivious, read-k-times, write-2-times, legth-(k + 1), time-((2k + 1) + 1), ad width-2 M0 brachig program B whose size is at most k2 M The brachig program B is costructed as follows. Let B be the read-oce write-2-times brachig program correspodig to the trellis of the rate- k k+1 covolutioal ecoder E (y) = (y 1,...,y, Q(y)), where ad y {0, 1} k. B is costructed from B by relabelig the iput variables of B accordig to the repetitio map r ad the permutatio π. Example 2.4 Ay biary code ca be trivially ecoded i the biary brachig program model i liear time ad liear space by a leveled ad oblivious brachig program B whose graph has a tree structure. The graph of B cosists of a height- complete biary tree whose leaves are grouped ito a sigle sik ode. The root of the tree is the start state of B. The biary iput labels of the edges correspod to the left ad right labels of the biary tree edges. Thus the simple paths from the root of the tree to the sik ode are i oe-to-oe correspodece with the the iput strigs i {0, 1}. O ay such iput strig, the brachig program follows the correspodig path ad outputs the whole codeword at the last trasitio from the th level to the sik ode. This liear ecodig time ad space makes sese for all codes i the radom-access machie ecodig model because we are ot restrictig the amout of read-oly memory, which is expoetial i this example, eeded to store the code descriptio (See Sectio 2.1). Example 2.5 Ay block-legth- biary liear code is aturally ecodable by a leveled, oblivious, width-2, O( 2 )-time, write-oce brachig program B. The reaso is that oe ca multiply (over GF(2)) a biary iput vector by a fixed biary matrix usig such a brachig program. The legth of B is D + 1, where D is the umber of oes i the matrix. The time of B is D = O( 2 ). 5

6 3 Mai result Theorem 3.1 Let E : {0, 1} {0, 1} m be a ijective ecodig map computable by a brachig program B of size S = S(), time t = t(), ad legth l = l(). Let C be the biary code associated with E, i.e., the image of E i {0, 1} m. If t() = Θ(), the the miimum distace of C is ( (log2 ) 1 ) l/ S O. Therefore, C is asymptotically bad whe S() = 2 o() ad t() = O(). More geerally, if t() = Ω(), the the miimum distace of C is ( ( ) 3 ( ) ) t log2 2t S O. Thus, C is asymptotically bad also whe S() = 2 O(1 ǫ 1) ad t() = O( log 1 ǫ2 2 ), for all ǫ 1, ǫ 2 > 0. I other words liear time ad subliear space for ecodig imply that the code is asymptotically bad, i.e., the miimum distace caot grow liearly with the block legth whe the rate is ovaishig. 3.1 Applicatio to turbo-like codes By applyig the first boud i Theorem 3.1 to parallel cocateated turbo codes ad repeatcovolute codes (See Examples 2.2 ad 2.3), we ca recover the correspodig boud i [3] as follows. The miimum distace of a parallel cocateated turbo code with a costat umber k braches, message legth, ad M 0 memory registers is O( 1 1/k (M 0 + log 2 ) 1/k ) because the legth l of the correspodig brachig program is l = k + 1 ad its size S is at most k2 M0 +1 (See Example 2.2), ad hece l/ = k ad log 2 S = O(M 0 +log 2 ). Note that this boud is slightly weaker tha the correspodig boud O( 1 1/k M 1/k 0 ) i [3]. To get rid of the additive log 2 term, apply Theorem 4.1 below which is stated i terms of the width W = 2 M0 of the brachig program. Similarly, the miimum distace of a repeat-covolute code with k repetitios, message legth, ad M 0 memory registers is O( 1 1/k M 1/k 0 ). So both types of codes will be asymptotically bad as log as M 0 is subliear i. Note that the situatio whe M 0 is subliear i correspods to the case whe the uderlyig trellis has subexpoetial size, i.e., whe the correspodig iterative turbo decodig algorithm has subexpoetial ruig time. 4 Proof of Theorem Ajtai proof techiques for the Hammig distace problem To prove the theorem we use brachig program techiques itroduced by Ajtai i [1]. More specifically, we are referrig to the brachig program techiques that Ajtai itroduced to show that there is o O()-time ad o( log 2 )-space R-way brachig program, where R = c ad c is some absolute costat, that solves the Hammig distace problem: give strigs i {0, 1} log 2 R, decide whether ay distict two of them are at Hammig distace λlog 2 R apart, where λ is aother absolute costat related to c. 6

7 Eve though this is a decisio problem i the settig of R-way brachig programs, while ours is ot a decisio problem ad is i the settig of 2-way brachig programs, the techiques itroduced by Ajtai lie behid the proof we describe below. We refer the reader to Ajtai s paper [1] for further details. 4.2 Objects uder cosideratio ad termiologies We will start by makig the brachig program B leveled. Recall from Sectio 2 that this meas that the states are partitioed ito l cosecutive sets L 0,...,L l 1 of states each called a level i such a way that edges, i.e., trasitios, occur oly betwee cosecutive levels. We will divide (meaig partitio) B ito blocks, meaig sets of cosecutive levels, i.e., sets of the form B(i, j) = {L i, L i+1,..., L j }. For a give block B(i, j), we will be lookig at states i the lower boudary level of the block; meaig the last level i the block with respect to the orderig of the levels i the block B(i, j), i.e., L j. Give a iput, we will be lookig at the computatio of the brachig program B o this iput, which, as explaied i Sectio 2, is defied to be the correspodig sequece of states startig with the start state ad edig with the ed state. So, i the leveled case, each computatio takes exactly l steps, i.e., it cotais exactly l states. Fix a iput strig x, ad cosider the computatio {s i } i of the brachig program B o x. Fix also a set L of levels or a set T of blocks. By a iput bit or variable x j beig accessed or read i L (or T) durig the computatio of B o x, we mea that there is a state s i0 i the computatio {s i } i of B o x that belogs to a level i L (or to a level i a block i T) such that the value of the iput variable x j is read i order to move from s i0 to s i0+1, i.e., j = i-var(s i0 ) i the termiology of Sectio 2. Similarly, by a output bit or variable y j beig set i L (or T) durig the computatio of B o x we mea that there is a state s i0 i the computatio {s i } i of B o x that belogs to a level i L (or to a level i a block i T) such that the value of the output variable y j is set at the trasitio form s i0 to s i0+1, i.e., j out-var(s i0, s i0+1) i the termiology of Sectio 2. Fially, by a computatio cotaiig a sequece of states, we mea that each state i this sequece appears i the computatio. Note that here the order does ot matter sice the states i a computatio are distict due to the fact that the brachig program graph is acyclic. 4.3 The oblivious case argumet Recall that a oblivious brachig program is aturally leveled i such a way that all the odes i the same level read the same iput variables, ad set the same output variables. Sice the proof of Theorem 3.1 is relatively log, it is istructive to look first at the very special case whe B is oblivious. This case is very restrictive compared to a geeral brachig program. To restrict the settig further, assume that B is read-k-times ad write-w-times, where k = O(1) ad w = O(1). This additioal restrictio will further simplify the argumet. The argumet we used i [3] to boud the miimum distace of repeat-covolute codes was i the settig of automata. More specifically, we studied the case of a repeat-covolute code where the covolutioal code is replaced by a arbitrary automato. Eve though the automata settig is less geeral tha the case we are cosiderig i this sectio, the argumet ca be aturally exteded as follows. Theorem 4.1 Let E : {0, 1} {0, 1} m be a ijective ecodig map computable by a read-ktimes, write-w-times, width-w oblivious brachig program B, where k = O(1) ad w = O(1). Let C be the biary code associated with E, i.e., the image of E i {0, 1} m. 7

8 Assume that W 2. The the miimum distace of C is ( ( ) ) 1/k log2 W O. Therefore, C is asymptotically bad whe W = 2 o(). Note that this boud is slightly sharper tha that of Theorem 3.1 sice it is i terms of the width W of the brachig program which is smaller tha its size S. Note also that the time t of the brachig program i Theorem 4.1 is at most wk, i.e., t = Θ(). Thus the more geeral case whe t is superliear i, which is hadled i the secod boud i Theorem 3.1, is ot applicable i the settig of Theorem 4.1. Proof: We will exhibit two distict iput strigs that map to two codewords at distace O(( log 2 W ) 1/k ). We will do this by fidig a oempty set of iput variables U, a subset J of levels, ad two distict strigs x 1 ad x 2 i {0, 1} such that x 1 ad x 2 agree outside U, ad the computatios of B o x 1 ad x 2 agree outside J. This will give us the desired boud o the miimum distace. J will be costructed as a uio of itervals from a partitio of B that we defie ext. Let l be the legth of the brachig program, ad ote that l k, where the first iequality follows form the fact each iput variable must be read i at least oe level sice E is ijective. Partitio B ito b cosecutive blocks, each cosistig of p 1 or p 2 levels, where p 1 = l/b ad p 2 = l/b k/b. Assume for ow that b is arbitrary as log as 1 b l. We will optimize o the iteger b later. Each of the iput variables is read by B i at most k blocks. Recall that B is oblivious. Thus, for ay specific variable, these blocks will be the same irrespective of the settig of the iput variables. Defie a k-set of blocks to be a set of at most k blocks. There are at most b k possible k-set of blocks. So there are at least /b k iput variables that are read by B i the same k-set of blocks. Let U be such a set of iput variables with U = /b k, T be such a k-set of blocks, thus 1 T k. The set J we metioed above is the uio of the blocks i T. Cosider the lower boudary levels (see the defiitio Sectio 4.2) L 1,..., L T of the blocks i T ordered by the level idex, ad let Q be the set of strigs i {0, 1} that are zero outside U, thus Q = 2 U. There are at most W k state sequeces i L 1... L T, ad for each x i Q the computatio of B o x cotais such a sequece. So if we ca guaratee that 2 U > W k, we get that there should be a sequece of states {s i } T i=1 i L 1... L T ad two differet strigs x 1 ad x 2 i Q such that the computatio of B o both x 1 ad x 2 cotais {s i } T i=1. Sice x 1 ad x 2 agree outside U, the computatio of B o x 1 ad x 2 are exactly the same outside the blocks i T. Thus E(x 1 ) ad E(x 2 ) ca oly differ i the blocks i T. This meas that the distace betwee E(x 1 ) ad E(x 2 ) is at most T p 2 w k k/b w, sice T k, ad p 2 k/b. This boud holds uder the assumptio that 2 U > W k, which ca be guarateed if 2 /bk > W k. We choose ( ) 1/k b = 1. k log 2 W to guaratee this requiremet. 8

9 Note that the oly other costraits we have o b are 1 b l. Sice k 1 ad W 2, the chose value of b satisfies b < l. The chose value of b satisfies b 1 whe W is ot expoetial i. Note also that if W is expoetial i (i.e., if log 2 W = Θ()), the statemet of Theorem 4.1 is trivial. By replacig the chose value of b i the upper boud k k/b w o the distace betwee E(x 1 ) ad E(x 2 ), we get ( ( ) ) 1/k k k ( k log 2 W )1/k 1 w = O log2 W, sice k = O(1) ad w = O(1). This is a upper boud o the miimum distace of C sice E(x 1 ) E(x 2 ) because x 1 x 2 ad E is ijective. This proof is short ad simple. But whe B is ot oblivious, this proof does ot go through. The mai reaso is that we caot costruct U ad T regardless of the values the iput variables assume as we did above. Moreover, i the o-oblivious case, the read-k-times ad the writew-times restrictios become restrictive. For example, i the geeral brachig program model, depedig o the iput, a very large umber of the output variables may be fixed i a particular trasitio, or a particular iput variable may be read a very large umber of times. We will sketch i the ext sectio how to hadle the geeral situatio. The proof is loger ad more sophisticated. This is to be expected sice the statemet we are provig is much more geeral. The reader is ecouraged to go over carefully the above argumet before proceedig to the geeral case. 4.4 Proof techique I this sectio we iformally overview the mai techiques used i the proof. The formal argumet is i the followig sectios. These techiques were itroduced by Ajtai [1] to study the Hammig distace problem. We wat to fid two iput strigs x 1 ad x 2 such that E(x 1 ) ad E(x 2 ) are close to each other. The first step is to make the brachig program leveled without affectig its iput-output behavior. Next, we divide the brachig program ito blocks each cosistig of cosecutive levels whose umber will be suitably selected later ad whose sizes are as uiform as possible. To exhibit x 1 ad x 2, we will fid a set T of blocks such that: the size of T is small, the computatios of B o x 1 ad x 2 are exactly the same i the blocks outside T, ad ot too may output bits of E(x 1 ) (respectively E(x 2 )) are set i ay of the blocks i T durig the computatio of B o x 1 (respectively x 2 ). Thus E(x 1 ) ad E(x 2 ) ca oly disagree o the few output bits that are set i T. To fid such x 1, x 2, ad T, we fid first T together with a set Q of iput strigs i {0, 1} ad a sequece {s i } i of states i the lower boudary levels of the blocks i T i such a way that for each x i Q : the computatio of B o x cotais {s i } i, ot too may output bits of E(x) are set i ay of the blocks i T durig the computatio of B o x, ad the umber of variables i x that are accessed oly i the blocks i T durig the computatio of B o x is large. 9

10 We will evetually fid the desired x 1 ad x 2 iside Q as follows. We modify the brachig program B agai so that B is forced to to pass through a state i the sequece {s i } i each time it attempts to leave a lower boudary level of a block i T, but without affectig its iput-output behavior o Q. Usig T, defie a equivalece relatio o {0, 1} by relatig two strigs if: they share the same the set of iput variables that are ot read durig the computatio of B i blocks outside T, ad they agree o the values of their bits outside this set. Thus each equivalece class [x] is determied by a set I [x] of iput variables ad a settig of the variables outside I [x]. We forced the computatio of B to cotai the sates {s i } i o all iputs so that we get [x] = 2 I [x], ad hece the size of each equivalece class [x] ca be guarateed to be large whe I [x] is large. Sice for each iput strig i Q, the umber of iput variables that are accessed oly i the block i T durig the computatio of B is large, we get that the equivalece class of each iput i Q is large. By cosiderig the set Ω of sufficietly large equivalece classes so that the equivalece classes of all the elemets of Q are guarateed to be elemets of Ω, our problem reduces to selectig the umber of blocks so that Q is strictly larger tha Ω, ad hece there are distict x 1 ad x 2 i Q that have the same equivalece class. The fact that [x 1 ] = [x 2 ] meas that the computatios of B o x 1 ad x 2 are exactly the same outside the blocks i T, ad hece E(x 1 ) ad E(x 2 ) ca oly disagree o the output bits that are set iside the blocks i T. By costructio, the umber of those output bits will be small. Moreover, E(x 1 ) ad E(x 2 ) are distict sice E is ijective. The distace betwee E(x 1 ) ad E(x 2 ) will be the desired boud o the miimum distace of C. 4.5 Proof steps Assume for the momet that t = Θ(). We will deal with the more geeral case whe we are doe by workig more carefully with the costats. So say that where a, c 1 are costats (a, c 1 because E is ijective). t = a ad l = c, (1) A) We modify the brachig program so that it is leveled. This ca be doe by a classical procedure. Costruct a leveled directed graph of l levels where each level cosists of a copy of all the odes of the origial brachig program together with the related output labels. Coect the odes i each two cosecutive levels accordig to the the graph of the origial brachig program. This results i a ed state i each level. Associate each ed state ot i the last level with a arbitrary iput variable ad coect it to the ed state i the ext level by two edges the first with zero-iputlabel ad the secod with oe-iput-label. Fially, remove all the odes (together with the i-goig ad out-goig edges) that are ot accessible from the start sate i the first level or ca ot reach the ed state i the last level. The start state of the ew brachig program is the remaiig state i the first level ad its ed state is the remaiig state i the last level. The modified brachig program computes the same fuctio, i.e., B computes E. The legth of the resultig brachig program B is l, its time is t, its size is at most Sl, ad its width is at most S. The differece is that ow edges are oly betwee cosecutive levels, ad each computatio takes exactly l steps. 10

11 B) Partitio B ito b cosecutive blocks, each cosistig of p 1 or p 2 levels, where def l c def l c p 1 = = ad p 2 = =. (2) b b b b Assume for ow that i geeral 1 b l so that 1 p 1, p 2 l. We will optimize o the iteger b later. C) Lemma 4.2 There exist: a) absolute costats h, α > 0, b) Q {0, 1} such that where c) a set of blocks T, Q 2 (Sb) k, (3) k def = c, (4) 1 T k, (5) d) ad a sequece {s i } T i=1 of states i the lower boudary levels of the blocks i T, such that for each x i Q : 1) the computatio of B o x cotais {s i } i, 2) at most w def = hp 1t (6) l output bits of E(x) are set i each block i T durig the computatio of B o x, ad 3) the umber of variables i x that are accessed oly i the blocks i T durig the computatio of B o x is at least α. b k Proof: See Sectio 4.6. D) Now we will modify the brachig program B agai so that B is forced to to pass through a state i {s i } i each time it attempts to leave a lower boudary level of a block i T, while guarateeig that B behaves exactly like the old B o the iputs i Q, i.e., it computes E(x) for each x i Q. We ca do this by simply coectig (o both iputs) all the states i the level above that of s i to s i, for each i. Note that B eed ot compute a ijective fuctio aymore, so it may ot read all the iput variables o some iputs. It may also leave some of the output variables uset, but this is ot a problem sice we ca assume that the output variables were arbitrarily preset. E) Fially, we boud i Sectio 4.7 the miimum distace of C by exhibitig distict x 1 ad x 2 i Q such that the distace betwee E(x 1 ) ad E(x 2 ) is O(( log 2 S ) 1 k ). F) I Sectio 4.8, we explai how to drop the assumptio t = Θ(). 4.6 Proof of Lemma 4.2 Cosider ay iput x i {0, 1}. Let k 1 be a iteger, ad let h > 0. We will choose k the h as we cotiue. Let R x be the set of those blocks that each fixes at most w def = hp 1t l bits of E(x) durig the computatio of B o x. 11

12 Let D x be the set of iput variables that are read i at most k states durig the computatio of B o x. Ad let D x be the set of iput variables i D x that are read oly i blocks i R x durig the computatio of B o x. First recall from (1) that a, c 1 are the costats satisfyig Recall also from (2) that Some bouds: p 1 def = t = a ad l = c. l c def = 1 ad p 2 = b b l c =. b b From the defiitio of R x, we must have (w + 1)(b R x ) t sice each of the b R x blocks ot i R x fixes at least w + 1 output variables durig the computatio of B o x. Thus w(b R x ) t, i.e., R x b t w = b l ( b 1 2 ), (7) hp 1 h where the equality follows from the defiitio of w as w = hp 1 t/l, ad the last iequality follows from the boud p 1 = l/b l/(2b). From the defiitio of D x, each of the D x iput variables outside D x must be read i at least k + 1 states i the computatio of B of x. Moreover, each iput variable i D x must be read i at least oe state i the computatio of B of x sice E is ijective. Thus D x + (k + 1)( D x ) l because the umber of states i the computatio of B o x is at most l. We ca rewrite this iequality as ( D x 1 l/ 1 ) ( = 1 c 1 ) k k sice l = c. Thus if we set we get k def = c, D x (1 ǫ), where ǫ def = c 1 < 1. (8) k The umber of iput variables read i blocks outside R x is at most (b R x )p 2 2bp 2 h 4c h, where the first iequality follows from (7), ad the secod from the boud p 2 = c/b 2c/b. Thus, by the defiitio of D x, we must have D x D x 4c ( h 1 ǫ 4c ), h where the secod iequality follows from (8). Let h be sufficietly large such that α def = 1 ǫ 4c > 0. (9) h 12

13 Note that this implies also that 1 2 h > 0 sice c 1, i.e by (7). R x > 0, To sum up, we have fixed some costats h, α > 0, ad specified k def = c, so that 1 R x b ad D x α. (10) Now, keep the defiitio of R x i mid, igore D x, ad recall that D x is a set of iput variables such that: each iput variable i D x is read i at most k levels durig the computatio of B o x, ad each of those levels belogs to a block i R x. Recall also that so far we are fixig a iput x i {0, 1}. Cosider all the k-sets i R x, i.e., the subsets of R x of cardiality at most k. Each iput variable i D x is read i such a k-set durig the computatio of B o x (recall the defiitio of a iput variable beig read i a set of blocks from Sectio 4.2), ad there are at most R x k such k-sets, so there are at least D x R x k α b k iput variables i D x that are read i the same k-set of blocks i R x durig the computatio of B o x, where we have used (10) to obtai the estimate. Let U x be such a set of variables i D x, ad let T x be such a k-set of blocks i R x. So 1 T x k ad U x α b k. Note that T x is oempty sice E is ijective. Associate each x i {0, 1} with ay such a U x ad T x. There are at most b k such T x, so there is a subset Q {0, 1} ad a k-set of blocks T such that Q 2 b k, ad T = T x for each x i Q. Now cosider the lower boudary levels L 1,...,L T of the blocks i T ordered by the level idex. There are at most S k state sequeces i L 1... L T, ad for each x i Q, the computatio of B o x cotais such a sequece, so there is a sequece {s i } T i=1 of states i L 1... L T ad a subset Q Q such that Q Q S k 2 (Sb) k, ad the computatio of B o x cotais {s i } i, for each x i Q. This completes the proof of Lemma Step E: boudig the miimum distace Now we are ready to fid the two distict messages x 1 ad x 2 that are mapped by E to codewords that are close to each other. Usig T, for each x i {0, 1}, let I x be the set of iput variable that are ot read durig the computatio of B o x i blocks outside T. Note that we eed a double egatio ( ot read 13

14 ad outside ) sice some of the iput variables may ot be read at all because we modified the brachig program i (D). So, from (3) i Lemma 4.2, for each x i Q, Usig T, defie the equivalece relatio o {0, 1} by x y if: I x = I y, ad x agree with y o the bits outside I x. I x α b k. (11) I other words, x I[x] = y I[y], where [x] meas the equivalece class of x. Give ay x i {0, 1}, each y x ca oly disagree with x o I x. Coversely, if y disagree with x oly iside I x, it must be the case that y x. To see why this is true, ote that we forced i (D) all the computatios of B to leave the blocks i T i the same states: the {s i } i that we exhibited i (1) i Lemma 4.2. So the computatios of B o x ad y are exactly the same outside the blocks i T, ad hece ay bit accessed o x outside T will be accessed o y outside T ad oe of the bits i I x will be accessed o y outside T. It follows that Thus, by (11), for each x i Q, [x] = 2 Ix. [x] 2 α/bk. Let Ω be the set of equivalece classes such that the size of each equivalece class is at least 2 α/bk. So, [x] is i Ω for each x i Q. Besides, sice the equivalece classes are disjoit, we must have Ω 2 α/bk 2, i.e., Ω 2. (12) 2 α/bk If we ca guaratee that Q > Ω, (13) it follows that there will exist x 1 x 2 i Q such that [x 1 ] = [x 2 ]. The fact that [x 1 ] = [x 2 ] meas that the computatios of B o x 1 ad x 2 are exactly the same outside the blocks i T, ad hece E(x 1 ) ad E(x 2 ) ca oly disagree o the output bits that are set iside the blocks i T. But, by (2) i Lemma 4.2, we costructed Q i such way that for o ay x i Q, each block i T ca set at most w bits of E(x) durig the computatio of B o x. Thus E(x 1 ) ad E(x 2 ) ca disagree o at most 2 T w 2k hp 1a c 2khca bc = 2kha b bits, where the first iequality follows from T k (by (5)) ad w = hp 1 t/l = hp 1 a/c (by (6) ad (1)), ad the secod follows from p 1 = l/b = c/b c/b (by (2)). Moreover, E(x 1 ) ad E(x 2 ) must disagree o at least oe bit sice x 1 x 2, ad E is ijective. Usig (3) ad (12), coditio (13) ca be guarateed to hold if 2 α/bk > (Sb) k, (14) which is fulfilled whe 1 b > ( ) 1/k k log2 (Sc), α 14

15 sice b l = c. If we ca select b so that this holds, we get that the miimum distace of C is at most 2kha/b. The oly restrictio we have o b is 1 b l, so use ( ) 1/k b def α = 1. (15) k log 2 (Sc) This is always less tha l, ad it caot be less tha 1 uless S 2 α/k2k log 2 (c) i which case the statemet of the theorem is trivial. Thus, via (14), the miimum distace of C is at most ( 2kha α k log 2 (Sc) ) 1/k 1 ( ( log2 S = O 4.8 Step F: droppig the liear time assumptio ) 1 ) k. Now we drop the assumptio that t = Θ(), thus a ad c eed ot be costats. Sice l t, we use a as a upper boud o c. Assume that a grows with ad assume also that it is O(log 2 ) sice otherwise the statemet of Theorem 3.1 is trivial. We will ot choose k = c. Goig back to (8) ad (9), we have α = 1 c 1 k 4c h > 1 a 1 4a k h, a value that we eed to keep bouded away from zero by a positive costat. Set k = 2(a 1) ad h = 16a, thus α > 1/4. By usig the same choice of b as i (15) ad the same boud o the miimum distace of C i (14), but with the ew values of k ad h ad the boud c a, we get that the miimum distace of C is at most 2 2(a 1) 16a a ( ) 1/ 2(a 1) /4 2(a 1) log 2 (Sa) 1 ( ( = O a 3 log2 S ) 1 ) 2a. 5 Whe the ecoder is a costat-depth AND-OR circuit To outlie the boudaries of the ecodig complexity versus miimum distace questio studied i this paper, we cosider the same problem but from the perspective of the circuit complexity of the ecoder. Here we ote that ot much ca be said other tha what is essetially expected. Sice we kow from [9] that there are asymptotically good codes that are ecodable by liear-size ad logarithmic-depth circuits, we are left with costat-depth circuits ecoders with ubouded fais. Let C {0, 1} m be a code. We say that C is ecodable by a depth-d AND-OR circuit with ubouded fais if C has a ecoder E : {0, 1} {0, 1} m (i.e., E is a ijective map such that C = Image(E)) such that each of the m output variables E 1 (x),..., E m (x) is computable by a depth-d AND-OR circuit o the iput variables x 1,..., x. We require the gates i the circuit to be AND or OR gates with possibly egated iputs, but we allow the umber of iputs per gate to be ubouded. The size of the circuit is defied as the total umber of gates. We argue by a direct applicatio of Hastad Switchig Lemma that a polyomial size costatdepth circuit caot ecode a asymptotically good code. We will actually show that a size-s costat-depth circuit caot ecode a asymptotically good code as log as log 2 S = o(m l/l ), where l is the depth of the circuit. This is ot surprisig sice i the special case of liear codes a small-depth circuit ecoder correspods to a code with a low desity geerator matrix. 15

16 Lemma 5.1 Hastad Switchig Lemma [7]: Let f : {0, 1} {0, 1} be computable by a ubouded-fais depth-d AND-OR circuit of size M. Cosider a radom restrictio ρ that idepedetly keeps each iput bit uset with a probability p = 1/(20k) d, sets it to 1 with a probability (1 p)/2, ad to 0 with a probability (1 p)/2. The the probability, over the choice of ρ, that f, whe restricted to the values set by ρ, caot be computed by a decisio tree of depth k is at most M2 2k. Note that a decisio tree computig a biary fuctio b : {0, 1} {0, 1} o variables is a biary tree where each ode is associated with oe of the iput variables, ad each leaf is associated with a 0 or 1 settig of the sigle output variable. This implies that if we fix ay settig of the iput variables, there are at most k variables that whe egated will affect the value of b, where k is the depth of the tree. I other words, whe k is small, b has low sesitivity. Thus, if a ecoder {0, 1} {0, 1} m is computable by m decisio trees each of depth k, a direct coutig argumet shows that its miimum distace ca be at most km/. Hastad Switchig Lemma essetially reduces the circuit case to this situatio. Theorem 5.2 Let E : {0, 1} {0, 1} m be a ijective ecodig map computable by a ubouded fais AND-OR circuit of size S ad depth l. Let C be the biary code associated with E, i.e., the image of E i {0, 1} m. Assume that m = Θ(), i.e., assume that the code-rate is ozero. The the miimum distace of C is O((20) l log l+1 2 (ms)). Thus, C is asymptotically bad whe l = O(1) ad S = 2 o(m1/l). Proof: Let x 1,..., x be the iput variables, A 1,..., A m the circuits that compute the output variables y 1,...,y m, ad a = m/. Thus a 1 is costat, the size of each A i is at most S, ad the depth of each A i is at most l. Hit the x i s with a radom restrictio ρ that keeps each x i uset with a probability p = 1/(20k) l, sets x i to 1 with a probability (1 p)/2, ad to 0 with a probability (1 p)/2. The, for each A i, from the Hastad Switchig Lemma, the probability that A i does ot collapse to a decisio tree of depth k is at most S2 2k. Thus the probability that oe of the A i s does ot collapse, or the umber of remaiig (uset) variables variables is less tha p/2 is at most P def = ms2 2k p(1 p) + (p/2) 2 = ms2 2k + 4(1 p), p where the secod term comes from the Chebychev Iequality. Fix k = log 2 (Sm) so that P 1/(Sm) + 4(20 log 2 (Sm)) l / < 1 whe is large eough ad S is subexpoetial i 1/l. Note that whe S is expoetial i 1/l, the statemet of the theorem is trivial. So, fix ay restrictio ρ with the property that: the set I of iput variables left uset by ρ has size at least p/2, ad each of the A i s collapses uder ρ to a decisio tree T i of depth k, where k = log 2 (Sm) ad p = 1/(20k) l. Cosider ay settig of the variables i I, ad let I i be the set of variables i I read by T i o this settig. Each I i cotais at most k variables, ad the output of T i ca oly be affected whe we chage some of the variables i I i. So there should be a variable i I that appears i at most m j=1 I j km I p/2 = 2ka p 16

17 of the I i s. By flippig this variable, we ca affect at most 2ka/p output bits, ad at least oe output bit sice E is ijective. Hece the miimum distace of C is at most 2ka p = 2 log 2 (Sm)a 1/(20 log 2 (Sm)) l = O((20)l log l+1 2 (Sm)). 6 Ope questios Usig brachig program techiques itroduced by Ajtai [1], we argued i Theorem 3.1 that there are o asymptotically good codes that are ecodable i liear time ad subliear space i the most geeral sese. Coversely, we kow that there are asymptotically good codes that are ecodable i liear time ad liear space (e.g., Spielma [9] explicitly costructed such codes. See also Example 2.4). Thus, whe the ecodig time is liear, the liear memory requiremet is asymptotically tight for ecodig asymptotically good codes. O the other extreme, ay liear code ca be ecoded by a quadratic time brachig program that uses miimal memory (See Example 2.5). We cojecture that i geeral Cojecture 6.1 Let E : {0, 1} {0, 1} m be a ijective ecodig map computable by a brachig program of memory M ad time T, where MT = o( 2 ). Let C be the biary code associated with E, i.e., the image of E i {0, 1} m. Assume that m = Θ(), i.e., assume that the code-rate is ozero. The the miimum distace of C must be o(), i.e., C caot be asymptotically good. Provig the cojecture or fidig the correct time-space tradeoffs for ecodig asymptotically good codes whe the ecodig time is superliear ad subquadratic is very desirable. 7 Ackowledgmet We would like to thak Daiel Spielma ad Ruediger Urbake for may stimulatig coversatios o this material. We would like to thak the aoymous reviewers for may useful commets o the paper. Refereces [1] Miklos Ajtai, Determiism versus No-Determiism for Liear Time RAMs, Thirty- First Aual ACM Symposium o Theory of Computig (STOC), 1999, pp [2] Louay Bazzi, Miimum Distace of Error Correctig Codes versus Ecodig Complexity, Symmetry, ad Pseudoradomess, PhD dissertatio, MIT, Cambridge, MA, [3] L. Bazzi, M. Mahdia, D. Spielma. The Miimum Distace of Turbo-Like Codes, to appear i IEEE Trasactios o Iformatio Theory, [4] C. Berrou, A. Glavieux, ad P. Thitimajshima, Near Shao Limit error-correctig codig ad decodig: Turbo-codes, i Proc., IEEE It. Cof. o Commu., 1993, pp [5] A. Borodi ad S. Cook, A Time-Space Tradeoff for Sortig o a Geeral Sequetial Model of Computatio, SIAM J. Comput. 11(2), 1982, pp

18 [6] D. Divsalar, H. Ji, R. J. McEliece, Codig Theorems for Turbo-Like Codes, Proc. of 36th Aual Allerto Coferece o Commuicatio, Cotrol ad Computig, 1998, pp [7] Joha Hastad, Computatioal Limitatios for Small Depth Circuits, PhD dissertatio, MIT, Cambridge, MA, [8] N. Kahale ad R. Urbake. O the Miimum Distace of Parallel ad Serially Cocateated Codes, to appear i IEEE Trasactios o Iformatio Theory, [9] Daiel A. Spielma, Liear-time ecodable ad decodable error-correctig codes, IEEE Trasactios o Iformatio Theory, 1996, Vol 42, No 6, pp