Asymptotically Tight Bounds for Performing. BMMC Permutations on Parallel Disk Systems. (Extended Abstract) Thomas H. Cormen. Leonard F.

Transcription

1 Asyptotically Tight Bounds for Perforing BMMC Perutations on Parallel Disk Systes (Extended Astract) Thoas H. Coren Leonard F. Wisniewski Departent of Matheatics and Coputer Science Dartouth College Astract We give asyptotically equal lower and upper ounds for the nuer of parallel I/O operations required to perfor BMMC perutations (dened y acharacteristic atrix that is nonsingular over GF (2)) on parallel disk systes. Under the Vitter-Shriver parallel-disk odel with records, D disks, lock sizeb, andm ; records ; of RAM, we show a universal lower ound of 1+ rank() parallel I/Os for perforing a BMMC perutation, where is the lower left lg(=b) lgb suatrix of the characteristic atrix. We adapt this lower ound to show that the algorith for BPC perutations in [Cor93] is asyptotically optial. ; We also present an algorith that uses at ost 2 6 rank() +5 parallel I/Os, which asyptotically atches the lower ound and iproves upon the BMMC algorith in [Cor93]. When rank() islow, this ethod is ; an iproveent over the general-perutation ound of lg(=b). Although any BMMC perutations of practical interest fall into suclasses that ight e explicitly invoked within the source code, we showhow to detect in at ost lg(=b)+1 =+ D parallel I/Os whether a given vector of target addresses species a BMMC perutation. Thus, one can deterine eciently at run tie whether a perutation to e perfored is BMMC and then avoid the generalperutation algorith and save parallel I/Os y using our algorith. 1 Introduction The I/O coplexity of operations with data on disk is ased on data oveent. As one of the ost asic data-oveent operations, peruting fors an iportant part of the theory of I/O coplexity for data on disk. Portions of this research were perfored while To Coren was at the MIT Laoratory for Coputer Science and appear in[cor92]. He was supported in part y the Defense Advanced Research Projects Agency under Grant J Otherportions of thisre- search were perfored while at Dartouth College andwere supported in part y funds fro Dartouth College. Len Wisniewski is supported in part y IFOSEC Grant Moreover, perforing perutations eciently when the data reside on disk is of great practical interest. The proles that we attack with supercoputers are ever-increasing in size, and we nd ore and ore that atrices and vectors exceed the eory provided y even the largest supercoputers. One solution is to store large atrices and vectors on parallel disk systes. The high latency of disk accesses akes it essential to iniize the nuer of disk I/O operations. Peruting the eleents of a atrix or vector is a coon operation, particularly in the data-parallel style of coputing, and good perutation algoriths can provide signicant savings over poor ones when the data reside on parallel disk systes. This paper exaines the class of it-atrix-ultiply/ copleent (BMMC) perutations for parallel disk systes and derives three iportant results: 1. universal lower ounds for BMMC and BPC perutations, 2. an algorith for perforing BMMC perutations whose I/O coplexity asyptotically atches the lower ound, thus aking it asyptotically optial, and 3. an ecient ethod for deterining at run tie whether a given perutation is BMMC, thus allowing us to use the BMMC algorith if it is. Depending on the exact BMMC perutation, this asyptotically optial ound ay e signicantly less than the asyptotically optial ound proven for general perutations. Model and previous results We use the parallel-disk odel rst proposed y Vitter and Shriver [VS90, VS92], who also gave asyptotically optial algoriths for several proles including sorting and general perutations. In the Vitter-Shriver odel, records are stored on D disks D 0 D 1 :::D D;1,with=D records stored on each disk. The records on each disk are organized in locks of B records each. When a disk is read fro or written to, an entire lock of records is transferred. Disk I/O transfers records etween the disks and a rando-access eory, orram, capale of holding M records. Each parallel I/Ooperation transfers up to D locks etween the disks and RAM, with at ost one lock transferred per disk, for a total of up to records transferred. The locks accessed in a single parallel I/O ay eatany locations on

2 D 0 D 1 D 2 D 3 D 4 D 5 D 6 D 7 track track track track Figure 1: The layout of =64records inaparallel disk syste with B =2and D =8.Eachoxrepresents one lock. The nuer of tracks needed is = = 4. uers indicate record indices. their respective disks we enforce no requireent onlockstep head oveent. We easure an algorith's eciency y the nuer of parallel I/O operations it requires. Although this cost odel does not account for the variation in disk access ties caused y head oveent and rotational latency, prograers often have no control over these factors. The nuer of disk accesses, however, can e iniized y carefully designed algoriths such as those in this paper. For convenience, we use the following notation extensively: =lgb d =lgd =lgm n =lg: We shall assue that, d,, andnare nonnegative integers, which iplies that B, D, M, and are exact powersof2. In order for the eory to accoodate the records transferred in a parallel I/O operation to all D disks, we require that M. Also, we assue that M <, since otherwise we can just perfor all operations in eory. These two requireents iply that + d <n. The Vitter-Shriver odel lays out data on a parallel disk syste as shown in Figure 1. A track consists of the D locks at the sae location on all D disks. Record indices vary ost rapidly within a lock, then aong disks, and nally aong tracks. We indicate the address of a record as an n-it vector x with the least signicant it rst: x =(x 0x 1:::x n;1). The oset within the lock is given y the least signicant its x 0x 1:::x ;1, the disk nuer y the next d its x x +1 :::x +d;1, and the track nuer y the reaining n ; ( + d) ost signicant its x +d x +d+1 :::x n;1. Since each parallel I/O operation accesses at ost records, any algorith that ust access all records requires (=) parallel I/Os, and so O(=) parallel I/Os is the analogue of linear tie in sequential coputing. ; ; Vitter and Shriver showed an upper ound of lg(=b) in D parallel I/Os for general perutations. The rst ter coes into play when the lock size ; B is sall, and the second ter is the sorting ound lg(=b), which was shown y Vitter and Shriver [VS90, VS92] for randoized sorting and y odine and Vitter [V90, V91, V92] for deterinistic sorting. These ounds are asyptotically tight, for they atch thelower ounds proven y Aggarwal and Vitter [AV88] using a odel with one disk and D independent read/write heads, which is at least as powerful as the Vitter-Shriver odel. Specic classes of perutations soeties require fewer parallel I/Os than general perutations. Vitter and Shriver showed ; how to transpose an R ; Satrix ( = RS) with only 1+ lg in(brs=b) parallel I/Os. Susequently, Coren [Cor93] studied several classes of itdened perutations that include atrix transposition as a special case. Tale 1 shows the classes of perutations exained and the corresponding upper ounds derived in [Cor93]. BMMC perutations The ost general class considered in [Cor93] is it-atrixultiply/copleent, or BMMC, perutations. 1 In a BMMC perutation, we have annncharacteristic atrix A =(a ij) whoseentries are drawn fro f0 1g and is nonsingular (i.e., invertile) over GF (2), 2 and we have acopleent vector c =(c 0c 1:::c n;1) of length n. Treating a source address x as an n-it vector, we perfor atrixvector ultiplication of GF (2) and then for the corresponding n-it target address y y copleenting soe suset of the resulting its: y = Ax c. The BMMC algorith in [Cor93] uses 2 H( M B) = 2l lg M ; r 8 lg B 4l >< >: + H(MB) parallel I/Os, where r is the rank of the leading lg M lg M suatrix of the characteristic atrix and +9 ifm p 4l lg(=b) 5 if p B M: +1 if p <M< p B (1) Onecanadaptthelower ; ound proven in this paper to show M;r that parallel I/Os are necessary (see Section 2.8 of [Cor92] for exaple), ut it has een unknown whether the (= + H(MB)) ter is necessary in all cases. This paper shows that it is not. BPC perutations By restricting the characteristic atrix A of a BMMC perutationtoeaperutation atrix having exactly one 1 in each row and each colun we otain the class of it-perute/copleent, orbpc, perutations. 3 One can think of a BPC perutation as foring each target address y applying a xed perutation to the source-address its and then copleenting a suset of the resulting its. The 1 Edelan, Heller, and Johnsson [EHJ92] call BMMC perutations ane transforations or, if there is no copleenting, linear transforations. 2 Matrix ultiplication over GF(2) is like standard atrix ultiplication over the reals ut with all arithetic perfored odulo 2. Equivalently, ultiplication is replaced y logical-and, and addition is replaced y OR. 3 Johnsson and Ho [JH91] call BPC perutations diension perutations, and Aggarwal, Chandra, and Snir [ACS87] call BPC perutations without copleenting rational perutations.

3 Perutation Characteristic atrix uer ofpasses BMMC (it-atrix-ultiply/ copleent) BPC (it-perute/ copleent) nonsingular atrix A 2l lg M ; r perutation atrix A 2l (A) + H(MB) +1 Block BMMC n ; h nonsingular 0 0 nonsingular i n ; 1 MRC (eoryrearrangeent/ copleent) n ; h nonsingular aritrary 0 nonsingular i n ; 1 Tale 1: Classes of perutations, their characteristic atrices, and upper ounds shown in [Cor93] on the nuer of passes needed to perfor the. A pass consists of reading and writing each record exactly once and therefore uses exactly 2= parallel I/Os. For lock BMMC and MRC perutations, suatrix diensions are shown on atrix orders. For BMMC perutations, r is the rank of the leading lg M lg M suatrix of A, andthefunction H(MB) isgiven y equation (1). For BPC perutations, the function (A) is dened in equation (3). class of BPC perutations includes any coon perutations such as atrix transposition, it-reversal perutations (used in perforing FFTs), vector-reversal perutations, hypercue perutations, and atrix relocking. We express the I/O coplexity of BPC perutations in ters of cross-ranks. For any n n perutation atrix A and for any k =0 1:::n; 1, we dene the k-cross-rank of A as k(a) = rank(a k::n;10::k;1) = rank(a 0::k;1k::n;1) (2) where, for exaple, A k::n;10::k;1 denotes the suatrix of A consisting of the intersection of rows k k +1:::n; 1 and coluns 0 1:::k; 1. The cross-rank of A is the axiu of the - and-cross-ranks: (A) =ax( (A) (A)) : (3) The BPC algorith in [Cor93] uses at ost l (A) parallel I/Os. As we shall see in Section 2, this algorith is asyptotically optial. One-pass perutations We shall use the reaining two classes in Tale 1 as suroutines in perforing BMMC perutations. Each class is a restricted BMMC perutation, descried y a characteristic atrix and a copleent vector, and is shown in[cor93] to require only one pass of = parallel reads and = parallel writes. Block BMMC perutations have the restrictions that oth the leading suatrix and the trailing (n ; ) (n ; ) suatrix are nonsingular and the rest of the characteristic atrix is 0. Meory-rearrangeent/copleent, ormrc, perutations have the restrictions that oth the leading and trailing (n ; ) (n ; ) suatrices are nonsingular, the upper right (n ; ) suatrix can contain any 0-1 values at all, and the lower left (n ; ) suatrix is all 0. Any MRC perutation can e perfored y reading in a eoryload (i.e., M records) of M= tracks, peruting the records read within eory, and writing the out to M= tracks. The class of MRC perutations includes those characterized y unit upper triangular atrices. As [Cor93] shows, oth the standard inary-reected Gray code and its inverse have characteristic atrices of this for, and so they are MRC perutations. Outline The reainder of this paper is organized as follows. Section 2 states and proves the lower ounds for BMMC and BPC perutations. In Section 3, we descrie an algorith for BMMC perutations whose I/O coplexity asyptotically atches the lower ound. Section 4 shows how to detect at run tie whether a vector of target addresses descries a BMMC perutation, thus enaling us to deterine whether the BMMC algorith is applicale. Finally, Section 5contains soe concluding rearks. We shall use several notational conventionsinthispaper. Matrix row and colun nuers are indexed fro 0 starting fro the upper left. As in equation (2), we index rows and coluns y sets to indicate suatrices, using \::" notation to indicate sets of contiguous nuers. When a suatrix index is a singleton set, we shall often oit the enclosing races. We denote an identity atrix y I and a atrix whose entries are all 0s y 0 the diensions of such atrices will e clear fro their contexts. All atrix and vector eleents are drawn fro f0 1g, and all atrix and vector arithetic is over GF(2). When convenient, we interpret it vectors as the integers they represent in inary. Vectors are treated as 1-colun atrices in context.

4 2 Lower ounds for BMMC and BPC perutations In this section, we state and prove thelower ounds for BMMC and BPC perutations. After stating the lower ounds, we riey discuss their signicance efore presenting the full proofs. The lower ounds are given y the following theores. Theore 1 Any algorith that perfors a BMMC perutation with characteristic atrix A requires 1+ rank() parallel I/Os, where is the suatrix A ::n;10::;1 of size (n ; ). Theore 2 Any algorith that perfors a BPC perutation with characteristic atrix A requires parallel I/Os. 1+ (A) These lower ounds are universal in the sense that they apply to all inputs. In contrast, lower ounds such asthe standard ( lg) lower ound for sorting ites on a sequential achine are existential: they apply to worst-case inputs, ut for soe inputs we ayealetodoetter. There are in fact algoriths that achieve the ounds given y Theores 1 and 2, and so these algoriths are asyptotically optial. Section 3 presents an algorith for BMMC perutations, and the BPC algorith of [Cor93] achieves the ound of Theore 2. The atrix-transposition parallel I/Os is ound of ; ; 1+ lg in(brs=b) consistent with the BPC forulation see [Cor92] for details. Proofs of the lower ounds To prove Theores 1 and 2, we rely heavily on the technique used y Aggarwal and Vitter [AV88] for a lower ound on I/Os in atrix transposition their proof is ased in turn on a ethod y Floyd [Flo72]. We prove the lower ound for the case in which D = 1, the general case following y dividing y D. We consider only I/Os that are siple. An input is siple if each record read is reoved fro the disk and oved into an epty location in RAM. An output is siple if the records are reoved fro the RAM and written to epty locations on the disk. When all I/Os are siple, exactly one copy of each record exists at any tie during the execution of an algorith. The following lea, proven y Aggarwal and Vitter, allows us to consider only siple I/Os when proving lower ounds. Lea 3 For each coputation that ipleents a perutation of records, there is a corresponding coputation strategy involving only siple I/Os such that the total nuer of I/Os is no greater. Potential function The asic schee of the proof of Theore 1 uses a potential function arguent. We call the tie interval starting when the qth I/O copletes and ending just efore the (q +1)st I/O starts tie q. We dene a potential function so that (q) is the potential at tie q. We copute the initial and nal potentials and ound the aount that the potential can increase in each I/O operation. The lower ound then follows. To e ore precise, we start with soe denitions. For i =0 1:::=B ; 1, we dene the ith target group to e the set of records that elong in lock i according to the given BMMC perutation. (Reeer that D = 1.) We denote y g lock (i k q) thenuerofrecordsinthe ith target group that are in lock k at tie q, andg e(i q) denotes the nuer of records in the ith target group that are in RAM at tie q. We dene the continuous function f(x) =n x lg x if x>0 0 if x =0 and we dene togetherness functions =B;1 G lock(kq) = f(g lock(i k q)) i=0 for each lock k at tie q and =B;1 G e(q) = f(g e(i q)) i=0 for RAM at tie q. Finally, we dene the potential at tie q, denoted (q), as the su of the togetherness functions: (q) =G e(q)+ =B;1 k=0 G lock(k q) : Aggarwal and Vitter eed the following leas in their lower-ound arguent. The rst one is ased on the oservation that the nuer of parallel I/Os is at least the total change in potential over all parallel I/Os divided y the axiu change in potential in any single parallel I/O. Lea 4 Let D =1, and consider any algorith that perfors a perutation. The increase in potential due to any parallel I/O operation is O(B ). Therefore, any algorith that uses t parallel I/Os to perfor a perutation ust have t = ; B + (t);(0) B. Lea 5 Let D =1,andconsider any perutation that can eperfored witht parallel I/Os. Then (t) =lg B. Therefore, any algorith ; that uses t parallel I/Os to perfor lg B;(0) aperutation uses B + B parallel I/Os. Oserve that these leas iply lower ounds that are universal. o atter what the input, the initial potential is (0), the nal potential is (t), the increase in potential ; per parallel I/O is ounded y O(B ), and so (t);(0) B parallel I/Os are required. The (=B) ter is the trivial lower ound due to reading the entire input.

5 Ranges and doains To prove the lower ounds, we shall exaine ranges of atrices and doains of vectors under atrix ultiplication. We oit the proofs of two siple leas. For a p q atrix A with 0-1 entries, we dene the range of A y R(A) =fy : y = Ax for soe x 2f0 1:::2 q ; 1gg that is, R(A) isthesetofvalues that can e produced y ultiplying all q-vectors with 0-1 entries (interpreted as integers in f0 1:::2 q ; 1g) y A over GF (2). We also adopt the notation R(A) x = fz : z = y x for soe y 2R(A)g that is, R(A) x is the exclusive-or of the range of A and a xed vector x. Lea 6 Let A e ap q atrix whose entries are drawn fro f0 1g, let x e any p-vector whose entries aredrawn fro f0 1g, andletr =rank(a). Then jr(a) xj =2 r. For a p q atrix A and a p-vector y 2R(A), we dene the doain of y under A y Do(A y) =fx : Ax = yg : That is, Do(A y) is the set of q-vectors x that ap to y when ultiplied y A. Lea 7 Let A e ap q atrix whose entries are drawn fro f0 1g, let y e any p-vector in R(A), and let r = rank(a). Then jdo(a y)j =2 q;r. Proof of Theore 1 To prove Theore 1, we prove the lower ound for the case in which D = 1, the general case following y dividing y D. We assue that all I/Os are siple and transfer exactly B records, soe possily epty. Since all records start on disk and I/Os are siple, RAM is initially epty. We need to copute the initial potential in order to apply Lea 5. The initial potential depends on the nuer of records that start in the sae source lock and are in the sae target group. A record with source address x =(x 0 x 1 :::x n;1 ) is in source lock k if and only if k = x ::n;1 (4) interpreting k as an (n ; )-it inary nuer. This record apstotarget lock i if and only if i = A ::n;10::n;1 x 0::n;1 = A ::n;10::;1 x 0::;1 A ::n;1::n;1 x ::n;1 (5) also interpreting i as an (n ; )-it inary nuer. The following lea gives the exact nuer of records that start in each source lock and are in the sae target group. Lea 8 Let r =rank(a ::n;10::;1 ), and consider any source lock k. There are exactly 2 r distinct target locks that soe record insource lock k aps to, and for each such target lock, exactly B=2 r records in source lock k ap to it. Proof: For a given source lock k, all source addresses fulll condition (4), and so they ap to target lock nuers given y condition (5) ut with x ::n;1 xed at k. The range of target lock nuers is thus R(A ::n;10::;1 ) A ::n;1::n;1 k which, y Lea 6, has cardinality 2 r. ow we deterine the set of source addresses in source lock k that ap to a particular target lock i in R(A ::n;10::;1 ) A ::n;1::n;1 k. Again xing x ::n;1 = k in condition (5) and exclusive-oring oth sides y A ::n;1::n;1 k, we see that this set is precisely Do(A ::n;10::;1i A ::n;1::n;1 k). By Lea 7, this set has cardinality exactly 2 ;r,whichequalsb=2 r. We can interpret Lea 8 as follows. Let r = rank(a ::n;10::;1 ), and consider a particular source lock k. Then there are exactly 2 r target locks i for which g lock (i k 0) is nonzero, and for each such nonzero target lock, we have g lock(i k 0) = B=2 r. ow we can copute (0). Since RAM is initially epty, g e(i 0) = 0 for all locks i, which iplies that G e(0) = 0. We have =B;1 (0) = G e(0) + G lock (k 0) = 0+ = =B;1 k=0 =B;1 k=0 = B B lg B 2 r k=0 =B;1 i=0 f(g lock(i k 0)) 2 r B 2 r lg B 2 r (y Lea 8) = (lg B ; r) : (6) Coining Lea 5 and equation (6), we get a lower ound of lg B ; (lg B ; r) + = B B 1+ rank(a::n;10::;1) B parallel I/Os. Dividing through y D yields a ound of 1+ rank(a ::n;10::;1) which copletes the proof of Theore 1. Proof of Theore 2 We use Theore 1 and the following lea to prove Theore 2. Lea 9 For any perutation atrix A, we have (A) ; (A) (A)+ : Proof: Since the rank of any suatrix of a perutation atrix is equal to the nuer of 1s in the suatrix, we

6 have that (A) = rank(a ::n;10::;1) = rank(a ::n;10::;1) ; rank(a ::;10::;1) + rank(a ::n;1::;1 ) = (A) ; rank(a ::;10::;1 ) + rank(a ::n;1::;1 ) : (7) The rank of a suatrix is nonnegative and is at ost the saller of the nuer of rows and the nuer of coluns, which iplies that 0 rank(a ::;10::;1 ) ; = 0 rank(a ::n;1::;1) ; = : Coining the aove inequalities with equation (7) yields (A) ; (A) ; rank(a ::;10::;1) (A) (A) + rank(a ::n;1::;1 ) (A) + which copletes the proof. To prove Theore 2, we note that Lea 9 iplies that rank(a ::n;10::;1) = (A) = (A) Coined with Theore 1, wegetalower ound of 1+ (A) parallel I/Os for BPC perutations. 3 An asyptotically optial BMMC algorith In this section, we present an algorith to perfor BMMC perutations y factoring the into sipler perutations. As usual, we assue that the BMMC perutationisgiven y annn characteristic atrix A and a copleent vector c; of length n. The nuer of parallel I/Os is at ost 2 6 rank() +5 parallel I/Os, where is the suatrix A ::n;10::;1. Our strategy is to factor the atrix A into a product of atrices, each ofwhich isthecharacteristic atrix of one of four types of perutations: MRC, lock BMMC, BPC, andafourth type descried in Appendix A. For now, we ignore the copleent vector c. We read the factors right to left to deterine the order in which to perfor these perutations. For exaple, if we factor A = VW, then we perfor A y rst peruting according to characteristic atrix W and then peruting according to characteristic atrix V. The reason for this right-to-left order is that if y = Ax, then we rstultiply Wx,giving y 0, and then ultiply Vy 0,givingy. Factoring the atrix A in this way will ake it easy to count how any passes the BMMC perutation takes. : Peruting coluns to create a nonsingular trailing suatrix We start y peruting the coluns of A so that the trailing (n; ) (n; ) suatrix is nonsingular. That is, we divide A etween the lower rows and coluns and the upper n; rows and coluns and factor it as A = A, where A = A = n ; n ; n ; n ; isaperutation atrix, and = A::n;1::n;1 is a nonsingular suatrix. We would like to perute the coluns of A so that the nuer of I/Os required to perfor is iniu. We call such a perutation a iniu-ipact perutation. The size of the largest set S of coluns of that are linearly independent is equal to rank(). We can coe up with a iniu-ipact perutation y choosingasett of n ; ; rank() coluns fro the leftost coluns of A that, along with S, provide the needed set of n ; linearly independent coluns. To nd these sets S and T, we can use Gaussian eliination, as descried in [Cor92]. Because the perutation characterized y exchanges n ; ; rank() coluns etween the leftost and the rightost n ; coluns, one can easily show that () =n ; ; rank() : (8) Moreover, ecause there are at least n ; ; rank() linearly independent coluns in that are also in,wehave rank() n ; ; rank() : (9) Coining properties (8) and (9) yields () rank(), andsowe can perfor the BPC perutation characterized rank() y with at ost passes. ote also that ecause the linearly independent coluns in coe fro either or fro the coluns of that are peruted to, we have that rank() rank()+n ; ; rank() 2rank() (10) Oserve that whenever a nonsingular atrix is expressed as a product of atrix factors, each of the factors ust itself e nonsingular. This fact follows ecause the rank of a atrix product is at ost the rank of any factor, so if the product of square atrices has full rank, each factor ust have full rank as well. Factoring the reaining atrix Our next task is to factor the atrix A,which is nonsingular y the aove arguent. We factor A as A = UV W

7 where U = V = W = n ; ;1 ;1 0 I n ; I 0 I n ; I 0 0 n ; n ; n ; andsothefactors U, V,andW are nonsingular. Since is nonsingular, W characterizes a lock BMMC perutation, and so it requires only one pass. The upper suatrix ;1 of U is nonsingular ecause if it contained linearly dependent coluns, then so would U, contradicting the nonsingularity ofu.thus, U characterizes an MRC perutation, which only requires one pass. We further divide the atrix V and factor it as where and P = Q = V = PQ 2 4 I 0 0 I I ; n; 2 4 I I 0 0 I ; n; Here, we have divided the suatrix as = ; n ; 3 5 ; n ; 3 5 ; n ; : : (11) The upper suatrix of P is unit lower triangular and, therefore, nonsingular. Thus, P characterizes an MRC perutation, requiring only one pass. It reains to perfor the perutation characterized y the atrix Q, whose lower left (n ; ) suatrix ay e nonzero. Our strategy is to factor Q into g = drank( )=( ; )e atrices Q (1) Q (2) :::Q (g), each of which characterizes a perutation that requires only two passes. ote that equation (11) iplies that rank() rank() and, using inequality (10), we have g d2 rank()=( ; )e. To factor Q, we need a colun asis C (a axial linearly independent set of coluns) for, which again we can nd y Gaussian eliination. Having found C, let C denote the suatrix of containing the coluns indexed y C. Let C 0 = f0 1::: ; 1g ;C index the coluns of not in the colun asis. We partition C into g sets C (1) C (2) :::C (g), where C (i) ; for i =1 2:::g. For each index j 2 C 0, the colun j is a linear coination of coluns in Cwe dene C j (i) to e the set of colun indices in C (i) that contriute to this su of coluns. That is, for each j 2 C 0,wehave gm M! j = k : i=1 k2c (i) j Each factor Q (i) is the atrix Q (i) = 2 4 I I 0 (i) 0 I ; n; 3 5 ; n ; (12) where the (n ; ) suatrix (i) has coluns dened as follows: For j 2 C (i),weset (i) j = j. For j 2 C ; C (i),weset (i) j M =0. For j 2 C 0,weset j (i) = k. k2c (i) j The atrices (i) are constructed so that rank( (i) ) = C (i) ; and that = (1) (2) (g). One can verify that Q = Q (1) Q (2) Q (g), as required. To recap,we have factored our original BMMC characteristic atrix A into A = UPQ (1) Q (2) Q (g) W. Both U and P characterize MRC perutations, and so does their product Z = UP. We can therefore express A with one fewer factor: A = ZQ (1) Q (2) Q (g) W. Appendix A sketches how to perfor each perutation characterized y a factor Q (i) in only two passes. (The procedure is conceptually coplex ut coputationally easy.) The factors Z and W require one pass each, and the BPC perutation rank() given y requires at ost passes. The BMMC perutation given y A, therefore, requires at ost 2g +2+2l rank() +1 2l 2 rank() 6l rank() l rank() passes. Earlier, we deferred discussion of the copleent vector c. To include it in the BMMC perutation, we perfor it as part of the last MRC perutation, which ischaracterized y Z. o additional pass is needed to incorporate the copleent vector into the BMMC perutation. 4 Detecting BMMC perutations at run tie In practice, wewish to run the BMMC algorith of Section 3 whenever possile to reap the savings over having run the ore costly algorith for general perutations. For that atter, we wish to run the BPC, MRC, and lock BMMC algoriths of [Cor93] whenever possile as well. We ust

8 know the characteristic atrix A and copleent vector c, however, to run any of these algoriths. If A and c are specied in the source code, efore running the algorith we onlyneedtocheck that A is of the correct for, e.g., that it is nonsingular for a BMMC perutation, a perutation atrix for a BPC perutation, etc. If instead the perutation is given y target addresses, we can detect at run tie whether it is a BMMC perutation y the following procedure: 1. Check that isapower of For a candidate characteristic atrix A and copleent vector c such that if the perutation is BMMC, then A and c ust e the correct characterizations. lg(=b)+1 This section shows how to dosowithonly D parallel reads. 3. Check that the characteristic atrix is of the correct for. 4. Verify that all target addresses are descried y the candidate characteristic atrix and copleent vector. If for any source address x and its corresponding target address y we have y 6= Ax c, the perutation is not BMMC and we can terinate verication. If y = Ax c for all source-target pairs, the perutation is BMMC. Verication uses at ost = parallel reads. The total nuer of parallel I/Os is at ost l lg(=b)+1 + D all of which are reads, and it is usually far fewer when the perutation turns out not to e BMMC. One enet of run-tie BMMC detection is that the prograer ight not realize that the perutation to perfor is BMMC. For exaple, as noted in Section 1, the standard inary reected Gray code and its inverse are oth MRC perutations. Yet the prograer ight not know to call a special MRC or BMMC routine. Even if the syste provides an entry point to perfor the standard Gray code perutation and this routine invokes the MRC algorith, variations on the standard Gray code ay foil this approach. For exaple, a standard Gray code with all its peruted the sae (i.e., a characteristic atrix of G, where is a perutation atrix and G is the MRC atrix that characterizes the standard Gray code) is BMMC ut not necessarily MRC. It ight noteovious enough that the perutation characterized y G is BMMC for the prograer to invoke the BMMC algorith explicitly. Foring the candidate characteristic atrix and copleent vector The ethod for foring the candidate characteristic atrix A and candidate copleent vector c is ased on two oservations. First, if the perutation is BMMC, thenthe copleent vector c ust e the target address corresponding to source address 0. This relationship holds ecause x = 0 and y = Ax c iply that y = c. The second oservation is as follows. Consider a source address x =(x 0x 1:::x n;1), and suppose that it position k holds a 1, i.e., x k = 1. Let us denote the jth colun for atrix A y A j. Also, let S k denote the set of it positions in x other than k that hold a 1: S k = fj : j 6= k and x j =1g. Ify = Ax c, thenwehave y = M j2s k A j! A k c (13) since only the it positions j for which x j = 1 contriute a colun of A to the su of coluns that fors the atrixvector product. If we know the target address y, the copleent vector c and the coluns A j for all j 6= k, we can rewrite equation (13) to yield the kth colun of A: A k = y! M A j c: (14) j2s k We shall copute the copleent vector c rst and then the coluns of the characteristic atrix A one at a tie, fro A 0 up to A n;1. When coputing A k,we will have already coputed A 0A 1:::A k;1, and these will e the only coluns we need in order to apply equation (14). In other words, S k f0 1:::k ; 1g. Recall that in an address, the lower its give the record's oset within its lock, the iddle d its give the disk nuer, and the upper n;(+d) its give the track nuer. Fro equation (14), it would e easy to copute A k if S k were epty. The set S k is epty if the source address is a unit vector, with its only 1 in position k. Ifwelookat these addresses, however, we nd that the target addresses for a disproportionate nuer all ut d of the reside on disk D 0. The lock whose disk and track eldsareallzero contains such addresses, so they can e fetched in one disk read. A prole arises for the n ; ( + d) source addresses with one 1 in the track eld: their target addresses all reside on dierent locks of disk D 0. Each ust e fetchedina separate read. The total nuer of parallel reads to fetch all the target addresses corresponding to all unit-vector source addresses is n ; ( + d)+1=lg(=)+1. lg(=b)+1 To achieve only D parallel reads, each read fetches one lock fro each of the D disks. The rst parallel read deterines the copleent vector, the rst + d coluns, and the next D ; d ; 1 coluns. Each susequent read deterines another D coluns, until all n coluns have een deterined. In the rst parallel read, we do the sae as aove for the rst + d its. That is, we fetch locks containing target addresses whose corresponding source addresses are unit vectors with one 1 in the rst + d positions. As efore, of the are in the sae lock ondisk D 0. This lock alsocontains address 0, which we need to copute the copleent vector. The reaining d are in track nuer 0 of disks D 1 D 2 D 4 D 8:::D D=2.Having fetched the corresponding target addresses, we have all the inforation we need to copute the copleent vector c and coluns A 0A 1:::A +d;1. The coluns we have yetto copute correspond to it positions in the track eld. If we were to copute these coluns in the sae fashion as the rst + d, wewould again encounter the prole that all the locks we need to read are on disk D 0. In the rst parallel read, the only unused disks reaining are those whose nuers are not a power of 2 (D 3 D 5 D 6 D 7 D 9:::). The key oservation is that we have already coputed all d coluns corresponding

9 to the disk eld, and we canthus apply equation (14). For exaple, let us copute colun A +d, which corresponds to the rst it of the track nuer. We read track 1on disk D 3 and nd the rst target address y in this lock. Disk nuer 3 corresponds to the rst two disk-nuer coluns, A and A +1. Applying equation (14) with S +d = f +1g, we copute A +d = y A A+1 c. The next colun we copute is A +d+1. Reading the lockattrack2ondiskd 5, we fetch a target address y and then copute A +d+1 = ya A +2 c. Continuing on in this fashion, we copute a total of D ; d ; 1 track-it coluns fro the rst parallel read. The reaining parallel reads copute the reaining track-it coluns. We follow the track-it pattern of the rst read, ut we use all disks, not just those whose disk nuers are not powersof2. Eachlockreadfetches a target address y, whichwe exclusive-or with a set of coluns fro the disk eld and with the copleent vector to copute a new colun fro the track eld. The rst parallel read coputes + D ; 1 coluns and all susequent parallel reads copute D coluns. The total nuer of parallel reads is thus 1+l n ; ( + D ; 1) D 5 Conclusions = 1+l lg(=b) ; D +1 D l lg(=b)+1 = : D This paper has shown an asyptotically tight ound on the nuer of parallel I/Os required to perfor BMMC perutations on parallel disk systes. It is particularly satisfying that the tight ound was achieved not y raising the lower ound proven here and in [Cor92], ut y decreasing the upper ound in [Cor93]. The constant factors in the I/O coplexity of our algorith are sall, which is especially fortunate in light of the expense of disk accesses. We have also shown how to detect BMMC perutations at run tie, given a vector of target addresses. Detection is inexpensive and, when successful, perits the execution of our BMMC algorith or possily a faster algorith for a ore restricted perutation class. An iportant open prole is deterining exact, rather than asyptotic, lower ounds. It ay yet e possile to \hack the constants" in the upper ound. Without exact lower ounds, however, we do not know when to stop hacking. Finally, we ask what other perutations can e perfored quickly? [Cor92] presents several O(1)-pass perutation classes, and this paper has added one ore (MLD perutations in Appendix A). What other useful perutation classes can we show to e BMMC? Can we generalize BMMC perutations further to useful classes that can also e perfored faster than the general perutation ound? Acknowledgents Thanks to To Sundquist, C. Esther Jeseru, and Michael Klugeran for their helpful suggestions. References [AV88] [ACS87] Alok Aggarwal, Ashok K. Chandra, and Marc Snir. Hierarchical eory with lock transfer. In Proceedings of the 28th Annual Syposiu on Foundations of Coputer Science, pages 204{216, Octoer Alok Aggarwal and Jerey Scott Vitter. The input/output coplexity ofsorting and related proles. Counications of the ACM, 31(9):1116{ 1127, Septeer [Cor92] Thoas H. Coren. Virtual Meory for Data- Parallel Coputing. PhD thesis, Departent of Electrical Engineering and Coputer Science, Massachusetts Institute oftechnology, Availale as Technical Report MIT/LCS/TR-559. [Cor93] Thoas H. Coren. Fast peruting in disk arrays. Journal of Parallel and Distriuted Coputing, 17(1{2):41{57, January and Feruary [EHJ92] Alan Edelan, Steve Heller, and S. Lennart Johnsson. Index transforation algoriths in a linear algera fraework. Technical Report LBL-31841, Lawrence Berkeley Laoratory, [Flo72] [JH91] [V90] [V91] [V92] [VS90] [VS92] Roert W. Floyd. Peruting inforation in idealized two-level storage. In Rayond E. Miller and Jaes W. Thatcher, editors, Coplexity ofcoputer Coputations, pages 105{109. Plenu Press, S. Lennart Johnsson and Ching-Tien Ho. Generalized shue perutations on oolean cues. Technical Report TR-04-91, Harvard University Center for Research in Coputing Technology, Feruary Mark H. odine and Jerey Scott Vitter. Greed sort: An optial external sorting algorith for ultiple disks. Technical Report CS-90-04, Departent of Coputer Science, Brown University, Feruary Mark H. odine and Jerey Scott Vitter. Largescale sorting in parallel eories. In Proceedings of the 3rd Annual ACM Syposiu on Parallel Algoriths and Architectures, pages 29{39, July Mark H. odine and Jerey Scott Vitter. Optial deterinistic sorting on parallel disks. Technical Report CS-92-08, Departent of Coputer Science, Brown University, Jerey Scott Vitter and Elizaeth A. M. Shriver. Optial disk I/O with parallel lock transfer. In Proceedings of the Twenty Second Annual ACM Syposiu on Theory of Coputing, pages 159{ 169, May Jerey Scott Vitter and Elizaeth A. M. Shriver. Algoriths for parallel eory I: Two-level eories. Technical Report CS-92-04, Departent of Coputer Science, Brown University, August Revised version of Technical Report CS

10 A Perforing perutations characterized y Q (i) in two passes This appendix sketches how to perfor each perutation characterized y a atrix Q (i), as dened in equation (12), in only two passes. A coplete description will appear in the full paper. Our ethod is to factor each atrix Q (i) into two atrices, each of which characterizes a one-pass perutation. One of these atrices in fact characterizes an MRC perutation. The other characterizes a new type of one-pass perutation, which we call a eoryload-dispersal, or MLD, perutation. For notational convenience, we assue fro here on that we are working withaatrixq (i) for a given value of i, and we drop the superscripts. Thus, we shall speak of Q, C, and with the iplicit understanding that we really ean agiven Q (i), C (i),and (i). Before presenting our factoring of a atrix Q, we need to dene soe further index sets for suatrix rows and coluns and soe special atrices. Each atrix Q has an associated index set C, where jcj ;, indexing a colun asis for the lower left (n ; ) suatrix. We further dene the following: r = jcj, so that rank() =r ;. C = f0 1:::; 1g ;C. R is a set of row indices in such that the rr suatrix RC is nonsingular. (In other words, we choose a row asis RC for the colun asis C.) I C is an r atrix fored y expanding an r r identity atrix out to coluns with the coluns in C eing 0 and the coluns in C eing coluns of the identity atrix. For exaple, if C = f0 2 3g (so that r =3)and = 5, then " # I C = : (We oit the proof that this factorization is correct.) Because Q is nonsingular, so are and Y. Oserve that the leading suatrix of ust e nonsingular, for otherwise there are linearly dependent coluns aong the leftost coluns of, and would then e singular. Therefore, characterizes an MRC perutation, as we claied aove would e the case. The atrix Y, however, characterizes a new type of perutation that we call eoryload-dispersal or MLD. We shall not delve into the details of perforing MLD perutations here, ut we sketch how to perfor one. Using Lea 6, one can show that ecause rank() =r ;, each source eoryload aps to exactly 2 r M=B target eoryloads. In addition, one can use Lea 7 to show that given a target eoryload, if a particular source eoryload aps any records to this target, then it aps exactly 2 ;r B such records. We can partition the source and target eoryload nuers into sets of 2 r source eoryloads and 2 r target eoryloads such thateach source set has a unique target set that all of its records ap to. We can perfor the MLD perutation y reading in a source eoryload at a tie, peruting it in eory according to the characteristic atrix Y,andwriting it out so that the records apping to each lock iage in RAM are destined for the sae target eoryload. Moreover, the structure of Y ensures that the locks are spread out evenly across the disks in the target eoryloads. We read the source records a eoryload at a tie, we write the target eoryloads a lock at a tie, and every lock ofevery target eoryload is written exactly once y the tie we are nished processing all the source eoryloads. This procedure takes one pass. After perforing the MLD perutation characterized y Y,every target eoryload has the records destined for it according to the atrix Q themrc perutation characterized y then puts each record into its correct location within its eoryload. Siilary, I C isa(;r) atrix fored y expanding a(; r) (; r) identity atrix out to coluns with the coluns in C eing 0 and the coluns in C eing coluns of the identity atrix. We factor the nonsingular atrix as Q = Y,where = Y = (I C) T 0 0 I I ; r ; r n ; ; n; I C I 0 ( RC ) ;1 R0::; I and the suatrix 0 of is given y =(I C) T ( RC) ;1 RC (I C ) T : 3 5 ; n ; ; r ; r n ;