Engineering a Lightweight Suffix Array Construction Algorithm (Extended Abstract)

Transcription

1 Engineering Lightweight Suffix Arry Constrution Algorithm (Extended Astrt) Giovnni Mnzini 1,2 nd Polo Ferrgin 3 1 Diprtimento di Informti, Università del Piemonte Orientle I Alessndri, Itly mnzini@mfn.unipmn.it 2 Istituto di Informti e Telemti, CNR I Pis, Itly 3 Diprtimento di Informti, Università dipis I Pis, Itly ferrgin@di.unipi.it 1 Introdution We onsider the prolem of omputing the suffix rry of text T [1,n]. This prolem onsists in sorting the suffixes of T in lexiogrphi order. The suffix rry [16] (orpt rry [9]) is simple, esy to ode, nd elegnt dt struture used for severl fundmentl string mthing prolems involving oth linguisti texts nd iologil dt [4, 11]. Reently, the interest in this dt struture hs een revitlized y its use s uilding lok for three novel pplitions: (1) the Burrows-Wheeler ompression lgorithm [3], whih is provly [17] nd prtilly [20] effetive ompression tool; (2) the onstrution of suint [10, 19] nd ompressed [7, 8] indexes; the ltter n store oth the input text nd its full-text index using roughly the sme spe used y trditionl ompressors for the text lone; nd (3) lgorithms for lustering nd rnking the nswers to user queries in we-serh engines [22]. In ll these pplitions the onstrution of the suffix rry is the omputtionl ottlenek oth in time nd spe. This motivted our interest in designing yet nother suffix rry onstrution lgorithm whih is fst nd lightweight in the sense tht it uses smll spe. The suffix rry onsists of n integers in the rnge [1,n]. This mens tht in theory it uses Θ(n log n) its of storge. However, in most pplitions the size of the text is smller thn 2 32 nd it is ustomry to store eh integer in four yte word; this yields totl spe oupny of 4n ytes. For wht onerns the ost of onstruting the suffix rry, the theoretilly est lgorithms run in Θ(n) time[5]. These lgorithms work y first uilding the suffix tree nd then otining the sorted suffixes vi n in-order trversl of the tree. However, suffix tree onstrution lgorithms re oth omplex nd spe onsuming sine they oupy t lest 15n ytes of working spe (or even more, depending on the Prtilly supported y Itlin MIUR projet on Tehnologies nd servies for enhned ontent delivery. R. Möhring nd R. Rmn (Eds.): ESA 2002, LNCS 2461, pp , Springer-Verlg Berlin Heidelerg 2002

2 Engineering Lightweight Suffix Arry Constrution Algorithm 699 text struture [14]). This mkes their use imprtil even for modertely lrge texts. For this reson, suffix rrys re usully uilt using lgorithms whih run in O(n log n) time ut hve smller spe oupny. Among these lgorithms the urrent leder is the qsufsort lgorithm y Lrsson nd Sdkne [15]. qsufsort uses 8n ytes 1 nd despite the O(n log n) worst se ound it is fster thn the lgorithms sed on suffix tree onstrution. Unfortuntely, the size of our douments hs grown muh more quikly thn the min memory of our omputers. Thus, it is desirle to uild suffix rry using s smll spe s possile. Reently, Itoh nd Tnk [12] nd Sewrd [21] hve proposed two new lgorithms whih only use 5n ytes. From the theoretil point of view these lgorithms hve Θ ( n 2 log n ) worst se omplexity. In prtie they re fster thn qsufsort when the verge lp is smll (the lp is the length of the longest ommon prefix etween two onseutive suffixes in the suffix rry). However, for texts with lrge verge lp these lgorithms n e slower thn qsufsort y ftor 100 or more. 2 In this pper we desrie nd extensively test new lightweight suffix sorting lgorithm. Our min ide is to use very smll mount of extr memory, in ddition to 5n ytes, to void the degrdtion in performne when the verge lp is lrge. To hieve this gol we mke use of engineered lgorithms nd d ho dt strutures. Our lgorithm uses 5n + n ytes, where is user tunle prmeter (in our tests ws t most 0.03). For files with verge lp smller thn 100 our lgorithm is fster thn Sewrd s lgorithm nd roughly two times fster thn qsufsort. The est lgorithm in our tests uses 5.03n ytes nd is fster thn qsufsort for ll files exept for the one with the lrgest verge lp. 2 Definitions nd Previous Results Let T [1,n] denote text over the lphet Σ. The suffix rry [16] (orpt rry [9]) for T is n rry SA[1,n] suh tht T [SA[1],n], T [SA[2],n], et. is the list of suffixes of T sorted in lexiogrphi order. For exmple, for T = then SA =[2, 1, 3, 5, 4] sine T [2, 5] = is the suffix with lower lexiogrphi rnk, followed y T [1, 5] =, followed y T [3, 5] = nd so on. 3 Given two strings v, w we write lp(v, w) to denote the length of their longest ommon prefix. The verge lp of text T is defined s the verge length of the longest ommon prefix etween two onseutive suffixes. The verge lp is rough mesure of the diffiulty of sorting the suffixes: if the verge lp is lrge we need in priniple to exmine mny hrters in order to estlish the reltive order of two suffixes. 1 Here nd in the following the spe oupny figures inlude the spe for the input text, for the suffix rry, nd for ny uxiliry dt struture used y the lgorithm. 2 This figure refers to Sewrd lgorithm [21]. We re in the proess of quiring the ode of the Itoh-Tnk lgorithm nd we hope we will e le to test it in the finl version of the pper. 3 Note tht to define the lexiogrphi order of the suffixes it is ustomry to ppend t the end of T speil end-of-text symol whih is smller thn ny symol in Σ.

3 700 Giovnni Mnzini nd Polo Ferrgin Tle 1. Files used in our experiments sorted in order of inresing verge lp Nme Ave. lp Mx. lp File size Desription ile ,047,392 The file ile of the Cnterury orpus e.oli ,815 4,638,690 The file E.oli of the Cnterury orpus wrld ,473,400 The file world192 of the Cnterury orpus sprot , ,617,186 Swiss prot dtse (sprot34.dt) rf , ,421,901 Contention of RFC text files howto ,720 39,422,105 Contention of Linux Howto text files reuters , ,711,151 Reuters news in XML formt linux , ,254,720 Linux kernel soure files (tr rhive) jdk ,334 69,728,899 html nd jv files from the JDK 1.3 do. hr22 1, ,999 34,553,758 Assemly of humn hromosome 22 g 8, ,970 86,630,400 g 3.0 soure files (tr rhive) Sine this is n lgorithmi engineering pper we mke the following ssumptions whih orrespond to the sitution most often fed in prtie. We ssume Σ 256 nd tht eh lphet symol is stored in one yte. Hene, the text T [1,n] tkes preisely n ytes. Furthermore, we ssume tht n 2 32 nd tht the strting position of eh suffix is stored in four yte word. Hene, the suffix rry SA[1,n] tkes preisely 4n ytes. In the following we use the term lightweight to denote suffix sorting lgorithm whih use 5n ytes plus some smll mount of extr memory (we re intentionlly giving n informl definition). Note tht 5n ytes re just enough to store the input text T nd the suffix rry SA. Although we do not lim tht 5n ytes re indeed required, we do not know of ny lgorithm using less spe. For testing the suffix rry onstrution lgorithms we use the olletion of filesshownintle1. These files ontin different kind of dt in different formts; they lso disply wide rnge of sizes nd of verge lp s. 2.1 The Lrsson-Sdkne qsufsort Algorithm The qsufsort lgorithm [15] is sed on the douling tehnique introdued in [13] nd first used for the onstrution of the suffix rry in [16]. Given two strings v, w nd t>0wewritev< t w if the length-t prefix of v is lexiogrphilly smller thn the length-t prefix of w. Similrly we define the symols t, = t nd so on. Let s 1,s 2 denote two suffixes nd ssume s 1 = t s 2 (tht is, T [s 1,n] nd T [s 2,n]hvelength-t ommon prefix). Let ŝ 1 = s 1 + t denote the suffix T [s 1 + t, n] nd similrly let ŝ 2 = s 2 + t. The fundmentl oservtion of the douling tehnique is tht s 1 2t s 2 ŝ 1 t ŝ 2. (1) In other words, we n derive the 2t order etween s 1 nd s 2 y looking t the rnk of ŝ 1 nd ŝ 2 in the < t order.

4 Engineering Lightweight Suffix Arry Constrution Algorithm 701 The lgorithm qsufsort works in rounds. At the eginning of the ith round the suffixes re lredy sorted ording to the 2 i ordering. In the ith round the lgorithm looks for groups of suffixes shring the first 2 i hrters nd sorts them ording to the 2 i ordering using Bentley-MIlroy ternry quiksort [1]. Beuse of (1) eh omprison in the quiksort lgorithm tkes O(1) time. After t most log n rounds ll the suffixes re sorted. Thnks to very lever dt orgniztion qsufsort only uses 8n ytes. Even more surprisingly, the whole lgorithm fits in two pges of len nd elegnt C ode. The experiments reported in [15] show tht qsufsort outperforms other suffix sorting lgorithm sed on either the douling tehnique or the suffix tree onstrution. The only lgorithm whih runs fster thn qsufsort, ut only for files with verge lp less thn 20, is the Bentley-Sedgewik multikey quiksort [2]. Multikey quiksort is diret omprison lgorithm sine it onsiders the suffixes s ordinry strings nd sorts them vi hrter-y-hrter omprison without tking dvntge of their speil struture. 2.2 The Itoh-Tnk two-stge Algorithm In [12] Itoh nd Tnk desrie suffix sorting lgorithm lled two-stge suffix sort (two-stge from now on). two-stge only uses the text T nd the suffix rry SA for totl spe oupny of 5n ytes. To desrie how it works, let us ssume Σ = {,,...,z}. Using ounting sort, two-stge initilly prtitions the suffixes into Σ ukets B,...,B z ording to their first hrter. Note tht uket is nothing more thn set of onseutive entries in the rry SA whih now is sorted ording to the 1 ordering. Within eh uket twostge distinguishes etween two types of suffixes: Type A suffixes in whih the seond hrter of the suffix is smller thn the first, nd Type B suffixes in whih the seond hrter is lrger thn or equl to the first suffix hrter. The ruil oservtion of lgorithm two-stge is tht when ll Type B suffixes re sorted we n derive the ordering of Type A suffixes. This is done with single pss over the rry SA; when we meet suffix T [i, n] welooktsuffix T [i 1,n]: if it is Type A suffix we move it to the first empty position of uket B T [i 1]. Type B suffixes re sorted using textook string sorting lgorithms: in their implementtion the uthors use MSD rdix sort [18] for sorting lrge groups of suffixes, Bentley-Sedgewik multikey quiksort for medium size groups, nd insertion sort for smll groups. Summing up, two-stge n e onsidered n dvned diret omprison lgorithm sine Type B suffixes re sorted y diret omprison wheres Type A suffixes re sorted y muh fster proedure whih tkes dvntge of the speil struture of the suffixes. In [12] the uthors ompre two-stge with three diret omprison lgorithms (quiksort, multikey quiksort, nd MSD rdix sort) nd with n erlier version of qsufsort. two-stge turns out to e roughly 4 times fster thn quiksort nd MSD rdix sort, nd 2 to 3 times fster thn multikey quiksort nd qsufsort. However, the files used for the experiments hve n verge lp of t

5 702 Giovnni Mnzini nd Polo Ferrgin most 31, nd we know tht the dvntge of douling lgorithms (like qsufsort) eome pprent for muh lrger verge lp s. 2.3 Sewrd opy Algorithm Independently of Itoh nd Tnk, in [21] Sewrd desries lightweight lgorithm, lled opy, whih is sed on onept similr to the Type A/Type B suffixes used y lgorithm two-stge. Using ounting sort, opy initilly sorts the rry SA ording to the 2 ordering. As efore we use the term uket to denote the ontiguous portion of SA ontining set of suffixes shring the sme first hrter. Similrly, we use the term smll uket to denote the ontiguous portion of SA ontining suffixes shring the first two hrters. Hene, there re Σ ukets eh one onsisting of Σ smll ukets. Note tht one or more (smll) ukets n e empty. opy sorts the ukets one t time strting with the one ontining the fewest suffixes, nd proeeding up to the lrgest one. Assume for simpliity tht Σ = {,,...,z}. To sort uket, let us sy uket B p, opy sorts the smll ukets p, p,..., pz. The ruil point of lgorithm opy is tht when uket B p is ompletely sorted, with simple pss over it opy sorts ll the smll ukets p, p,..., zp. These smll ukets re mrked s sorted nd therefore opy will skip them when their prent uket is sorted. As further improvement, Sewrd shows tht even the sorting of the smll uket pp n e voided sine its ordering n e derived from the ordering of the smll ukets p,..., po nd pq,..., pz. This trik is extremely effetive when working on files ontining long runs of identil hrters. Algorithm opy sorts the smll ukets using Bentley-MIlroy ternry quiksort. During this sorting the suffixes re onsidered tomi, tht is, eh omprison onsists of the omplete omprison of two suffixes. The stndrd trik of sorting the lrger side of the prtition lst nd eliminting til reursion ensures tht the mount of spe required y the reursion stk grows, in the worst se, logrithmilly with the size of the input text. In [21] Sewrd ompres tuned implementtion of opy with the qsufsort lgorithm on set of files with verge lp up to 400. In these tests opy outperforms qsufsort for ll files ut one. However, Sewrd reports tht opy is muh slower thn qsufsort when the verge lp exeeds thousnd, nd for this reson he suggests the use of qsufsort s fllk when the verge lp is lrge. 4 Sine the soure ode of oth qsufsort nd opy is ville 5,wehvetested oth lgorithms on our suite of test files whih hve n verge lp rnging from to (see Tle 1). The results of our experiments re reported in the top two rows of Tle 2 (for AMD Athlon proessor) nd Tle 3 (for Pentium III proessor). In ordne with Sewrd s results, opy is fster thn 4 In [21] Sewrd desries nother lgorithm, lled he,whihisfsterthnopy for files with lrger verge lp. However, lgorithm he uses 6n ytes. 5 Algorithm opy ws originlly oneived to split the input file into 1MB loks. We modified it to llow the omputtion of the suffix rry for the whole file.

6 Engineering Lightweight Suffix Arry Constrution Algorithm 703 qsufsort when the verge lp is smll, nd it is slower when the verge lp is lrge. The turning point ppers to e when the verge lp is in the rnge However, this is not the omplete story. For exmple for ll files the running time of qsufsort on the Pentium is smller thn the running time for the Athlon; this is not true for opy (see for exmple files jdk13 nd g). We onjeture tht differene in the he rhiteture nd ehvior ould explin this differene, nd we pln to investigte it in the full pper (see lso Setion 4). We n lso see tht the differene in performne etween the two lgorithms does not depend on the verge lp lone. The DNA file hr22 hs very lrge verge lp, nevertheless the two lgorithms hve similr running times. The file linux hs muh greter verge lp thn reuters nd roughly the sme size. Nevertheless, the differene in the running times etween qsufsort nd opy is smller for linux thn for reuters. The most strikingdt in Tles2 nd 3 re the running times for g:for this file lgorithm opy is times slower thn qsufsort. This is not eptle sine g is not pthologil file uilt to show the wekness of opy, onthe ontrry it is file downloded from very usy site nd we n expet tht there re other files like it on our omputers. 6 In the next setion we desrie new lgorithm whih uses severl tehniques for voiding suh tstrophi ehvior nd t the sme time retining the nie fetures of opy: the5n ytes spe oupny nd the good performne for files with moderte verge lp. 3 Our Contriution: Deep-Shllow Suffix Sorting Our strting point for the design of n effiient suffix rry onstrution lgorithm is Sewrd opy lgorithm. Within this lgorithm we reple the proedure used for sorting the smll ukets (i.e. the groups of suffixes hving the first two hrters in ommon). Insted of using Bentley-MIlroy ternry quiksort we use more sophistited tehnique. More preisely, we sort the smll ukets using Bentley-Sedgewik multikey quiksort nd we stop the reursion when we reh predefined depth L (tht is, when we hve to sort group of suffixes with length-l ommon prefix). At this point we swith to different string sorting lgorithm. This pproh hs severl dvntges: (1) it provides simple nd effiient men to detet the groups of suffixes with long ommon prefix; (2) euse of the limit L, the size of the reursion stk is ounded y predefined onstnt whih is independent of the size of the input text nd n e tuned y the user; (3) if the suffixes in the smll uket hve ommon prefixes whih never exeed L, ll the sorting is done y multikey quiksort whih is n extremely effiient string sorting lgorithm. We ll this pproh to suffix sorting deep-shllow sorting sine we mix n lgorithm for sorting suffixes with smll lp (shllow sorter) with n lgorithm (tully more thn one, s we shll see) for sorting suffixes with lrge lp (deep 6 As we hve lredy pointed out, lgorithm opy ws oneived to work on loks of dt of size t most 1MB. The reder should e wre tht we re using n lgorithm outside its intended domin!

7 704 Giovnni Mnzini nd Polo Ferrgin Tle 2. Running times (in seonds) for 1400 MHz AMD Athlon proessor, with 1GB min memory nd 256K L2 he. The operting system ws Dein GNU/Linux Dein 2.2. The ompiler ws g ver with options -O3 - fomit-frme-pointer. The tle reports (user + system) time verged over five runs. The running times do not inlude the time spent for reding the input files ile e.oli wrld sprot rf howto reuters linux jdk13 hr22 g qsufsort opy ds0 L = ds0 L = ds0 L = ds0 L = ds1 L = ds1 L = ds1 L = ds1 L = ds1 L = ds2 d = ds2 d = ds2 d = ds2 d = sorter). In the next setions we desrie severl deep sorting strtegies, i.e. lgorithms for sorting suffixes whih hve length-l ommon prefix. 3.1 Blind Sorting Let s 1,s 2,...,s m denote group of m suffixes with length-l ommon prefix tht we need to deep-sort. If m is smll (we will disuss lter wht this mens) we sort them using n lgorithm, lled lind sort, whih is sed on the lind trie dt struture introdued in [6, Set. 2.1] (see Fig. 1). Blind sorting simply onsists in inserting the strings s 1,...,s m one t time in n initilly empty lind trie; then we trverse the trie from left to right thus otining the strings sorted in lexiogrphi order. The insertion of string s i in the trie requires first phse in whih we sn s i nd simultneously trverse the trie until we reh lef l. Then we ompre s i with the string ssoited to lef l nd we determine the length of their ommon prefix. Finlly, we updte the trie dding the lef orresponding to s i (see [6] for detils nd for the merits of lind tries with respet to stndrd ompted tries). Oviously in the onstrution of the trie we ignore the first L hrters of eh suffix euse they re identil. Our implementtion of the lind sort lgorithm uses t most 36m ytes of memory. Therefore, we use it when the numer of suffixes to e sorted is less

8 Engineering Lightweight Suffix Arry Constrution Algorithm 705 Tle 3. Running times (in seonds) for 1000MHz Pentium III proessor, with 1GB min memory nd 256K L2 he. The operting system ws GNU/Linux Red Ht 7.1. The ompiler ws g ver with options -O3 -fomit-frmepointer. The tle reports (user + system) time verged over five runs. The running times do not inlude the time spent for reding the input files ile e.oli wrld sprot rf howto reuters linux jdk13 hr22 g qsufsort opy ds0 L = ds0 L = ds0 L = ds0 L = ds1 L = ds1 L = ds1 L = ds1 L = ds1 L = ds2 d = ds2 d = ds2 d = ds2 d = n 2000 thn B = If the text is 100MB long, this overhed is 1.8MB whih should e ompred with the 500MB required y the text nd the suffix rry. 7 If the numer of suffixes to e sorted is lrger thn B = n we sort them. Thus, the spe overhed of using lind sort is t most 9n 500 ytes. using Bentley-MIlroy ternry quiksort. However, with respet to lgorithm opy, we introdue the following two improvements: 1. As soon s we re working with group of suffixes smller thn B we stop the reursion nd we sort them using lind sort; 2. during eh prtitioning phse we ompute L S (resp. L L ) whih is the longest ommon prefix etween the pivot nd the strings whih re lexiogrphilly smller (resp. lrger) thn the pivot. When we sort the strings whih re smller (resp. lrger) thn the pivot we n skip the first L S (resp. L L ) hrters sine we know they onstitute ommon prefix. We hve lled the ove lgorithm ds0 nd its performnes re reported in Tles 2 nd 3 for severl vlues of the prmeter L (the depth t whih we 7 Although we elieve this is smll overhed, we point out tht the limit B = n 2000 ws hosen somewht ritrrily. Preliminry experimentl results show tht there is only mrginl degrdtion in performne when we tke B = n,orb = n In the future we pln to etter investigte the spe/time trdeoff introdued y this prmeter nd its impt on the he performne. 2000

9 706 Giovnni Mnzini nd Polo Ferrgin Compted Trie Blind Trie Fig. 1. A stndrd ompted trie (left) nd the orresponding lind trie (right) for the strings:,,,,,,. Eh internl node of the lind trie ontins n integer nd set of outgoing lelled rs. A node ontining the integer k represent set of strings whih hve length-k ommonprefixnddifferinthe(k +1)st hrter. The outgoing rs re lelled with the different hrters tht we find in position k +1. Note tht sine the outgoing rs re ordered lphetilly, y visiting the trie leves from left to right we get the strings in lexiogrphi order stop multikey quiksort nd we swith to the lind sort/quiksort lgorithms). We n see tht lgorithm ds0 is slower thn qsufsort only for the files jdk13 nd g. Ifweompreopy nd ds0 we notie tht our deep-shllow pproh hs redued the running time for g y ftor 10. This is ertinly good strt. As we shll see, we will e le to redue it gin y the sme ftor tking dvntge of the ft tht the strings we re sorting re ll suffixes of the sme text. 3.2 Indued Sorting One of the nie fetures of two-stge nd opy lgorithms is tht some of the suffixes re not sorted y diret omprison: insted their reltive order is derived in onstnt time from the ordering of other suffixes whih hve een lredy sorted. We use generliztion of this tehnique in the deep-sorting phse of our lgorithm. Assume we need to sort the suffixes s 1,...,s m whih hve length- L ommon prefix. We sn the first L hrters of s 1 looking t eh pir of onseutive hrters (e.g. T [s 1 ]T [s 1 +1], T [s 1 +1]T [s 1 +2], up to T [s 1 + L 2]T [s 1 + L 1]). As soon s we find pir of hrters, sy αβ, elonging to n lredy sorted smll uket αβ the ordering of s 1,...,s m n e derived from the ordering of αβ s follows.

10 Engineering Lightweight Suffix Arry Constrution Algorithm 707 Assume α = T [s 1 +t]ndβ = T [s 1 +t+1] for some t<l 1. Sine s 1,...,s m hve length-l ommon prefix, every s i ontins the pir αβ strting from position t. Hene αβ ontins m suffixes orresponding to s 1,...,s m (tht is, αβ ontins the suffixes strting t s 1 + t, s 2 + t,...,s m + t). Note tht these suffixes re not neessrily onseutive in αβ.sinethefirstt 1hrters of s 1,...,s m re identil, the ordering of s 1,...,s m n e derived from the ordering of the orresponding suffixes in αβ. Summing up, the ordering is done s follows: 1. We sort the suffixes s 1,...,s m ording to their strting position in the input text T [1,n]. This is done so tht in Step 3 we n use inry serh to nswer memership queries in the set s 1,...,s m. 2. Let ŝ denote the suffix strting t the text position T [s 1 + t]. We sn the smll uket αβ in order to find the position of ŝ within αβ. 3. We sn the suffixes preeding nd following ŝ in the smll uket αβ.for eh suffix s we hek whether the suffix strting t the position T [s t] is in the set s 1,...,s m ;ifsowemrkthesuffixs When m suffixes in αβ hve een mrked, we sn them from left to right. Sine αβ is sorted this gives us the orret ordering of s 1,...,s m. Oviously there is no gurntee tht in the length-l ommon prefix of s 1,..., s m there is pir of hrters elonging to n lredy sorted smll uket. In this se we simply resort to the quiksort/lind sort omintion. We ll this lgorithm ds1 nd its performnes re reported in Tles 2 nd 3 for severl vlues of L. Wenseethtds1 with L = 500 runs fster thn qsufsort for ll files exept g. In generl, ds1 ppers to e slightly slower thn ds0 for files with smll verge lp ut it is lerly fster for the files with lrge verge lp: forg it is 8-9 times fster. 3.3 Anhor Sorting Profiling shows tht the most ostly opertion of indued sorting is the snning of the smll uket αβ in serh of the position of suffix ŝ (Step 2 ove). We now show tht we n void this opertion if we re willing to use smll mount of extr memory. For fixed d>0 we prtition the text T [1,n]into n/d segments of length d: T [1,d],T[d +1, 2d] nd so on up to T [n d +1,n](for simpliity let us ssume tht d divides n). We define two rrys Anhor[ ] nd Offset[ ] ofsizen/d suh tht, for i =1,...,n/d: Offset[i] ontins the position of leftmost suffix whih strts in the ith segment nd elongs to n lredy sorted smll uket. If in the ith segment does not strt ny suffix elonging to n lredy sorted smll uket then Offset[i] =0. Let ŝ i denote the suffix whose strting position is stored Offset[i]. Anhor[i] ontins the position of ŝ i within its smll uket. 8 The mrking is done setting the most signifint it of s. This mens tht we n work with texts of size t most The sme restrition holds for qsufsort s well.

11 708 Giovnni Mnzini nd Polo Ferrgin The use of the rrys Anhor[ ] ndoffset[ ] is firly simple. Assume tht we need to sort the suffixes s 1,...,s m whih hve length-l ommon prefix. For j =1,...,m,lett j denote the segment ontining the strting position of s j.if ŝ tj (tht is, the leftmost lredy sorted suffix in segment t j ) strts within the first L hrters of s j (tht is, s j < ŝ tj <s j + L) thenwensorts 1,...,s m using the indued sorting lgorithm desried in the previous setion. However, wenskipstep2sinethepositionofŝ tj within its smll uket is stored in Anhor[t j ]. Oviously, it is possile tht for some j ŝ tj does not exist or nnot e used. However, sine the suffixes s 1,...,s m usully elong to different segments, we hve m possile ndidtes. In our implementtion mong the ville sorted suffixes ŝ tj s we use the one whose strting position is losest to the orresponding s j (tht is, we hoose j whih minimizes ŝ tj s j ; this helps Step 3 of indued sorting). If there is no ville sorted suffix, then we resort to the lind sort/quiksort omintion. For wht onerns the spe oupny of nhor sorting, we note tht in Offset[i] we n store the distne etween the eginning of the ith segment nd the leftmost sorted suffix. Hene Offset[i] <d.ifwetked<2 16 the rry Offset requires 2n/d ytes of storge. Sine eh entry of Anhor requires four ytes, the overll spe oupny is 6n/d ytes. In our tests we used t lest d = 500 whih yields n overhed of 6n 9n 500 ytes. If we dd the 500 ytes required y lind sorting with B = n 3n 2000, we get mximum overhed of t most 100 ytes. Hene, for 100MB text the overhed is t most 3MB, whih we onsider smll mount ompred with the 500MB used y the text nd the suffix rry. In Tles 2 nd 3 we report the running times of nhor sorting (under the nme ds2) ford rnging from 500 to 5000 nd L = d We see tht for the files with moderte verge lp ds2 with d = 500 is signifintly fster thn opy nd roughly two times fster thn qsufsort. For the files with lrge verge lp ds2 is fster thn qsufsort for ll files exept g. Forg ds2 is 15% slower thn qsufsort on the Athlon nd 50% slower on the Pentium. In our opinion this slowdown on single file is n eptle prie to py in exhnge for the redution in spe oupny hieved over qsufsort (5.03n ytes vs. 8n ytes). We elieve tht the possiility of uilding suffix rrys for lrger files hs more vlue thn greter effiieny in hndling files with very lrge verge lp. 4 Conlusions nd Further Work In this pper we hve presented novel lgorithm for uilding the suffix rry of text T [1,n]. Our lgorithm uses 5.03n ytes nd is fster thn ny other tested lgorithm. Only on single file our lgorithm is outperformed y qsufsort whih however uses 8n ytes. For pthologil inputs, i.e. texts with n verge lp of Θ(n), ll lightweight lgorithms tke Θ ( n 2 log n ) time. Although this worst se ehvior does not our in prtie, it is n interesting theoretil open question whether we n hieve O(n log n) timeusingo(n) spe in ddition to the spe required y the input text nd the suffix rry.

12 Referenes Engineering Lightweight Suffix Arry Constrution Algorithm 709 [1] J. L. Bentley nd M. D. MIlroy. Engineering sort funtion. Softwre Prtie nd Experiene, 23(11): , [2] J. L. Bentley nd R. Sedgewik. Fst lgorithms for sorting nd serhing strings. In Proeedings of the 8th ACM-SIAM Symposium on Disrete Algorithms, pges , [3] M. Burrows nd D. Wheeler. A lok sorting lossless dt ompression lgorithm. Tehnil Report 124, Digitl Equipment Corportion, [4] M. Crohemore nd W. Rytter. Text Algorithms. Oxford University Press, [5] M. Frh-Colton, P. Ferrgin, nd S. Muthukrishnn. On the sorting-omplexity of suffix tree onstrution. Journl of the ACM, 47(6): , [6] P. Ferrgin nd R. Grossi. The string B-tree: A new dt struture for string serh in externl memory nd its pplitions. Journl of the ACM, 46(2): , [7] P. Ferrgin nd G. Mnzini. Opportunisti dt strutures with pplitions. In Pro. of the 41st IEEE Symposium on Foundtions of Computer Siene, pges , [8] P. Ferrgin nd G. Mnzini. An experimentl study of n opportunisti index. In Pro. 12th ACM-SIAM Symposium on Disrete Algorithms, pges , [9] G. H. Gonnet, R. A. Bez-Ytes, nd T. Snider. New indies for text: PAT trees nd PAT rrys. In B. Frkes nd R. A. Bez-Ytes nd, editors, Informtion Retrievl: Dt Strutures nd Algorithms, hpter 5, pges Prentie-Hll, , 699 [10] R. Grossi nd J. Vitter. Compressed suffix rrys nd suffix trees with pplitions to text indexing nd string mthing. In Pro. of the 32nd ACM Symposium on Theory of Computing, pges , [11] D. Gusfield. Algorithms on Strings, Trees, nd Sequenes: Computer Siene nd Computtionl Biology. Cmridge University Press, [12] H. Itoh nd H. Tnk. An effiient method for in memory onstrution of suffix rrys. In Proeedings of the sixth Symposium on String Proessing nd Informtion Retrievl, SPIRE 99, pges IEEE Computer Soiety Press, , 701 [13] R. Krp, R. Miller, nd A. Rosenerg. Rpid Identifition of Repeted Ptterns in Strings, Arrys nd Trees. In Proeedings of the ACM Symposium on Theory of Computtion, pges , [14] S. Kurtz. Reduing the spe requirement of suffix trees. Softwre Prtie nd Experiene, 29(13): , [15] N. J. Lrsson nd K. Sdkne. Fster suffix sorting. Tehnil Report LU- CS-TR:99-214, LUNDFD6/(NFCS-3140)/1-43/(1999), Deprtment of Computer Siene, Lund University, Sweden, , 700, 701 [16] U. Mner nd G. Myers. Suffix rrys: new method for on-line string serhes. SIAM Journl on Computing, 22(5): , , 699, 700 [17] G. Mnzini. An nlysis of the Burrows-Wheeler trnsform. Journl of the ACM, 48(3): , [18] P. M. MIlroy nd K. Bosti. Engineering rdix sort. Computing Systems, 6(1):5 27,

13 710 Giovnni Mnzini nd Polo Ferrgin [19] K. Sdkne. Compressed text dtses with effiient query lgorithms sed on the ompressed suffix rry. In Proeeding of the 11th Interntionl Symposium on Algorithms nd Computtion, pges Springer-Verlg, LNCS n. 1969, [20] J. Sewrd. The zip2 home pge, zip2/index.html. 698 [21] J. Sewrd. On the performne of BWT sorting lgorithms. In DCC: Dt Compression Conferene, pges IEEE Computer Soiety TCC, , 702 [22] O. Zmir nd O. Etzioni. Grouper: A dynmi lustering interfe to we serh results. Computer Networks, 31(11-16): ,