Validating XML Documents in the Streaming Model with External Memory

Transcription

1 Vlidting XML Douments in the Steming Model with Extenl Memoy Chistin Kond LIAFA, Univ. Pis Dideot; Pis, Fne; nd Univ. Pis-Sud; Osy, Fne. Fédéi Mgniez LIAFA, Univ. Pis Dideot, CNRS; Pis, Fne. ABSTRACT We study the poblem of vlidting XML douments of size N ginst genel DTDs in the ontext of steming lgoithms. The stting point of this wok is well-known spe lowe bound. Thee e XML douments nd DTDs fo whih p-pss steming lgoithms equie Ω(N/p) spe. We show tht when llowing ess to extenl memoy, thee is deteministi steming lgoithm tht solves this poblem with memoy spe O(log 2 N), onstnt numbe of uxiliy ed/wite stems, nd O(log N) totl numbe of psses on the XML doument nd uxiliy stems. An impotnt intemedite step of this lgoithm is the omputtion of the Fist-Child-Next-Sibling (FCNS) enoding of the initil XML doument in steming fshion. We study this poblem independently, nd we lso povide memoy effiient steming lgoithms fo deoding n XML doument given in its FCNS enoding. Futhemoe, vlidting XML douments enoding biny tees in the usul steming model without extenl memoy n be done with subline memoy. Thee is one-pss lgoithm using O( N log N) spe, nd bidietionl twopss lgoithm using O(log 2 N) spe pefoming this tsk. Ctegoies nd Subjet Desiptos F.2.2 [Theoy of Computtion]: Anlysis of lgoithms nd poblem omplexity Nonnumeil Algoithms nd Poblems Genel Tems Algoithms, Theoy 1. INTRODUCTION Suppoted by the Fenh ANR SeSu nd Defis pogms unde ontts ANR-07-SESU-013 (VERAP pojet) nd ANR-08-EMER-012 (QRAC pojet). Chistin Kond is suppoted by Fondtion CFM-JP Aguil gnt. Pemission to mke digitl o hd opies of ll o pt of this wok fo pesonl o lssoom use is gnted without fee povided tht opies e not mde o distibuted fo pofit o ommeil dvntge nd tht opies be this notie nd the full ittion on the fist pge. To opy othewise, to epublish, to post on seves o to edistibute to lists, equies pio speifi pemission nd/o fee. ICDT 2012, Mh 26 30, 2012, Belin, Gemny. Copyight 2012 ACM /12/03...$10.00 The e of steming lgoithms hs expeiened temendous gowth ove the lst dede in mny pplitions. Steming lgoithms sequentilly sn the whole input piee by piee in one pss, o in smll numbe of psses (i.e., they do not hve ndom ess to the input), while using subline memoy spe, idelly polylogithmi in the size of the input. The design of steming lgoithms is motivted by the explosion in the size of the dt tht lgoithms e lled upon to poess in eveydy el-time pplitions. Exmples of suh pplitions ou in bioinfomtis fo genome deoding, in Web dtbses fo the seh of douments, o in netwok monitoing. The nlysis of Intenet tffi [1], in whih tffi logs e queied, ws one of the fist pplitions of this kind of lgoithm. Thee e vious extensions of this bsi steming model. One of them gives the steming lgoithm ess to n extenl memoy onsisting of sevel ed/wite stems [8, 10, 6]. Then the steming lgoithm is lso elxed to pefom multiple psses in ny dietion ove the input stem nd the uxiliy stems. In most of the pplitions, the numbe of uxiliy stems is onstnt nd the totl numbe of psses is logithmi in the input size. Veifying popeties o evluting queies of mssive dtbses is n tive nd hllenging topi. Fo eltionl lgeb queies ginst eltionl dtbses, the sitution is quite le. Thee e bidietionl O(log N)-pss deteministi steming lgoithms with onstnt memoy spe nd onstnt numbe of uxiliy stems [7]. Moeove, the logithmi numbe of psses is neessy ondition in ode to keep the memoy spe subline, even if ndomiztion is llowed. The ltte ws initilly stted fo one-sided eo [7] nd then extended to two-sided eo [4, 3]. In the ontext of dt exhnge, espeilly on the Web, Extended Mkup Lnguge (XML) is emeging s the stndd, nd is uently dwing muh ttention in dt mngement eseh. Only few is known on XML quey poessing when only steming ess is llowed to the XML doument. Fo evluting XQuey nd XPth queies ginst XML douments of size N, only the lowe bound hs been extended [7, 4, 3], mening tht Ω(log N) psses e neessy. Fo the uppe bound, only simple efinements of the diet lgoithm e known: no uxiliy stem, one pss nd line memoy in the height of the XML doument, whih in the wost se is s lge s N. This ppe onsides the poblem of vlidting XML douments ginst given Doument Type Definition (DTD) in steming fshion without estitions on the DTD. Pio

2 woks on tht topi [16, 15] essentilly ty to hteize those DTDs fo whih vlidity n be heked by finitestte utomton, tht is one-pss deteministi steming lgoithm with onstnt memoy. Conening bity DTDs, two ppohes hve been onsideed in [16]. The fist one leds to n lgoithm with memoy spe line in the height of the XML doument [16]. The seond one onsists in onstuting efined DTD of t most qudti size, whih defines simil fmily of tee douments s the oiginl one, nd ginst whih vlidtion n be done with onstnt spe. Nonetheless, fo n existing doument nd DTD, the lte equies tht both, douments nd DTD, e onveted befoe vlidtion. One of the obstles pio woks hd to ope with ws to hek well-fomedness of XML douments, tht is evey opening tg mthes its sme-level losing tg. Due to the pst wok of [12], we n now pefom suh veifition with onstnt-pss ndomized steming lgoithm with subline memoy spe nd no uxiliy stems. In onepss the memoy spe is O( N log N), nd ollpses to O(log 2 N) with n dditionl pss in evese dietion. The stting point of this wok is the ft tht heking vlidity is hd without uxiliy stems. Thee e DTDs defining teny XML douments fo whih ny p-pss bidietionl ndomized steming lgoithm equies Ω(N/p) spe. This lowe bound omes fom enoding well known ommunition omplexity poblem, Set-Disjointness, s n XML vlidity poblem. This lowe bound should be wellknown, howeve we e not we of omplete poof in the litetue. In [9], simil ppoh using teny tees with edution to Set-Disjointness is used fo poving lowe bounds fo queies. Fo the ske of ompleteness we povide poof in Appendix A. An XML doument is vlid ginst DTD if fo eh node, the sequene of the lbels of thei hilden fulfills egul expession defined in the DTD. Fo the se of XML douments enoding biny tees, we pesent in Setion 3 two deteministi steming lgoithms fo heking vlidity with subline spe. As onsequene, the pesene of nodes of degee t lest 3 is indeed neessy ondition fo the line spe lowe bound fo genel douments. We fist show how to design one-pss lgoithm with spe O( N log N) (Theoem 1). We onjetue tht thee is Ω(N α ) lowe bound fo one-pss lgoithms fo some 1 3 α 1. With seond pss in evese dietion the memoy 2 ollpses to O(log 2 N) (Theoem 2). These two lgoithms mke use of the simple, but fundmentl ft tht in one pss ove n XML doument eh node is seen twie in fom of its opening nd losing tg. Hene, it is not neessy to emembe ll opening tgs in the stem sine thee is seond hne to get the sme infomtion by thei losing tgs. Ou lgoithms exploit this obsevtion. Then, in Setion 4 we pesent ou min esult. Coolly 1 sttes tht vlidity of ny XML doument ginst ny DTD n be heked in the steming model with extenl memoy with poly-logithmi spe, onstnt numbe of uxiliy stems, nd O(log N) psses ove these stems. Vlidity of node depends on its hilden, hene is uil to hve esy ess to the sequene of hilden of ny node. The fundmentl ide to estblish this is, fistly, to ompute the Fist-Child-Next-Sibling (FCNS) enoding, n enoding s 2-nked tee of the XML doument. In this enoding, the sequene of losing tgs of the hilden of node e onseutive. The omputtion of this enoding is the hd pt of the vlidtion poess, nd the esoue equiements of ou vlidtion lgoithm stem fom this opetion (Theoem 3). Sine the FCNS enoding n be seen s eodeing of the tgs of the oiginl doument, ou sttegy is to see this poblem s soting poblem with ptiul ompison funtion. Mege sot n be implemented s steming lgoithm, nd we mke use of it ustomized by n dpted mege funtion. The sme ide n be used fo deoding with simil omplexity (Theoem 7). Then, bsed on the FCNS enoding, veifition n be done eithe in one pss nd O( N log N) spe (Theoem 4), o in two bidietionl psses nd O(log 2 N) spe (Theoem 5). Conening FCNS enoding nd deoding, we show line spe lowe bound fo one-pss lgoithms. Fo deoding, we pesent O( N log N) lgoithm (Theoem 6) tht pefoms one pss ove the input, but two psses ove the output. We onjetue fo enoding tht memoy spe emins Ω(N) fte ny onstnt numbe of psses, whih would show tht deoding is esie thn enoding. This suggests systemti use of the FCNS enoding fo lge douments sine vlidity n be heked esily without uxiliy stems nd in subline spe. Fo use intetions, the oiginl doument n be obtined by the subline spe 3-pss lgoithm. The pplibility of this ide is left s n open question. 2. PRELIMINARIES Fom now Σ is finite lphbet. The k-th lette of X Σ N is denoted by X[k], fo 1 k N, nd the onseutive lettes of X between positions i nd j by X[i, j]. A subsequene of X is ny sting X[i 1]X[i 2] X[i k ], whee 1 i 1 < i 2 <... < i k N. 2.1 Steming model In steming lgoithms, pss ove input X Σ N mens tht X is given s input stem X[1], X[2],, X[N], whih ives sequentilly, i.e., lette by lette in this ode. Steming lgoithms hve ess to ndom ess memoy spe, nd, s the se my be, to ed-wite extenl memoy s in [7, 5]. See lso the eview in [8]. We ssume tht ny lette of Σ fits into one ell of intenl/extenl memoy. The extenl memoy is olletion of uxiliy stems, tht we see s ed/wite stems with, gin, sequentil ess. When needed, we ugment the lphbet of uxiliy stems fom Σ by k-tuples of elements in Σ [0, 2N], fo some fixed onstnt k, whih theefoe fit in one ell of uxiliy stems. At the beginning of eh pss on ed/wite stem, the lgoithm deides whethe it pefoms ed o wite pss. The input stem is ed-only. On witing pss, the lgoithm n eithe wite lette, nd then move to the next ell, o move dietly to the next ell. Fo the se of bidietionl steming lgoithms, opposed to unidietionl steming lgoithm whee eh pss is in the sme ode, the lgoithm n deide the dietion of the sequentil ess. Fo simpliity, we ssume thoughout this tile tht the length of the input is known in dvne by the lgoithm. Nonetheless, ll ou lgoithms n be dpted to the se in whih the length is unknown until the end of pss. See [13] fo n intodution to steming lgoithms.

3 Definition 1 (Steming lgoithm). A p(n)- pss steming lgoithm A with s(n) spe, k(n) uxiliy stems, t(n) poessing time pe lette is n lgoithm suh tht fo evey input stem X Σ N : 1. A hs ess to k(n) uxiliy ed/wite stems, 2. A pefoms in totl t most p(n) psses on X nd uxiliy stems, 3. A mintins memoy spe of size s(n) lettes of Σ nd bits while eding X nd uxiliy stems, 4. A does not exeed unning time of t(n) between two wite o ed opetions. We sy tht A is bidietionl if it pefoms t lest one pss in eh dietion. Othewise A is impliitly unidietionl. We do not mention the numbe of uxiliy stems when thee e none (k(n) = 0). Futhemoe, we ssume tht opetions on numbes N [0, 2N] n be done in onstnt time. 2.2 XML douments We onside finite unnked odeed lbeled tees t, whee eh tee node is lbeled by some lbel in Σ, nd its oot hs distinguished lbel. Moeove, the hilden of evey non-lef node e odeed. Fom now, we omit the tems odeed nd lbeled. Then k-nked tees e speil se whee eh node hs t most k hilden. Biny tees e speil type of 2-nked tees, whee eh node is eithe lef o hs extly 2 hilden. Fo eh lbel Σ, we ssoite its oesponding opening tg nd losing tg, stnding fo nd / in the usul XML nottions. An XML sequene is sequene ove the lphbet Σ = {, : Σ}. The XML sequene of tee t is the sequene of opening tgs nd losing tgs in the ode of depth fist tvesl of t (Figue 1): when t step i we visit node with lbel top-down (espetively bottom-up), we let X[i] = (espetively X[i] = ). Hene X is wod ove Σ = {, : Σ} of size twie the numbe of nodes of t. The XML file desibing t is unique, nd we denote it s XML(t). b b b Figue 1: Let Σ = {, b, }, nd let t be the tee s bove. Then XML(t) = bbbbbb. We ssume tht the input XML sequenes X e wellfomed, nmely X = XML(t), fo some tee t. The pst wok of [12] legitimtes this ssumption sine heking wellfomedness is t lest s esy s ny of ou lgoithms fo heking vlidity. Hene we ould un n lgoithm fo wellfomedness in pllel without inesing the esoue equiements. Note, tht ndomness is neessy fo heking well-fomedness with subline spe, whees we will show tht ndomness is useless fo vlidtion. Let us intodue moe useful nottions. Sine the length of well-fomed XML sequene is known in dvne, we will denote it by 2N insted of N. Eh opening tg X[i] nd mthing losing tg X[j] in X = XML(t) oesponds to unique tee node v of t. We sometimes denote v eithe by X[i] o X[j]. We lso wite (mbiguously) v fo its oesponding opening tg, nd v fo its oesponding losing tg. Then, the position of v in X is pos(v) = i. Similly, pos(v) = j. We onside XML vlidity ginst some DTD. A DTD is mpping D fom Σ to egul expessions ove Σ. Let t be tee. Then node v t with lbel v nd hilden v 1, v 2,, v k with espetive lbels v 1, v 2,, v k is vlid ginst D if v 1, v 2,, v k stisfies the egul expession D(v). In ptiul, v n be lef if nd only if the empty wod ε stisfies the egul expession D(v). Then t is vlid ginst D if ll its nodes e vlid ginst D. Thoughout the doument we ssume tht DTDs e onsidebly smll nd ou lgoithms hve full ess to them without ounting this to thei spe equiements. Definition 2 (Vlidity). Let D be some DTD. The poblem Vlidity onsists of deiding whethe n input tee t given by its XML sequene XML(t) on n input stem is vlid ginst D. We denote by Vlidity(2) the poblem Vlidity estited to input XML sequenes desibing biny tees. 3. VALIDITY OF BINARY TREES Fo simpliity, we only onside biny tees in this setion. A left opening/losing tg (espetively ight opening/losing tg) of n XML sequene X is tg whose oesponding node is the fist hild of its pent (espetively seond hild). Ou lgoithms fo biny tees n be extended to 2- nked tees. This equies few hnges in the one-pss Algoithm 1 nd the two-pss Algoithm 2 (indeed in the suboutine Algoithm 3), tht we do not desibe hee. We fix now DTD D, nd ssume tht, in ou lgoithms, we hve ess to poedue hek(, b, ) tht signlizes invlidity nd bots if b is not vlid ginst the egul expession D(). Othewise it etuns without ny tion. 3.1 One-pss lgoithm In ode to vlidte n XML doument, we ensue vlidity of ll tee nodes. Fo heking vlidity of node v with two hilden, we hve to elte 3 lbels, tht is the lbel v of the node itself, nd the lbels of the two hilden nodes v 1, v 2. In top-down veifition we use the opening tg v of the pent node v fo veifition, in bottom-up veifition we use the losing tg v of the pent node v. Algoithm 1 mkes use of the ft tht thee e these two hnes to veify node. It uses stk onto whih it pushes ll opening tgs in ode to pefom top-down veifitions one the infomtion of the hilden nodes ives on the stem. v 1v 2 foms substing of the input, hene top-down veifition equies only the stoge of the opening tg v sine the lbels of the hilden ive in blok. The lgoithm s spe equiements depend on pmete K (we optimize by setting K = N log N). One the numbe of opening tgs on the stk is bout to exeed K, we emove the bottom-most opening tg. The oesponding node will then be veified bottom-up. Note tht v 2v foms substing of the input. Hene fo bottom-up veifitions it is enough to stoe the lbel of the left hild v 1 on the stk sine the lbel of the ight hild ives in fom of losing tg ight befoe the losing tg of the pent node. See Algoithm 1 fo detils. Fo the unique identifition of losing tgs on the stk, we hve to stoe them with thei depth in the tee. A stk

4 item oesponding to losing tg equies hene O(log N) spe. Opening tgs don t equie the stoge of thei depth (we stoe depth of 1 whih we ssume to equie only onstnt spe). Algoithm 1 Vlidity of biny tees in 1-pss Requie: input stem is well-fomed XML doument 1: d 0, S empty stk 2: K N log N 3: while stem not empty do 4: x next tg on stem 5: if x is n opening tg then 6: if x is lef then hek(, ɛ, ɛ) end if 7: if S hs on top (, 1), (b, d) then 8: hek(, b, ); pop S {Top-down veifition} 9: end if 10: if {(, 1) S opening } K then 11: emove bottom-most (, 1) in S, opening 12: end if 13: d d : push (x, 1) 15: else if x is losing tg then 16: d d 1 17: if S hs on top (, d + 1), (b, d + 1) then 18: hek (,, b) {Bottom-up veifition} 19: pop S, pop S 20: else if S hs on top (b, d + 1) then pop S 21: end if 22: if S hs on top (, 1) then pop S end if 23: push (x, d) 24: end if 25: end while 4. A left losing tg is only emoved fom S upon veifition of its pent node. Lemm 1. Let S = (x 1, d 1), (x k, d k ) be the stk t the beginning of the while loop in line 3. Let ( i, d i), ( i+1, d i+1) be two onseutive left losing tgs in S suh tht ( i+1, d i+1) is not the topmost one. Then pos( i+1) pos( i) + 2K. Poof. Denote by X = X[1]X[2] X[2N] the input stem. Sine i+1 is not the topmost left losing tg in S, the lgoithm hs ledy poessed the ight sibling opening tg X[pos( i+1) + 1] of i+1. By Item 4 of Ft 1, no veifition hs been done of the pent of i+1, sine i+1 is still in S. Theefoe, the pent s opening tg X[k] of i+1 hs been deleted fom S, whee pos( i) < k < pos( i+1). This n only hppen if t lest K opening tgs hve been pushed on S between X[k] nd i+1. Sine these K opening tgs must hve been losed between X[k] nd i+1 we obtin pos( i+1) pos( i) + 2K. 1 2 k The quey in line 6 n be implemented by lookhed of 1 on the stem. The opening tg x oesponds to lef only if the subsequent tg in the stem is the oesponding losing tg x. Figue 2 visulizes the diffeent ses with thei stk modifitions ppeing in Algoithm 1. b 1 b 3 b l b 2 line 6 line 7 line 17 line 20 line 22 b b b X X Xb X Xb X Xb X X X Figue 2: Visuliztion of the diffeent onditions in Algoithm 1 with the pplied stk modifitions. X epesents the bottom pt of the stk. Note tht Algoithm 1 pushes the uently teted tg o on the stk in Line 14 o Line 23. o oesponds to the highlighted node. Ft 1 (not poved hee) nd Lemm 1 onen the stutue of the stk S used in Algoithm 1. Ft 1. Let S = (x 1, d 1), (x k, d k ) be the stk t the beginning of the while loop in line 3. Then: 1. pos(x 1) < pos(x 2) < pos(x k ), 2. depth(x 1) depth(x 2) depth(x k ) d. Moeove, if depth(x i) = depth(x i+1) then x i is the left sibling of x i+1, 3. The sequene x 1 x k stisfies the egul expession b (ɛ de), whee e left losing tgs, b e opening tgs, is losing tg, d is left losing tg, nd e is ight losing tg. Figue 3: Visuliztion of the stutue of the stk used in Algoithm 1. The stk fulfills the egul expession b (ɛ de), ompe Item 3 of Ft 1. The ( i ) i=1...k e losing tgs whose pents nodes wee not veified top-down. Fo j > i, j is onneted to i by the ight sibling of i. The (b i ) i=1...l fom sequene of opening tgs suh tht b i is the pent node of b i+1. On top of the stk might be one o two losing tgs depending on the uent stte of the veifition poess. Ft 1 nd Lemm 1 povide moe insight in the stk stutue nd e used in the poof of Theoem 1. Item 3 of Ft 1 sttes tht the stk bsilly onsists of sequene of left losing tgs whih e the left hilden tht e needed fo bottom-up veifitions of nodes tht ould not be veified top-down. This sequene is followed by sequene of opening tgs fo whih we still im top-down veifition. The poof of Lemm 1 explins the ft tht the two sequenes e stitly septed: left-losing tg v 1 only emins on the stk if t the moment of insetion thee e no opening tgs on the stk. Theoem 1. Algoithm 1 is one-pss steming lgoithm fo Vlidity(2) with spe O( N log N) nd O(1) poessing time pe lette. Poof. To pove oetness, we hve to ensue vlidity of ll nodes. Leves e oetly vlidted upon ivl of its opening tg in line 6. Conening non-lef nodes, fistly,

5 note tht ll losing tgs e pushed on S in line 23, in ptiul ll losing tgs of left hilden ppe on the stk. The lgoithm emoves left losing tgs only fte vlidtion of its pent node, no mtte whethe the veifition ws done top-down o bottom-up, ompe Item 4 of Ft 1. Emptiness of the stk fte the exeution of the lgoithm follows fom Item 2 of Ft 1 nd implies hene the vlidtion of ll non-lef nodes. Fo the spe bound, Line 10 guntees tht the numbe of opening tgs in S is lwys t most K. We bound the numbe of losing tgs on the stk by N +2. Item 3 of Ft 1 K sttes tht the stk ontins t most one ight losing tg. Fom Item 4 of Ft 1 we dedue tht S ompises t most N K +1 left losing tgs, sine the stem is of length 2N, nd the distne in the stem of two onseutive left losing tgs tht eside on S exept the top-most one is t lest 2K. A losing tg with depth (, d) Σ [N] equies O(log N) spe, n opening tg equies only onstnt spe. Hene the totl spe equiements e O(( N +2) log N +K) whih K is minimized fo K = N log N. Conening the poessing time pe lette, the lgoithm only pefoms onstnt numbe of lol stk opetions in one itetion of the while loop. Remk Algoithm 1 n be tuned into n lgoithm with spe omplexity O( D log D), whee D is the depth of the XML doument. If D is known befoehnd, it is enough to set K = D log D in line 2. If D is not known in dvne, we mke use of n uxiliy vible D stoing guess fo the doument depth. Initilly we set D = C, C > 0 some onstnt, we set K = D log D, nd we un Algoithm 1. Eh time d exeeds D, we double D, nd we updte K odingly. This guntees tht the numbe of opening tgs on the stk is limited by O( D log D). Sine we stted with too smll guess fo the doument depth, we my hve emoved opening tgs tht would hve emined on the stk if we hd hosen the depth oetly. This leds to futhe bottom-up veifitions, but no moe thn O( D/ log D) gunteeing O( D log D) spe. 3.2 Two-pss lgoithm The bidietionl two-pss Algoithm 2 uses suboutine tht heks in one-pss vlidity of ll nodes whose left subtee is t lest s lge s its ight subtee. Feeding into this suboutine the XML doument ed in evese dietion nd intepeting opening tgs s losing tgs nd vie ves, it heks vlidity of ll nodes whose ight subtee is t lest s lge s its left subtee. In this wy ll tee nodes get veified. The suboutine pefoms only heks in bottom-up fshion, tht is, the veifition of node v with hilden 1, 2 mkes use of the tgs 1 nd 2 (whih e djent in the XML doument nd hene esy to eognize) nd the losing tg of v. When 1, 2 ppes in the stem, 4-tuple onsisting of 1, 2, depth( 1) nd pos( 1) gets pushed on the stk. Upon ivl of v, depth( 1) is needed to identify 1, 2 s the hilden of v. pos( 1) is needed fo lening the stk: with the help of the pos vlues of the stk items, we identify stk items whose pents nodes hve lge ight subtees thn left subtees, nd these stk items get emoved fom the stk. In so doing, we guntee tht the stk size does not exeed log(n) elements whih is n exponentil impovement ove the one-pss lgoithm. Note tht the evese pss n be done independently of the fist one, nmely eventully in pllel. Algoithm 2 Two-pss lgoithm vlidting biny tees un Algoithm 3 eding the stem fom left to ight un Algoithm 3 eding the stem fom ight to left, whee opening tgs e intepeted s losing tgs, nd vie ves. Algoithm 3 Vlidting nodes with size(left subtee) size(ight subtee) 1: l 0; n 0; S empty stk 2: while stem not empty do 3: x next tg on stem (nd move stem to next tg) 4: y next tg on stem, without onsuming it yet 5: n n + 1 6: if x is n opening tg then 7: l l + 1 8: if y = then hek(, ɛ, ɛ) end if 9: else {x is losing tg } 10: l l 1 11: if S hs on top (,, l + 1, ) then 12: (, b,, ) pop fom S; hek(,, b) 13: end if 14: if y is n opening tg d then 15: push (, d, l, n) to S 16: end if 17: end if 18: while thee is s 1 = (,,, n 1 ) just below s 2 = (,,, n 2 ) in S with n n 2 > n 2 n 1 do 19: suppess s 2 fom S 20: end while 21: end while Figue 4 visulizes the diffeent ses in Algoithm 3. line 8 line 11 line 14 b d Figue 4: Visuliztion of the diffeent onditions in Algoithm 3. The inoming tg x oesponds to the highlighted node. We highlight some popeties onening the stk used in Algoithm 3. Ft 2. S in Algoithm 3 stisfies the following: 1. If ( 2, b 2, depth( 2), pos( 2)) is below ( 1, b 1, depth( 1), pos( 1)) in S, then pos( 2) < pos( 1), depth( 2) < depth( 1), nd 1, b 1 e in the subtee of b Conside l t the end of the while loop in line 20. Then thee e no stk elements (,, l, ) with l > l. Figue 5 illusttes the eltionship between two onseutive stk elements s disussed in Item 1 of Ft 2. Lemm 2. Algoithm 3 veifies ll nodes v whose left subtee is t lest s lge s its ight subtee. Poof. Let q be suh node. Let 1, b 1 be the hilden of q. Then it holds tht pos( 1) pos( 1) pos(b 1) pos(b 1), (1)

6 p 2 b 2 q 1 b 1 Stk S. ( 1, b 1, pos( 1), depth( 1)) ( 2, b 2, pos( 2), depth( 2)). n 1 1, we obtin tht n i 2i 1 n + 1, nd, in ptiul, n t 1 (n 1) + 1. Sine ll ni e integes, it holds 2 i 2 i n tht n t 1 n. Futhemoe, sine n t > n t 1, we obtin n log n+1 n + 1 whih is ontdition, sine the element t position n + 1 hs not yet been seen. Sine n 2N nd the size of stk element is in O(log n), Algoithm 3 uses spe O(log 2 N). This lso implies tht the while-loop t line 18 of Algoithm 3 n only be iteted O(log n) times duing the poessing of tg on the stem. The poessing time pe lette is then O(log N), sine we ssume tht opetions on the stk un in onstnt time. Figue 5: is the uent element unde onsidetion in Algoithm 3. 1, b 1 is in the subtee of b 2, ompe Item 1 of Ft 2.. sine the size of the left subtee of q is t lest s lge s the size of the ight subtee. Upon ivl of 1 Algoithm 3 pushes the 4-tuple t = ( 1, b 1, pos( 1), depth( 1)) onto the stk S. We hve to show tht t emins on the stk until the ivl of q. Moe peisely, we hve to show tht the ondition in line 18 is neve stisfied fo s 2 = t. Sine the lgoithm neve deletes the bottom-most stk item, we onside the se whee thee is stk item ( 2, b 2, pos( 2), depth( 2)) just below t. Item 1 of Ft 2 tells us tht 1, b 1 e in the subtee of b 2. Let be the uent tg unde onsidetion suh tht pos(b 1) < pos() < pos(q). The sitution is visulized in Figue 5. Aoding to the ondition of line 18, t gets emoved fom the stk if pos() pos( 1) > pos( 1) pos( 2). (2) Note tht the left side of Inequlity 2 is lowe bound on the size of the ight subtee of q. Futhemoe, the ight side of Inequlity 2 uppe bounds the size of the left subtee of q. Using pos() pos( 1) pos(b 1) pos(b 1) + 1 nd pos( 1) pos( 2) > pos( 1) pos( 1), Inequlity 2 ontdits Inequlity 1 whih shows tht t emins on the stk until the ivl of q. Item 2 of Ft 2 guntees tht thee is no othe stk element on top of t upon ivl of q. This guntees the veifition of node q nd poves the lemm. Theoem 2. Algoithm 2 is bidietionl two-pss steming lgoithm fo Vlidity(2) with spe O(log 2 N) nd O(log N) poessing time pe lette. Poof. To pove oetness of Algoithm 2, we ensue tht ll nodes get veified. By Lemm 2, in the fist pss, ll nodes with left subtee being t lest s lge s its ight subtee get veified. The seond pss ensues then veifition of nodes with ight subtee tht is t lest s lge s its left subtee. Next, we pove by ontdition tht fo ny uent vlue of vible n in Algoithm 3, the stk ontins t most log(n) elements. Assume tht thee is stk onfigution of size t log(n) + 1. Let (n 1, n 2, n t) be the sequene of the fouth pmetes of the stk elements. Sine these elements e not yet emoved, due to line 18 of Algoithm 3, it holds tht n n i n i n i 1, o equivlently n i 1/2(n + n i 1), fo ll 1 < i t. Sine 4. VALIDITY OF GENERAL TREES 4.1 Peption The FCNS enoding (see fo instne [14]) is n enoding of unnked tees s extended 2-nked tees, whee we distinguish left hild fom ight hild. This is n extension of odeed 2-nked tees, sine node my hve left hild but no ight hild, nd vie ves. We theefoe duplite the lbels Σ to L nd R, fo espetively left nd ight opening/losing tgs. The FCNS tee is obtined by keeping the sme set of tee nodes. The oot node of the unnked tee emins the oot in the FCNS tee, nd we nnotte it by defult left. The left hild of ny intenl node in the FCNS tee is the fist hild of this node in the unnked tee if it exists, othewise it does not hve left hild. The ight hild of node in the FCNS tee is the next sibling of this node in the unnked tee if it exists, othewise it does not hve ight hild. Fo tee t, we denote FCNS(t) the FCNS tee, nd XML(FCNS(t)) the XML sequene of the FCNS enoding of t. Insted of nnotting by left/ight, nothe wy to uniquely identify node s left o ight is to inset dummy leves with lbel. Fo tee t, we denote the biny vesion without nnottions nd insetion of leves by FCNS. The two epesenttions n be esily tnsfomed into eh othe. In this setion, we ompute the FCNS enoding with nnottions. In the next setion, we pesent lgoithms fo the vlidtion of the enoded fom tht mke use of the epesenttion using dummy leves. See Figue 6 fo n exmple. b b b b Figue 6: Left: intodutoy exmple tee t ledy shown in Figue 1. Middle: FCNS enoding of t: XML(FCNS(t)) = L b L L R R R R L b R b R L R R L R R b R b R b L L. Right: FCNS enoding of t: XML(FCNS (t)) = b b b bbb. In the following subsetions we povide steming lgoithms fo the tnsfomtion of XML(t) to XML(FCNS(t)), tht we ll the FCNS enoding, nd its invese, the FCNS deoding. The FCNS enoding n be seen s eodeing of the tgs of XML(t) nd n nnottion of the tgs with left/ight. b b b b b

7 We stte sevel popeties bout the eltionship of the odeing of the tgs in XML(t) nd XML(FCNS(t)). Ft 3 onens the stutue of the subsequene of opening tgs in XML(FCNS(t)), Ft 4 onens the stutue of the subsequene of losing tgs in XML(FCNS(t)), nd Ft 5 onens the inteply of the subsequenes of opening nd losing tgs in XML(FCNS(t)). Ft 3. The opening tgs in XML(t) e in the sme ode s the opening tgs in XML(FCNS(t)). Fo node v of some tee t, let pos (v) nd pos ( v) be the espetive positions of the opening nd losing tgs of v in XML(FCNS(t)). Ft 4. Nodes v 1, v 2 of t stisfy pos (v 1) < pos (v 2) iff one of the following onditions holds: 1. v 1 is in the subtee of v 2 in t; 2. o v 1 is ight sibling of v 2 in t; 3. o thee is node u with depth(u) depth(v 1) 2 suh tht pos(v 1) < pos(u) pos(v 2). Ft 5. Nodes v 1, v 2 of t stisfy pos (v 1) < pos (v 2) iff thee is node u with depth(u) depth(v 1) 2 suh tht pos(v 1) < pos(u) pos(v 2). 4.2 FCNS enoding In this setion, we e inteested in omputing the tnsfomtion XML(t) XML(FCNS(t)). Ou sttegy is to ompute the subsequene of opening tgs of XML(FCNS(t)) (using Ft 3 nd disussed in subsetion 4.2.1) nd the subsequene of losing tgs (using Ft 4 nd disussed in 4.2.2) of XML(FCNS(t)) independently, nd mege them ftewds (using Ft 5 nd disussed in subsetion 4.2.3) Computing the sequene of opening tgs Conening the opening tgs, sine due to Ft 3 the subsequenes of opening tgs in XML(t) nd XML(FCNS(t)) oinide, we extt the subsequene of opening tg of XML(t), nd we nnotte them with left o ight s they should be in XML(FCNS(t)). Remind tht n opening tg is left if it is the opening tg of fist hild, othewise it is ight. Futhemoe, fo lte use we nnotte eh opening tg with depth() in t nd the position in the stem pos(). We summize this s ft: Ft 6. Thee is steming lgoithm with spe O(log N) tht, given XML(t) s input, outputs on n uxiliy stem the sequene of opening tgs of XML(FCNS(t)) with left/ight nnottions, nd futhemoe, nnottes eh tg with depth() nd pos(), pefoming one pss on eh stem Computing the sequene of losing tgs Fo omputing the sequene of losing tgs, we stt with the sequene of opening tgs of XML(FCNS(t)) s podued by the output of the lgoithm of Ft 6, tht is, oetly nnotted with left/ight nd with depth nd position nnottions. To obtin the oet subsequene of losing tgs s in XML(FCNS(t)), we intepet the opening tgs s losing tgs nd we sot them with mege sot lgoithm. Mege sot n be implemented s steming lgoithm with O(log(N)) psses nd 3 uxiliy stems [7]. Fo the ske of simpliity, Algoithm 4 ssumes n input of length 2 l fo some l > 0. Algoithm 4 Mege sot Requie: unsoted dt of length 2 l on stem 1 1: fo i = 0 l 1 do 2: opy dt in bloks of length 2 i fom stem 1 ltentely onto stem 2 nd stem 3 3: fo j = 1 2 l i 1 mege(2 i ) end fo 4: end fo mege(b) eds simultneously the next b vlues fom stem 2 nd stem 3, nd meges them onto stem 1. The whole loop in Line 3 of Algoithm 4 equies one ed pss on stem 2, one ed pss on stem 3, nd one wite pss on stem 1. See Figue 7 fo n illusttion. st 1: st 2: st 3: line 2 (opy) B 1 B 2 B 3 B 4 B 2 l i B 1 B 3 B l i 1 B 2 B 4 B l i line 3 (mege) B 12 B 34 B 2 l i 1 2 l i B 1 B 2 B 3 B 4 B l i 1 B l i Figue 7: In Line 2, bloks fom stem 1 e opied onto stem 2 nd stem 3. The B i e soted bloks. In line 3, ll bloks B i nd B i+1 e meged into soted blok B i(i+1). In ode to use mege sot, we hve to define ompto funtion tht, given two losing tgs 1, 2, deides whethe pos ( 1) < pos ( 2). Fistly, onside nodes v 1, v 2 with pos(v 1) < pos(v 2) to be s in Point 1 o Point 2 of Ft 4, tht is, eithe v 1, v 2 e siblings o one node is ontined in the subtee of the othe one. Obviously, thei odeing with espet to pos n esily be deided by thei depth: pos (v 1) < pos (v 2) iff depth(v 1) > depth(v 2). If neithe v 1, v 2 e siblings, no v 2 is in the subtee of v 1 (Point 3 of Ft 4), then pos (v 1) < pos (v 2), independently of thei depths. A ompison funtion hene should be ble to infe the eltionship of the two nodes, howeve, this seems to be diffiult in the steming model. To oveome this poblem, insted of defining ompison funtion, we design omplete mege funtion in Lemm 3 tht, by onstution, only ompes two nodes of the fist kind. The key ide is to intodue septo tgs whih we denote by new tg outside Σ. They e initilly inseted ight fte eh losing tg of lst hild u, tht is extly befoe the depth deeses. We denote by u the septo we intodue when seeing the lst hild u, nd we define depth(u) = depth(u). Ft 7. Thee is steming lgoithm with spe O(log N) tht, given sequene XML(t) on stem, omputes on nothe stem the sequene of opening tgs XML(FCNS(t)) togethe with thei septos, nd nnotted with depth, pos nd left/ight, pefoming one pss on eh stem. We hve to define the wy we integte the septos into ou soting. Let v 1, v 2,, v k be the odeed sequene of the hilden of some node. Fo the septo v k we sk thei position mong the losing tgs to stisfy fo eh node v: pos (v) < pos (v k ) iff pos (v) pos (v 1); (3) nd fo ny othe septo w k : pos (v k ) < pos (w k ) iff pos (v k ) < pos (w k ). (4) Bloks ppeing in mege sot fulfill popety tht we ll well-soted. A blok B of losing tgs is well-soted

8 if the oesponding tgs in XML(FCNS(t)) ppe in the sme ode, nd fo ll v 1, v 2 B with pos(v 1) < pos(v 2), ll losing tgs v of nodes v with pos(v 1) < pos(v) < pos(v 2) e in B s well. In ddition, fo two bloks B 1, B 2 of losing tgs, we sy tht (B 1, B 2) is well-soted djent pi, if B 1 nd B 2 e well-soted, fo eh losing tg v 1 B 1 nd eh losing tg v 2 B 2 pos(v 1) < pos(v 2) is stisfied, nd futhemoe, ll losing tgs v of nodes v with pos(v 1) < pos(v) < pos(v 2) e eithe in B 1 o B 2. The only funtion to design is ompto deiding fo two losing tgs v 1, v 2 fom well-soted djent pi (B 1, B 2) whethe pos (v 1) < pos (v 2). The following lemm shows tht we n mege wellsoted djent pi oetly. Lemm 3. Let (B 1, B 2) be well-soted djent pi, nd let v 1 = B 1[p 1] nd v 2 = B 2[p 2] fo some p 1, p 2. Assume tht pos (v) < pos (v 1) nd pos (v) < pos (v 2), fo ll v B 1[1, p 1 1] B 2[1, p 2 1]. Then: 1. If v 1 is septo, o thee is septo in B 1 fte v 1, then pos (v 1) < pos (v 2); 2. Else if v 2 is septo then: () if depth(v 1) < depth(v 2) then pos (v 1) < pos (v 2), (b) else pos (v 1) > pos (v 2); 3. Else (neithe v 1 no v 2 is septo): () if depth(v 1) depth(v 2) then pos (v 1) < pos (v 2), (b) else pos (v 1) > pos (v 2). Poof. Let (B 1, B 2) be well-soted djent pi. Let l = mx{i : B 1[i] is septo}. If thee e no septos in B 1, let l = 0. Fist, we pove Point 1. Sine B 1 is well-odeed, we only need to hek tht pos (B 1 [l]) < pos (B 2 [1]). Denote by u the lst hild tht ws esponsible fo the insetion of the septo tg B 1[l]. Let u be the left-most sibling of u. Due to Eqution (3) it suffies to show tht pos (u ) < pos (B 2[1]). Clely, the shotest pth fom u to B 2[1] psses by ommon nesto p of u nd B 2[1] whih is not the pent of u sine the septo B 1[l] indites tht the lst hild u hs been seen. Then, by the thid ondition of Ft 4, we get pos (u ) < pos (B 2[1]). Fo poving Points 2 nd 3 we use the obsevtion tht if the pemises to Point 1 e not fulfilled, v 1, v 2 do not hve ommon nesto p s.t. pos(v 1) < pos(p) < pos(v 2) nd p is not the pent node of v 1. Futhemoe, this obsevtion implies tht depth(v 2) depth(v 1) 1 nd hene, if depth(v 2) > depth(v 1) then v 2 is in the subtee of v 1. This nd Ft 4 pove Points 2, 2b, 3 nd 3b. We pove the obsevtion by ontdition. Assume tht thee is suh node p. Sine (B 1, B 2) is well-odeed djent pi nd pos(v 1) < pos(p) < pos(v 2), node p would be in B 1 B 2. Theefoe, the septo u inseted fte the ightmost sibling of v 1 would be lso in B 1 B 2 s well. Moe peisely, this septo would be in B 2[1 p 2 1] sine othewise Point 1 would hve been pplied. This, howeve, is ontdition to the ssumption tht pos (v) < pos (v 1) v B 1[1 p 1 1] B 2[1 p 2 1] sine it holds tht pos (v 1) < pos (u). Hene suh node does not exist. Lemm 4. Thee is O(log N)-pss steming lgoithm with spe O(log N) nd 3 uxiliy stems tht omputes the subsequene of losing tgs of the FCNS enoding of ny XML doument given in the input stem. Poof. Using Ft 7, we ompute on the fist uxiliy stem the sequene of opening tgs with oesponding nnottions, togethe with septos, nd intepet opening tgs s losing tgs. We show tht we n do mege sot lgoithm with mege funtion inspied by Lemm 3 on the fist thee uxiliy stems with O(log N) spe nd psses. Fo tht ssume tht the fist stem ontins sequene (B 1, B 2,, B M ) of bloks of size 2 i. Fo simpliity we ssume tht M is even, othewise we dd n empty blok. We ltentely opy odd bloks on the seond stem, nd even bloks on the thid stem. Fo blok B 2i tht we wite on the thid stem, we wite befoe eh of them, the numbe of septos tht ou in the blok B 2i 1 tht ws opied on the seond stem. Then we mege sequentilly ll pis of bloks (B 2k 1, B 2k ) fo 1 k M/2 using Lemm 3. Note tht (B 2k 1, B 2k ) i e ll well-fomed pis. Let l = mx{i : B 2k 1 [i] is septo}. Fistly, we opy elements B 2k 1 [1, l] onto uxiliy stem 1. Knowing the numbe of septos in B 2k 1 llows us to pefom this opetion. The oetness of this step follows fom Point 1 of Lemm 3. Then, we mege bloks B 2k 1 [l+1, 2 i ] nd B 2k by using the ompison funtion defined in Points 2 nd 3 of Lemm Meging opening nd losing tgs Meging the subsequene of opening tgs of XML(FCNS(t)) nd the subsequene of losing tgs of XML(FCNS(t)) n be done using one dditionl pss. Lemm 5. Thee is steming lgoithm with spe O(log N) tht, given the sequene of opening tgs of XML(FCNS(t)) on stem, nd the sequene of losing tgs of XML(FCNS(t)) on nothe stem, omputes XML(FCNS(t)) on thid stem using one pss on eh stem. Poof. We dietly pply Ft 5, so tht we know when to ltente fom the sequene of opening tgs to the one of losing tgs, nd onvesely. Fom Ft 6, Lemm 4 nd Lemm 5 we obtin Theoem 3. Theoem 3. Thee is O(log N)-pss steming lgoithm with spe O(log N) nd 3 uxiliy stems nd O(1) poessing time pe lette tht omputes on the thid uxiliy stem the FCNS tnsfomtion of ny XML doument given in the input stem. Poof. Fistly, we ompute oding to Lemm 4 the sequene of losing tgs nd we stoe them on uxiliy stem 1. Then, by Ft 6 we extt the sequene of opening tgs, nd we stoe them on uxiliy stem 2. By Lemm 5 we n mege the tgs of uxiliy stem 1 nd uxiliy stem 2 oetly onto stem 3. The spe equiements of these opetions do not exeed O(log N). The poessing time pe lette of these opetions is onstnt. 4.3 Cheking Vlidity on the enoding fom

9 In this setion, we euse the lgoithms fo vlidting biny tees fo the vlidtion of the enoded fom. We disuss one-pss ed/wite steming lgoithms (one fo leftto-ight psses, nd one fo ight-to-left psses) tht ed XML(FCNS (t)) nd output n XML doument with nnottions on losing tgs tht n be fed into Algoithm 1 o Algoithm 2. This equies little modifitions in Algoithm 1 nd Algoithm 2 sine vlidity then depends on the nnottions. Sine we wnt to euse the lgoithms fo the vlidtion of biny tees, we suppose tht XML(FCNS (t)) is vilble s input. The lgoithm stted in Theoem 3 n be esily dpted suh tht it outputs XML(FCNS (t)) insted of XML(FCNS(t)). The poblem of vlidting the enoded fom nd the poblem of vlidting biny tees e simil. Note tht the hilden v 1, v k of node v fom substing in XML(FCNS (t)), see Figue 8. Hene, fo vlidting node v, the lbel v hs to be elted to the blok of hilden nodes v k v 1. This is simil to the tsk of the vlidtion of biny tees whee the pent lbel v hs to be elted to the blok of hilden nodes v 1v 2. t : v v 1 v 2 v k t 1 t 2 t k FCNS (t) : t 1 v 1 t 2 v v 2 Figue 8: A tee t nd its FCNS enoding. While the opening nd losing tgs of the hilden of node v e septed by the subtees t 1, t k in XML(t), the losing tgs of the hilden of v e onseutive in XML(FCNS (t)) in evese ode, tht is v k v k 1 v 2 v 1 is substing of XML(FCNS (t)). Fo node v, we gthe the infomtion of the hilden nodes v 1,, v k by the help of finite utomt A 1 (fo leftto-ight psses) nd A 2 (fo ight-to-left psses) tht we define lte, nd the infomtion - stte of the utomt - is nnotted t the losing tg of lef v 1 (left-to-ight) o v k (ight-to-left). Then, by the help of Algoithm 1 o Algoithm 2, this infomtion is elted to the pent lbel v. We define utomt A 1 nd A 2 now. A 1 is onstuted fom utomton A. Let A = (Σ, Q, q 0, δ, F ) be deteministi finite utomton whee Σ is its input lphbet, Q is the stte set, q 0 is its initil stte, δ : Q Σ Q is the tnsition funtion, nd F is set of finl sttes. Fo Σ nd the input DTD D, denote by A deteministi finite utomton tht epts the egul expession D(). We ompose the A s in the left illusttion of Figue 9 to n utomton A tht epts wods ω suh tht ω = ω, Σ, ω Σ if ω D(). Let A 1 = (Σ, Q 1, (q 0) 1, δ 1, F 1) be deteministi finite utomton tht epts wod ω, iff ω ev is epted by A, whee ω ev denotes ω ed fom ight to left. Let A 2 = (Σ, Q 2, (q 0) 2, δ 2, F 2) be deteministi finite utomton tht epts wod ω suh tht ω = ω, Σ, ω Σ if ω D(). A 2 is vesion of the utomton in the ight illusttion of Figue 9 without ɛ tnsitions. In the following, we ssume fo evey stte q 1 Q 1 nd t k v k q Σ A 1 A 2. A Σ q 0 Figue 9: Left: Automton A. A 1 epts wods ω if A epts ω ev. Right: Automton A 2 is vesion of the illustted utomton without ɛ tnsitions. q 2 Q 2 tht δ 1(q 1, ) = q 1 nd δ 2(q 2, ) = q 2. Given XML(FCNS (t)), fo left-to-ight pss, we nnotte losing tgs v by stte fom the stte set Q 1 of utomton A 1. We denote the nnottion fo left-to-ight psses of v by nn 1(v). if v is lef then nn 1(v) = δ 1((q 0) 1, v), othewise let u be the ight hild of v, then nn 1(v) = δ 1(nn 1(u), v). Fo ight-to-left pss, we nnotte losing tgs v by stte fom the stte set Q 2 of utomton A 2. We denote the nnottion of ight-to-left psses of v by nn 2(v). ǫ ǫ ǫ A 1 A 2. A Σ if v is left hild then nn 2(v) = δ 2((q 0) 2, v), if v is ight hild of u then nn 2(v) = δ 2(nn 2(u), v). Fo the ske of ompleteness, the oot n be nnotted by nn 2() = (q 0) 2, though this nnottion will not be used fo heking vlidity. Figue 10 shows the nnottions of the hilden nodes v 1,, v k of node v. v 1 v v 2 v k v v 1(δ 1(nn 1(v 2), v 1)) v 2(δ 1(nn 1(v 3), v 2)) v k(δ 1((q 0) 1, v k)) v v 1(δ 2((q 0) 2, v 1)) 1 2 Σ v 2(δ 2(nn 2(v 1), v 2)) q f v k(δ 2(nn 2(v k 1), v k)) Figue 10: Left: node v with its hilden v 1, v k in the FCNS tee. Middle: nnottions fo left-to-ight psses. v is vlid if δ 1 (nn 1 (v 1 ), v) esults in n epting stte of A 1. Right: nnottions fo ight-to-left psses. v is vlid if δ 2 (nn 2 (v k ), v) is n epting stte of A 2. The nnottion opetions n be seen s steming lgoithms pefoming one ed pss ove the input nd one wite pss ove nothe stem using onstnt spe, sine the nnottion of losing tg v only depends on the nnottion of its ight hild (fo left-to-ight psses) o its pent (fo ightto-left psses). The espetive losing tg is in both ses the tg pio to v in the input stem XML(FCNS (t)). Remembe tht we onside ight-to-left pss fo the nnottion with nn 2. In the following, we pove tht given the nnottions, Algoithm 1 nd Algoithm 2 n be dpted to deide vlidity of the enoded fom. Theoem 4. Thee is one-pss deteministi lgoithm fo Vlidity with spe O( N log N) nd O(1) poessing time pe lette when the input is given in its FCNS enoding. Poof. Append the ule D( ) = ɛ to the input DTD

10 D. Compute utomton A 1. Compute new XML stem with nnottions nn 1 nd feed this stem dietly into Algoithm 1. In ode to veify node v, Algoithm 1 uses the losing tg v 1 of the left hild v 1 of v. Sine v is vlid if δ 1(nn 1(v 1), v) is n epting stte, we only hve to eple the hek funtion used in Algoithm 1. The new hek funtion omputes δ 1(nn 1(v 1), v) nd bots if the esulting stte is not epting. The spe equiements nd the poessing time pe lette inheit fom Algoithm 1. Theoem 5. Thee is bidietionl two-pss deteministi lgoithm fo Vlidity with spe O(log 2 N) nd O(log N) poessing time pe lette when the input is given in its FCNS enoding. Poof. Append the ule D( ) = ɛ to the input DTD D. Compute utomt A 1 nd A 2. Fistly, in left-to-ight pss, s in the poof of Theoem 4 we ompute new XML stem with nnottions nn 1 nd feed this stem dietly into Algoithm 3. We dpt the hek funtion s bove. Conening the ight-to-left pss, we ompute the nnottions nn 2, nd feed this stem dietly into Algoithm 3 intepeting opening tgs s losing tgs, nd vie ves. Note tht the nnottions nn 2 e hene nnotted on opening tgs. Let v be node with hilden v 1,, v k. We hve to show how Algoithm 3 n be dpted to elte the nnottion of the opening tg v k to v. When Algoithm 3 eds the losing tg w nd the opening tg v 1, it pushes (w, v 1,, ) on the stk, whee w is the sibling of v 1 in the FCNS enoding. Note tht the subsequent tgs on the stem e v 2, v 3, v k. Node v k n be identified sine v k is eithe lef o followed by tg with lbel. Hene, the stk item (w, v 1,, ) n be nnotted with nn 2(v k ) when v k is seen. Agin by dpting the hek outine, we n ompute δ 2(nn 2(v k ), v 1) nd bot if the esult is not n epting stte. The spe equiements nd the poessing time pe lette inheit fom Algoithm 2. Applying the bidietionl lgoithm of Theoem 5 on the enoded fom XML(FCNS (t)), we obtin tht vlidity of genel tees n be deided memoy effiiently in the steming model with uxiliy stems. Coolly 1. Thee is bidietionl O(log N)-pss deteministi steming lgoithm fo Vlidity with spe O(log 2 N), O(log N) poessing time pe lette, nd 3 uxiliy stems. 4.4 Deoding In the following, we pesent steming lgoithm fo FCNS deoding, tht is, given XML(FCNS(t)) of some tee t, output XML(t). We stt with non-steming lgoithm, Algoithm 5 pefoming this tsk. We desibe how this lgoithm n be onveted into steming lgoithm. Fo some opening tg X[i], heking the ondition in Line 4 n esily be done by investigting X[i + 1]. If X[i + 1] is ight opening tg o equls X[i], X[i] does not hve left subtee. The diffiulty in onveting this lgoithm into steming lgoithm is in Line 8, it is diffiult to keep tk of opening tgs until the espetive losing tgs of thei left hilden e seen, nd indeed, this n not be done with subline spe in one pss (Ft 10). Algoithm 5 offline lgoithm fo FCNS deoding 1: fo i = 1 2N do 2: if X[i] is n opening tg then 3: wite X[i] 4: if X[i] does not hve left subtee then 5: wite X[i] 6: end if 7: else if X[i] is left losing tg then {See Figue 11} 8: let p be the pent node of X[i] 9: wite p 10: end if 11: end fo p v 1 v 2 subtee of v 1 subtee of v 2 Figue 11: The min diffiulty of the FCNS deoding is to wite the losing tg of node p when the losing tg of its left hild is seen. This is diffiult when the subtees of v 1 nd v 2 e lge. In the following, we pesent steming lgoithm tht pefoms one pss ove the input, but two psses ove the output, nd uses O( N log N) spe, nd steming lgoithm tht pefoms O(log N) psses ove the input nd 3 uxiliy stems using O(log 2 (N)) spe One ed-pss nd two wite-psses We ed bloks of size N log N nd exeute Algoithm 5 on tht blok. In Lemm 6 we show tht in ny blok thee is t most one left losing tg fo whih the pent s opening nd losing tg e not in tht blok. Hene pe blok thee is t most one left losing tg fo whih we n not obtin the lbel of the pent node. We ll this losing tg itil. In this se we wite dummy symbol on the output stem tht will be ovewitten by the pent losing tg in the seond pss. The losing tg of the pent node will ive in subsequent blok, nd it n esily be identified s this sine it is the next losing tg iving t depth 1 of the itil losing tg. We stoe it upon its ivl in ou ndom ess memoy. Sine thee is t most one itil losing tg pe blok nd we hve blok size of N log N, we hve to eove t most O( N/ log N) pent nodes. At the end of the pss ove the input stem we hve eoveed ll losing tgs of pent nodes fo whih we wote dummy symbols on the output stem. In seond pss ove the output stem we ovewite the dummy symbols by the oet losing tgs. The omplexity deives fom the following lemm demonstting tht in blok thee is t most one itil left losing tg. Lemm 6. Let X[i, j] be blok. Then thee is t most one left losing tg with pent node p suh tht: pos(p) < i pos() j < pos(p). (5) Poof. By ontdition, ssume tht thee e 2 left losing tgs, b with p being the pent node of, nd q being the pent node of b, fo whih Inequlity 5 holds. Wlog we ssume tht pos(p) < pos(q). Sine pos(p) < pos(q) < pos(), q is ontined in the subtee of o q =. This, howeve, implies tht pos(q) pos() < j ontditing

11 pos(q) > j. Theoem 6. Thee is steming lgoithm using O( N log N) spe nd O(1) poessing time pe lette whih pefoms one pss ove the input stem ontining XML(t) nd two psses ove the output stem onto whih it outputs XML(FCNS(t)) Logithmi numbe of psses Agin, we use the offline Algoithm 5 s stting point fo the lgoithm we design now. Fo oping with the poblem tht it is hd to emembe ll opening pent tgs when thei oesponding losing tg ought to be witten on the output, we wite tegoilly dummy symbols on the output stem fo ll pent losing tgs. The ux then is the following obsevtion: Ft 8. Let 1L N L be the subsequene of losing tgs of left hilden of XML(FCNS(t)). Then the sequene p 1 p N is subsequene of XML(t) whee p i is the pent node of i in FCNS(t). We pply modified vesion of ou bidietionl two-pss Algoithm 2 to eove the missing tgs. Insted of heking vlidity, one the hek funtion is lled in Algoithm 3 with vibles (, b, ), we output the pent lbel onto n uxiliy stem, nnotted with pos(b). We do the sme in evese pss ove the input stem ounting positions fom 2N downwds to 1. In so doing, the uxiliy stem ontins ll pent lbels fo whih dummy symbols e witten on the output stem. Ft 8 shows tht it is enough to sot by mens of two futhe uxiliy stems the uxiliy stem with espet to the nnotted position of the losing tgs of the left hilden of these nodes. In lst pss we inset the pent losing tgs into the output stem. Theoem 7. Thee is O(log N)-pss steming lgoithm with spe O(log 2 N) nd O(log N) poessing time pe lette nd 3 uxiliy stems tht omputes on the thid uxiliy stem the FCNS deoding of ny FCNS enoded doument given in the input stem. 5. LOWER BOUNDS FOR FCNS ENCOD- ING AND DECODING We define fmily of hd instnes of length N = Θ(n) fo the omputtion of the FCNS enoding of tee s in Figue 12. x 1 x 2 x n x 1 x 2 Figue 12: Left: hd instne. Right: its FCNS enoded fom. It is esy to see tht omputing the sequene of losing tgs in the FCNS enoding equies to invet stem. Let t be hd instne. Then XML(t) = x 1x 1x 2x 2 x nx n, nd XML(FCNS(t)) = Lx 1Lx 2R x nrx nrx n 1R x 2Rx 1L L. Sine the witing x n on the stem n only stt fte eding x n, we dedue tht memoy spe Ω(n) is equied, in ode to stoe ll pevious tgs. Ft 9. Evey one-pss ndomized steming lgoithm fo FCNS enoding with bounded eo equies Ω(N) spe. We onjetue tht this gument n be extended s follows: Conjetue 1. Any p-psses ndomized steming lgoithm fo FCNS enoding with bounded eo equies spe Ω(N/p). We now define nothe fmily of hd instnes of length N = Θ(n) fo deoding FCNS enoded tee s in Figue 13. x n x k+1 x k x k 1 subtee y x 1 x k x k+1 x n x 1 x k 1 subtee z Figue 13: Left: hd instne in FCNS fom, whee y is ny tee of size Θ(n). Right: its deoded fom. Intuitively, deoding the tee of ny hd instne equies to put the full tee y into memoy. Let XML(FCNS(t)) denote hd instne whih we im to deode into XML(t). Then: XML(FCNS(t)) = x 1L x nlx nl x k+1l Y x kl x 1L L nd the deoded fom is XML(t) = x 1 x nx n x k Zx k 1 x 1 whee Z is the deoded fom of Y. Sine Z n only be witten fte x k, nd sine x k nnot be memoized beuse k ws unknown until we eh Y, memoy spe Ω(n) is equied. This gument n be esily fomlized using stndd infomtion theoy guments. Ft 10. Evey one-pss ndomized steming lgoithm fo FCNS deoding with bounded eo equies spe Ω(N). This gument n be extended to two-pss ndomized steming lgoithms. Constut hd instne of size Θ(n 2 ) by gluing n pevious instnes (x i, k i, y i ) 1 i n s follows: instne (x i+1, k i+1, y i+1 ) is bnhed to the left most lef of instne (x i, k i, y i ). Afte the fist pss, the lgoithm is not ble to wite n losing tgs of fom x i k i. Theefoe he needs to stoe them in ode to wite them t the seond pss. This equies some fomliztion tht we omit hee. Ft 11. Evey ndomized steming lgoithm fo FCNS deoding with bounded eo, one pss on the input stem, nd two psses on the output stem equies spe Ω( N). Aknowledgements The uthos would like to thnk Mihel de Rougemont, who, mong othe things, intodued the uthos to the poblem of vlidting steming XML douments.