Tree-structured Data Regeneration in Distributed Storage Systems with Regenerating Codes

Transcription

1 Tee-stuctue Data Regeneation in Distibute Stoage Systems with Regeneating Coes Jun Li, Shuang Yang, Xin Wang School of Compute Science Fuan Univesity, China { , , Baochun Li Depatment of Electical an Compute Engineeing Univesity of Toonto, Canaa Abstact Distibute stoage systems povie lage-scale eliable ata stoage by stoing a cetain egee of eunancy in a ecentalize fashion on a goup of stoage noes To ecove fom ata losses ue to the instability of these noes, wheneve a noe leaves the system, aitional eunancy shoul be egeneate to compensate such losses In this context, the geneal objective is to minimize the volume of actual netwok taffic cause by such egeneations A class of coes, calle egeneating coes, has been popose to achieve an optimal taeoff cuve between the amount of stoage space equie fo stoing eunancy an the netwok taffic uing the egeneation In this pape, we jointly consie the choices of egeneating coes an netwok topologies We popose a new esign, efee to as RCTREE, that combines the avantage of egeneating coes with a tee-stuctue egeneation topology Ou focus is the efficient utilization of netwok links, in aition to the euction of the egeneation taffic With the extensive analysis an quantitative evaluations, we show that RCTREE is able to achieve a both fast an stable egeneation, even with epatues of stoage noes uing the egeneation Inex Tems Distibute Stoage System, Data Regeneation, Regeneating Coes I INTRODUCTION Lage-scale istibute stoage systems ae esigne to povie eliable sevices of ata stoage, by stoing a egee of ata eunancy in a ecentalize manne, acoss a lage numbe of stoage noes in the system [1] These stoage noes may be off-the-shelf cluste noes in lage-scale ata centes, isk aays in stoage aea netwoks, o even oinay en hosts acoss the Intenet, oganize in a pee-to-pee fashion Regaless of thei eliability, howeve, stoage noes in istibute stoage systems may fail, leaing to the ata loss In fact, lage ata centes ae esigne to teat stoage noe failues as the ule, not the exception With the pesence of noe failues, it is esiable to maintain a egee of ata eunancy, such that a subset of stoage noes is sufficient to ecove the oiginal ata When a stoage noe oes fail, it is necessay to egeneate ata in a eplacement noe, calle a newcome, in oe to estoe the equie egee of ata eunancy How such egeneation is to be pefome epens on the esign objectives of coes to achieve eunancy If the objective is to minimize the stoage space neee fo eunancy, it has been shown that aximum Distance Sepaable (DS) coes ae optimal fo such minimum-stoage egeneation [2] It has been use in the liteatue to maintain a much smalle egee of eunancy than simple eplication fo the same eliability [3] Fo example, a file of size bits can be ivie into k blocks, each of size /k, an then be encoe into n coe blocks with an (n, k) DS coe, to be stoe in n istinct stoage noes, an any k blocks can be use to ecove the oiginal file Howeve, if minimizing netwok banwith use to egeneate ata on the newcome becomes the objective instea, with DS coes, a newcome nees a minimum of k blocks, ie a total of bits, to egeneate its new coe block of size /k bits; while only /k bits ae equie if eplication is use Dimakis et al [2] an Wu et al [4] have shown the supising esult that, eteministic linea netwok coing (efine ove a sufficiently lage finite fiel) can be use to esign a class of minimum-banwith egeneating coes to minimize banwith equie fo the egeneation, as long as moe than k stoage noes, calle povies, can be contacte by the newcome uing the egeneation Aceański et al [5] has also evaluate the ole of anom linea coing, athe than eteministic linea coes, in istibute stoage systems Though it is encouaging to esign minimum-banwith egeneating coes to minimize banwith, the existing liteatue has not focuse on the ole of the netwok topology, within which the egeneation pocess takes place It has conventionally been assume that the egeneation pocess is pefome on a simple sta-stuctue topology, ie, the newcome eceives coe blocks iectly fom each of povies In the ovelay mesh connecting stoage noes, howeve, not all ovelay links enjoy the same available banwith If we take into account the heteogeneity of banwith on links between stoage noes, a tee-stuctue topology natually ensues, in which povies ae allowe to elay egeneation taffic to the newcome How shoul we constuct such a tee to efficiently utilize available banwith on each link between stoage noes? How shoul we jointly consie the constuction of egeneation tees an the esign of egeneating coes? In this pape, we consie the geneal case of constucting such tee-stuctue egeneation topologies, with a vaiable numbe of povies in a tee topology, as well as the use of egeneating coes to achieve the stoage-banwith optimal tae-off cuve Ou new esign, efee to as RCTREE, is able to wok effectively with the banwith heteogeneity RCTREE even consies the case that the stoage noe may fail

2 2 50bps 10bps 50bps 10bps 50bps 10bps 50bps 10bps 50bps 10bps (a) netwok moel with one newcome an fou stoage noes (b) STAR with V 1,V 3 an V 4 as povies (c) RCSTAR with V 1,V 2,V 3 an V 4 as povies () TREE with V 1,V 3 an V 4 as povies (e) RCTREE with V 1,V 2,V 3 an V 4 as povies Fig 1: Examples of fou egeneation schemes: STAR, RCSTAR, TREE an RCTREE uing the egeneation pocess With the extensive analysis an quantitative evaluations base on statistical ata in PlanetLab, we ae able to show that RCTREE helps to be one step close towas a pactical epai of faile stoage noes in istibute stoage systems The emaine of the pape is oganize as follows Sec II shows the avantages of RCTREE with an illustative example In Sec III we intouce the netwok moel, an pesent ou extensive analysis on the egeneation tee Sec IV poposes RCTREE with etaile analysis Sec V analyzes the stability of RCTREE in compaison with some existing schemes Sec VI conclues this pape II BEYOND REGENERATING CODES: A OTIVATING EXAPLE We now intouce an illustative example of ata egeneation in the istibute stoage system in Fig 1 Fig 1(a) shows the netwok moel Thee ae five stoage noes, enote by V 0, V 1, V 2, V 3, an V 4 The banwith capacity of the link between two povies is heteogeneous We assume that the eunancy is coe by a (5, 3) DS coe, stoe in V 1, V 2, V 3, V 4, an a epating stoage noe Each stoage noe stoes a coe block of 3 bits, if the size of the oiginal ata is bits In oe to egeneate the lost eunancy, V 0 is selecte to be the newcome Since (5, 3) DS coe is use, V 0 nees to eceive eunancy fom at least thee povies Fig 1(b) Fig 1(e) show illustations of fou egeneation schemes Fo the conventional sta-stuctue egeneation (STAR) in Fig 1(b), if V 0 selects V 1, V 3, an V 4 as povies, it eceives ata iectly fom the thee povies, illustate by the akene eges Consieing the egeneation time, ie, the time that the newcome spens on egeneating a new coe 3 block, STAR costs secons to accomplish the egeneation, because the tansmission is bottlenecke by the link between V 0 an V 4 We ignoe the encoing time of DS coe because the pocessos usually pefom encoing opeations much faste than the netwok tansmission, an the encoing can be pefome simultaneously with the tansmission On the othe han, if moe than thee povies, fo example, V 1, V 2, V 3, an V 4, ae use as povies in the egeneation, egeneating coes [2], [4] povie a way to euce the banwith usage in the egeneation Apat fom minimum-banwith egeneating coes, Wu et al also popose minimum-stoage egeneating (SR) coes, which cost the same stoage space on stoage noes with DS coes Fo an (n, k) SR coe, each stoage noe stoes k bits an only k( k+1) bits ae tansmitte on each link in the egeneation, whee is the numbe of povies Diffeent fom DS coes, the oiginal file is ivie into moe than k blocks The coe blocks ae thei eteministic [4] o anom [6] linea combinations an each stoage noe stoes moe than one coe blocks Even though SR coes ae not able to each the minimum egeneation taffic of egeneating coes, they cost the least amount of stoage space in stoage noes Othe kins of egeneating coes can futhe euce the egeneation taffic with an incease stoage cost, but SR coes use the stoage space most effectively Theefoe, we consie SR coes in this pape In Fig 1(c), with the employment of SR coes in STAR (RCSTAR), thee will be only half of 3 bits tansfee on each akene ege Theefoe, the egeneation time can be 6 euce to 20bps secons STAR an RCSTAR, howeve, suffe fom the bottleneck links of (V 0, V 4 ) an (V 0, V 2 ), espectively If we consie the links between povies, we can utilize these links to bypass the slow bottleneck link in STAR In Fig 1(), we show an example of the tee-stuctue egeneation (TREE), with thee povies of V 1, V 3, an V 4 A spanning tee, calle egeneation tee, is constucte ove V 0, V 1, V 2, an V 4 V 1 eceives ata fom V 3 an V 4, encoes the eceive ata with the ata it stoes, an sens the encoe ata to V 0 By steamlining the elay on V 1, ie, V 1 encoes the ata byte-by-byte athe than afte eceiving the whole block, the egeneation time will be bottlenecke by the link between V 1 an V 4, an thus the egeneation time is secons only 3 In ou pevious wok [7], we have analyze the bottleneck banwith that the tee-stuctue egeneation can achieve, yet with the constaint of exactly k povies, namely thee povies in Fig 1() In fact, as shown by Fig 1(c), if thee is only one stoage noe losing its ata among a total of five stoage noes, thee ae fou stoage noes available to be use as povies in the egeneation In Fig 1(e), we constuct a egeneation tee with V 1, V 2, V 3, an V 4 as povies, an use SR coes in the system As a esult, the egeneation 6 time can be futhe euce to secons Compae with STAR in Fig 1(b), the egeneation time is euce by 583% in RCTREE In this pape, we pesent an in-epth analysis of the geneal case that the numbe of povies is vaiable, an popose RCTREE, a combine scheme of egeneating coes an TREE

3 3 In aition, we also consies noe failues uing the egeneation We compae the stability of STAR, TREE an RCTREE fom the pespective of lifetime of egeneation tees III CONSTRUCTING REGENERATION TREES In this section, we pesent an in-epth analysis of TREE, the tee-stuctue egeneation in the geneal case with a vaiable numbe of povies We fist intouce ou netwok moel fo the egeneation pocess in istibute stoage systems Then we valiate Pim s algoithm to obtain the optimal egeneation tee, analyze its bottleneck banwith, an show the stategy of eciing the numbe of povies A Netwok oel We assume that in a istibute stoage system, eunant ata is pouce by an (n, k) DS coe, which ivies the oiginal file into k blocks, F 1, F 2,, F k, an encoes them into n coe blocks B 1, B 2,, B n In the netwok, with espect to one file, thee ae n stoage noes, V 1, V 2,, V n, stoing the n coe blocks We assume that B i is stoe in V i, i = 1, 2,,n, fo this stoage space allocation scheme will lea to the optimal ecovey ate [8] Without loss of geneality, assume that B n gets lost Anothe coe block will then be egeneate in a newcome V 0 Assume stoage noes ae active in the egeneation, eg V 1, V 2,, an V In oe to maintain the DS popety, V 0 shoul eceive ata fom at least k noes of the active stoage noes, calle povies, k < n Let (V i, V j ) be the uniecte ege connecting V i an V j, an ω(v i, V j ), the weight of (V i, V j ), epesent the banwith capacity of (V i, V j ) In this pape, we pesent the netwok moel in the egeneation as an uniecte complete gaph G(; n, k) = {V ( + 1), E( + 1), ω}, k < n, whee V ( + 1) = {V 0, V 1,,V }, an E( + 1) = {(V i, V j ) 0 i < j } V 0 is the newcome an othe noes in V ( + 1) ae stoage noes, at least k noes of which shoul be selecte as povies We assume that the weight of each ege in E( + 1) is iffeent fom othe eges In the wie-aea netwok o Intenet, this hols with high pobability Fig 1(a) is an example of G(4; n, k), n > 4 k B The (, ) Regeneation Tee Given the netwok moel, the following efinition escibes the tee-stuctue egeneation Definition 1: In G(; n, k) = {V ( + 1), E( + 1), ω}, an (, ) egeneation tee is a tee whose oot is V 0 an coves povies in V ( + 1), k In an (, ) egeneation tee, the non-leaf povies eceive ata fom thei chilen noes, encoe the eceive ata with the ata they stoe, an elay the encoe ata to thei paent noes byte-by-byte By the elay of povies, the newcome will get a linea combination of coe blocks of povies, though it may pobably connect to fewe than povies iectly On each ege in the egeneation tee, k bits of ata ae tansmitte The bottleneck ege is the least weighte ege in the (, ) egeneation tee In oe to get an optimal (, ) egeneation tee, which has the maximum bottleneck banwith, we can use Pim s algoithm, which constucts a maximum spanning tee stating fom the oot inuctively If the oot has been selecte, in the th step of Pim s algoithm, thee ae + 1 noes in the pouce tee, whose bottleneck banwith is optimal among all (, ) egeneation tees in G(; n, k) Theoem 1: Afte the th step, Pim s algoithm can pouce an optimal (, ) egeneation tee in G(; n, k) Poof: When = 1, the poof is clea The Pim s algoithm will select the maximum ege incient to V 0 at the fist step Suppose this statement is tue when = k 0, 0 < k 0 < Afte the k0 th step, an optimal (k 0, ) egeneation tee T k0 is pouce Assume that a (k 0 + 1, ) egeneation tee T k0+1 is pouce afte the (k 0 + 1) th step, an e is the selecte ege at the (k 0 + 1) th step If ω(e) B(T k0 ), T k0+1 is an optimal (k 0 + 1, ) egeneation tee, othewise it will contaict with that T k0 is an optimal (k 0, ) egeneation tee If ω(e) < B(T k0 ), clealy B(T k0 ) > B(T k0+1) Assume that thee exists a tee T k with oot V 0+1 0, an B(T k ) > 0+1 B(T k0+1) Removing one of the leaf noes except V 0 an the ege incient to this noe in T k, we obtain anothe tee T 0+1 k 0, an thus B(T k0 ) B(T k 0 ) B(T k ) > B(T 0+1 k 0+1) If the numbe of the noes in T k0 T k 0 is moe than k 0 + 1, then the weight of the bottleneck ege of the (k 0 + 1, ) egeneation tee constucte by Pim s algoithm in T k0 T k 0 is no less than B(T k 0+1 ), an thus must be moe than B(T k0+1) This contaicts with the fact that B(T k0+1) is constucte by Pim s algoithm If the numbe of the noes in T k0 T k 0 is exactly k 0 + 1, the noe set of T k0 is the same with that of T k 0 Since the uniqueness of the ege weight leas to the uniqueness of the maximum spanning tee, we have T k0 = T k 0 By Pim s algoithm, we thus have B(T k ) = B(T 0+1 k 0+1) This leas to the contaiction Fom the poof of Theoem 1, we can easily get the following coollay, which eveals the stategy of eciing the numbe of povies in TREE Coollay 1: Given G(; n, k), the bottleneck banwith of an optimal ( + 1, ) egeneation tee is no bette than an optimal (, ) egeneation tee Fig 2 shows the output of Pim s algoithm on the netwok moel in Fig 1(a) The fou steps coespon to an optimal (, 4) egeneation tee, = 1, 2, 3, 4, whose bottleneck banwith ae 55bps, 50bps,, an, espectively We can see the employment of moe povies will not impove the bottleneck banwith in TREE C Bottleneck Banwith of the (, ) Regeneation Tee Since we have obtaine the optimal (, ) egeneation tee by Pim s algoithm, we analyze its bottleneck banwith in this section We epesent the bottleneck ege in the optimal (, ) egeneation tee by its sequential inex in the ege set Base on the pobability of the sequential inex an the expecte banwith of the ege with the coesponing sequential inex, we can obtain the expecte bottleneck banwith

4 4 1 i ( 1) + 1, whee 50bps 10bps 50bps 10bps 50bps 10bps (a) step 1 (b) step 2 50bps 10bps (c) step 3 () step 4 Fig 2: Regeneation tees afte 1 st 4 th steps of Pim s algoithm on the netwok moel in Fig 1(a) The bottleneck banwith eceases with the incease numbe of povies Definition 2: Let e be the bottleneck ege of an optimal (, ) egeneation tee in G(; n, k), pouce by Pim s algoithm If e is the i th maximum ege in E( + 1), σ TREE (, G(; n, k)) = i The following popety gives the uppe an lowe bouns of σ TREE (, G(; n, k)) Popety 1: σ TREE (, G(; n, k)) +1 +1, whee = ( 1) 2 Poof: Thee ae eges in the (, ) egeneation tee, so σ TREE (, G(; n, k) Because G(; n, k) is a complete gaph, it is -ege-connecte G(; n, k) will still be connecte afte emoving 1 eges, so σ TREE (, G(; n, k)) The following lemma shows the pobability of σ TREE (, G(k; n, k)), a special case of the netwok moel that the numbe of povies is exactly k Lemma 1: [7] Let Q(l, j) enote the numbe of the connecte gaphs which contain l labele noes an j eges, an P(k+1, i) enote the pobability that σ TREE (k, G(k; n, k)) = i Thus in G(k; n, k) = (V (k + 1), E(k + 1), ω), if k < i < k+1 k + 1, P(k + 1, i) = k C l 1 i 1 k 1 l=1 j=0 Q(l, j)q(k + 1 l, i 1 j) C i 1 k+1 1, (1) an 0 j < l 1; j Q(l, j) = C j l P(l, i) l 1 j l ; (2) i=l 1 0 j > l Base on Lemma 1, we show the pobability of σ TREE in the geneal moel, G(; n, k) Theoem 2: Let p( + 1, + 1; i) be the pobability that σ TREE (, G(; n, k)) = i We have p(, ; i) = 1 2 l=1 C l 1 1 2( 2) 2R(,,i, l) + C i 1 1 l=2 C l 2 3R(,, i, l), (3) R(,, i, l) = i 1 i 1 =l 1 Q(l, i 1) l t= l C t 1 l 1 i 1 i 1 i 2 =t 1 Q(t, i 2)C i 1 i 1 i 2 l t (4) Poof: In G( 1; n, k), given an optimal ( 1, 1) egeneation tee pouce by Pim s algoithm, we assume that (V a, V b ) is its bottleneck ege an it is the i th maximum ege in E(), 0 a < b < Removing all the eges in E() whose weight is less than (V a, V b ), we can see that p(, ; i) equals the pobability that any i eges connecting noes in V () can fom an ( 1, 1) egeneation tee If the position of (V a, V b ) has been etemine, appaently the numbe of ways to select othe i 1 eges in E() is C i 1 1 We now consie the numbe of egeneation tees The pobability of a = 0 is 2 Divie V into two goups, V (a) an V (b) Let V a belong to V (a) an V b belong to V (b) Assume V (a) contains l noes, 1 l 1 Since V 0 an V b belong to V (a) an V (b), espectively, the numbe of ways to assign othe 2 noes into the two goups is C l 1 2 On the othe han, the pobability of a 0 is 2 Howeve, now thee ae two possibilities that V a can belongs to eithe V (a) o V (b) oeove, since now V a V 0, we have 2 l 1 As we nee to assign othe 3 noes into the two goups, thee ae C l 2 3 ways to achieve this Since we have etemine the noes in V (a) an V (b), we nee to assign othe i 1 eges except (V a, V b ) into V (a) an V (b) to fom an ( 1, 1) egeneation tee Suppose thee ae R(,, i, l) ways to achieve this, we get Eq (3) Now we consie R(,, i, l) Without loss of geneality, we assume that V 0 V (a) Assume thee ae i 1 eges in V (a) an i 1 i 1 eges in V (b) The i 1 eges in V (a) have to constuct a spanning tee on V (a) By Eq (1), the numbe of ways to achieve this is Q(l, i 1 ) Fo the i 1 i 1 eges in V (b), an spanning tee has to be constucte ove V b an at least othe t 1 noes in V (b), l t l, othewise the ( 1, 1) egeneation tee can not be constucte by Pim s algoithm Thee ae C t 1 l 1 ways to select the noes covee by the spanning tee, since V b has been selecte Assign i 2 eges into these t noes, t 1 i 2 i 1 i 1, an thee ae Q(t, i 2 ) possibilities by Eq (1) Fo the emaining l t noes an i 1 i 1 i 2 eges, just assign such numbe of eges as the eges connecting these l t noes an the numbe of ways is C i 1 i1 i2 l t In summay, R(,, i, l) = i 1 i 1=l 1 Q(l, i 1 ) l C t 1 i 1 i 1 l 1 t= l i 2=t 1 Q(t, i 2 )C i 1 i1 i2 l t When = = k, Eq (1) is a special case of Eq (3) Let E (i:+1 ) enote the expecte banwith of the i th maximum ege in E() We can obtain the expecte bottleneck banwith of the optimal (, ) egeneation tee in G(; n, k)

5 5 by Eq (5): E TREE(,G(; n, k)) = i= p( + 1, + 1; i)e (i:+1) (5) IV REGENERATION WITH REGENERATING CODES A Regeneating Coes In Sec III, we show a geneal analysis of the egeneation tee using (n, k) DS coes Though the bottleneck banwith can be impove if we have moe stoage noes as the caniates of povies, employing moe povies oes not povie a substantial impovement Fist, accoing to Coollay 1, the incease numbe of povies oes not elieve the bottleneck futhe Secon, it incus moe netwok taffic in the egeneation, because moe eges ae employe in the (, ) egeneation tee an the taffic on each ege has not been euce Theefoe, we popose RCTREE, combining minimum-stoage egeneating (SR) coes with the teestuctue egeneation (TREE) Compae with DS coes, SR coes can euce the egeneation taffic Since the numbe of povies is vaiable in this pape, the coing scheme of SR coes shoul aapt to this Fo the egeneation with povies, the file shoul be ivie into at least k( k + 1) blocks to achieve the lowe boun of egeneation taffic [6] Assume that the oiginal file ae ivie into L blocks, an is the maximum intege that satisfies k( k + 1) L L shoul be lage enough so that k, ie, thee can be at least k povies in the egeneation Each stoage noe stoes L k coe blocks In the egeneation with povies, k, fo RCTREE, each povie encoes its L L k coe blocks into k k+1 blocks an then sens them to its paent noe Then the newcome L eceives a total of blocks an finally encoes them k k+1 into L k blocks, so the newcome has to eceive ata iectly fom at least k + 1 povies The taffic on each link is k( k+1) bits appoximately if L is lage enough Definition 3: An (,, k) egeneation tee in G(; n, k) is a tee with oot V 0 whose egee is at least k + 1, an coves povies in V ( + 1), k Algoithm 1 shows how to get an optimal (,, k) egeneation tee We fist constuct an optimal (, ) egeneation tee by Pim s algoithm (Line 1 Line 5) If the egee of the (, ) egeneation tee is invali, we ajust the eges in the tee by aing the ege in E oot inuctively (Line 6 Line 10) B Bottleneck Banwith of the Optimal (,, k) egeneation tee In this section, we iscuss the bottleneck banwith of the optimal (,, k) egeneation tee pouce by Algoithm 1 Simila to Definition 2, we give the efinition of σ RCTREE of the optimal (,, k) egeneation tee in G(; n, k) Algoithm 1 Fin an optimal (,, k) egeneation tee T in G(; n, k), k Define E oot = {(V 0, V i ) i = 1, 2,, }, an D(T) = E oot {eges in T } 1: T 2: fo i 1 to o 3: e i the lagest ege making T {e i } a oote tee 4: T T {e i } 5: en fo 6: fo i D(T) + 1 to k + 1 o 7: e 1 the lagest ege E oot T 8: e 2 any ege T E oot making T {e 1 } {e 2 } a tee oote by V 0 9: T T {e 1 } {e 2 } 10: en fo Definition 4: Let e be the bottleneck ege of an optimal (,, k) egeneation tee in G(; n, k), pouce by Algoithm 1 If e is the i th maximum ege in E( + 1), σ RCTREE (, G(; n, k)) = i Notice that an (,, k) egeneation tee is still an (, ) egeneation tee By Popety 1, σ RCTREE (, G(; n, k)) Now we show the pobability of σ RCTREE (, G(; n, k)) In Algoithm 1, some eges may be ae into the optimal (, ) egeneation tee if the egee constaint of the oot is not satisfie (Line 6 Line 10) We fist iscuss whethe this will ecease the bottleneck banwith of the egeneation tee Lemma 2: Let C n (k1,k2) = n! k, 0 k 1!k 2! 1, k 2 n Let p(+ 1, + 1, c; i) be the pobability that σ RCTREE (, G(; n, k)) = i an c eges in E oot have weights moe than the i th ege in E( + 1) p(,, c; i) = 2( 2) 1 l=2 C l 2 3 S(,, i, l, c) C i 1 1 l=1 1 i ( 1) + 1, whee S(,,i, l, c) = an Q (l, j, c) = i 1 i 1 =l 1 Q (l, i 1, c) l t= l i 1 i 1 i 2 =t 1 C l 1 2S(,, i, l, c 1) C t 1 l 1, (6) Q(t, i 2)C i 1 i 1 i 2 l t, (7) 1 l = 1, j = c = 0; j c p(l,l, c, i) C(l 1 c,j) l C c c j i C l 1 c i l 1 c =max{1,c+i j} l i 1 c l 1, l 2 j c l 1 ; 0 othewise (8) Poof: Eq (6) can be pove similaly with the poof of Eq (3) In G( 1; n, k), given an optimal ( 1, 1, k) egeneation tee pouce by Algoithm 1, its bottleneck ege (V a, V b ) is the i th maximum ege in E(), 0 a < b < We emove all the eges in E() whose weight is less than (V a, V b ) Divie V into two goups, V (a) an V (b)

6 6 Let V a belong to V (a) an V b belong to V (b) Assume V (a) contains l noes Let S(,, i, l, c) epesent the numbe of ways of assigning i 1 eges into V (a) an V (b) to fom an ( 1, 1, k) egeneation tee Thus eplacing R(,, i, l) by S(,, i, l, c) in Eq (3), we can obtain the poof of Eq (6) Let V (a) an V (b) contain V a an V b, espectively Without loss of geneality, we assume that V (a) contains V 0 Note that V a may equal V 0 We efine Q (l, j, c) as the numbe of connecte gaphs on V (a) (l noes with one given oot V 0 ) with j eges in which thee ae c eges in E oot having weights lage than the i th ege in E() Thus Eq (7) can be obtaine fom Eq (4) by eplacing Q(l, i 1 ) with Q (l, i 1, c) Now we pove Eq (8) We efine Q (l, j, c) = 1 when l = 1 an j = c = 0 Othewise Q (l, j, c) > 0 if an only the following two conitions ae both satisfie: 1) Since thee ae l noes in V (a), thee shoul be at most l 1 eges with weights no less than the i th maximum ege in E eanwhile, in oe to guaantee the connectivity of the (,, k) egeneation tee, thee is at least one ege in E oot with weight lage than the i th maximum ege in E, ie, 1 c l 1 2) Noes in V (a) V 0 ae connecte Thus l 2 j c l 1 When the two conitions above ae satisfie, all the l noes ae connecte, so l 1 j l Assume that the bottleneck ege of the maximum spanning tee in such a gaph satisfying the conitions above is the i th lagest ege among the j eges, an the egee of the oot in the maximum spanning tee is c We have max{1, c + i j} c c The numbe of such kin of maximum spanning tees is C i l p(l, l, c, i) Thee ae still j i eges to be assigne into the gaph Among the j i eges, c c eges shoul be assigne to connect the oot (l 1 c caniates of positions), an j i c + c eges to be assigne to connect the non-oot noes ( l i l + c + 1 caniates of positions) Thus the numbe of gaphs is C i l p(l, l, c, i)c c c l 1 c C j i c+c l i l+c +1 Because j i c + c 0, c c + i j, Q (l, j, c) = j c i=l 1 max{1,c+i j} j c i l 1 c =max{1,c+i j} C i l p(l, l, c, i)c c c l 1 c C j i c+c l i l+c +1 = p(l, l, c, i) C(l 1 c,j) l C l 1 c l i C c c j i Base on Lemma 2, we can obtain the pobability of σ RCTREE (, G(; n, k)), by simply checking whethe thee ae enough non-selecte eges in E oot so that the bottleneck banwith will not be affecte Theoem 3: Let p RCTREE(k) (+1, +1; i) be the pobability that σ RCTREE (, G(; n, k)) = i in G(; n, k) p RCTREE(k) (,; i) = k 1 1 c= k j<i p(,, c; i)+ p(,,c; j) C k c 1 i 1 j C 1 +k i C 1 c j (9) Poof: In G( 1; n, k), accoing to Algoithm 1, if the optimal ( 1, 1) egeneation tee pouce by Pim s algoithm (Line 1 Line 5) is also an ( 1, 1, k) egeneation tee, o thee ae enough non-selecte eges in E oot with weights lage than the i th eges in E, the bottleneck ege of the optimal ( 1, 1) egeneation tee will still be the bottleneck ege of the optimal ( 1, 1, k) egeneation tee These cases occu with pobability 1 p(,, c; i) c=( 1) k+1 Howeve, if the egee of the oot of the ( 1, 1) egeneation tee pouce by Pim s algoithm is less than k, an thee is no enough non-selecte ege to be ae into the tee (Line 7 Line 9 in Algoithm 1), the bottleneck banwith of the optimal ( 1, 1, k) egeneation tee is less than that of the optimal ( 1, 1) egeneation tee Assume that thee ae c eges in E oot with weights lage than the i th maximum ege in E, 1 c k 1 Because of the egee constaint of the oot, k c eges with weights less than the i th maximum ege shoul be ae into the optimal ( 1, 1) egeneation tee If the minimum ege ae is the j th ege in E, j > i, this is equivalent to selecting 1 c eges, in which the ( k c 1) th ege is the j th ege in E oot, fom a total numbe of j eges This pobability is C k c 1 i 1 j C 1 +k i C 1 c j kin of cases is k 1 j<i Thus, the pobability of this p(,, c; j) C k c 1 i 1 j C 1 c j C 1 +k i Simila to Eq (5), we obtain E RCTREE (, G(; n, k)), the expecte bottleneck banwith of the optimal (,, k) egeneation tee in G(; n, k): E RCTREE(, G(; n, k)) = i= C Quantitative Results p RCTREE(k) ( + 1, + 1; i)e (i:+1) (10) In this section, we compae the egeneation schemes of STAR, TREE, an RCTREE by a quantitative evaluation We assume that in G(; k, n) = (V ( + 1), E( + 1), ω), ω, the weight of the ege in E(+1), satisfies a unifom istibution U[03bps, 120bps], which eveals the banwith capacity between noes in PlanetLab [9] By the theoy of oe statistics [10], we obtain the value of E (i:k+1 ) une the istibution of U[a, b], whee a = 03bps, an b = 120bps: E (i:k+1) = (b a)( k+1 i + 1) k a (11) Since the bottleneck ege of STAR with povies in G(; n, k) shoul be the th maximum ege in E oot, we obtain its bottleneck banwith by Eq (12): E STAR(,G(; n, k)) = (b a) a (12) We compae B(, G(; n, k)), the vitual bottleneck banwith of STAR, TREE, RCSTAR (STAR with

7 7 vitual bottleneck banwith (bps) B RCTREE (,G(15;n,3)) B RCSTAR (,G(15;n,3)) B TREE (,G(15;n,3)) B TREE (,G(;n,3)) B STAR (,G(15;n,3)) (#use povies) Fig 3: Vitual bottleneck banwith of five egeneation schemes egeneating coes), an RCTREE The egeneation time is k B(,G(;n,k)) Fo STAR an TREE, B STAR(, G(; n, k)) = E STAR (, G(; n, k)) an B TREE (, G(; n, k)) = E TREE (, G(; n, k)) Fo RCSTAR an RCTREE, B RCSTAR (, G(; n, k)) = ( k + 1)E STAR (, G(; n, k)) an B RCTREE (, G(; n, k)) = ( k + 1)E RCTREE (, G(; n, k)), because the amount of tansfee ata on each ege in 1 G(; n, k) is bits Fig 3 shows the evaluation esult in G(15; n, 3), n 15 is the numbe of povies With the powe of egeneating coes, the vitual bottleneck banwith of RCTREE an RCSTAR is impove significantly, compae with TREE an STAR On the othe han, even though the netwok taffic on each ege is euce by egeneation coes, the vitual bottleneck banwith of RCTREE an RCSTAR can not incease monotonically Fo RCTREE, its topology is constaine by the egee of the oot Fo RCSTAR, moeove, its bottleneck banwith eceases with the incease numbe of povies, since it is base on STAR When = 10(9), the cuve of the vitual bottleneck banwith of RCTREE (RCSTAR) eaches its peak When = 10, the vitual bottleneck banwith of RCSTAR, TREE, an STAR ae 75%, 22%, an 9% of RCTREE, espectively RCTREE outpefoms all othe schemes by combining teestuctue egeneation, which utilizes high-banwith links moe efficiently, with egeneating coes, which euces the egeneation taffic significantly k+1 k V LIFETIE OF REGENERATION TREES A Regeneation with Noe Depatues We have analyze the tee-stuctue egeneation an its combination with egeneating coes Howeve, we have not consiee that noes may leave uing the egeneation Fig 4 shows some examples of noe epatues uing the egeneation In Fig 4, Case 1, 2, an 3 ae thee examples that a leaf noe, a non-leaf noe, an the newcome in a egeneation tee leave the netwok, espectively In Case 1, afte the leaf noe leaves the netwok, a egeneation tee with 3 povies emains In Case 2, afte the non-leaf noe leaves the netwok, all its chilen noes shoul be egae as leaving the netwok, because the ata can not be tansfee to V 0 until anothe egeneation tee has been constucte In Case 3, the newcome (a) Case 1: a leaf noe leaves (b) Case 2: a non-leaf noe leaves (c) Case 3: the newcome leaves Fig 4: Examples of noe epatues in a egeneation tee leaves the system Appaently the egeneation fails, since no ata can be egeneate at the newcome any moe Since STAR can be egae as a special fom of TREE, an RCTREE constucts a egeneation tee with the egee constaint of the oot, we iscuss the continuous tansmission time in the egeneation tee with noe epatues Assume that in a egeneation tee with povies, V 0 is still connecte with 0 povies afte a noe leaves the netwok The connecte component containing V 0 is a subtee of the oiginal egeneation tee Fo RCTREE, the subtee also satisfies the egee constaint of the oot because the egee of V 0 eceases by one at most If 0 k, the egeneation tee is still alive, because the newcome can still eceive ata fom at least k povies Definition 5: The lifetime of a egeneation tee in G(; n, k) is the time between when the tee is constucte an when less than k povies emain in the subtee We assume that noes o not leave simultaneously in the egeneation In the egeneation tee, we assume that all the noes may leave the netwok with the same pobability All the noes ae awae of the epatues of thei chilen noes Specifically, when the paent noe oes not eceive ata fom one of its chilen noes, it egas this noe as a teminate noe an stops the ata tansmission If the eunancy is coe by egeneating coes, the encoing coefficients of the egeneating coes may change with the epatues of the povies Diviing coe blocks into geneations with suitable size may solve this poblem B Lifetime of STAR an TREE Assume that in a egeneation tee with noes, t noes emain afte one noe leaves the netwok This occus with pobability P (, t) The lifetime is the time the egeneation tee keeps stable plus the lifetime of the emaining subtee with t noes Then we obtain a ecusion of lifetime Fist we consie the value of P(, t) Lemma 3: Fo STAR, { 1 t = 0; P STAR (,t) = 0 0 < t < 1; (13) 1 t = 1

8 8 Poof: When the newcome leaves the netwok, t = 0 Since any noe may leave the netwok with the same pobability, P STAR (, 0) = 1 Othewise, t can only become t 1, because any povies ae leaf noes in STAR Thus P(, 1) = 1 1 Lemma 4: Fo TREE, { 1 t = 0; P TREE (, t) = 1 C t 1 t t 1 ( t) t < t < 2 (14) Poof: When t = 0, the poof can be seen in Lemma 3 When 0 < t <, let e be the ege connecting the leaving noe with its paent noe The epatue of this noe can be egae as emoving e fom the egeneation tee Removing e will ivie the egeneation tee into two subtees Thee ae C 2 t 1 ways to select t noes in the subtee containing the newcome V 0, because V 0 an the leaving noe have been selecte By Cayley s fomula [11], the numbe of spanning tees on n labele noes is n n 2 Thus the numbe of egeneation tees which will have t povies emaining afte emoving e is t t 2 ( t) t 2 t Since the numbe of egeneation tees ove C 2 t 1 tt 1 ( t) t 2 2 noes is 2, P TREE (, t) = 1 Compaing Theoem 3 with Theoem 4, we can see STAR can be moe stable than TREE, because if the newcome oes not leave, the egeneation tee will only lose at most one povie in STAR, but it may lose moe than one povies in TREE, if a non-leaf povie leaves Now we give the ecusion of lifetime fo STAR an TREE Theoem 4: Let L() be the expecte lifetime of a egeneation tee with povies E() is the expecte time that all the noes emain in the egeneation tee In G(; n, k), fo STAR, t= 1 L STAR(k) ( 1) = E() + P STAR (,t)l STAR(k) (t 1) (15) t=k+1 Fo TREE, eplace L STAR(k) ( 1) an P STAR (, t) with L TREE(k) ( 1) an P TREE (, t), espectively Poof: Fo a egeneation tee in STAR o TREE with noes ( 1 povies), the expecte time that the tee keeps stable is E() When a noe leaves, the expecte lifetime of the subtee is L(t 1) if t noes emain, k + 1 t < We get the expecte lifetime by aing E() with the expecte lifetime of the emaining subtee C Lifetime of RCTREE Now we iscuss the lifetime of RCTREE We also fin a ecusion of lifetime, by iscussing how many povies emain afte the noe epatue We fist intouce a lemma as follows Lemma 5: [11] Ove n labele noes in which k noe have been esignate as oots, the numbe of foests containing k oote tees is kn n k 1 Coollay 2: Ove n labele noes in which one noe has been esignate as oot, the numbe of spanning tees in which the egee of the oot is k, is T(n, k) = Cn 2 k 1 (n 1)n k 1, 1 k < n Poof: Given a spanning tee in the statement, we can emove the oot noe to make it become a foest with k tees Thus the numbe of spanning tees in the statement equals the numbe of ways to select k noes fom n 1 noes, multiplying the numbe of such foests by Lemma 5, ie, Cn 1 k k(n 1)n 1 k 1 = Cn 2 k 1 (n 1)n k 1 Specifically, if n = 1, let T(n, k) = 1 when k = 0, an T(n, k) = 0 othewise Lemma 6: In G(; n, k), given an ( 1,, k) egeneation tee in which the egee of V 0 is c, afte a noe epatue, a subtee with t noes emains The egee of V 0 is still c with pobability P0 RCTREE (, t, c), an become c 1 with pobability P1 RCTREE (, t, c) When 0 < t <, P RCTREE 0 (,t, c) = 1 P RCTREE 1 (,t, c) = 1 C t 1 2 (t 1)T(t, c)( t) t 2 (16) T(,c) C t 1 2 T(t,c 1)( t) t 2 T(,c) (17) when c = t = 1 o 1 < c 1, an equals 0 othewise Poof: To pove this lemma, we efe to the poof of Lemma 4 The poofs of Eq (16) an Eq (17) ae the same with the poof of Eq (14) except fo thee points Fist, since V 0 in the emaining subtee afte emoving e satisfies the egee constaint, the numbes of such subtees ae T(t, c) an T(t, c 1), espectively Secon, fo P0 RCTREE (, t, c), e can connect to any t 1 noes in the subtee, but fo P RCTREE 1 (, t, c), e can only connect to V 0 Thi, ove labele noes, thee ae T(, c) egeneation tees whee the egee of V 0 is c oeove, fo P RCTREE 1 (, t, c), when c = 1, it is impossible that t > 1, because if the only noe connecting V 0 leaves, only the newcome will emain in the subtee Simila to Theoem 4, we obtain the following lemma Lemma 7: Let L RCTREE(k) (, c) be the lifetime of an (,, k) egeneation tee in which the egee of V 0 is c, c k + 1 L RCTREE(k) ( 1, c) = E()+ t= 1 [P RCTREE 0 (, t, c)l RCTREE(k) (t 1, c)+ t=k+1 P RCTREE 1 (,t, c)l RCTREE(k) (t 1, c 1)] (18) Theoem 5: Given an optimal (,, k) egeneation tee pouce by Algoithm 1 in G(; n, k), its lifetime is L RCTREE(k) () = + k T( + 1, c) L RCTREE(k) (, k + 1) T( + 1, c) = k+1 T( + 1, ) L RCTREE(k) (, ) (19) T( + 1, c) Poof: In Algoithm 1, if the egee of V 0 in the optimal (, ) egeneation tee pouce by Pim s algoithm (Line 1

9 9 Line 5) is invali, this happens with pobability k T(+1,c) T(+1,c) Then the eges in E oot will be ae to the tee until the egee of the oot is k+1, so the lifetime is L RCTREE(k) (, k+1) On the othe han, if the egee of V 0 is, k+1, which happens with pobability T(+1,), the lifetime is L RCTREE(k) (, ) D Compaison T(+1,c) If the active time of a noe in the netwok between one join an one epatue satisfies an exponential istibution exp(1/λ), E() = λ We let λ = secons accoing to the use behavios in PlanetLab [12], an then obtain the expecte lifetime of STAR, TREE, an RCTREE in G(15; n, 3), n 15, illustate by Fig 5 expecte life time L() (sec) x 105 STAR(3) TREE(3) RCTREE(3) (#use povies) Fig 5: Expecte lifetime of STAR, TREE, an RCTREE in G(15; n, 3), n 15 In Fig 5, the lifetime of all thee schemes inceases with, because moe povies can esist bette towas noe epatues STAR has the best lifetime, because the epatue of one povie will not incu the loss of any othe povies Howeve, fo TREE, since the epatue of one povie usually leas to the loss of some othe povies in the egeneation tee, its lifetime is less than 60% of STAR when 7 Due to the egee constaint of V 0, RCTREE is much moe stable than TREE When is lage enough, RCTREE is quite simila to STAR fom the pespective of the egee of V 0 When 8, the lifetime of RCTREE is moe than 90% of STAR an still continues appoaching STAR Since the cuve of the vitual bottleneck banwith of RCTREE eaches its peak in Fig 3 when = 10, an the vitual bottleneck banwith is impove significant by RCTREE compae with STAR, we can ignoe the mino impovement of lifetime of STAR in pactice its bottleneck banwith of the tee-stuctue egeneation with a vaiable numbe of povies (TREE) Base on this analysis, we iscuss the tee-stuctue egeneation combine with egeneating coes (RCTREE) an analyze its bottleneck banwith Consieing the noe chun in istibute stoage systems, we make an analysis of the lifetime of TREE, RCTREE, an the conventional sta-stuctue egeneation (STAR) Ou analysis esults show that RCTREE is not only the fastest scheme, but also a vey stable scheme Theefoe, RCTREE is suitable fo istibute stoage systems, especially fo the system with a substantial egee of banwith heteogeneity In ou futue wok, we will valiate the theoetical avantage of RCTREE by eal-platfom base simulations an expeiments ACKNOWLEDGENT We thank the anonymous eviewes fo thei helpful avices This wok was suppote in pat by NSFC une Gant No , Shanghai unicipal R&D Founation une Gant No , the Shanghai Rising-Sta Pogam une Gant No 08QA14009, NSERC Discovey Gant RGPIN an NSERC Stategic Gant STPGP Xin Wang is the coesponing autho REFERENCES [1] R Bhagwan, K Tati, Y-C Cheng, S Savage, an G Voelke, Total ecall: system suppot fo automate availability management, in Poc of the 1st Symposium on Netwoke Systems Design an Implementation (NSDI) Bekeley, CA, USA: USENIX Association, 2004 [2] A Dimakis, P Gofey, Wainwight, an K Ramchanan, Netwok coing fo istibute stoage systems, in Poc of INFOCO, pp , ay 2007 [3] R Roigues an B Liskov, High availability in hts: Easue coing vs eplication, in Poc of 4th Intenational Wokshop on Pee-to-Pee Systems (IPTPS), 2005 [4] Y Wu, R Dimakis, an K Ramchanan, Deteministic egeneating coes fo istibute stoage, Alleton Confeence on Contol, Computing, an Communication, 2007 [5] S Aceański, S Deb, éa, an R Koette, How goo is anom linea coing base istibute netwoke stoage? in Poc of 1st Wokshop on Netwok Coing, WiOpt, Ap 2005 [6] A Duminuco an E W Biesack, A pactical stuy of egeneating coes fo pee-to-pee backup systems, in Poc of IEEE Intenational Confeence on Distibute Computing Systems (ICDCS), Jun 2009 [7] J Li, S Yang, X Wang, X Xue, an B Li, Tee-stuctue ata egeneation with netwok coing in istibute stoage systems, in Poc of 17th IEEE Intenational Wokshop on Quality of Sevice (IWQoS), 2009 [8] D Leong, A G Dimakis, an T Ho, Distibute stoage allocation poblems, in Poc of Fifth Wokshop on Netwok Coing, Theoy an Applications (NetCo), 2009 [9] S-J Lee, P Shama, S Banejee, S Basu, an R Fonseca, easuing banwith between PlanetLab noes, Passive an Active Netwok easuement, pp , 2005 [10] H A Davi an H N Nagaaja, Oe Statistics, 3 e Wiley- Intescience, Aug 2003 [11] A Cayley, A theoem on tees, Quat J ath 23 (1889), pp [12] J Stibling Planetlab all pais ping [Online] Available: VI CONCLUSION In this pape, we aess challenges in constucting the egeneation tee in istibute stoage systems with egeneating coes We fist analyze the constuctive algoithm an