The Fundamental Theorem of Distributed Storage Systems Revisited

Transcription

1 The Fundamental Theorem of Dstrbuted Storage Systems Revsted Png Hu Department of Electronc Engneerng Cty Unversty of Hong Kong Emal: Kenneth W. Shum Insttute of Networ Codng the Chnese Unversty of Hong Kong, Shenzhen Key Lab. of Networ Codng Key Tech. and Applcaton, Chna. Emal: Ch Wan Sung Department of Electronc Engneerng Cty Unversty of Hong Kong Emal: Abstract The fundamental theorem of dstrbuted storage systems characterzes the maxmum fle sze that can be stored wth certan assumptons on fle retreval and node repar. The result s composed of two parts, namely, the mn-cut bound and that the bound can be acheved by lnear networ code wth bounded feld sze. The dervaton of the mn-cut bound s reexamned and llumnated by mang an mplct step explct. Furthermore, a smple alternatve proof for the achevablty of the mn-cut bound s presented, whcs based on the constructon of the generc storage code, a restrcted form of generc networ code. The proof technques n ths paper are expected to be extensble to other more complex models of dstrbuted storage systems. I. INTRODUCTION Dstrbuted storage systems (DSS) have attracted a lot of research attenton n recent years, partly because of the successful applcaton of networ codng theory to the desgn of DSS wth repar consderatons. In the semnal wor of [1], the dynamcs of node falures and repars n DSS s modeled as a sngle-source multcast acyclc networ, and the fundamental tradeoff between storage effcency and repar bandwdtn DSS s revealed by characterzng the mnmum-cut of the correspondng nformaton flow graph. Le the study of any communcaton channel or networ, the fundamental queston to as n relaton to DSS s ths: What s ts capacty n the nformaton-theoretc sense? The answer to ths type of questons naturally conssts of two parts, an upper bound of the capacty and the achevablty of the bound. The wor of [1] gves the frst part of the answer by dervng the mn-cut bound for the nformaton flow graph of DSS and provde a partal answer to the second by usng results from networ codng theory, whch can be appled to DSS wth a bounded number of node falures and repars. A more complete answer to the second part s gven n [2], whch proves that the mn-cut bound can be acheved by a lnear networ code wth a feld sze whch depends only on Ths wor was partally supported by a grant from the Unversty Grants Commttee of the Hong Kong Specal Admnstratve Regon, Chna (Project No. AoE/E-02/08), and Shenzhen Key Laboratory of Networ Codng Key Technology and Applcaton (ZSDY ), and Peaco Scheme (KQCX ), Shenzhen, Chna. The authors are lsted alphabetcally. the number of storage nodes, but not on the number of falures and repars that occur n the system. The contrbuton of ths paper s two-fold. Frst, we prove a combnatoral result (n Lemma 1) about the sze of the boundary of a cut n the nformaton flow graph. Ths result s crucal n computng the mn-cut bound, but s assumed mplctly n the proof of the mn-cut bound n [1]. We rederve the mn-cut bound on the capacty of a DSS n the frst part of ths paper. Second, we provde an alternatve proof for the achevablty of the fundamental tradeoff between storage and repar bandwdth by lnear networ codes n [2]. The proof of [2] reles on a path-weavng procedure, whch s hghly dependent on the structure of the nformaton flow graph. In ths paper, we gve a determnstc algorthm for constructng a lnear regeneratng code, based on a restrcted form of generc networ code [3]. We call t a generc storage code. The procedure of constructng a generc storage code s conceptually smpler than the one n [2], but requres a larger feld sze. Our constructon s easly transplantable to other models n dstrbuted storage systems wth dfferent assumptons. We remar that the results n ths paper cannot be drectly obtaned from the orgnal lnear networ codng theory [3], [4], whch consders multcast networs that are fnte and fxed n advance. For DSS, the flow graps unbounded because there s no lmt on the number of node falures, but the results n [3], [4] assume fnte networs and fntely many recevng nodes. Our result, as well as that n [2], shows the exstence of a capacty-achevng lnear code, whose feld sze grows exponentally on the number of storage nodes. It s an open queston whether the feld sze can be reduced to polynomal growth. II. DISTRIBUTED STORAGE SYSTEMS In a dstrbuted storage system (DSS), there are n storage nodes. A fle of B symbols from an alphabet set Σ has to be stored n the DSS. Ths fle can later be retreved by a data collector, whch connects to any storage nodes. Each storage node s able to store α symbols. These storage nodes are not relable and may fal after prolonged use. If a node fals, a replacement node, called newcomer, s created by /14/$ IEEE 65

2 downloadng data from any d of the survvng nodes. These d nodes are called helper nodes, and the number of symbols a newcomer downloads from each of them s denoted by β. The total number of symbols downloaded, dβ, s called the repar bandwdth. The newcomer, just le the orgnal faled node, s allowed to store at most α symbols. The newcomer s not requred to store exactly the same symbols as the orgnal faled node. Ths repar mechansm s commonly called functonal repar. A DSS that satsfes the above requrement s denoted by DSS(n,, d, α, β). As tme evolves, nodes fal and are replaced by newcomers. The partcular falure order of the storage nodes and the choce of helper nodes for each newcomer defnes an nstance of a DSS. Each DSS nstance can be represented by a drected acyclc graph (DAG), called nformaton flow graph, whch conssts of three types of vertces: a source node S, storage nodes, In s and Out s, and data collectors DC s. For = 1, 2,..., n, there s a drected edge from S to In wtnfnte capacty, and a drected edge from In to Out wth capacty α. When the -th node falure n the DSS occurs, where = 1, 2,..., a correspondng newcomer jons the system. We denote t by node, where = n +. As the orgnal storage nodes, t s represented by In and Out, whch are connected by a drected edge from the former to the latter wth a capacty of α. Furthermore, In s connected to d helper nodes by addng a drected edge from each of Out 1, Out 2,..., Out d to In, where j < for j = 1, 2,..., d. For all possble choces of survvng storage nodes, a correspondng DC s added. Let the nodes to whch DC connects be nodes 1, 2,...,. Then there s an edge of nfnte capacty from each of Out 1, Out 2,..., Out, to DC. For each DSS nstance, there s a correspondng nformaton flow graph. The collecton of graphs for all possble nstances of DSS(n,, d, α, β) s denoted by G(n,, d, α, β), or smply G f no ambguty could occur. Let I(v) and O(v) denote the set of ncomng edges and the set of outgong edges of v V. We defne the mportant concepts of cut and flow below: Defnton 1. An s-t cut K = (S, T ) of a DAG, G = (V, E), s a partton of V nto two dsjont subsets, S and T, such that s S, t T and there s at least one edge jonng S and T. The cut-set of K s {(u, v) E : u S, v T }. The cut-value of K s the sum of the capacty of all edges n the cut-set of K. Defnton 2. A flow f = {f e : e E} n a DAG, G = (V, E), from S to DC s a vald assgnment of a nonnegatve nteger f e to every edge e E such that f e s less than or equal to the capacty of e and e I(v) f e = e O(v) f e for any e V \ {S, DC}. The flow-value of f s e O(S) f e. The maxmum flow-value from S to DC s denoted by maxflow(dc). Fgure 1 shows an example of a grapn G(4, 2, d, α, β) wth only one DC shown. (The other DC s are omtted for clarty.) We are nterested to fnd the mnmum value of all the cuts, wth respect to all DC s, n any G G. A possble Fg. 1. An example of a grapn G(4, 2, d, α, β). cut wth value α + 2β s also shown n the fgure. A rgorous mn-cut analyss wll be done n the next secton. III. MIN-CUT BOUND Gven an nformaton flow graph G 0 G, we construct an auxlary graph G = (V, E) as follows: Frst, we combne In and Out nto a sngle vertex v, whch represents storage node, for all. Denote the source node and an arbtrary data collector by s and t, respectvely, where s, t V. Furthermore, denote the set of all chld vertces of s by V s and the set of all parent vertces of t by V t. Snce the data collector connects to storage nodes for fle retreval, we have V t =. Note that the n-degree of each vertex v V \(V s {s, t}) s equal to d, snce v s a newcomer and has contacted d storage nodes for repar. To smplfy the proof below, for every v V s, we splt ts ncomng edge nto d parallel edges, so that all vertces, except s and t, n the auxlary graph, G, have n-degree d. It s clear that G s also a DAG. Defnton 3. Gven an s-t cut K = (S, T ) of a DAG, G = (V, E), a vertex v T s sad to be on the boundary of the cut, denoted by K, f v s the head of any edge n the cut-set of K. Note that a cut K = (S, T ) can also be specfed by the cutset of K, or by K, as there s a one-to-one correspondence between (S, T ) and the cut-set of K, and also between (S, T ) and K. Lemma 1. For any s-t cut K = (S, T ) n an auxlary graph G, f t K, then K mn{d, }. Proof: Consder an arbtrary cut K = (S, T ) whch satsfes t K. It s clear that V t T, and therefore T. Frst, consder the case d. Snce G s a DAG, there s a topologcal orderng of the vertces, whch means that the vertces are ordered one after another n a way such that (v, v j ) E mples < j. The frst d vertces n T must belong to K, snce ther n-degrees are all equal to d. Hence, K d = mn{d, }. Next, consder the case d >. If T d, then by the same reasonng as before, K d > = mn{d, }. If T < d, then K = T 66

3 and hence K = mn{d, }. The statement s proved by combnng the two cases. The above result can be used to analyze the mn-cut n the nformaton flow graph. The proof s essentally the same as n [1, Lemma 2]. We nclude the proof for the sae of completeness. Lemma 2 (Mn-Cut Bound [1]). Gven any G 0 G(n,, d, α, β), the value of any s-t cut of G 0 s bounded below by mn{d,} 1 =0 mn{(d )β, α}. (1) Fg. 2. Graphcal llustraton of the bound. Furthermore, the lower bound s tght, meanng that there exsts a DC n G 0 such that the correspondng s-t cut has value equal to (1). Proof: Gven an nformaton flow graph G 0, consder an arbtrary cut K 0 = (S 0, T 0 ). We can exclude the case where t belongs to K 0, snce all ncomng edges of t have nfnte capacty. We can also exclude the case where In T and Out S for some, snce the cut-value can be reduced by movng Out from S to T. Now gven K 0, we can construct a correspondng cut K n the auxlary graph as follows: If ether In or Out s n K 0, then let v be n K. By Lemma 1, there are at least mn{d, } vertces n K, whcmples that there are also at least mn{d, } vertces n K 0. 1 Snce G 0 s a DAG, there s a topologcal orderng of ts vertces. Let Out 1 be the topologcally frst output node n T. Then ether In 1 K 0 or Out 1 K 0. If Out 1 K 0, then ts sngle ncomng edge contrbutes α to the cut-value. If In 1 K 0, we can exclude the case where In 1 V s, snce the ncomng edge of In 1 has nfnte capacty. Therefore, we only need to consder the case where In 1 V s. Its d ncomng edges contrbute a total of dβ to the cut-value. Combnng the two cases, a value of mn{dβ, α} s contrbuted to the cut-value. Next, consder Out j, the topologcally j-th output node n T. Agan ether In j K 0 or Out j K 0. If Out j K 0, then ts sngle ncomng edge contrbutes α to the cut-value. If In j K 0, agan we can exclude the case where In j V s. Then at least d (j 1) of the ncomng edges of In j must be n the cut-set of K 0, snce at most j 1 of ts ncomng edges can be connected to other out-vertces n T, namely Out 1, Out 2,..., Out j 1. Therefore, ts ncomng edges contrbute at least (d j + 1)β to the cut-value. Combnng the two cases, a value of mn{(d j + 1)β, α} s contrbuted to the cut-value. Snce there are at least mn{d, } vertces n K 0 and addng any one or more vertces to T wll not decrease the cut-value, the mnmum cut-value s therefore gven by the expresson n (1). 1 The fact that there cannot be less than mn{d, } vertces n K 0 was taen for granted n the last paragraph of the proof of [1, Lemma 2]. Now we show that the lower bound n (1) s tght by constructng an nformaton flow graph and a cut achevng equalty n (1). Intally, there are n storage nodes connected to S. Consder newcomers ndexed by n+1, n+2,..., n+. For = 1, 2,...,, In n+ connects to Out n+ d,..., Out n+ 1. Consder a data collector that connects to the last Out s. A cut K that acheves the lower bound can be constructed as follows: For = 0, 1,..., 1, f (d )β α, then In K; otherwse, Out K. A graphcal llustraton s shown n Fgure 2. Followng [1], we assume wthout loss of generalty that d. By Lemma 2, we have the followng theorem: Theorem 3. Let B be the sze of the fle stored n DSS(n,, d, α, β). If d, then 1 B mn{(d )β, α}. (2) =0 Proof: By the max-flow bound [5, Theorem 18.3], we have B mn maxflow(dc ). (3) Snce d, by Lemma 2, the mnmum cut-value s bounded below by the rght-hand sde of (2). By the well-nown maxflow mn-cut theorem [6], we have 1 mn maxflow(dc ) mn{(d )β, α}. (4) =0 The statement then follows from (3) and (4). IV. GENERIC STORAGE CODE In ths secton, we frst present some basc concepts n networ codng theory from [7]. Then, we ntroduce the refned nformaton flow graph for DSS. Next, we defne generc storage code, a concept n relaton to the refned nformaton flow graph. A. Basc Concepts of Lnear Networ Code Consder a sngle-source acyclc communcaton networ and ts correspondng graph, G = (V, E). Let the alphabet Σ be the fnte feld GF (q). A lnear networ code can be 67

4 specfed by the set of all local encodng ernels, {l d,e Σ : d I(v), e O(v), v V }. Suppose the message to be transmtted form the source node s an ω-dmensonal column vector x over GF (q). We add ω magnary edges termnatng at S and assgn each of them wth a dstnct column vector n the ω-dmensonal standard bass. These vectors are referred to as the global encodng ernels of the magnary edges. For each edge e = (u, v) E, we teratvely defne ts global encodng ernel by g e l d,e g d. d I(u) The transmtted symbol on edge e s x T g e. For an edge set P, denote the set of the correspondng global encodng ernels by er(p ) {g e : e P }, and the lnear span of er(p ) by For a vertex u, defne vspace(p ) span(er(p )). vspace(u) span(er(i(u))). A sequence of edges e 1, e 2,..., e n forms a patf Head(e ) = Tal(e +1 ) for 1 n 1. Two paths are edge-dsjont f they do not have any edge n common. A set of edges s sad to be path-ndependent f each edge n ths set s on a path orgnatng from an magnary edge and these paths are edgedsjont. An edge set P s sad to be regular wth respect to a lnear networ code f the global encodng ernels n er(p ) are lnearly ndependent. B. Refned Informaton Flow Graph Gven any nformaton flow graph for a DSS, we construct a refned nformaton flow graph by ntroducng the concept of repar stage. In regard to the refned nformaton flow graph, the repar process of a node s called a repar stage. From S to In s s called stage 1. The orgnal n storage nodes s sad to be n stage 0. In stage s > 0, the out-vertex of each storage node, except the one faled n stage s 1, s connected to an auxlary out-vertex by a drected edge of capacty α. The out-vertex of node n stage s s denoted by Out (s) and that n stage s + 1 s denoted by Out (s+1). Let the ndex of the newcomer n stage s + 1 be. We re-label the Out vertex of the newcomer as Out (s+1). Snce each storage node has capacty α, every edge from S to In wtnfnte capacty s replaced by an edge of capacty α. Furthermore, each edge of capacty c s replaced by c parallel edges of unt capacty. Denote the set of all the edges n stage s, except the ncomng edges of data collectors, by E s. Note that the refned nformaton flow graph represents a sngle-source mult-cast acyclc networ. An example wth n = 4, d = 3 s shown n Fgure 3. In the example, node 1 fals n stage 0 and node 2 fals n stage 1. C. Generc Storage Code Snce a DC can connect to nodes n the same stage, we only need to ensure the path-ndependent sets of edges n the same stage are regular. A code that satsfes ths requrement may Fg. 3. An example of a refned nformaton flow graph. be regarded as a restrcted form of a generc networ code [3], [7]. We call t generc storage code, whcs formally defned below: Defnton 4. An ω-dmensonal lnear networ code on a refned nformaton flow graps sad to be an ω-dmensonal generc storage code f every path-ndependent ω-subsets of E s s regular, for any stage s = 0, 1, 2,.... V. ACHIEVABILITY In ths secton, we prove the achevablty of the capacty bound n Theorem 3 by generc storage code. Lemma 4. Let G be a refned nformaton flow graph G such that there s at least one path-ndependent set of edges of sze ω n each stage. An ω-dmensonal generc storage code on G over GF(q) can be constructed, provded that q > ( ) ω. Proof: Let q be a prme power greater than ( ) ω. We prove the statement by mathematcal nducton on the number of repar stages n the refned nformaton flow graph. We want to mantan the nductve nvarant that, n any stage, any pathndependent set of edges s regular. Consder a refned nformaton graph wth no repar stage, (.e. wth only stages 1 and 0). Frst, note that any ω-subset of the edges n stage 1 s path-ndependent. We clam that there exsts a lnear code such that all these ω-subsets are regular. For the frst ω edges n stage 1, t s clear that they can be assgned lnearly ndependent global encodng ernels. For each of the subsequent edges n stage 1, we can pc a vector x vspace(ζ), where ζ s any (ω 1)-subset of edges that have already been assgned global encodng ernels. Ths can be done by pcng a generator matrx of an ω-dmensonal Reed-Solomon code of length nα. We can also assgn the global encodng ernels sequentally, snce ζ vspace(ζ) ( nα ω 1 ) q ω 1 < q ω. Let the global encodng ernels of α ncomng edges of every In, = 1, 2,..., n be the same as those of ts α outgong 68

5 edges. Snce n stage 1, any ω-subset of the nα edges s regular, so s any ω-subset of the nα edges n stage 0. Assume that for any refned nformaton graph wth s 0 repar stages, (.e., s + 1 stages ncludng stage 0), a generc storage code has been constructed. By defnton, any pathndependent ω-subset of E s s regular wth respect to the constructed networ code. In stage s + 1, there are n 1 auxlary out-vertces of the survvng nodes n stage s, and one newcomer. Let the set of ndces of the n 1 survvng nodes be S, and the ndex of the newcomer be. For S, Out (s) has α ncomng edges and α outgong edges connectng to. Let the global encodng ernels of these α outgong edges be the same as those of the α ncomng edges. Let the d helper nodes of newcomer be ndexed by h 1, h 2,..., h d S, where h 1 < h 2 < < h d. Let U be the edge set whch Out (s+1) for. It remans to determne the global encodng ernels for all the ncomng edges and outgong edges of In n such a way that any pathndependent ω-subset of U s regular. Ths can be done by Algorthm 1, whcs adapted from [5, Algorthm 19.34]. conssts of all ncomng and outgong edges of Out (s) S and all ncomng edges of Out (s+1) Algorthm 1 Assgn Global Encodng Kernels for the Newcomer Input: {g e : e I(Out (s) ) for all S} and h 1, h 2,..., h d Output: {g e : e I(In ) or e I(Out (s+1) )} 1: U 0 := {e I(Out (s) ) for all S}; 2: for := 1, 2,... d do 3: for j := 1, 2,... β do 4: e := the j-tncomng edges of In from Out (s) ; 5: Choose a vector x vspace(out (s) ) such that x vspace(ζ), where ζ s any ω-subset of U 0 such that ζ s regular and vspace(out (s) ) vspace(ζ); 6: g e := x and U 0 := U 0 {e}; 7: end for 8: end for 9: for j := 1, 2,... α do 10: e := the j-tncomng edges of Out (s+1) ; 11: Choose a vector x vspace(in ) such that x vspace(ζ), where ζ s any ω-subset of U 0 such that ζ s regular and vspace(in ) vspace(ζ); 12: g e := x and U 0 := U 0 {e}; 13: end for By constructon, t can be seen that any path-ndependent ω-subset of U s regular. In the algorthm, the vector x n lne 5 can always be found. To see ths, notce that there are at most (nα + dβ) edges n U 0 and the number of possble ). Denote the dmenson of ) vspace(ζ), the dmenson of vspace(out (s) ) vspace(ζ) s less than or equal choces of ζ s at most ( ω vspace(out (s) ) by ν. Snce vspace(out (s) to ν 1. Thus, vspace(out (s) ) ( ( ) nα + dβ vspace(ζ)) q ν 1 < q ν. ω ζ Lewse, the vector x n lne 11 can also be found. Theorem 5. A fle wth sze B = ω 1 =0 mn{(d )β, α} can be stored n a DSS(n,, d, α, β) by an ω-dmensonal generc storage code over GF(q), where q > ( ) ω. Proof: We have shown the exstence of generc storage code n Lemma 4. Now we show any collector can retreve the fle based on the generc storage code. By Lemma 2, there are at least ω dsjont paths termnatng at any out-vertces, n every stage s 0. Thus there are at least one path-ndependent set, say P wth sze ω wthn the ncomng edges of these out-vertces. By the defnton of generc storage code, the dmenson of er(p ) s ω, and the fle wth sze B = ω can be decoded. VI. CONCLUDING REMARKS We present a more rgorous dervaton of the mn-cut bound, and mae explct that fndng the mnmum number of nodes on the cut-boundary s a necessary step of the proof. We hope that ths provdes more nsght to the problem, whch may be useful for analyzng mn-cut bounds for future DSS models wth dfferent assumptons. To prove the achevablty of the mn-cut bound, a new concept called generc storage code s ntroduced. We remar that the correspondng nducton nvarant n Lemma 4 s larger than necessary n provng the achevablty result. Instead, the nducton nvarant found n [2] requres a smaller nductve nvarant. Hence, the feld sze requrement n [2] s smaller than that of generc storage code. Another approach, whch uses exact-repar regeneratng codes, can also obtan feld szes that reman constant for an unbounded number of falures. Exact-repar regeneratng codes, however, do not exst for the entre tradeoff curve [8]. Among all these approaches, the use of generc storage code s more flexble n the sense that t does not rely on the structure of the nformaton flow graph, renderng t more applcable to other DSS models. REFERENCES [1] A. G. Dmas, P. B. Godfrey, Y. Wu, M. J. Wanwrght, and K. Ramchandran, Networ codng for dstrbuted storage systems, IEEE Trans. Inf. Theory, vol. 56, no. 9, pp , Sep [2] Y. Wu, Exstence and constructon of capacty-achevng networ codes for dstrbuted storage, IEEE J. on Selected Areas n Commun., vol. 28, no. 2, pp , Feb [3] S.-Y. R. L, R. W. Yeung, and N. Ca, Lnear networ codng, IEEE Trans. Inf. Theory, vol. 49, no. 2, pp , Feb [4] R. Koetter and M. Médard, An algebrac approach to networ codng, IEEE/ACM Trans. Networng, vol. 11, no. 5, pp , Oct [5] R. W. Yeung, Informaton Theory and Networ Codng. Sprnger, [6] L. K. Ford Jr. and D. R. Fulerson, Flows n Networ. Prnceton Unversty Press, [7] M. Tan, R. W. Yeung, S.-T. Ho, and N. Ca, A unfed framewor for lnear networ codng, IEEE Trans. Inf. Theory, vol. 57, no. 1, pp , Jan [8] C. Tan, Characterzng the rate regon of the (4,3,3) exact-repar regeneratng codes, IEEE J. on Selected Areas n Commun., vol., no. 5, pp , May