Product Matrix MSR Codes with Bandwidth Adaptive Exact Repair

Transcription

1 1 Product Matrix MSR Codes with Bandwidth Adaptive Exact Repair Kaveh Mahdaviani, Soheil Mohajer, and Ashish Khisti ECE Dept, University o Toronto, Toronto, ON M5S3G4, Canada ECE Dept, University o Minnesota, Minneapolis, MN 55404, USA {kaveh, akhisti}@commutorontoca, soheil@umnedu Abstract In a distributed storage systems DSS) with k systematic nodes, robustness against node ailure is commonly provided by storing redundancy in a number o other nodes and perorming repair mechanism to reproduce the content o the ailed nodes Eiciency is then achieved by minimizing the storage overhead and the amount o data transmission required or data reconstruction and repair, provided by coding solutions such as regenerating codes [1] Common explicit regenerating code constructions enable eicient repair through accessing a predeined number, d, o arbitrary chosen available nodes, namely helpers In practice, however, the state o the system dynamically changes based on the request load, the link traic, etc, and the parameters which optimize system s perormance vary accordingly It is then desirable to have coding schemes which are able to operate optimally under a range o dierent parameters simultaneously Speciically, adaptivity in the number o helper nodes or repair is o interest While robustness requires capability o perorming repair with small number o helpers, it is desirable to use as many helpers as available to reduce the transmission delay and total repair traic In this work we ocus on the minimum storage regenerating MSR) codes, where each node is supposed to store α inormation units, and the source data o size kα could be recovered rom any arbitrary set o k nodes We introduce a class MSR codes that realize optimal repair bandwidth simultaneously with a set o dierent choices or the number o helpers, namely D = {d 1,,d δ } We show that comparing to the only other explicit constructions with such capabilities, presented in [2], the required value or α is exponentially smaller In our construction, particularly or a DSS with n nodes, and k systematic nodes, the required value or α is reduced rom s n to sk, where s = lcmd 1 k+1,,d δ k+1) We also show the required ield size is signiicantly smaller in the proposed construction I INTRODUCTION Distributed storage systems DSS) are compelling solutions to the ast growth o the demand in storage and accessibility o digital content The main eature o such systems is to provide accessibility and durability or the stored data by introducing redundancy In parallel, the number o storage components as well as the number o users connecting to these servers are dramatically increasing These increase the chance o ailures in the system, due to storage device ailures or inaccessible nodes with overloaded traic Replication is the simplest approach to make the distributed storage system robust against such ailures, which is implemented in systems such as [3] This approach however, provides simplicity in the cost o huge storage overhead In the literature o erasure codes, there exists solutions, such as the Reed-Solomon RS) code [4], which oer similar aulttolerance level as the replication does, with signiicantly less storage overhead For instance, a replication based DSS needs to accommodate two mirrors or every single storage node in order to achieve reliability guarantee against any simultaneous two node ailure, which arises 200% storage overhead, while maximum distance separable MDS) codes, such as RS code, achieve the same level o ault-tolerance guarantee by adding only two redundant storage nodes or the whole system In other words, an erasure code with MDS property can guarantee to recover the whole source data rom any subset o stored encoded segments o collective size equal to the source data size On the other hand, when a node ails in a DSS, it needs to be replaced by a new node in order to maintain the system s perormance Such procedure is reerred to as a repair To perorm a repair, the system downloads some data rom a subset o the surviving nodes, called helper nodes The amount o data downloaded or a repair is reerred to as the repair bandwidth In conventional MDS erasure codes, such as the RS codes, one has to reconstruct the entire source data and re-encode it to recover a single lost segment Hence, the repair bandwidth o these codes is at least as large as the size o the whole source data Considering both storage overhead and repair bandwidth simultaneously, the amily o regenerating codes [1], [5] oers a very eicient coding mechanism or distributed storage systems More precisely, a regenerating code on a Galois ield F q or a DSS with n storage nodes, maps the source data o size F symbols into n pieces o size α symbols each, and stores them in n separate nodes, such that any k out o n nodes suice to recover the data Such a system is capable to tolerate up to n k) node ailures Moreover, upon ailure o one node, it can be replaced by a new node whose content o size α symbols is determined by connecting to an arbitrary set o d where d k) helper nodes, and downloading β symbols orm each where β α) Hence, the total repair bandwidth in regenerating codes is dβ, which is denoted by γ Also, the parameter α is reerred to as the sub-packetization level

2 2 Ideally, one would like to minimize the storage overhead, and repair bandwidth simultaneously It turns out that or a given ile size F, there is a trade-o between the sub-packetization level α and the repair bandwidth γ, and one can be minimized only at the cost o a penalty or the other [1] In particular, at one extreme point o this trade-o, one could irst minimize the sub-packetization level, α, and then minimize the per-node repair bandwidth, β, to obtain a minimum storage regenerating MSR) code As a result, MSR codes have the MDS property, and also minimize the repair bandwidth or the given α [1], which means or an MSR code we have F = kα, and β = F kd k +1) 1) The total repair bandwidth dβ) o an MSR code is then upper-bounded by that o RS code, and only coincides with that when d = k In other word, the total repair bandwidth is decreasing super-linearly as d grows in MSR codes Reversing the order o minimization between α, and β results in another extreme point o the trade-o Such codes are not MDS and hence have more storage overhead but provide the smallest possible repair bandwidth and reerred to as minimum repair bandwidth MBR) codes Our ocus in this work is on MSR codes as they minimize the storage cost In general, two distinct types o repair can be identiied: in an exact repair scenario the replacement generated node will contain the same data as stored in the ailed node In the so called unctional-repair, however, the replacement node may store a dierent content, provided that the entire new system maintain the same properties as o the original one In practice, exact repair regenerating codes are much more appealing, mainly due to the act that they could provide systematic encoding, which is a signiicant advantage in practice It is shown that or unctional-repair MSR code design could be translated into a linear network coding problem [1] Design o such codes with exact repair property is more challenging, due to a large number o constraints need to be simultaneously satisied Various types o MSR code constructions are proposed or various set o parameters [2], [6] [16] Among the numerous available regenerating code designs, the common adopted model considers a predetermined number d where k d n 1) o helpers required or any repair procedure Each o these helpers is also assumed to provide β = γ/d repair symbols This sets a single threshold or the system s capability to perorm the repair On the other hand, in practice the state o the system dynamically changes as a unction o various actors including availability o nodes, traic load, available bandwidth, etc It has been shown that when such characteristics changes in the system the optimal number o helper nodes or minimizing the cost o repair changes [17] Thereore, runtime adaptation would be o great value towards optimizing the perormance For instance, when the system is heavily loaded by many read requests, many nodes in the network might not be capable to provide the required repair bandwidth and hence would be considered unavailable or the repair This may result in system s ailure to perorm the repair, while there might be a ew nodes, less than d, which are capable o providing even more than β repair symbols A natural question is whether one can download more data rom the available helpers, and accomplish the repair without the busy helpers Moreover, rom 1) it is clear that in optimal repair o MSR codes, increasing the number o helpers reduces both β, and γ = dβ One could consider a situation in which there are many nodes, more than d, which are capable o contributing to the repair It would be o great practical value then i the system could increase the number o helpers and reduce both the per-node and total repair traic Note that this will also reduce the transmission delay, which is one o the main bottlenecks in the DSS s perormance We reer to such property as bandwidth adaptivity Note that the dynamic capability o service or storage nodes is a well-known characteristic or many practical distributed systems such as peer-to-peer systems or heterogeneous network topologies [18] [23] The design o such codes has been o interest and the signiicance o bandwidth adaptivity in the perormance o the system has been emphasised in [7], [24] [26] However, it is a challenging problem to design such coding scheme with a large lexibility degree since it needs to satisy many optimality conditions simultaneously As a result, this problem has only been considered or the MSR [2], [7], and MBR [27] extreme points o the tradeo For the MBR case, [27] provided a solution or a wide range o practical parameters based on the Product Matrix ramework introduced in [6] In [7] a solution is provided based on intererence alignment, which only achieves the MSR characteristics when both α and β tend to ininity The irst explicit exact repair MSR code constructions which satisy the bandwidth adaptivity are introduced in [2] These constructions work or any parameters k, n, and all values o d such that k < d < n Although these constructions can achieve optimality or inite values o α and β, but the required value or these parameters are still very huge ie exponentially large in n), and hence they only achieve optimality or extremely large contents Recently, [15] introduced a modiied version o the code constructions in [2] which achieves MSR optimality or much lower values o α, at the cost o loosing bandwidth adaptivity Indeed the MSR code in [15] works only or d = n 1 In this work we will address the design problem o MSR codes with bandwidth adaptive exact repair or small α, and β, ollowing the Product Matrix ramework [6] The code allows us to choose the number o helper nodes or each repair scenario based on availability o the nodes, network traic and load state o the nodes, and it is capable to adjust the per-node repair bandwidth, β, to its optimum value based on the number o selected helpers as in 1) Compared to the constructions proposed in [2] or a DSS with n storage nodes the required values or α and β in the presented code is reduced exponentially or the same set o other parameters We also show that the required iled size is smaller in the presented coding scheme The main contributions o this work are explained in the next section, ater ormally deining the problem setup

3 3 The rest o this paper is organized as ollows: The ollowing section ormally introduced the problem setup and summarizes the main contributions Section III briely reviews the most relevant works in the literature Coding scheme and a ew examples are presented in Section IV, which is ollowed by a discussion on the properties o the code in Section V Conclusion and abstracts which contain more lengthy proos and examples are provided at the end A Model II MODEL AND MAIN RESULTS In this section we will briely introduce a setup or the distributed storage system and the coding scheme o our interest This model is a modiied version o the original setup considered in [1], and is very similar to the model considered in [27] The irst element we consider or the model o our bandwidth adaptive distributed storage system is a predeined Galois iled alphabet, F q o size q Hereater we assume all the symbols stored or transmitted through the network are elements o F q Besides, we consider a homogeneous group o n storage nodes, each capable o storing α symbols Deinition 1 Bandwidth Adaptive Regenerating Code) Consider parameters α, n, k, δ, a set D = {d 1,,d δ }, with d 1 < < d δ, and a total repair bandwidth unction γ : D [α, ) A bandwidth adaptive regenerating code Cn, k, D, α, γ) is a regenerating code with sub-packetization level α, such that ollowing mechanisms are guaranteed Repair: In each repair process the number o helpers, d, can be chosen arbitrarily rom the set D The choice o helper nodes is also arbitrary, and each o the chosen helpers then provides βd) = γd)/d repair symbols Data Reconstruction: The data collector recovers the whole source data by accessing any arbitrary set o k nodes Note that the lexibility o the repair procedure depends on the parameter δ, such that or a larger δ, there are more options to select the number o helpers In general, it is appealing to have small choices such as d 1, to guarantee the capability o code to perorm repair when the number o available helpers is small, and also large choices such as d δ, to provide the capability o reducing the total as well as per-node repair bandwidth, and hence the transmission delay, whenever a larger number o helpers are available The coding scheme we present in this work allows to design such a range or the elements in D Deinition 2 Total Storage Capacity) For a set o parameters α, n, k, δ, a set D = {d 1,,d δ }, and a given unction γ : D [α, ), the total storage capacity o a bandwidth adaptive regenerating code, Cn, k, D, α, γ), is the maximum size o a ile that could be stored in a network o n storage nodes with sub-packetization level α, using a bandwidth adaptive regenerating code Cn, k, D, α, γ) We will denote the storage capacity o such a system by Fn, k, D, α, γ), or simply F, when the parameters could be inerred rom the context Deinition 3 Bandwidth Adaptive MSR Codes, and Flexibility Degree) For any choice o parameters α, n, k, δ, and set D = {d 1,,d δ }, the bandwidth adaptive regenerating codes that realize both the MDS property deined by as well as the the MSR characteristic equation simultaneously or all d D, given as Fn,k,D,α,γ) = kα, 2) α = d k +1)βd), d D, 3) are reerred to as bandwidth adaptive MSR codes Moreover, the number o elements in the set D is reerred to as lexibility degree o the code, and is denoted by δ B Main Results The main contribution o this work is to provide a bandwidth adaptive MSR coding scheme with small sub-packetization level, and ield size requirement This coding scheme also guarantees exact repair o any ailed node with many dierent choices o the number o helpers This result is ormally stated in the ollowing theorem In this paper lcm) denotes the least common multiple Theorem 1 For arbitrary positive integers n, k, and δ, there exists an adaptive bandwidth MSR code over a Galois ield F q o size q n, with sub-packetization level α and total storage capacity F, satisying which is capable o perorming exact-repair using any arbitrary d i helpers, or α = k 1)lcm1,2,,δ), F = kα, 4) d i = i+1)k 1), i {1,,δ}, and simultaneously satisies the MSR characteristic equation 3) or any d i ie, α βd i ) =, i {1,,δ} d i k +1) We provide a constructive proo or this theorem based on the bandwidth adaptive MSR coding scheme that could be designed to achieve any arbitrary inite lexibility degree, introduced in Section IV

4 4 III RELATED WORKS The property o bandwidth adaptivity has been o interest in the literature o regenerating codes or the last ew years There has been a number o researchers who have addressed this problem under dierent settings Some, such as [24], [25], [28], have considered unctional-repair, in which code design problem reduces to linear network coding, and mainly ocused on the theoretical limitations and properties o a bandwidth adaptive regenerating codes There are, however, a ew other works which have considered exact-repair as well [2], [7], [27] In this section we briely review these works and their relevance to the problem setup introduced in the previous section In the MBR case, [27], [29] addressed a similar setup For given integers d min, d max, authors presented a bandwidth adaptive exact-repair MBR regenerating code with D = {d min,d min +1,,d max }, which is simultaneously error resilient against up to a certain number o erroneous nodes in the network Moreover, [29] also addresses the non-symmetric bandwidth adaptive exact-repair, in which dierent helpers may participate non-equally in the repair procedure, according to their available resources For the case o MSR, in [28] Wang et al considered a unctional-repair coding scheme which supports bandwidth adaptivity Later [24] also considered a similar setup, while in both these works the main ocus is on derivation o the optimal trade-o between the storage overhead and repair bandwidth or the unctional-repair in a coordinated setting, where more than one node ailure is considered to be repaired together Note, however, that none o these works address the exact-repair with bandwidth adaptivity in regenerating codes Aggrawal et al [25] also considered a unctional-repair setup to achieve bandwidth adaptivity in regenerating codes They analysed the mean-time-to-ailure MTTF) in the regenerating codes with and without bandwidth adaptivity Their analysis is based on a birth-death process model in which the population o available storage node randomly changes with appropriately chosen rates They showed that bandwidth adaptivity provides a signiicant gain in terms o MTTF When considering exact-repair regenerating codes, Cadambe et al were the irst to address the bandwidth adaptivity [7] They present an intererence alignment based regenerating coding scheme which is capable o perorming exact-repair with bandwidth adaptivity in the repair procedure The code presented by Cadambe et al is the irst exact-repair regenerating code with bandwidth adaptivity, however, their coding scheme only asymptotically achieves the MSR optimality, when α and β tend to ininity with proper ratio The importance o this result, however, is to show that bandwidth adaptivity could be implemented without extra cost in the optimal trade-o between the storage overhear and repair bandwidth, at least or the MSR codes, even when the exact-repair is required In [26], a similar setup, reerred to as progressive engagement, is considered or regenerating codes with bandwidth adaptive exact-repair, and two coding schemes are provided While the two coding schemes both preserve the MDS property, none o them satisy the MSR characteristic equation simultaneously or dierent choices o d Another main dierence between the progressive engagement setup and the one considered in this work is that the authors in [26] relax the property that any subset o surviving nodes could be considered as helpers by assuming that all the remaining systematic nodes are always participating as helpers Moreover, they require the set o available choices or the number o helpers, namely D, to be D = {k,k +1,,n 1}, while in our ormulation, D does not need to contain all integers between k and n 1 Indeed, we will provide a dierent view point o our result in Section IV-D, which shows our ormulation could indeed consider D to be arbitrary The irst two explicit constructions or exact-repair MSR codes with bandwidth adaptivity with inite sub-packetization level were introduced in [2] Both o these constructions work or any arbitrary values o k and n, and the set o choices or the number o helpers, D, could be designed to contain any value d i such that k < d i < n However, the required value or the sub-packetization level, α, in both constructions is still considerably huge In particular, or a DSS with n storage nodes, and the set D = {d 1,,d δ }, constructions in [2] require a ield size o n k)n and α = [lcmd 1 k +1,,d δ k +1)] n 5) Thereore, these constructions only achieve optimality or storage o contents which are exponentially large in terms o the number o storage nodes in the system, which is still impractical or common parameters in practice For instance, the coding schemes suggested or Facebook and Windows Azure have n = 14 [30], and n = 10 [31] respectively, while other industrial solutions even report hundreds o nodes in their DSS [21] For such schemes the smallest realization o constructions in [2] with bandwidth adaptivity require sub-packetization levels o order or higher 1, hence, leaving the problem o exact-repair bandwidth adaptive regenerating code design yet open or practical parameters In this work, as presented in Theorem 1, we address the exact-repair bandwidth adaptive MSR code design problem with small subpacketization level, ollowing the product matrix PM) ramework introduced in [6] Comparing 5) with 4), one could see that the presented scheme reduces the required α and β) values exponentially However, this scheme works only or 2k 2 d i, d i D As a result, the design o high-rate bandwidth adaptive MSR codes with small α and β still remains an open problem 1 Considering the required ield size or constructions in [2] this is equivalent to per-node storage capacity o order at least tens o peta-bytes

5 5 IV CODING SCHEME In this section we introduce a bandwidth adaptive exact-repair MSR coding scheme, which could be designed to provide any required lexibility degree with elements o the set D evenly located between the smallest and the largest element, namely d 1 and d δ We describe the encoding and decoding schemes or storage, repair, and data reconstruction procedures or the adaptive bandwidth MSR code in the ollowing subsections This coding scheme is closely related to the product matrix MSR code introduced in [6], and could be considered as an extension o the product matrix code that achieves bandwidth adaptivity As mentioned in the previous section, we will assume all the source and encoded symbols, are elements o a Galois ield o an appropriately large ield size q n, denoted by F q Moreover, all the operations hereater are considered to be iled operations o F q We will reer to F q as the code alphabet In the design o the proposed coding scheme, we chose a design parameter µ, and the required lexibility degree δ All the other parameters o the code, including α, F, k, D = {d 1,,d δ }, and βd i ) will be then determined based on µ, and δ as ollows The sub-packetization level is α = µ lcm1,2,,δ) 6) Moreover, we have k = µ+1, and F = kα = µ+1)α, which satisies the MDS property Finally, or D we have D = {d 1,,d δ }, d i = i+1)µ, i {1,,δ} 7) and or any d i D, the associated per-node and total repair bandwidths denoted by βd i ), and γd i ) respectively are A Coding or Storage βd i ) = α iµ, γd i) = d i βd i ) = i+1)α 8) i We begin the introduction o the coding scheme by describing the process o encoding the source symbols and deriving the encoded symbols to be stored in the storage nodes Similar to the product matrix codes, the irst step in encoding or storage in this scheme is to arrange the inormation symbols in a matrix, denoted by M, which we reer to hereater as the data matrix Let z δ = lcm1,2,,δ) 9) The data matrix in our coding scheme is structured as ollows, S 1 S 2 O O O O O S 2 S 3 S 4 O O O O O S 4 S 5 S 6 O O O O O S 6 S 7 S 8 O O M =, 10) O O S 2zδ 4 S 2zδ 3 S 2zδ 2 O O O S 2zδ 2 S 2zδ 1 O O O O S 2zδ where, each S i, i {1,,2z δ } is a symmetric µ µ matrix illed with µµ+1)/2 source symbols, and O is a µ µ zero matrix Thereore, M has z δ +1)µ rows and z δ µ columns Note that the total number o distinct source symbols in the data matrix M is F = µµ+1)z δ = kα 11) Example 1 Consider the design parameters µ = 2, and δ = 2 Thereore, rom 9) we have z δ = 2, and using 11), the maximum number o source symbols that we can arrange in the data matrix M is 12 Denoting the source symbols by s 1,,s 12, then we have s1 s 2 s4 s 5 M = S s 1S 2 2 s 3 s 5 s 6 [ = s4 s 5 12) S 2 S 3 O S 4 ] s7 s 8 s 5 s 6 s 8 s s10 s s 11 s 12

6 6 Once the data matrix is ready, the source encoder creates the vector o coded symbols or each o the n storage nodes, by calculating the product o a node-speciic coeicient vector and the data matrix To describe this process, we irst need the ollowing deinition Deinition 4 [Generalized Vandermonde Matrix] For distinct and non-zero elements e 1,,e m o F q, and some integer c 0, a matrix A m l with entries is reerred to as a generalized Vandermonde matrix A i,j = e c+j 1 i, or i {1,,m}, j {1,,l}, The ollowing lemma about these matrices will be used in the proos o the ollowing theorems, Lemma 1 Consider distinct and non-zero elements e 1,,e l in F q, and an integer c 0 Then a square l l generalized Vandermonde matrix, A, as deined in Deinition 4, is invertible in F q The proo o the above lemma, simply ollows rom the act that, e c e 1 e 2 0 e c 2 A = e 2 e e c l 1 e l e 2 l el 1 1 el 1 2 el 1 l and the two matrices on the right hand side are both ull-rank, as one o them is diagonal with non-zero diagonal elements and the other one is a square Vandermonde matrix Back to the description o our coding scheme, or distinct and non-zero elements e i s in F q, with i {1,,n} we set c = 0, and deine a generalized Vandermonde matrix o size n z δ +1)µ as Ψ = 1 e 1 e 2 1 e z δ+1)µ e 2 e 2 2 ez δ+1)µ e n e 2 n ez δ+1)µ 1 n Note that all o the submatrices o Ψ are also generalized Vandermonde matrices We reer to Ψ as the coeicient matrix and denote the j th row o Ψ by ψ j The vector o encoded symbols to be stored on node j, j {1,,n}, denoted by x j, is calculated as, x j = ψ j M 13) The vector ψ j is the node-speciic coeicient vector or storage node j Note that the per-node storage capacity requirement or this coding scheme is then z δ µ as given by 6) Example 2 Let s consider the setting in Example 1 again, and assume we have n = 7 nodes in the network Assume that the code alphabet is the Galois ield F 11 The coeicient matrix could be ormed based on e 1,e 2,,e 7 ) = 1,2,,7) as ollows, Ψ = ) The encoded content o nodes 1 to 7 can be calculated using 12) and 13) For instance, or node 1 we have which gives x 1 = [x 1,1,x 1,2,x 1,3,x 1,4 ] = [1,1,1,1,1,1]M, x 1,1 = s 1 +s 2 +s 4 +s 5, x 1,2 = s 2 +s 3 +s 5 +s 6, x 1,3 = s 4 +s 5 +s 7 +s 8 +s 10 +s 11, x 1,4 = s 5 +s 6 +s 8 +s 9 +s 11 +s 12,

7 7 and similarly or node 7 we have x 7 = [x 7,1,x 7,2,x 7,3,x 7,4 ] = [1,7,5,2,3,10]M, which gives x 7,1 = s 1 +7s 2 +5s 4 +2s 5, x 7,2 = s 2 +7s 3 +5s 5 +2s 6, x 7,3 = s 4 +7s 5 +5s 7 +2s 8 +3s s 11, x 7,4 = s 5 +7s 6 +5s 8 +2s 9 +3s s 12 B Data Reconstruction In order to reconstruct all the inormation stored in the system, the data collector accesses k arbitrary nodes in the network and downloads all their contents To describe the details o the decoding we use the ollowing lemma Lemma 2 Let X and Φ be two known matrices o size µ + 1) µ, such that Φ is a generalized Vandermonde matrix, and assume is a known diagonal matrix o size µ+1) µ+1), with distinct and non-zero diagonal elements Then the equation is uniquely solvable or unknown µ µ symmetric matrices A and B X = ΦA+ ΦB, 15) The proo o this lemma is based on the data reconstruction scheme o the product matrix MSR codes, introduced in [6], and is provided in Appendix A to help keeping this paper sel-contained The ollowing theorem explains the data reconstruction procedure in this coding scheme Theorem 2 For the coding scheme presented in subsection IV-A, there exists a decoding scheme to reconstruct all the source symbols arranged in the data matrix M rom the encoded content o any arbitrary group o k = µ+1 storage nodes Proo Let s assume the set o accessed nodes is {l 1,,l k } Moreover, let s denote the k z δ + 1)µ submatrix o Ψ associated with the nodes l 1,,l k, by Ψ DC We will urther denote the submatrix o Ψ DC consisting o columns i 1)µ+1 through iµ, by Ψ DC i) In other words, we have a partitioning o Ψ DC s columns as Ψ DC = ψ l1 ψ lk = [Ψ DC 1),,Ψ DC z δ +1)] 16) As a result, deining the diagonal matrix e µ l e µ l Λ DC =, or each k µ submatrix Ψ DC i) we have e µ l k Ψ DC i+1) = Λ DC Ψ DC i) 17) Similarly, let s denote the matrix consisting o the collected encoded vectors by X DC, and its partitioning to k µ submatrices X DC i), i {1,,z δ } as ollows X DC = x l1 x lk = [X DC 1),,X DC z δ )] 18) Example 3 Following the setting described in Example 1 and Example 2, we have k = 3, and the coeicient matrix Ψ is given in 14) Let s assume the data collector accesses the storage nodes 1, 2, and 4 Then, with z δ = 2, we have, Ψ DC = [Ψ DC 1),Ψ DC 2),Ψ DC 3)] = ,

8 8 and, and Moreover, with µ = 2 we have and inally, Ψ DC 1) = ,Ψ DC 2) = ,Ψ DC 3) = Λ DC = = , 19) X DC = [X DC 1),X DC 2)] = x 1 x 2 = x 1,1 x 1,2 x 1,3 x 1,4 x 2,1 x 2,2 x 2,3 x 2,4, x 4 x 4,1 x 4,2 x 4,3 x 4,4 X DC 1) = x 1,1 x 1,2 x 2,1 x 2,2,X DC 2) = x 1,3 x 1,4 x 2,3 x 2,4 20) x 4,1 x 4,2 x 4,3 x 4,4 The decoding procedure or data reconstruction consists o z δ consecutive steps The irst step uses only the submatrix X DC 1) Using 17) we have, S1 X DC 1) = [Ψ DC 1),Ψ DC 2)] S 2 = Ψ DC 1)S 1 +Λ DC Ψ DC 1)S 2 Using Lemma 2, the decoder recovers both S 1, and S 2, using X DC 1), in step one Then, or i {2,,z δ }, the decoder perorms step i by using submatrix X DC i), and decodes submatrices S 2i 1, and S 2i, as ollows In step i, i {2,,z δ }, o the data reconstruction decoding, the decoder uses the submatrix X DC i) Note that X DC i) = [Ψ DC i 1),Ψ DC i),ψ DC i+1)] S 2i 1) S 2i 1 S 2i = Ψ DC i 1)S 2i 1) +[Ψ DC i),ψ DC i+1)] [ S2i 1 Having the submatrix S 2i 1) already recovered rom step i 1, the decoder irst calculates Then rom 17), and 21), we have ˆX DC i) = X DC i) Ψ DC i 1)S 2i 1) S2i 1 = [Ψ DC i),ψ DC i+1)] 21) S 2i ˆX DC i) = Ψ DC i)s 2i 1 +Λ DC Ψ DC i)s 2i Again using Lemma 2, decoder recovers S 2i 1, and S 2i at the end o the step i o the decoding Thereore, by inishing step z δ, the decoder reconstructs all the submatrices in the data matrix M, and recovers the whole data stored in the network Example 4 Following the setting o Examples 1 to 3, consider µ = 2, k = 3, and the data matrix as provided in 12) The irst step o decoding in the data reconstruction process based on the encoded data stored in nodes 1, 2 and 4, starts by using only X DC 1), as given in 20) Using Lemma 2, The decoder then recovers S 1, and S 2 submatrices o the data matrix M In the second step then the decoder irst calculates ˆX DC 2) = X DC 2) Ψ DC 1)S 2, S 2i ] which is equal to ˆX DC 2) = Ψ DC 2)S 3 +Λ DC Ψ DC 2)S 4, and hence is o the desired orm o 15) Thereore, again using lemma 2, the decoder recovers S 3, and S 4, which completes the data reconstruction Algorithm IV-B summarizes the data reconstruction mechanism in this coding scheme

9 9 Algorithm 1 Data Reconstruction 1: Input: µ, z δ, x li, ψ li, and e li or i {1,,k} 2: Output: Submatrices S 1,,S 2zδ 3: Form matrices Ψ DC i), or i {1,,z δ +1}, using 16) 4: Form matrices X DC i), or i {1,,z δ }, using 18) 5: Form matrix Λ DC, using 19) 6: Recover submatrices S 1, S 2 rom X DC 1), Ψ DC 1), and Λ DC, using Lemma 2 7: or i = 2 to z δ do 8: Calculate ˆX DC i) = X DC i) Ψ DC i 1)S 2i 1) 9: Recover submatrices S 2i 1, S 2i rom ˆX DC i), Ψ DC i), and Λ DC, using Lemma 2 10: end or C Bandwidth Adaptive Exact-Repair We now describe the bandwidth adaptive repair procedure, by assuming that node is ailed and the set o helpers selected or the repair are H = {h 1,,h d }, or some arbitrary d D The ollowing theorem describes the repair procedure in this bandwidth adaptive MSR code Theorem 3 Consider the coding scheme presented in subsection IV-A, with design parameters µ, and δ, and D as deined in 7) For any arbitrary ailed node, and any arbitrary set o helpers H = {h 1,,h d }, or some d D, there exists a repair scheme or recovering the content o node with per-node repair bandwidth, βd) = α d µ 22) Remark 1 Note that 7) and 22) are consistent with 8), which satisies the MSR characteristic equation 3) or any d D Proo Without loss o generality let d = m+1)µ, or some m {1,,δ} Note that 6), and 7) guarantee that or any d D, α is an integer multiple o d µ, hence βd) is an integer Each helper node h H, creates βd) = α/d µ) repair symbols to repair node as ollows First helper node h partitions its encoded content into βd) equal segments, such that or i {1,,βd)}, the segment x h i) is o size α/βd) = d µ = mµ, and contains elements x h,i 1)mµ+1 through x h,imµ Then we have x h = [x h 1),,x h βd))] 23) Similarly, or any node l, we split the irst α entries o a coeicient vector assigned to node l, namely ψ l, into βd) equal segments as ψ l 1 : α) = ψ l 1),,ψ l βd)), 24) where each segment ψ l i) is o size d µ = mµ Now each helper node h H, creates its βd) repair symbols as rh,) = [ r 1 h,),,r βd) h,) ] = [ x h 1) ψ 1),,xh βd)) The repair decoder then stacks d repair vectors rh,), or h H, into a d βd) matrix rl 1,) Υ H = rl d,) ] ψ βd)) 25) 26) We then introduce the ollowing partitioning o the matrix Υ H, into βd) submatrices, as ollows Υ H = [Υ H 1),,Υ H βd))], 27) where Υ H i), i {1,,β} is the i th column in Υ H, o size d 1 Beore starting to describe the repair decoding procedure, we need to introduce some notations associated to a given repair scenario Consider a repair procedure with d = m + 1)µ For the corresponding βd) = α/d µ), we will partition matrix M as depicted in Fig 1, and equation 28) Note that this results in βd) non-overlapping diagonal submatices M i, i {1,,βd)}, each o size mµ mµ, along with µ µ symmetric submatrices S 2m,S 4m,,S 2βd)m = S 2zδ as shown in the igure It is worth mentioning that the general pattern o the partitioning shown in igure 1 is preserved the same or all d D, and only the size, and number o the M i diagonal blocks changes or dierent choices o the parameter d From

10 10 mµ mµ mµ mµ M 1 S 2m S 2m M 2 S 4m 0 S 4m S 2i 1)m M = S 2i 1)m M i S 2im S 2β 1)m 0 S 2im S 2β 1)m M β S 2zδ Fig 1 In the above igure β represents βd), or some chosen d D, such that, d = m + 1)µ Moreover, the white area is illed by zeros, and each coloured squares represent a non-zero symmetric submatrix o M S 2i 1)m+1 S 2i 1)m+2 O O O O S 2i 1)m+2 S 2i 1)m+3 S 2i 1)m+4 O O O O S 2i 1)m+4 S 2i 1)m+5 S 2i 1)m+6 O O M i = 28) O O S 2im 4 S 2im 3 S 2im 2 O O O S 2im 2 S 2im 1 the construction o the data matrix, introduced in 10), each M i submatrix will be symmetric As a result, the data matrix M could be interpreted in terms o the submatrices M i, and S 2i or i {1,,βd)}, associated to a repair procedure with d = m+1)µ, d D Finally the last notations we use to describe the adaptive repair decoding scheme, using a given set o helpers H = {h 1,,h d }, is, Ω H i) = e i 1)mµ h 1 e i 1)mµ h d e i 1)mµ+1 h 1 e i 1)mµ+1 h d e im+1)µ 1 h 1, i {1,,βd)} 29) e im+1)µ 1 h d Note that, Ω H i), i {1,,βd)}, is a d d generalized Vandermonde matrix and hence is invertible as shown in Lemma 1 We denote the upper d µ) d submatrix o Ω H i)) 1 by Θ H i), and the lower µ d submatrix by Ξ H i) Then we have, Ω H i)) 1 ΘH i) = 30) Ξ H i) The decoding procedure or the repair o node is perormed in βd) sequential steps as will be described in the ollowing Let s begin with the irst step For the ailed node, let φ denote the 1 µ vector, [ φ = 1, e,, e µ 1 ] 31)

11 11 Using a partitioning similar to 23) or x, then we have, x 1) = ψ M 1 O O S 2m [ = ψ 1)M 1 + O,,O,e mµ φ S 2m ]µ mµ 32) In the irst step, the decoder recovers the two terms on the right in the above equation to reconstruct x 1) using only the irst repair symbol received rom each o the helpers, namely r 1 h i,), or i {1,,d}, as ollows Using 23) to 27), and the partitioning denoted in Fig 1, the submatrix Υ H 1), introduced in 27) can be written as, Then using 29), we have, x h1 1) Υ H 1) = x hd 1) = ψ 1) ψ 1) x h1 1) = x hd 1) 1 e h1 e 2 h 1 e m+1)µ 1 h 1 1 e hd e 2 h d e m+1)µ 1 h d ψ 1) M 1 O O S 2m ψ 1) 33) Υ H 1) = Ω H 1) M 1 ψ 1)) 34) O O S 2m Multiplying the inverse o Ω H 1) rom right to the both sides o 34), and using 30) the decoder derives That gives, and similarly, using 31), M 1 ψ 1) O O S 2m Since both M 1, and S 2m are symmetric, rom 35) we have, and rom 36), by multiplying the scalar e µ, we get ) [ ΘH i) = Ξ H i) ] Υ H 1) M 1 ψ 1) = ΘH 1)Υ H 1), 35) S 2m e m 1)µ φ = ΞH 1)Υ H 1) 36) ψ 1)M 1 = Θ H 1)Υ H 1), 37) e mµ φ S 2m = e µ Ξ H1)Υ H 1) 38) From 37), and 38), and using 32) the decoder then recovers x 1) as, [ x 1) = ψ 1)M 1 + O,,O,e mµ φ S 2m ]µ mµ, 39) where, the rightmost term in the above expression is derived by padding m 1, µ µ zero matrices, O, to the let o the matrix calculated in 38)

12 12 In step i or i = 2 through βd) o the repair decoding, the decoder then recovers x i), using Υ H i) received rom the helpers, along with e i 1)mµ φ S 2i 1)m, recovered rom the step i 1 o decoding To this end, irst note that similar to 33) the repair symbols in Υ H i) can be written as, x h1 i) ψ i) x h1 i) ) Υ H i) = = ψ i) x hd i) ψ i) x hd i) S 2i 1)m O O e i 1)mµ h 1 e i 1)mµ+1 h 1 e im+1)µ 1 h 1 = e i 1)mµ h d e i 1)mµ+1 h d e im+1)µ 1 M i ψ i)) h d O O S 2m Using 29) we can rewrite the above equation as, Υ H i) = e i 1)mµ µ l 1 e i 1)mµ µ l d φ l1 φ ld S 2i 1)m e i 1)mµ φ +ΩH i) M i O O S 2im ψ i) The decoder irst removes the contribution o the S 2i 1)m submatrix in the repair symbols in Υ H i) by calculating In the above expression, S 2i 1)m ˆΥ H i)=υ H i) e i 1)mµ e i 1)mµ µ l 1 e i 1)mµ µ l d Thereore, similar to 35) through 38) the decoder derives, and Finally, using 41) and 42), we have φ l1 φ ld S 2i 1)m e i 1)mµ i 1)mµ φ is itsel derived by transposing e ˆΥ H i) = Ω H i) M i ψ i)) O O S 2im φ 40) φ S 2i 1)m As a result we have, ψ i)m i = Θ H i)υ H i), 41) e imµ φ S 2im = e µ Ξ Hi)Υ H i) 42) x i) = ψ i)m i + [ O,,O,e imµ φ S 2im ]µ mµ 43) The ollowing algorithm summarizes the bandwidth adaptive repair procedure Remark 2 Note that calculating the inverse matrices Ω H i)) 1, or i {1,,β}, can be carried out recursively since rom 29) we have, e i 1)mµ h e i 1)mµ Ω H i) = h 2 0 Ω H1) 0 0 e i 1)mµ h d

13 13 Algorithm 2 Bandwidth Adaptive Repair 1: Input:, e, β, m, µ and rh,), ψ h, e h, or h H 2: Form matrices Υ H i), or i {1,,β}, using 26), and 27) 3: Form vectors φ using 31) 4: Calculate matrices Ω H i), and Ω H i)) 1, or i {1,,β}, using 29) 5: Calculate matrices ψ 1)M 1 using 37) 6: Calculate matrices e ma φ S 2m using 38) 7: Recover x 1) using 39) 8: or i = 2 to β do 9: Calculate ˆΥ H i) using 40) 10: Calculate ψ i)m i using 35) 11: Calculate e imµ φ S 2im using 42) 12: Recover x i) using 43) 13: end or 14: Form x = [x 1),,x β)] Remark 3 In a DSS with n nodes, or D = {d 1,,d δ }, the bandwidth adaptive MSR codes presented in [2] require α = lcmd 1 k +1,,d δ k +1)) n 44) Comparing 44) with 6), one could see that the presented scheme reduces the requiredαandβ) values exponentially However, this scheme works only or 2k 2 d i, d i D Hence, the design o high-rate bandwidth adaptive MSR codes with small α and β still remains an open problem The ollowing example provides a detailed illustration o the MSR bandwidth adaptive exact-repair procedure in the same setup as described in Examples 1 to 4 Example 5 As in the previous examples we will consider µ = 2, and δ = 2, which gives k = 3, and using 6), α = 4 The code alphabet is F 11, and the data matrix M as given in 12) As a result, using 7), we have D = {d 1,d 2 } = {4,6}, and rom 22) their associated per-node repair bandwidths are β 1 = 2, and β 2 = 1 Without loss o generality, let s assume node 7 is ailed, ie = 7 In this setup, the ollowing two repair scenarios are then possible A: One option is to use d = d 2 = 6 helpers, and download only β 2 = 1 repair symbol rom each o them, which means d = m +1)µ or m = 2 Consider H = {1,2,,6} In this case, using the coeicient matrix Ψ or this setup, which is given in 14), rom 24), with α = 4,β = 1, or any node l we have ψ l 1) = ψ l 1 : 4), which is a row vector consisting o the irst α = 4 elements o the coeicient vector assigned to node l As a result, using 23), each helper node h H will use all o its encoded content x h to create a single repair symbol as [ ] rh,) = [r 1 h,)] = x h 1) ψ 1) The repair decoder will then receive Υ H = [Υ H 1)] = r1,) r6,) = = [x h [1,7,5,2] ] x 1,1 +7x 1,2 +5x 1,3 +2x 1,4 x 2,1 +7x 2,2 +5x 2,3 +2x 2,4 x 3,1 +7x 3,2 +5x 3,3 +2x 3,4 x 4,1 +7x 4,2 +5x 4,3 +2x 4,4 x 5,1 +7x 5,2 +5x 5,3 +2x 5,4 x 6,1 +7x 6,2 +5x 6,3 +2x 6,4 Moreover, using 31), and with e = 7 rom 14) as also used in all previous examples, or = 7 we get φ = [1,7]

14 14 Finally notice that in this case the repair decoding will only have one single step Thereore, or step i = 1, with m = 2, and µ = 2 we get e imµ l = e 4 l, and hence, Ω H 1) = ) Then, based on the partition represented in Fing 1), or the decoder has access to M 1 = S1 S 2, S 2 S 3 Υ H = Ω H 1) M 1 ψ 1) O S 4 s 1 s 2 s 4 s 5 s 2 s 3 s 5 s 6 = Ω H 1) s 4 s 5 s 7 s 8 [ s 5 s 6 ][ s 8 s 9 ψ 1)) ] 0 0 s10 s s 11 s 12 Note that the Ω H 1), given in 45) is an invertible matrix in the code alphabet F 11, and we have Ω H 1)) 1 ΘH 1) = = Ξ H 1) The decoder then calculates the lost data x, using 39), with e µ = 72 = 5 in F 11, as ollows x = Θ H 1)Υ H +[0,0,5Ξ H 1)Υ H ] B: The second option is to use d = d 1 = 4 helpers With µ = 2 we have d = m+1)µ or m = 1 In this case, using 22) we will have β 1 = 4/4 2) = 2 repair symbols per helper node Let s without loss o generality assume node 7 is ailed, ie = 7, and H = {1,2,3,4} is the set o helper nodes chosen to perorm the repair As a result, using the coeicient matrix in the code alphabet F 11, given in 14), rom equation 24) or the helper nodes we have and or the ailed node = 7, ψ 1 1) = [1,1], ψ 1 2) = [1,1], ψ 2 1) = [1,2], ψ 2 2) = [4,8], ψ 3 1) = [1,3], ψ 3 2) = [9,5], ψ 4 1) = [1,4], ψ 4 2) = [5,9], ψ 1) = [1,7], ψ 2) = [5,2] Similarly, or the coded content o each o these nodes we consider the ollowing partition x l = [x l 1),x l 2)] Each helper node h H then creates two repair symbols = [[x l,1,x l,2 ],[x l,3,x l,4 ]], or l {1,2,3,4,7} rh,) = [r 1 h,),r 2 h,)] [ = x l 1) ψ 1),xl 2) ] ψ 2) = [x h,1 +7x h,2, 5x h,3 +2x h,4 ],

15 15 and the repair decoder receives Υ H = [Υ H 1),Υ H 2)] r 1 1,) r 2 1,) = r 1 4,) r 2 4,) x 1,1 +7x 1,2 5x 1,3 +2x 1,4 = x 2,1 +7x 2,2 5x 2,3 +2x 2,4 x 3,1 +7x 3,2 5x 3,3 +2x 3,4 x 4,1 +7x 4,2 5x 4,3 +2x 4,4 Moreover, using 31), and with e = 7 rom 14) as also used in all previous examples, or = 7 we get φ = [1,7] Finally notice that in this case the repair decoding will have two steps Thereore or step i = 1, with m = 1, and µ = 2 we get e imµ l = e 2 l, and hence, using equation 29), we have Ω H 1) = ) Then, according to the partition represented in Fig 1), or the repair decoder has access to M 1 = S 1, M 2 = S 3, M1 Υ H 1) = Ω H 1) ψ S 2 1) s1 s 2 = Ω H 1) [ s 2 s 3 ] s4 s 5 s 5 s 6 [ 1 7 As expected Ω H 1) is an invertible matrix in the code alphabet F 11, and we have Ω H 1)) 1 ΘH 1) = = Ξ H 1) The decoder then calculates the lost data x 1), using 39), with e mµ ] x 1) = Θ H 1)Υ H1) +5ΞH 1)Υ H 1) = 7 2 = 5 in F 11, as ollows To start the second step, i = 2, o the repair decoding, then the repair decoder irst uses e i 1)mµ S 2 e i 1)mµ φ = 5ΞH 1)Υ H 1)), = 7 2 = 5, to calculate where 5Ξ H 1)Υ H 1)) is already derived in the step 1 Then the decoder calculates ˆΥ H 2) using equation 40) as, ˆΥ H 2) = Υ H 2) e 2 1)m 1)µ 1 φ 1 e 2 1)m 1)µ 4 φ 4 S 2 e 2 1)mµ φ r 2 1,) 1 1 = r 2 2,) r 2 3,) S 25Ξ H 1)Υ H 1))) r 2 4,) 1 4

16 16 From 41), with the repair decoder then has access to Ω H 2) = , = Ω H 2) ] M2 ˆΥ H 2) = Ω H 2)[ ψ S 4 2) s7 s 8 s 8 s 9 s10 s 11 s 11 s 12 [ 5 2 As expected Ω H 2) is also an invertible matrix in the code alphabet F 11, and we have Ω H 2)) 1 ΘH 2) = = Ξ H 2) ] Finally, the decoder calculates the lost data x 2), using 39), with e 2mµ x 2) = Θ H 2)Υ H2) +3ΞH 2)Υ H 2) = 7 4 = 3 in F 11, as ollows D A Reversed Viewpoint In this subsection we provide a dierent view to the main contribution o this work in the orm o the ollowing Corollary This result is indeed a direct corollary o the Theorem 1 Corollary 1 Consider a inite set o δ positive integers D = {d 1,,d δ }, 1 < d 1 < < d δ, and let µ be a positive common divisor o all the elements o D, such that µ / D Then we introduce an adaptive bandwidth MSR code, with k = µ+1, and a small sub-packetization level α and total storage capacity F, satisying and α = lcm1,2,, )µ, F = kα, which is capable o perorming exact-repair using any arbitrary d i helpers, or any d i D, and simultaneously satisying the MSR characteristic equation 3) or all d i D Proo Simply note that setting the code design parameter,µ, andδ, or the coding scheme presented in the previous subsections guarantees that the sub-packetization level and total storage capacity requirements are satisied Moreover, the presented coding scheme will provide the set o possible choices or the number o helpers or exact-repair as given in 7), which contains the required set D Remark 4 Note that or the case d 1 = 1, which is excluded in the above theorem, since any node should be repairable by any single helper, the trivial repetition code which stores the same α symbols in all the nodes in the network is the only possible solution, and it is easy to see that it is a bandwidth adaptive exact-repair MSR with k = 1, as it satisies the MDS property 2), and the MSR characteristic equation 3) could be satisied with setting β i = α/d i, or any i {1,,δ}, as long as d i α V DISCUSSION AND PROPERTIES In this section we will briely review some o the technical requirements and properties o the coding scheme presented in this work Particularly, we will show that the ield size and sub-packetization level requirements o the presented coding scheme are not signiicantly limiting actors in the practical implementations

17 17 A Field Size Requirement The only actor that inluences the choice o the code alphabet F q in the presented coding scheme is the existence coeicient matrix Ψ, and all the inverses o its square submatrices To satisy this requirement or a network with n storage nodes, it is enough to have q n [32], which is the same as the ield size requirement o many other coding schemes such as the Product Matrix codes [6] or the commonly used Reed-Solomon codes [4] Hence, the presented coding scheme in this work is providing the bandwidth adaptivity property at no extra cost in the ield size requirements, and any ield o size larger than n could be used as the code alphabet It worth mentioning that the ield size requirement o the only other bandwidth adaptive exact-repair MSR constructions, introduced in [2] is lower bounded by n 2 kn which is signiicantly larger or large distributed storage networks n Moreover, techniques such as those presented in [33] could easily be applied in the presented coding scheme to reduce the ield size to any arbitrary eg binary) small ield However, this discussion is out o the scope o this work B Subpacketization Level The requirement or the sub-packetization level α or the presented coding scheme is given in by 6) A natural question that arises is how ast does α grow as a unction o the code design parameters, µ, and δ From 6) it is clear that α is proportional to the design parameter a Regarding the dependency o α on δ, rom [34] we have, and, recently [35] showed that As a result, or the presented coding scheme, we have lcm1,2,,δ) 4 δ, lcm1,2,,δ) 2 δ, or δ 7 µ2 δ α µ4 δ, or δ 7 VI CONCLUSION We presented an alternative solution or exact-repair MSR codes in which optimal exact-repair is guaranteed simultaneously with a range o choices, D = {d 1,,d δ }, or the number o helpers The introduced coding scheme is based on the Product Matrix ramework, introduced irst in [6] The repair mechanism in this ramework is based on speciic symmetries in the structure o the data matrix We proposed a novel structure or the data matrix, which preserves the required symmetries in many dierent submatrices Corresponding repair mechanisms are also introduced to utilise these symmetries to perorm optimal repair in dierent scenarios, whit varying number o helpers In addition, the data reconstruction procedure is enhanced based on a novel successive intererence cancellation scheme to perorm optimally under the new design o data matrix Comparing to the only other explicit constructions, presented in [2], we showed that when d i 2k 2, d i D, the required values or α, and β are reduced to rom z n δ to kz δ or a DSS with n nodes, and k systematic nodes Multiplying both sides o 15) by Φ rom right we get Following the notation in [6], we introduce Then using 47) and the above equations we have APPENDIX A PROOF OF LEMMA 2 XΦ = ΦAΦ + ΦBΦ 47) P = ΦAΦ, Q = ΦBΦ XΦ = P + Q 48) Note that both P and Q are symmetric µ + 1) µ + 1) matrices Recall that is a diagonal matrix, with non-zero and distinct diagonal elements Hence, or any 1 i < j µ+1), we now have both and XΦ ) i,j = P i,j + i,i Q i,j, 49) XΦ ) j,i = P j,i + j,j Q j,i = P i,j + j,j Q i,j 50)