A Fundamental Tradeoff between Computation and Communication in Distributed Computing

Transcription

1 1 A Fundamental Tadeoff between Computation and Communication in Ditibuted Computing Songze Li, Student embe, IEEE, ohammad Ali addah-ali, embe, IEEE, Qian Yu, Student embe, IEEE, and A. Salman Avetimeh, Senio embe, IEEE axiv: v2 [c.it] 23 Sep 2017 Abtact How can we optimally tade exta computing powe to educe the communication load in ditibuted computing? We anwe thi quetion by chaacteizing a fundamental tadeoff between computation and communication in ditibuted computing, i.e., the two ae inveely popotional to each othe. oe pecifically, a geneal ditibuted computing famewok, motivated by commonly ued tuctue like apreduce, i conideed, whee the oveall computation i decompoed into computing a et of ap and Reduce function ditibutedly aco multiple computing node. A coded cheme, named Coded Ditibuted Computing CDC, i popoed to demontate that inceaing the computation load of the ap function by a facto of i.e., evaluating each function at caefully choen node can ceate novel coding oppotunitie that educe the communication load by the ame facto. An infomation-theoetic lowe bound on the communication load i alo povided, which matche the communication load achieved by the CDC cheme. A a eult, the optimal computation-communication tadeoff in ditibuted computing i exactly chaacteized. Finally, the coding technique of CDC i applied to the Hadoop TeaSot benchmak to develop a novel CodedTeaSot algoithm, which i empiically demontated to peed up the oveall job execution by , fo typical etting of inteet. Index Tem Ditibuted Computing, apreduce, Computation-Communication Tadeoff, Coded ulticating, Coded TeaSot I. ITRODUCTIO We conide a geneal ditibuted computing famewok, motivated by pevalent tuctue like apreduce [4] and Spak [5], in which the oveall computation i decompoed into two tage: ap and Reduce. Fitly in the ap tage, ditibuted computing node poce pat of the input data S. Li, Q. Yu and A.S. Avetimeh ae with the Depatment of Electical Engineeing, Univeity of Southen Califonia, Lo Angele, CA, 90089, USA ongzeli@uc.edu; qyu880@uc.edu; avetimeh@ee.uc.edu.. A. addah-ali i with Depatment of Electical Engineeing, Shaif Univeity of Technology, Tehan, 11365, Ian maddah_ali@haif.edu. A peliminay pat of thi wok wa peented in 53d Annual Alleton Confeence on Communication, Contol, and Computing, 2015 [1]. A pat of thi wok wa peented in IEEE Intenational Sympoium on Infomation Theoy, 2016 [2]. A pat of thi wok wa peented in the 6th Intenational Wokhop on Paallel and Ditibuted Computing fo Lage Scale achine Leaning and Big Data Analytic, 2017 [3]. Thi wok i in pat uppoted by SF gant CCF , ETS , OR awad , SA Awad o. H C-0255, and a eeach gift fom Intel. Thi mateial i baed upon wok uppoted by Defene Advanced Reeach Poject Agency DARPA unde Contact o. HR001117C0053. The view, opinion, and/o finding expeed ae thoe of the autho and hould not be intepeted a epeenting the official view o policie of the Depatment of Defene o the U.S. Govenment. locally, geneating ome intemediate value accoding to thei deigned ap function. ext, they exchange the calculated intemediate value among each othe a.k.a. data huffling, in ode to calculate the final output eult ditibutedly uing thei deigned Reduce function. Within thi famewok, data huffling often appea to limit the pefomance of ditibuted computing application, including elf-join [6], tea-ot [7], and machine leaning algoithm [8]. Fo example, in a Facebook Hadoop clute, it i obeved that 33% of the oveall job execution time i pent on data huffling [8]. Alo a i obeved in [9], 70% of the oveall job execution time i pent on data huffling when unning a elf-join application on an Amazon EC2 clute [10]. A uch motivated, we ak thi fundamental quetion that if coding can help ditibuted computing in educing the load of communication and peeding up the oveall computation? Coding i known to be helpful in coping with the channel uncetainty in telecommunication and alo in educing the toage cot in ditibuted toage ytem and cache netwok. In thi wok, we extend the application of coding to ditibuted computing and popoe a famewok to ubtantially educe the load of data huffling via coding and ome exta computing in the ap phae. oe pecifically, we fomulate and chaacteize a fundamental tadeoff elationhip between computation load in the ap phae and communication load in the data huffling phae, and demontate that the two ae inveely popotional to each othe. We popoe an optimal coded cheme, named Coded Ditibuted Computing CDC, which demontate that inceaing the computation load of the ap phae by a facto of i.e., evaluating each ap function at caefully choen node can ceate novel coding oppotunitie in the data huffling phae that educe the communication load by the ame facto. To illutate ou main eult, conide a ditibuted computing famewok to compute Q abitay output function fom input file, uing ditibuted computing node. A mentioned ealie, the oveall computation i pefomed by computing a et of ap and Reduce function ditibutedly aco the node. In the ap phae, each input file i poceed locally, in one of the node, to geneate Q intemediate value, each coeponding to one of the Q output function. Thu, at the end of thi phae, Q intemediate value ae calculated, which can be plit into Q ubet of intemediate value and each ubet i needed to calculate one of the output function. In the Shuffle phae, fo evey output function to be calculated, all intemediate value

2 2 coeponding to that function ae tanfeed to one of the node fo eduction. Of coue, depending on the node that ha been choen to educe an output function, a pat of the intemediate value ae aleady available locally, and do not need to be tanfeed in the Shuffle phae. Thi i becaue that the ap phae ha been caied out on the ame et of node, and the eult of mapping done at a node can emain in that node to be ued fo the Reduce phae. Thi offe ome aving in the load of communication. To educe the communication load even moe, we may map each input file in moe than one node. Appaently, thi inceae the faction of intemediate value that ae locally available. Howeve, a we will how, thee i a bette way to exploit thi edundancy in computation to educe the communication load. The main meage of thi pape i to how that following a paticula patten in epeating ap computation along with ome coding technique, we can ignificantly educe the load of communication. Pehap upiingly, we how that the gain of coding in educing communication load cale with the ize of the netwok. To be moe pecie, we define the computation load, 1, a the total numbe of computed ap function at the node, nomalized by. Fo example, = 1 mean that none of the ap function ha been e-computed, and = 2 mean that on aveage each ap function can be computed on two node. We alo define communication load L, 0 L 1, a the total amount of infomation exchanged aco node in the huffling phae, nomalized by the ize of Q intemediate value, in ode to compute the Q output function dijointly and unifomly aco the node. Baed on thi fomulation, we now ak the following fundamental quetion: Given a computation load in the ap phae, what i the minimum communication load L, uing any data huffling cheme, needed to compute the final output function? We popoe Coded Ditibuted Computing CDC that achieve a communication load of L coded = 1 1 fo = 1,...,, and the lowe convex envelop of thee point. CDC employ a pecific tategy to aign the computation of the ap and Reduce function aco the computing node, in ode to enable novel coding oppotunitie fo data huffling. In paticula, fo a computation load {1,..., }, CDC utilize a caefully deigned epetitive mapping of data block at ditinct node to ceate coded multicat meage that delive data imultaneouly to a ubet of 1 node. Hence, compaed with an uncoded data huffling cheme, which a we how late achieve a communication load L uncoded = 1, CDC i able to educe the communication load by exactly a facto of the computation load. Futhemoe, the popoed CDC cheme applie to a moe geneal ditibuted computing famewok whee evey output function i computed by moe than one, o paticulaly {1,..., } node, which povide bette fault-toleance in ditibuted computing. We numeically compae the computation-communication tadeoff of CDC and uncoded data huffling cheme i.e., L coded and L uncoded in Fig. 1. A it i illutated, in the uncoded cheme that achieve a communication load L uncoded = 1, inceaing the computation load offe only a modet eduction in communication load. In Communication Load L Uncoded Scheme Coded Ditibuted Computing Computation Load Fig. 1: Compaion of the communication load achieved by Coded Ditibuted Computing L coded with that of the uncoded cheme L uncoded, fo Q = 10 output function, = 2520 input file and = 10 computing node. Fo {1,..., }, CDC i time bette than the uncoded cheme. fact fo any, thi gain vanihe fo lage numbe of node. Conequently, it i not jutified to tade computation fo communication uing uncoded cheme. Howeve, fo the coded cheme that achieve a communication load of L coded = 1 1, inceaing the computation load will ignificantly educe the communication load, and thi gain doe not vanih fo lage. Fo example a illutated in Fig. 1, when mapping each file at one exta node = 2, CDC educe the communication load by 55.6%, while the uncoded cheme only educe it by 11.1%. We alo pove an infomation-theoetic lowe bound on the minimum communication load L. To pove the lowe bound, we deive a lowe bound on the total numbe of bit communicated by any ubet of node, uing induction on the ize of the ubet. To deive the lowe bound fo a paticula ubet of node, we fit etablih a lowe bound on the numbe of bit needed by one of the node to ecove the intemediate value it need to calculate it aigned output function, and then utilize the bound on the numbe of bit communicated by the et of the node in that ubet, which i given by the inductive agument. The deived lowe bound on L matche the communication load achieved by the CDC cheme fo any computation load 1. A a eult, we exactly chaacteize the optimal tadeoff between computation load and communication load in the following: L = L coded = 1 1, {1,..., }. Fo geneal 1, L i the lowe convex envelop of the above point {, L coded : {1,..., }}. ote 1 that fo lage, 1 1, hence L 1. Thi eult eveal a fundamental inveely popotional elationhip between computation load and communication load in ditibuted computing. Thi alo illutate that the gain of 1 achieved by CDC i optimal and it cannot be impoved by any othe cheme ince L coded i an infomation-theoetic lowe bound on L that applie to any data huffling cheme.

3 3 Having theoetically chaacteized the optimal computationcommunication tadeoff achieved by the popoed CDC cheme, we alo empiically demontate the pactical impact of thi tadeoff. In paticula, we apply the coding technique of CDC to a widely ued Hadoop oting benchmak TeaSot [11], developing a novel coded ditibuted oting algoithm CodedTeaSot [3]. We pefom extenive expeiment on Amazon EC2 clute, and obeve that fo typical etting of inteet, CodedTeaSot peed up the oveall execution of the conventional TeaSot by a facto of Finally, we dicu ome futue diection to extend the eult of thi wok. In paticula, we conide topic including heteogeneou netwok with aymmetic tak, taggling/failing computing node, multi-tage computation tak, multi-laye netwok and tuctued topology, joint toage and computation optimization, and coded edge/fog computing. Related Wok. The poblem of chaacteizing the minimum communication fo ditibuted computing ha been peviouly conideed in eveal etting in both compute cience and infomation theoy communitie. In [12], a baic computing model i popoed, whee two paitie have x and y and aim to compute a boolean function fx, y by exchanging the minimum numbe of bit between them. Alo, the poblem of minimizing the equied communication fo computing the modulo-two um of ditibuted binay ouce with ymmetic joint ditibution wa intoduced in [13]. Following thee two eminal wok, a wide ange of communication poblem in the cope of ditibuted computing have been tudied ee, e.g., [14] [19]. The key diffeence ditinguihing the etting in thi pape fom mot of the pio one ae 1 We focu on the flow of communication in a geneal ditibuted computing famewok, motivated by apreduce, athe than the tuctue of the function o the input ditibution. 2 We do not impoe any containt on the numbe of output eult, input data file and computing node they can be abitaily lage, 3 We do not aume any pecial popety e.g. lineaity of the computed function. The idea of efficiently ceating and exploiting coded multicating wa initially popoed in the context of cache netwok in [20], [21], and extended in [22], [23], whee cache pefetch pat of the content in a way to enable coding duing the content delivey, minimizing the netwok taffic. In thi pape, we popoe a famewok to tudy the tadeoff between computation and communication in ditibuted computing. We demontate that the coded multicating oppotunitie exploited in the above caching poblem alo exit in the data huffling of ditibuted computing famewok, which can be ceated by a tategy of epeating the computation of the ap function pecified by the Coded Ditibuted Computing CDC cheme. Finally, in a ecent wok [24], the autho have popoed method fo utilizing code to peed up ome pecific ditibuted machine leaning algoithm. The conideed poblem in thi pape diffe fom [24] in the following apect. We popoe a geneal methodology fo utilizing coding in data huffling that can be applied to any ditibuted computing famewok with a apreduce tuctue, egadle of the undelying application. In othe wod, any ditibuted computing algoithm that fit in the apreduce famewok can benefit fom the popoed CDC olution. We alo chaacteize the infomation-theoetic computation-communication tadeoff in uch famewok. Futhemoe, the coding ued in [24] i at the application laye i.e., applying computation on coded data, while in thi pape we focu on applying code diectly on the huffled data. II. PROBLE FORULATIO In thi ection, we fomulate a geneal ditibuted computing famewok motivated by apreduce, and define the function chaacteizing the tadeoff between computation and communication. We conide the poblem of computing Q abitay output function fom input file uing a clute of ditibuted computing node eve, fo ome poitive intege Q,,, with. oe pecifically, given input file w 1,..., w F 2 F, fo ome F, the goal i to compute Q output function φ 1,..., φ Q, whee φ q : F 2 F F 2 B, q {1,..., Q} map all input file to a length-b binay team u q = φ q w 1,..., w F 2 B, fo ome B. otivated by apreduce, we aume that a illutated in Fig. 2 the computation of the output function φ q, q {1,..., Q} can be decompoed a follow: whee φ q w 1,..., w = h q g q,1 w 1,..., g q, w, 1 The ap function g n = g 1,n,..., g Q,n : F 2 F F 2 T Q, n {1,..., } map the input file w n into Q length-t intemediate value v q,n = g q,n w n F 2 T, q {1,..., Q}, fo ome T. 1 The Reduce function h q : F 2 T F 2 B, q {1,..., Q} map the intemediate value of the output function φ q in all input file into the output value u q = h q v q,1,..., v q,. Remak 1. ote that fo evey et of output function φ 1,..., φ Q uch a ap-reduce decompoition exit e.g., etting g q,n to identity function uch that g q,n w n = w n fo all n = 1,...,, and h q to φ q in 1. Howeve, uch a decompoition i not unique, and in the ditibuted computing liteatue, thee ha been quite ome wok on developing appopiate decompoition of computation like join, oting and matix multiplication ee, e.g., [4], [25], fo them to be pefomed efficiently in a ditibuted manne. Hee we do not impoe any containt on how the ap and Reduce function ae choen fo example, they can be abitay linea o nonlinea function. 1 When mapping a file, we compute Q intemediate value in paallel, one fo each of the Q output function. The main eaon to do thi i that paallel poceing can be efficiently pefomed fo application that fit into the apreduce famewok. In othe wod, mapping a file accoding to one function i only maginally moe expenive than mapping accoding to all function. Fo example, fo the canonical Wod Count job, while we ae canning a document to count the numbe of appeaance of one wod, we can imultaneouly count the numbe of appeaance of othe wod with maginally inceaed computation cot.

4 4 i.e., fo ome encoding function ψ k : F 2 T Q k F 2 l k ode k, we have at X k = ψ k { g n : w n k }. 2 ap Function Reduce Function Fig. 2: Illutation of a two-tage ditibuted computing famewok. The oveall computation i decompoed into computing a et of ap and Reduce function. The above computation i caied out by ditibuted computing node, labelled a ode 1,..., ode. They ae inteconnected though a multicat netwok. Following the above decompoition, the computation poceed in thee phae: ap, Shuffle and Reduce. ap Phae: ode k, k {1,..., } compute the ap function of a et of file k, which ae toed on ode k, fo ome deign paamete k {w 1,..., w }. Fo each file w n in k, ode k compute g n w n =v 1,n,..., v Q,n. We aume that each file i mapped by at leat one node, i.e., k=1,..., k = {w 1,..., w }. Definition 1 Computation Load. We define the computation load, denoted by, 1, a the total numbe of ap function computed aco the node, nomalized by the k=1 numbe of file, i.e., k. The computation load can be intepeted a the aveage numbe of node that map each file. Shuffle Phae: ode k, k {1,..., } i eponible fo computing a ubet of output function, whoe indice ae denoted by a et W k {1,..., Q}. We focu on the cae Q, and utilize a ymmetic tak aignment aco the node to maintain load balance. oe peciely, we equie 1 W 1 = = W = Q, 2 W j W k = fo all j k. Remak 2. Beyond the ymmetic tak aignment conideed in thi pape, chaacteizing the optimal computationcommunication tadeoff allowing geneal aymmetic tak aignment i a challenging open poblem. A the fit tep to tudy thi poblem, in ou follow-up wok [26] in which the numbe of output function Q i fixed and the computing eouce ae abundant e.g., numbe of computing node Q, we have hown that aymmetic tak aignment can do bette than the ymmetic one, and achieve the optimum un-time pefomance. To compute the output value u q fo ome q W k, ode k need the intemediate value that ae not computed locally in the ap phae, i.e., {v q,n : q W k, w n / k }. Afte ode k, k {1,..., } ha finihed mapping all the file in k, the node poceed to exchange the needed intemediate value. In paticula, each node k, k {1,..., }, ceate an input ymbol X k F 2 l k, fo ome l k, a a function of the intemediate value computed locally duing the ap phae, Having geneated the meage X k, ode k multicat it to all othe node. By the end of the Shuffle phae, each of the node eceive X 1,..., X fee of eo. Definition 2 Communication Load. We define the communication load, denoted by L, 0 L 1, a L l1+ +l QT. That i, L epeent the nomalized total numbe of bit communicated by the node duing the Shuffle phae. 2 Reduce Phae: ode k, k {1,..., }, ue the meage X 1,..., X communicated in the Shuffle phae, and the local eult fom the ap phae { g n : w n k } to contuct input to the coeponding Reduce function of W k, i.e., fo each q W k and ome decoding function χ q k : F 2 l 1 F 2 l F 2 T Q k F 2 T, ode k compute v q,1,..., v q, = χ q k X 1,..., X, { g n : w n k }. 3 Finally, ode k, k {1,..., }, compute the Reduce function u q = h q v q,1... v q, fo all q W k. We ay that a computation-communication pai, L R 2 i feaible if fo any δ > 0 and ufficiently lage, thee exit 1,...,, W 1,..., W, a et of encoding function {ψ k } k=1, and a et of decoding function {χq k : q W k} k=1 that achieve a computation-communication pai, L Q 2 uch that δ, L L δ, and ode k can uccefully compute all the output function whoe indice ae in W k, fo all k {1,..., }. Definition 3. We define the computation-communication function of the ditibuted computing famewok L inf{l :, L i feaible}. 4 L chaacteize the optimal tadeoff between computation and communication in thi famewok. Example Uncoded Scheme. In the Shuffle phae of a imple uncoded cheme, each node eceive the needed intemediate value ent uncodedly by ome othe node. Since a total of Q intemediate value ae needed aco the node and Q of them ae aleady available afte the ap phae, the communication load achieved by the uncoded cheme L uncoded = 1 /. 5 = Q Remak 3. Afte the ap phae, each node know the intemediate value of all Q output function in the file it ha mapped. Theefoe, fo a fixed file aignment and any ymmetic aignment of the Reduce function, pecified by 2 Fo notational convenience, we define all vaiable in binay extenion field. Howeve, one can conide abitay field ize. Fo example, we can conide all intemediate value v q,n, q = 1,..., Q, n = 1,...,, to be in the field F p T, fo ome pime numbe p and poitive intege T, and the ymbol communicated by ode k i.e., X k, to be in the field F l k fo ome pime numbe and poitive intege l k, fo all k = 1,...,. In thi cae, the communication load can be defined a L l 1+ +l log. QT log p

5 5 W 1,..., W, we can atify the data equiement uing the ame data huffling cheme up to elabelling the Reduce function. In othe wod, the communication load i independent of the aignment of the Reduce function. In thi pape, we alo conide a genealization of the above famewok, which we call cacaded ditibuted computing famewok, whee afte the ap phae, each Reduce function i computed by moe than one, o paticulaly node, fo ome {1,..., }. Thi genealized model i motivated by the fact that many ditibuted computing job equie multiple ound of ap and Reduce computation, whee the Reduce eult of the peviou ound eve a the input to the ap function of the next ound. Computing each Reduce function at moe than one node admit data edundancy fo the ubequent ap-function computation, which can help to impove the fault-toleance and educe the communication load of the next-ound data huffling. We focu on the cae Q, and enfoce a ymmetic aignment of the Reduce tak to maintain load balance. Paticulaly, we equie that Q evey ubet of node compute a dijoint ubet of Reduce function. The feaible computation-communication tiple,, L R R i defined imila a befoe. We define the computation-communication function of the cacaded ditibuted computing famewok L, inf{l :,, L i feaible}. 6 III. AI RESULTS Theoem 1. The computation-communication function of the ditibuted computing famewok, L i given by L = L coded 1 1, {1,..., }, 7 fo ufficiently lage T. Fo geneal 1, L i the lowe convex envelop of the above point {, 1 1 : {1,..., }}. We pove the achievability of Theoem 1 by popoing a coded cheme, named Coded Ditibuted Computing, in Section V. We demontate that no othe cheme can achieve a communication load malle than the lowe convex envelop of the point {, 1 1 : {1,..., }} by poving the convee in Section VI. Remak 4. Theoem 1 exactly chaacteize the optimal tadeoff between the computation load and the communication load in the conideed ditibuted computing famewok. Remak 5. Fo {1,..., }, the communication load achieved in Theoem 1 i le than that of the uncoded cheme in 5 by a multiplicative facto of, which equal the computation load and can gow unboundedly a the numbe of node inceae if e.g. = Θ. A illutated in Fig. 1 in Section I, while the communication load of the uncoded cheme deceae linealy a the computation load inceae, L coded achieved in Theoem 1 i inveely popotional to the computation load. Remak 6. While inceaing the computation load caue a longe ap phae, the coded achievable cheme of Theoem 1 maximize the eduction of the communication load uing the exta computation. Theefoe, Theoem 1 povide an analytical famewok to optimally tading the computation powe in the ap phae fo moe bandwidth in the Shuffle phae, which help to minimize the oveall execution time of application whoe pefomance ae limited by data huffling. Theoem 2. The computation-communication function of the cacaded ditibuted computing famewok, L,, fo {1,..., }, i chaacteized by L, = L coded, min{+,} l=max{+1,} l l 2 l 1 l, 8 fo ome {1,..., } and ufficiently lage T. Fo geneal 1, L, i the lowe convex envelop of the above point {, L coded, : {1,..., }}. We peent the Coded Ditibuted Computing cheme that achieve the computation-communication function in Theoem 2 in Section V, and the convee of Theoem 2 in Section VII. Remak 7. A peliminay pat of thi eult, in paticula the achievability fo the pecial cae of = 1, o the achievable cheme of Theoem 1 wa peented in [1]. We note that when = 1, Theoem 2 povide the ame eult a in Theoem 1, i.e., L, 1 = 1 1, fo {1,..., }. Communication Load L =1 =2 = Computation Load Fig. 3: inimum communication load L, = L coded, in Theoem 2, fo Q = 360 output function, = 2520 input file and =10 computing node. Remak 8. Fo any fixed {1,..., } numbe of node that compute each Reduce function, a illutated in Fig. 3, the communication load achieved in Theoem 2 outpefom the linea elationhip between computation and communication, i.e., it i upelinea with epect to the computation load. Befoe we poceed to decibe the geneal achievability cheme fo the cacaded ditibuted computing famewok alo the ditibuted computing famewok a a pecial cae of = 1, we fit illutate the key idea of the popoed Coded

6 6 Ditibuted Computing cheme by peenting two example in the next ection, fo the cae of = 1 and > 1 epectively. IV. ILLUSTRATIVE EXAPLES: CODED DISTRIBUTED COPUTIG In thi ection, we peent two illutative example of the popoed achievable cheme fo Theoem 1 and Theoem 2, which we call Coded Ditibuted Computing CDC, fo the cae of = 1 Theoem 1 and > 1 Theoem 2 epectively. Example 1 CDC fo = 1. We conide a apreducetype poblem in Fig. 4 fo ditibuted computing of Q = 3 output function, epeented by ed/cicle, geen/quae, and blue/tiangle epectively, fom = 6 input file, uing = 3 computing node. ode 1, 2, and 3 ae epectively eponible fo final eduction of ed/cicle, geen/quae, and blue/tiangle output function. Let u fit conide the cae whee no edundancy i impoed on the computation, i.e., each file i mapped once and computation load = 1. A hown in Fig. 4a, ode k map File 2k 1 and File 2k fo k = 1, 2, 3. In thi cae, each node map 2 input file locally, computing all thee intemediate value needed fo the thee output function fom each mapped file. In Fig. 4, we epeent, fo example, the intemediate value of the ed/cicle function in File n uing a ed cicle labelled by n, fo all n = 1,..., 6. Simila epeentation follow fo the geen/quae and the blue/tiangle function. Afte the ap phae, each node obtain 2 out of 6 equied intemediate value to educe the output function it i eponible fo e.g., ode 1 know the ed cicle in File 1 and File 2. Hence, each node need 4 intemediate value fom the othe node, yielding a communication load of = 2 3. ow, we demontate how the popoed CDC cheme tade the computation load to lah the communication load via in-netwok coding. A hown in Fig. 4b, we double the computation load uch that each file i now mapped on two node = 2. It i appaent that ince moe local computation ae pefomed, each node now only equie 2 othe intemediate value, and an uncoded huffling cheme would achieve a communication load of = 1 3. Howeve, we can do much bette with coding. A hown in Fig. 4b, intead of unicating individual intemediate value, evey node multicat a bit-wie XOR, denoted by, of 2 locally computed intemediate value to the othe two node, imultaneouly atifying thei data demand. Fo example, knowing the blue/tiangle in File 3, ode 2 can cancel it fom the coded packet ent by ode 1, ecoveing the needed geen/quae in File 1. Theefoe, thi coding incu a communication load of = 1 6, achieving a 2 gain fom the uncoded huffling. Fom the above example, we ee that fo the cae of = 1, i.e., each of the Q output function i computed on one node and the computation of the Reduce function ae ymmetically ditibuted aco node, the popoed CDC cheme only equie pefoming bit-wie XOR a the encoding and decoding opeation. Howeve, fo the cae of > 1, a we File Ha eed File Ha eed ode 1 ap ode 3 ap File Ha eed ap File Ha ap eed 1 2 a Uncoded Ditibuted Computing Scheme. ode File Ha eed ap Ha eed ode ap 1 2 ode 1 ode 2 6 File 24 b Coded Ditibuted Computing Scheme. 3 Fig. 4: Illutation of the conventional uncoded ditibuted computing cheme with computation load = 1, and the popoed Coded Ditibuted Computing cheme with computation load = 2, fo computing Q = 3 function fom = 6 input on = 3 node. will how in the following example, the popoed CDC cheme equie computing linea combination of the intemediate value duing the encoding poce. Example 2 CDC fo > 1. In thi example, we conide a job of computing Q = 6 output function fom = 6 input file, uing = 4 node. We focu on the cae whee the computation load = 2, and each Reduce function i computed by = 2 node. In the ap phae, each file i mapped by = 2 node. A hown in Fig. 5, the et of the file mapped by the 4 node ae 1 = {w 1, w 2, w 3 }, 2 = {w 1, w 4, w 5 }, 3 = {w 2, w 4, w 6 }, and 4 = {w 3, w 5, w 6 }. Afte the ap phae, ode k, k {1, 2, 3, 4}, know the intemediate value of all Q = 6 output function in the file in k, i.e., {v q,n : q {1,..., 6}, w n k }. In the Reduce phae, we aign the computation of the Reduce function in a ymmetic manne uch that evey ubet of = 2 node compute a common Reduce function. oe pecifically a hown in Fig. 5, the et of indice of the Reduce function computed by the 4 node ae W 1 ={1, 2, 3},

7 7 W 2 = {1, 4, 5}, W 3 = {2, 4, 6}, and W 4 = {3, 5, 6}. Theefoe, fo example, ode 1 till need the intemediate value {v q,n : q {1, 2, 3}, n {4, 5, 6}} though data huffling to compute it aigned Reduce function h 1, h 2, h 3. ode 1 ode 2 ode 3 ode 4 ulticat ulticat ulticat ulticat ode 1 ode 2 ode 3 ode 4 Fig. 5: Illutation of the CDC cheme to compute Q = 6 output function fom = 6 input file ditibutedly at = 4 computing node. Each file i mapped by = 2 node and each output function i computed by = 2 node. Afte the ap phae, evey node know 6 intemediate value, one fo each output function, in evey file it ha mapped. The Shuffle phae poceed in two ound. In the fit ound, each node multicat bit-wie XOR of intemediate value to ubet of two node. In the econd ound, each node plit an intemediate value v q,n evenly into two egment v q,n = v q,n, 1 v q,n, 2 and multicat two linea combination of the egment that ae contucted uing coefficient α 1, α 2, and α 3 to the othe thee node. The data huffling poce conit of two ound of communication ove the multicat netwok. In the fit ound, intemediate value ae communicated within each ubet of 3 node. In the econd ound, intemediate value ae communicated within the et of all 4 node. In what follow, we decibe thee two ound of communication epectively. Round 1: Subet of 3 node. We fit conide the ubet {1, 2, 3}. Duing the data huffling, each node whoe index i in {1, 2, 3} multicat a bit-wie XOR of two locally computed intemediate value to the othe two node: ode 1 multicat v 1,2 v 2,1 to ode 2 and ode 3, ode 2 multicat v 4,1 v 1,4 to ode 1 and ode 3, ode 3 mulicat v 4,2 v 2,4 to ode 1 and ode 2, Since ode 2 know v 2,1 and ode 3 know v 1,2 locally, they can epectively decode v 1,2 and v 2,1 fom the coded meage v 1,2 v 2,1. We employ the imila coded huffling cheme on the othe 3 ubet of 3 node. Afte the fit ound of huffling, ode 1 ecove v 1,4, v 1,5, v 2,4, v 2,6 and v 3,5, v 3,6, ode 2 ecove v 1,2, v 1,3, v 4,2, v 4,6 and v 5,3, v 5,6, ode 3 ecove v 2,1, v 2,3, v 4,1, v 4,5 and v 6,3, v 6,5, ode 4 ecove v 3,1, v 3,2, v 5,1, v 5,4 and v 6,2, v 6,4. Round 2: All 4 node. We fit plit each of the intemediate value v 6,1, v 5,2, v 4,3, v 3,4, v 2,5, and v 1,6 into two equalized egment each containing T/2 bit, which ae denoted by v q,n 1 and v q,n 2 fo an intemediate value v q,n. Then, fo ome coefficient α 1, α 2, α 3 F T 2 2, ode 1 multicat the following two linea combination of thee locally computed egment to the othe thee node. v 1 4,3 + v1 5,2 + v1 6,1, 9 α 1 v 1 4,3 + α 2v 1 5,2 + α 3v 1 6,1. 10 Similaly, a hown in Fig. 5, each of ode 2, ode 3, and ode 4 multicat two linea combination of thee locally computed egment to the othe thee node, uing the ame coefficient α 1, α 2, and α 3. Having eceived the above two linea combination, each of ode 2, ode 3, and ode 4 fit ubtact out one egment available locally fom the combination, o moe pecifically, fo ode 2, v1 5,2 fo ode 3, and v1 4,3 fo ode 4. Afte the ubtaction, each of thee thee node ecove the equied v 1 6,1 egment fom the two linea combination. oe pecifically, ode 2 ecove v 1 4,3 and v1 5,2, ode 3 ecove v1 4,3 and v1 6,1, and ode 4 ecove v 1 5,2 and v1 6,1. It i not difficult to ee that the above decoding poce i guaanteed to be ucceful if α 1, α 2, and α 3 ae all ditinct fom each othe, which equie the field ize 2 T 2 3 e.g., T = 4. Following the imila pocedue, each node ecove the equied egment fom the linea combination multicat by the othe thee node. oe pecifically, afte the econd ound of data huffling, ode 1 ecove v 1,6, v 2,5 and v 3,4, ode 2 ecove v 1,6, v 4,3 and v 5,2, ode 3 ecove v 2,5, v 4,3 and v 6,1, ode 4 ecove v 3,4, v 5,2 and v 6,1. We finally note that in the econd ound of data huffling, each linea combination multicat by a node i imultaneouly ueful fo the et of the thee node. V. GEERAL ACHIEVABLE SCHEE: CODED DISTRIBUTED COPUTIG In thi ection, we fomally pove the uppe bound in Theoem 1 and 2 by peenting and analyzing the Coded Ditibuted Computing CDC cheme. We focu on the moe geneal cae conideed in Theoem 2 with 1, and the cheme fo Theoem 1 imply follow by etting = 1. We fit conide the intege-valued computation load {1,..., }, and then genealize the CDC cheme fo any 1. When =, evey node can map all the input file and compute all the output function locally, thu no communication i needed and L, = 0 fo all {1,..., }. In what follow, we focu on the cae whee <.

8 8 conide ufficiently lage numbe of input file, and We η1 1 < η1, fo ome η 1. We fit inject η1 empty file into the ytem to obtain a total of = η1 file, which i now a multiple of of. We note that = 1. ext, we poceed to peent the achievable lim cheme fo a ytem with input file w 1,..., w. A. ap Phae Deign In the ap phae the input file ae evenly patitioned into dijoint batche of ize η1, each coeponding to a ubet T {1,..., } of ize, i.e., {w 1,..., w } = T {1,...,}, T = B T, 11 whee B T denote the batch of η 1 file coeponding to the ubet T. Given thi patition, ode k, k {1,..., }, compute the ap function of the file in B T if k T. O equivalently, B T k if k T. Since each node i in 1 1 ubet of ize, each node compute 1 1 η1 = ap function, i.e., k = fo all k {1,..., }. Afte the ap phae, ode k, k {1,..., }, know the intemediate value of all Q output function in the file in k, i.e., {v q,n : q {1,..., Q}, w n k }. B. Coded Data Shuffling We ecall that we focu on the cae whee the numbe of the output function Q atifie Q, and enfoce a ymmetic aignment of the Reduce function uch that evey ubet of Q node educe function. Specifically, Q = η2 fo ome η 2, and the computation of the Reduce function ae aigned ymmetically aco the node a follow. Fitly the Q Reduce function ae evenly patitioned into dijoint batche of ize η 2, each coeponding to a unique ubet P of node, i.e., {1,..., Q} = P {1,...,}, P = D P, 12 whee D P denote the indice of the batch of η 2 Reduce function coeponding to the ubet P. Given thi patition, ode k, k {1,..., }, compute the Reduce function whoe indice ae in D P if k P. O equivalently, D P W k if k P. A a eult, each node compute 1 1 η2 = Q Reduce function, i.e., W k = Q fo all k {1,..., }. Fo a ubet S of {1,..., } and S with =, we denote the et of intemediate value needed by all node in S\, no node outide S, and known excluively by node in a V S\S1. oe fomally: V S\S1 {v q,n :q \ W k, q / k / S W k, w n k, w n / k } k / S1 We obeve that the et V S\S1 defined above contain intemediate value of S η2 output function. Thi i becaue that the output function whoe intemediate value ae included in V S\S1 hould be computed excluively by the node in S\ and a ubet of S node in. Theefoe, V S\S1 contain the intemediate value of a total of S η2 = S η2 output function. Since evey ubet of node map a unique batch of η 1 file, V S\S1 contain V S\S1 = S η1 η 2 intemediate value. ext, we fit concatenate all intemediate value in V S\S1 to contuct a ymbol U S\S1 F2. Then fo S η 1 η 2T = {σ 1,..., σ }, we abitaily and evenly plit U S\S1 into egment, each containing η1η 2T bit, i.e., S U S\S1 = U S\S1,σ 1, U S\S1,σ 2,..., U S\S1,σ, 14 whee U S\S1,σ i F2 S η 1 η 2 T denote the egment aociated with ode σ i. Fo each k S, thee ae a total of S 1 1 ubet of S with ize that contain the element k. We index thee ubet a S k [1], S k [2]..., S k [ S 1 1 ]. Within a ubet S k [i], the egment aociated with ode k i U S\S k[i] S k [i],k, fo all i = 1,..., S 1 S\S 1. We note that each egment U k [i] S k [i],k, i = 1,..., S 1 1, i known by all node whoe indice ae in S k [i], and needed by all node whoe indice ae in S\S k [i]. 1 Encoding: The huffling cheme of CDC conit of multiple ound, each coeponding to all ubet of the node with a paticula ize. Within each ubet, each node multicat linea combination of the egment that ae aociated with it to the othe node in the ubet. oe pecifically, fo each ubet S {1,..., } of ize max{+1, } S min{ +, }, we define n 1 S 1 1 and n2 S 2 1. Then fo each k S, ode k compute n 2 meage ymbol, denoted by Xk S[1], XS k [2],..., XS k [n 2] a follow. Fo ome coefficient α 1,..., α n1 whee α i F2 fo all i = 1,..., n 1, ode k compute X S k [1]=U S\S k[1] S k [1],k S η 1 η 2 T + U S\S k[2] S k [2],k + + U S\S k[n 1] S k [n 1],k, X S k [2]=α 1 U S\S k[1] S k [1],k +α 2U S\S k[2] S k [2],k + +α n 1 U S\S k[n 1] S k [n 1],k,. Xk S [n 2 ]=α n2 1 1 U S\S k[1] S k [1],k + αn2 1 2 U S\S k[2] S k [2],k o equivalently, X S k [1] X S k [2]. X S k [n 2] + + α n2 1 n 1 U S\S k[n 1] S k [n 1],k, 15 = α 1 α 2 α n α n2 1 1 α n2 1 2 αn n2 1 }{{ 1 } A S U S\S k[1] S k [1],k U S\S k[2] S k [2],k. U S\S k[n 1] S k [n 1],k. 16

9 9 We note that the above encoding poce i the ame at all node whoe indice ae in S, i.e., each of them multiplie the ame matix A S in 16 with the egment aociated with it. Having geneated the above meage ymbol, ode k multicat them to the othe node whoe indice ae in S. Remak 9. When = 1, i.e., evey output function i computed by one node, the above huffling cheme only take one ound fo all ubet S of ize S = + 1. Intead of multicating linea combination, evey node in S can imply multicat the bit-wie XOR of it aociated egment to the othe node in S. 2 Decoding: Fo j S and j k, thee ae a total of S 2 2 ubet of S that have ize and imultaneouly contain j and k. Hence, among all n 1 egment U S\S k[1] S k [1],k, U S\S k[2] S k [2],k,..., U S\S k[n 1] S k [n 1],k aociated with ode k, S 2 2 of them ae aleady known at ode j, and the et of n 1 S 2 2 = S 1 1 S 2 2 = S 2 1 = n2 egment ae needed by ode j. We denote the indice of the ubet that contain the element k but not the element j a b 1 jk, b2 jk,..., bn2 jk, uch that 1 b1 jk < b2 jk < < bn2 jk n 1, and j / S k [b i jk ] fo all i = 1, 2,..., n 2. Afte eceiving the ymbol X S k [1], XS k [2],..., XS k [n 2] fom ode k, ode j fit emove the locally known egment fom the linea combination to geneate n 2 ymbol Y S jk [1], Y S jk [2],..., Y S jk [n 2], uch that Yjk S [1] Yjk S [2]. = α b 1 jk α b 2 jk α n b 2 jk Yjk S [n 2] α n2 1 α n2 1 α n2 1 b 1 jk b 2 jk b n 2 jk }{{} whee B S jk Fn2 n2 2 S η 1 η 2 T B S jk U S\S k[b 1 jk ] S k [b 1 jk ],k U S\S k[b 2 jk ] S k [b 2 jk ],k. U S\S k[b n 2 jk ] S k [b n 2 jk ],k, 17 i a quae ub-matix of A S in 16 that contain the column with indice b 1 jk, b2 jk,..., bn2 jk of A S k. ode j can decode the deied egment fom ode k if the matix B S jk i invetible. We note that BS jk i a Vandemonde matix, and it i invetible if α b 1 jk, α b 2 jk,..., α n b 2 ae all jk ditinct. Thi hold fo all j S\{k} if thee exit n 1 ditinct coefficient in F2 S η 1 η 2 T, which equie 2 S η 1 η 2 T n 1 = S 1 log 1, o equivalently T S 1 1. Finally, the S η 1η 2 popoed coded huffling cheme can uccefully delive all the equied intemediate value within all ubet S with max{ + 1, } S min{ +, }, if T i ufficiently lage, i.e., T max max{+1,} S min{+,} C. Coectne of CDC log S η1 η 2 S We demontate the coectne of the above huffling cheme by howing that afte the Shuffle phae, each node can decode all of the equied intemediate value to compute it aigned Reduce function. We ue ode 1 a an example, and imila agument apply to all othe node. WLOG we aume that the Reduce function h 1 i to be computed by ode 1. ode 1 will need a total of 1 η1 ditinct intemediate value of h 1 fom othe node it aleady know = 1 η1 intemediate value of h 1 by mapping the file in 1. By the aignment of the Reduce function, thee exit a ubet S 2 of ize containing ode 1 uch that all node in S 2 need to compute h 1. Then, duing the data huffling poce within each ubet S containing S 2 note that by the definition of V S\S1 in 13, the intemediate value of h 1 will not be communicated to ode 1 if S 2 S, and thi i becaue that ome node outide S alo want to compute h 1, thee ae 1 S 1 ubet S1 of S with ize = uch that 1 / and S\ S 2, and thu ode 1 decode 1 S 1 η1 ditinct intemediate value of h 1. Theefoe, the total numbe of ditinct intemediate value of h 1 ode 1 decode ove the entie Shuffle phae i min{+,} l=max{+1,} 1 l 1 l η 1 = 1 η 1, 19 which matche the equied numbe of intemediate value fo h 1. Thi i alo tue fo all the othe Reduce function aigned to ode 1. D. Communication Load In the above huffling cheme, fo each ubet S {1,..., } of ize max{ + 1, } S min{ +, }, each ode k S communicate n 2 = S 2 1 meage ymbol. Each of thee ymbol contain η1η 2T S bit. Hence, all node whoe indice ae in S communicate a total of S S 2 η1η 2T 1 S bit. The oveall communication load achieved by the popoed CDC cheme i L coded, = lim min{+,} l=max{+1,} = lim = min{+,} l l l l=max{+1,} min{+,} l l 2 l 1 l=max{+1,} E. on-intege Valued Computation Load l l 2 1 l η1 η 2 T QT l 2 1 l l. 20 Fo non-intege valued computation load 1, we genealize the CDC cheme a follow. We fit expand the computation load = α 1 +1 α 2 a a convex combination of 1 and 2, fo ome 0 α 1. Then we patition the et of input file {w1,..., w } into two dijoint ubet I 1 and I 2 of ize I 1 = α and I 2 = 1 α. We next apply the CDC cheme decibed above epectively to the file in I 1 with a computation load 1 and the file in I 2 with a computation load 2, to compute

10 10 each of the Q output function at the ame et of node. Thi eult in a communication load of Qα lim L coded 1, T + Q1 α L coded 2, T QT =αl coded 1, + 1 αl coded 2,, 21 whee L coded, i the communication load achieved by CDC in 20 fo intege-valued, {1,..., }. Uing thi genealized CDC cheme, fo any two integevalued computation load 1 and 2, the point on the line egment connecting 1, L coded 1, and 2, L coded 2, ae achievable. Theefoe, fo geneal 1, the lowe convex envelop of the achievable point {, L coded, : {1,..., }} i achievable. Thi pove the uppe bound on the computation-communication function in Theoem 2 alo the achievability pat of Theoem 1 by etting = 1. Remak 10. The idea of efficiently ceating and exploiting coded multicating oppotunitie have been intoduced in caching poblem [20] [22]. In thi ection, we illutated how coding oppotunitie can be utilized in ditibuted computing to lah the load of communicating intemediate value, by deigning a paticula aignment of exta computation aco ditibuted computing node. We note that the calculated intemediate value in the ap phae mimic the locally toed cache content in caching poblem, poviding the ide infomation to enable coding in the following Shuffle phae o content delivey. Fo the cae of = 1 whee no two node ae inteeted in computing a common Reduce function, the coded data huffling of CDC i imila to a coded tanmiion tategy in wiele D2D netwok popoed in [22], whee the ide infomation enabling coded multicating ae pe-fetched in a pecific epetitive manne in the cache of wiele node in CDC uch infomation i obtained by computing the ap function locally. When i lage than 1, i.e., evey Reduce function need to be computed at multiple node, ou CDC cheme ceate novel coding oppotunitie that exploit both the edundancy of the ap computation and the commonality of the data equet fo Reduce function aco node, futhe educing the communication load. Remak 11. Geneally peaking, we can view the Shuffle phae of the conideed ditibuted computing famewok a an intance of the index coding poblem [27], [28], in which a cental eve aim to deign a boadcat meage code with minimum length to imultaneouly atify the equet of all the client, given the client ide infomation toed in thei local cache. ote that while a andomized linea netwok coding appoach ee e.g., [29] [31] i ufficient to implement any multicat communication whee meage ae intended by all eceive, it i geneally ub-optimal fo index coding poblem whee evey client equet diffeent meage. Although the index coding poblem i till open in geneal, fo the conideed ditibuted computing cenaio whee we ae given the flexibility of deigning ap computation thu the flexibility of deigning ide infomation, we pove in the next two ection tight lowe bound on the minimum communication load fo the cae = 1 and > 1 epectively, demontating the optimality of the popoed CDC cheme. VI. COVERSE OF THEORE 1 In thi ection, we pove the lowe bound on L in Theoem 1. Fo k {1,..., }, we denote the et of indice of the file mapped by ode k a k, and the et of indice of the Reduce function computed by ode k a W k. A the fit tep, we conide the communication load fo a given file aignment 1, 2..., in the ap phae. We denote the minimum communication load unde the file aignment by L. We denote the numbe of file that ae mapped at j node unde a file aignment, a a j, fo all j {1,..., }: a j = File 1 3 J {1,...,}: J =j k J k \ i / J i. 22 File File ode 1 ode 2 ode 3 Fig. 6: A file aignment fo = 6 file and = 3 node. Fo example, fo the paticula file aignment in Fig. 6, i.e., = {1, 3, 5, 6}, {4, 5, 6}, {2, 3, 4, 6}, a 1 = 2 ince File 1 and File 2 ae mapped on a ingle node i.e., ode 1 and ode 3 epectively. Similaly, we have a 2 = 3 File 3, 4, and 5, and a 3 = 1 File 6. Fo a paticula file aignment, we peent a lowe bound on L in the following lemma. Lemma 1. L a j j j. ext, we fit demontate the convee of Theoem 1 uing Lemma 1, and then give the poof of Lemma 1. Convee Poof of Theoem 1. It i clea that the minimum communication load L i lowe bounded by the minimum value of L ove all poible file aignment which admit a computation load of : L Then by Lemma 1, we have L inf : = L. 23 inf : = a j j j. 24 Fo evey file aignment uch that =, {a j } atify a j 0, j {1,..., }, 25 =, 26 a j ja j =. 27

11 11 Then ince the function j j in 24 i convex in j, and by 26 = 1, 24 become a j L inf : = j aj j aj a =, 28 whee a i due to the equiement impoed by the computation load in 27. The lowe bound on L in 28 hold fo geneal 1. We can futhe impove the lowe bound fo nonintege valued a follow. Fo a paticula /, we fit find the line p+qj a a function of 1 j connecting the two point, and,. oe pecifically, we find p, q R uch that p + qj j= =, 29 p + qj j= =. 30 Then by the convexity of the function j j in j, we have fo intege-valued j = 1,...,, j j Then 24 educe to L p + qj, j = 1,...,. 31 inf : = = inf : = a j a j p + qj 32 p + ja j q 33 b = p + q, 34 whee b i due to the containt on {a j } in 26 and 27. Theefoe, L i lowe bounded by the lowe convex envelop of the point {, : {1,..., }}. Thi complete the poof of the convee pat of Theoem 1. Remak 12. Although the model popoed in thi pape only allow each node ending meage independently, we can how that even if the data huffling poce can be caied out in multiple ound and dependency between meage ae allowed, the lowe bound on L emain the ame. We devote the et of thi ection to the poof of Lemma 1. To pove Lemma 1, we develop a lowe bound on the numbe of bit communicated by any ubet of node, by induction on the ize of the ubet. In paticula, fo a ubet of computing node, we fit chaacteize a lowe bound on the minimum numbe of bit equied by a paticula node in the ubet, which i given by a cut-et bound epaating thi node and all the othe node in the ubet. Then, we combine thi bound with the lowe bound on the numbe of bit communicated by the et of the node in the ubet, which i given by the inductive agument. Poof of Lemma 1. Fo q {1,..., Q}, n {1,..., }, we let V q,n be i.i.d. andom vaiable unifomly ditibuted on F 2 T. We let the intemediate value v q,n be the ealization of V q,n. Fo ome Q {1,..., Q} and {1,..., }, we define V Q, {V q,n : q Q, n }. 35 Since each meage X k i geneated a a function of the intemediate value that ae computed at ode k, the following equation hold fo all k {1,..., }. HX k V :,k = 0, 36 whee we ue : to denote the et of all poible indice. The validity of the huffling cheme equie that fo all k {1,..., }, the following equation hold : HV Wk,: X :, V :,k = Fo a ubet S {1,..., }, we define Y S V WS,:, V :,S, 38 which contain all the intemediate value equied by the node in S and all the intemediate value known locally by the node in S afte the ap phae. Fo any ubet S {1,..., } and a file aignment, we denote the numbe of file that ae excluively mapped by j node in S a a j,s a j,s : J S: J =j k J k \ i / J i, 39 and the meage ymbol communicated by the node whoe indice ae in S a X S = {X k : k S}. 40 Then we pove the following claim. Claim 1. Fo any ubet S {1,..., }, we have S HX S Y S c T a j,s Q S j, 41 j whee S c {1,..., }\S denote the complement of S. We pove Claim 1 by induction. a. If S = {k} fo any k {1,..., }, obviouly HX k Y {1,...,}\{k} 0 = T a 1,{k} Q b. Suppoe the tatement i tue fo all ubet of ize S 0. Fo any S {1,..., } of ize S = S 0 +1 and any k S, we have HX S Y S c = 1 HX S, X k Y S c 43 S = 1 HX S X k, Y S c + HX k Y S c 44 S 1 HX S X k, Y S c + 1 S S HX S Y S c. 45