The Three-Terminal Interactive. Lossy Source Coding Problem

Transcription

1 The Three-Termial Iteractive 1 Lossy Source Codig Problem Leoardo Rey Vega, Pablo Piataida ad Alfred O. Hero III arxiv: v3 cs.it] 18 Ja 2016 Abstract The three-ode multitermial lossy source codig problem is ivestigated. We derive a ier boud to the geeral rate-distortio regio of this problem which is a atural extesio of the semial work by Kaspi 85 o the iteractive two-termial source codig problem. It is show that this rather ivolved ier boud cotais several rate-distortio regios of some relevat source codig settigs. I this way, besides the o-trivial extesio of the iteractive two termial problem, our results ca be see as a geeralizatio ad hece uificatio of several previous works i the field. Specializig to particular cases we obtai ovel rate-distortio regios for several lossy source codig problems. We fiish by describig some of the ope problems ad challeges. However, the geeral three-ode multitermial lossy source codig problem seems to offer a formidable mathematical complexity. Idex Terms Multitermial source codig, Wyer-Ziv, rate-distortio regio, Berger-Tug ier boud, iteractive lossy source codig, distributed lossy source codig. The material i this paper was partially published i the IEEE Iteratioal Symposium o Iformatio Theory, Hoolulu, Hawaii, USA, Jue 29 - July 4, 2014 ad i the cieee Iteratioal Symposium o Iformatio Theory, Hog-Kog, Chia, Jue 14 - Jue 19, The work of L. Rey Vega was partially supported by project UBACyT BA. The work of P. Piataida was partially supported by the FP7 Network of Excellece i Wireless COMmuicatios NEWCOM#. The work of A. Hero was partially supported by a DIGITEO Chair from 2008 to 2013 ad by US ARO grat W911NF L. Rey Vega is with the Departmets of Electroics FIUBA ad CSC-CONICET, Bueos Aires, Argetia lrey@fi.uba.ar, cgalar@fi.uba.ar. P. Piataida is with the Laboratoire des Sigaux et Systèmes L2S, CetraleSupelec, Gif-sur-Yvette, Frace pablo.piataida@supelec.fr. Alfred O. Hero III is with the Departmet of Electrical Eg. & CompSci Uiversity of Michiga, A Arbor, MI, USA hero@umich.edu.

2 2 I. INTRODUCTION A. Motivatio ad related works Distributed source codig is a importat brach of study i iformatio theory with eormous relevace for the preset ad future techology. Efficiet distributed data compressio may be the oly way to guaratee acceptable levels of performace whe eergy ad lik badwidth are severely limited as i may real world sesor etworks. The distributed data collected by differet odes i a etwork ca be highly correlated ad this correlatio ca be exploited at the applicatio layer, e.g., for target localizatio ad trackig or aomaly detectio. I such cases cooperative joit data-compressio ca achieve a better overall rate-distortio trade-off that idepedet compressio at each ode. Complete aswers to the optimal trade-offs betwee rate ad distortio for distributed source codig are scarce ad the solutio to may problems remai elusive. Two of the most importat results i iformatio theory, Slepia-Wolf solutio to the distributed lossless source codig problem 1] ad Wyer-Ziv 2] sigle letter solutio for the rate-distortio regio whe side iformatio is available at the decoder provided the kick-off for the study of these importat problems. Berger ad Tug 3], 4] geeralized the Slepia-Wolf problem whe lossy recostructios are required at the decoder. It was show that the regio obtaied, although ot tight i geeral, is the optimal oe i several special cases 5] 8] ad strictly suboptimal i others 9]. Heegard ad Berger 10] cosidered the Wyer-Ziv problem whe the side iformatio at the decoder may be abset or whe there are two decoders with degraded side iformatio. Timo et al 11] correctly exteded the achievable regio for may > 2 decoders. I 12] ad the refereces therei, the complemetary delivery problem closely related to the Heegard- Berger problem is also studied. The use of iteractio i a multitermial source codig settig has ot bee so extesively studied as the problems metioed above. Through the use of multiple rouds of iteractive exchages of iformatio explicit cooperatio ca take place usig distributed/successive refiemet source codig. Trasmittig reduced pieces of iformatio, ad costructig a explicit sequetial cooperative exchage of iformatio, ca be more efficiet that trasmittig the total iformatio i oe-shot. The value of iteractio for source codig problems was first recogized by Kaspi i his semial work 13], where the iteractive two-termial lossy source codig problem was itroduced

3 3 ad solved uder the assumptio of a fiite umber of commuicatio rouds. I 14] it is show that iteractio strictly outperforms i term of sum rate the Wyer-Ziv rate fuctio. There are also several extesios to the origial Kaspi problem. I 15] the iteractive source codig problem with a helper is solved whe the sources satisfy a certai Markov chai property. I 16] 18] other iterestig cases where iteractive cooperatio ca be beeficial are studied. To the best of our kowledge, a proper geeralizatio of this settig to iteractive multitermial > 2 lossy source codig has ot yet bee observed. B. Mai cotributios I this paper, we cosider the three-termial iteractive lossy source codig problem preseted i Fig. 1. We have a etwork composed of 3 odes which ca iteract through a broadcast ratelimited error free chael. Each ode measures the realizatio of a discrete memoryless source DMS ad is required to recostruct the sources from the other termials with a fidelity criterio. Nodes are allowed to iteract by iterchagig descriptios of their observed sources realizatios over a fiite umber of commuicatio rouds. After the iformatio exchage phase is over, the odes try to recostruct the realizatio of the sources at the other odes usig the recovered descriptios. The geeral rate-distortio regio seems to pose a formidable mathematical problem which ecompass several kow ope problems. However, several properties of this problem are established i this paper. Geeral achievable regio We derive a geeral achievable regio by assumig a fiite umber of rouds. This regio is ot a trivial extesio of Kaspi s regio 13] ad the mai ideas behid its derivatio are the exchage of commo ad private descriptios betwee the odes i the etwork i order to exploit optimality the differet side iformatios at the differet odes. As i the origial Kaspi s formulatio, the key to obtaiig the achievable regio is the atural cooperatio betwee the odes iduced by the geeratio of ew descriptios based o the past exchaged descriptio. However, i compariso to Kaspi s 2 ode case, the 3 odes iteractios make sigificat differeces i the optimal actio of each ode at the ecodig ad decodig procedure i a give roud. At each ecodig stage, each ode eed to commuicate to two odes with

4 4 ˆX 12,D ˆX 13,D ˆX 21,D ˆX 23,D R 2 R 1 Ecoder 1 Ecoder 2 X 1 R 3 R 1 R 3 R 2 X 2 X 3 Ecoder 3 ˆX 31,D 31 ˆX 32,D 32 Figure 1: Three-Termial Iteractive Source Codig. There is a sigle oiseless rate-limited broadcast chael from each termial to the other two termials. D ij deotes the average perletter distortio betwee the source Xj ad ˆX ij measured at the ode i for each pair i j. differet side iformatio. This is remiiscet of the Heegard-Berger problem 10], 11], whose complete solutio is ot kow, whe the side iformatio at the decoders is ot degraded. Moreover, the situatio is a bit more complex because of the presece of 3-way iteractio. This similarity betwee the Heegard-Berger problem leads us to cosider the geeratio of two sets of messages at each ode: commo messages destied to all odes ad private messages destied to some restricted sets of odes. O the other had, whe each ode is actig as a decoder the odes eed to recover a set of commo ad private messages geerated at differet odes i.e. at roud l ode 3, eeds to recover the commo descriptios geerated at odes 1 ad 2 ad the private oes geerated also at odes 1 ad 2. This is remiiscet of the Berger- Tug problem 4] 6], 19] which is also a ope problem. Agai, the situatio is more ivolved because of the cooperatio iduced by the multiple rouds of exchaged iformatio. Particularly importat is the fact that, i the case of the commo descriptios, there is cooperatio based o the coditioig o the previous exchaged descriptios i additio to cooperatio aturally iduced by the ecodig-decodig orderig imposed by the etwork. This explicit cooperatio for the exchage of commo messages is accomplished through the use of a special biig techique to be explaied i Appedix B.

5 5 Besides the complexity of the achievable regio, we give a ier boud to the rate-distortio regio that allows us to recover the two ode Kaspi s regio. We also recover several previous ier bouds ad rate-distortio regios of some well-kow cooperative ad iteractive as well as o-iteractive lossy source codig problems. Special cases As the full problem seems to offer a formidable mathematical complexity, icludig several special cases which are kow to be log-stadig ope problems, we caot give a full coverse provig the optimality of the geeral achievable regio obtaied. However, i Sectio V we provide a complete aswer to the rate-distortio regios of several specific cooperative ad iteractive source codig problems: 1 Two ecoders ad oe decoder subject to lossy/lossless recostructio costraits without side iformatio see Fig Two ecoders ad three decoders subject to lossless/lossy recostructio costraits with side iformatio see Fig Two ecoders ad three decoders subject to lossless/lossy recostructio costraits, reversal delivery ad side iformatio see Fig Two ecoders ad three decoders subject to lossy recostructio costraits with degraded side iformatio see Fig Three ecoders ad three decoders subject to lossless/lossy recostructio costraits with degraded side iformatio see Fig. 6. Iterestigly eough, we show that for the two last problems, iteractio through multiple rouds could be helpful. Whereas for the other three cases, it is show that a sigle roud of cooperatively exchaged descriptios suffices to achieve optimality. Table I summarizes the characteristics of each of the above metioed cases. Next we summarize the cotets of the paper. I Sectio II we formulate the geeral problem. I Sectio III we preset ad discuss the ier boud of the geeral problem. I Sectio IV we show how our ier boud cotais several results previously obtaied i the past. I Sectio V we preset the coverse results ad their tightess with respect to the ier boud for the special cases metioed above providig the optimal characterizatio for them. I Sectio VI we preset a discussio of the obtaied results ad their limitatios ad some umerical results

6 6 Cases R 1 R 2 R 3 Costraits at Node 1 Costraits at Node 2 Costraits at Node = 0 is ot recostructig is ot recostructig Pr ˆX 31 X1 ɛ ay source ay source E d ˆX ] 32, X2 D = 0 E d ˆX ] 12, X2 D 12 Pr ˆX 21 X1 ɛ Pr ˆX 31 X1 ɛ E d ˆX ] 32, X2 D = 0 E d ˆX ] 12, X2 D 12 Pr ˆX 21 X1 ɛ Pr ˆX 32 X2 ɛ E d ˆX ] 31, X1 D = 0 E d ˆX ] 12, X2 D 12 E d ˆX ] 21, X1 D 21 E d ˆX ] 31, X1 D 31 E d ˆX ] 32, X2 D is ot decodig Pr ˆX 21 X1 ɛ Pr ˆX 31 X1 ɛ E d ˆX ] 23, X3 D 23 E d ˆX ] 32, X2 D 32 Table I: Special cases fully characterized i Sectio V. cocerig the ew optimal cases from the previous Sectio. Fially i Sectio VII we provide some coclusios. The major mathematical details are relegated to the appedixes. Notatio: We summarize the otatio. With x ad upper-case letters X we deote vectors ad radom vectors of compoets, respectively. The i-th compoet of vector x is deoted as x i. All alphabets are assumed to be fiite. Etropy is deoted by H ad mutual iformatio by I ;. H 2 p deotes the etropy associated with a Beroulli radom variable with parameter p. With h we deote differetial etropy. Let X, Y ad V be three radom variables o some alphabets with probability distributio p XY V. Whe clear from cotext we will simple deote p X x with px. If the probability distributio of radom variables X, Y, V satisfies px yv = px y for each x, y, v, the they form a Markov chai, which is deoted by X Y V. The probability of a evet A is deoted by Pr {A, where the measure used to compute it will be uderstood from the cotext. Coditioal probability of a set A with respect to a set B is deoted as Pr { A B. The set of strog typical sequeces associated with radom variable X see appedix A is deoted by TX]ɛ, where ɛ > 0. We simply deote these sets as Tɛ whe clear from the cotext. The cardial of set A is deoted by A. The complemet of a set is deoted by Ā. With Z α ad R β we deote the itegers ad reals umbers greater tha α ad

7 7 β respectively. co {A deotes the covex hull of a set A R N, where N N. II. PROBLEM FORMULATION Assume three discrete memoryless sources DMS s with alphabets ad pmfs give by X 1 X 2 X 3, p X1 X 2 X 3 ad arbitrary bouded distortio measures: dj : X j ˆX j R 0, j M {1, 2, 3 where { ˆX j j M are fiite recostructio alphabets 1. We cosider the problem of characterizig the rate-distortio regio of the iteractive source codig sceario described i Fig. 1. I this settig, through K rouds of iformatio exchage betwee the odes each oe of them will attempt to recover a lossy descriptio of the sources that the others odes observe, e.g., ode 1 must recostruct while satisfyig distortio costraits the realizatio of the sources X 2 ad X 3 observed by odes 2 ad 3. Ideed, this settig ca be see as a geeralizatio of the well-kow Kaspi s problem 13]. Defiitio 1 K-step iteractive source code: A K-step iteractive -legth source code, deoted for the etwork model i Fig. 1, is defied by a sequece of ecoder mappigs: f1: l X1 J2 1 J3 1 J2 l 1 J3 l 1 J l 1, 1 f2: l X2 J1 1 J3 1 J3 l 1 J1 l J l 2, 2 f3: l X3 J1 1 J2 1 J1 l J2 l J l 3, 3 with l 1 : K] ad message sets: J l i { 1, 2,..., I l i, I l i Z 0, i M, ad recostructio mappigs: g ij : X i m M, m i J 1 m J K m ˆX ij, i j. 4 The average per-letter distortio ad the correspodig distortio levels achieved at the ode i with respect to source j are: E d j Xj, ˆX ] ij D ij i, j M, i j 5 with d x, y 1 dx m, y m. 6 m=1 1 The problem ca be easily geeralized to the case i which there are differet recostructio alphabets at the termials. It ca also be show that all the results are valid if we employ arbitrary bouded joit distortio fuctios, e.g. at ode 1 we use dx 2, X 3; ˆX 2, ˆX 3.

8 8 I compact form we deote a K-step iteractive source codig by, K, F, G where F ad G deote the sets of ecoders ad decoders mappigs. Remark 1: The code defiitio depeds o the ode orderig i the ecodig procedure. Above we defied the ecodig fuctios { f1, l f2, l f3 l K assumig that i each roud ode 1 l=1 acts first, followed by ode 2, ad fially by ode 3, ad the process begiig agai at ode 1. Defiitio 2 Achievability ad rate-distortio regio: Cosider R R 1, R 2, R 3 ad D D 12, D 13, D 21, D 23, D 31, D 32. The rate vector R is D, K-achievable if ε > 0 there is 0 ε, K N such that > 0 ε, K there exists a K-step iteractive source code, K, F, G with rates satisfyig: 1 K l=1 log J l i R i + ɛ, i M 7 ad with average per-letter distortios at ode i with respect to source j: E d j Xj, ˆX ] ij D ij + ɛ, i, j M, i j, 8 where ˆX ij g ij X i, m M, m i J 1 m J K m R 3 D, K is defied by: R 3 D, K =, i j M. The rate-distortio regio { R : R is D, K-achievable Similarly, the D-achievable regio R 3 D is give by R 3 D = K=1 R 3D, K 2, that is: R 3 D = {R : R is D, K-achievable for some K Z Remark 2: By defiitio R 3 D, K is closed ad usig a time-sharig argumet it is easy to show that it is also covex K Z 1. Remark 3: R 3 D, K depeds o the ode orderig i the ecodig procedure. Above we defied the ecodig fuctios { f1, l f2, l f3 l K assumig that i each roud ode 1 acts first, l=1 followed by ode 2, ad fially by ode 3, ad the process begiig agai at ode 1. I this paper we restrict the aalysis to the caoical orderig However, there are 3! = 6 differet orderigs that geerally lead to differet regios ad the D, K-achievable regio 9 2 Notice that this limit exists because it is the uio of a mootoe icreasig sequece of sets.

9 9 defied above is more explicitly deoted R 3 D, K, σ c, where σ c is the trivial permutatio for M. The correct D, K-achievable regio is: R 3 D, K = R 3 D, K, σ 11 σ ΣM where ΣM cotais all the permutatios of set M. The theory preseted i this paper for determiig R 3 D, K, σ c ca be used o the other permutatios σ σ c to compute III. INNER BOUND ON THE RATE-DISTORTION REGION I this Sectio, we provide a geeral achievable rate-regio o the rate-distortio regio. A. Ier boud We first preset a geeral achievable rate-regio where each ode at a give roud l will geerate descriptios destied to the other odes based o the realizatio of its ow source, the past descriptios geerated by a particular ode ad the descriptios geerated at the other odes ad recovered by the ode up to the preset roud. I order to precisely describe the complex rate-regio, we eed to itroduce some defiitios. For a set A, let C A = 2 A \ {A, be the set of all subsets of A mius A ad the empty set. Deote the auxiliary radom variables: U i S,l, S C M, i / S, l = 1,..., K. 12 Auxiliary radom variables {U i S,l will be used to deote the descriptios geerated at ode i ad at roud l ad destied to a set of odes S C M with i / S. For example, U 1 23,l deote the descriptio geerated at ode 1 ad at roud l ad destied to odes 2 ad 3. Similarly, {U 1 2,l will be used to deote the descriptios geerated at ode 1 at roud l ad destied oly to ode 2. We defie variables: W i,l] Commo iformatio 4 shared by the three odes available at ode i at roud l before ecodig 3 It should be metioed that this is ot the most geeral settig of the problem. The most geeral ecodig procedure will follow from the defiitio of the trasmissio order by a sequece t 1, t 2, t 3,..., t M K with t i M. This will cover eve the situatio i which the order ca be chaged i each roud. To keep the mathematical presetatio simpler we will ot cosider this more geeral settig. 4 Not to be cofused with the Wyer s defiitio of commo iformatio 20].

10 10 V S,l,i] Private iformatio shared by odes i S C M available at ode i S, at roud l, before ecodig I precise terms, the quatities itroduced above for our problem are defied by: W 1,l] ={U 1 23,k, U 2 13,k, U 3 12,k l 1 k=1, W 2,l] =W 1,l] U 1 23,l, W 3,l] =W 2,l] U 2 13,l, V 12,l,1] ={U 1 2,k, U 2 1,k l 1 k=1, V 12,l,2] = V 12,l,1] U 1 2,l, V 13,l,1] ={U 1 3,k, U 3 1,k l 1 k=1, V 13,l,3] = V 13,l,1] U 1 3,l, V 23,l,2] ={U 2 3,k, U 3 2,k l 1 k=1, V 23,l,3] = V 23,l,2] U 2 3,l. Before presetig the geeral ier boud, we provide the basic idea of the radom codig scheme that achieves the rate-regio i Theorem 1 for the case of K commuicatio rouds. Assume that all codebooks are radomly geerated ad kow to all the odes before the iformatio exchage begis ad cosider the ecodig orderig give by so that we begi at roud l = 1 i ode 1. Also, ad i order to maitai the explaatio simple ad to help the reader to grasp the essetials of the codig scheme employed, we will cosider that all termial are able to recover the descriptios geerated at other odes which will be the case uder the coditios i our Theorem 1. From the observatio of the source X 1, ode 1 geerates a set of descriptios for each of the other odes coected to it. I particular it geerates a commo descriptio to be recovered at odes 2 ad 3 i additio to two private descriptios for ode 2 ad 3, respectively, geerated from a coditioal codebook give the commo descriptio. The, ode 2 tries to recover the descriptios destied to it the commo descriptio geerated at 1 ad its correspodig private descriptio, usig X 2 ow descriptios, based o source X 2 as side iformatio, ad geerates its ad the recovered descriptios from ode 1. Agai, it geerates a commo descriptio for odes 1 ad 3, a private descriptio for ode 3 ad aother oe for ode 1. The same process goes o util ode 3, which tries to recover joitly the commo descriptios geerated by ode 1 ad ode 2, ad the the private descriptios destied to him by ode 1 ad 2. The geerates its ow descriptios commo ad private oes destied to odes 1 ad 2. Fially, ode 1 tries to recover all the descriptios destied to it geerated by odes 2 ad 3 i the same way as previously doe by ode 3. After this, roud l = 1 is over,

11 11 ad roud l = 2 begis with ode 1 geeratig ew descriptios usig X 1, its ecodig history from previous roud ad the recovered descriptios from the other odes. The process cotiues i a similar maer util we reach roud l = K where ode 3 recovers the descriptios from the other odes ad geerates its ow oes. Node 1 recovers the last descriptios destied to it from odes 2 ad 3 but does ot geerate ew oes. The same holds for ode 2 who oly recovers the descriptios geerated by ode 3 ad thus termiatig the iformatio exchage procedure. Notice that at the ed of roud K the decodig i ode 1 ad ode 2 ca be doe simultaeously. This is due to the fact that ode 1 is ot geeratig a ew descriptio destied to ode 2. However, i order to simplify the aalysis ad otatio i the appedix we will cosider that the last decodig of ode 2 occurs i roud K After all the exchages are doe, each ode recovers a estimate of the other odes, source realizatios by usig all the available recovered descriptios from the K previous rouds. Theorem 1 Ier boud: Let R3 D, K be the closure of set of all rate tuples satisfyig: R 1 = R 2 = R 3 = R 1 + R 2 = R 1 + R 3 = R 2 + R 3 = K l=1 K l=1 K l=1 K l=1 K+1 l=1 K l=1 l R R l R l 1 3 l R R l R l 2 3 l R R l R l 3 2 l R R l R l R l R l R l 2 1 l R R l R l R l R l R l 3 1 l R R l R l R l R l R l This is clearly a fictitious roud, i the sese that there is ot descriptios geeratio o it. I this way, there is ot modificatio of the fial rates achieved by the procedure described if we cosider this additioal roud.

12 12 where 6 for each l 1 : K]: R l 1 23 > I R l 2 13 > I R l 3 12 > I R l R l 2 13 > I R l R l 3 12 > I R l R l > I R l R l 1 2 > I R l R l > I R l 1 3 > I R l 2 3 > I R l R l 2 3 > I R l 2 1 > I R l 3 1 > I R l R l 3 1 > I X 1 ; U 1 23,l X2 W 1,l] V 12,l,1] V 23,l 1,3] X 2 ; U 2 13,l X3 W 2,l] V 13,l,1] V 23,l,2] X 3 ; U 3 12,l X1 W 3,l] V 12,l,2] V 13,l,3] X 1 X 2 ; U 1 23,l U 2 13,l X3 W 1,l] V 13,l,1] V 23,l,2] X 2 X 3 ; U 2 13,l U 3 12,l X1 W 2,l] V 12,l,2] V 13,l,3] X 1 X 3 ; U 1 23,l U 3 12,l 1 X2 W 3,l 1] V 12,l,1] V 23,l 1,3] X2 > I X 3 ; U 3 2,l 1 W 2,l] V 23,l 1,3] V 12,l,2] X2 X 1 ; U 1 2,l W 2,l] V 23,l,2] V 12,l,1] X 1 X 3 ; U 1 2,l U 3 12,l 1 X2 W 2,l] V 23,l 1,3] V 12,l,1] X 1 ; U 1 3,l X3 W 3,l] V 23,l,3] V 13,l,1] X 2 ; U 2 3,l X3 W 3,l] V 23,l,2] V 13,l,3] X 1 X 2 ; U 1 3,l U 2 3,l X3 W 3,l] V 23,l,2] V 13,l,1] X 2 ; U 2 1,l X1 W 1,l+1] V 12,l,2] V 13,l+1,1] X 3 ; U 3 1,l X1 W 1,l+1] V 12,l+1,1] V 13,l,3] X 2 X 3 ; U 2 1,l U 3 1,l X1 W 1,l+1] V 12,l,2] V 13,l,3] with R 0 i S = RK+1 i S = 0 ad U i S,0 = U i S,K+1 = for S C M ad i / S. With these defiitios the rate-distortio regio satisfies 7 : p PD,K R 3 D, K R 3 D, K, 34 6 Notice that these defiitios are motivated by the fact that at roud 1, ode 2 oly recovers the descriptios geerated by ode 1 ad at roud K + 1 oly recovers what ode 3 already geerated at roud K. 7 It is straightforward to show that the LHS of equatio 34 is covex, which implies that the covex hull operatio is ot eeded.

13 13 where PD, K deotes the set of all joit probability measures associated with the followig Markov chais for every l 1 : K]: 1 U 1 23,l X 1, W 1,l] X 2, X 3, V 12,l,1], V 13,l,1], V 23,l,2], 2 U 1 2,l X 1, W 2,l], V 12,l,1] X 2, X 3, V 13,l,1], V 23,l,2], 3 U 1 3,l X 1, W 2,l], V 13,l,1] X 2, X 3, V 12,l,2], V 23,l,2], 4 U 2 13,l X 2, W 2,l] X 1, X 3, V 12,l,2], V 13,l,3], V 23,l,2], 5 U 2 1,l X 2, W 3,l], V 12,l,2] X 1, X 3, V 13,l,3], V 23,l,2], 6 U 2 3,l X 2, W 3,l], V 23,l,2] X 1, X 3, V 12,l+1,1], V 13,l,3], 7 U 3 12,l X 3, W 3,l] X 1, X 2, V 12,l+1,1], V 13,l,3], V 23,l,3], 8 U 3 1,l X 3, W 1,l+1], V 13,l,3] X 1, X 2, V 12,l+1,1], V 23,l,3], 9 U 3 2,l X 3, W 1,l+1], V 23,l,3] 1X 1, X 2, V 12,l+1,1], V 13,l+1,1], ad such that there exist recostructio mappigs: g ij Xi, V ij,k+1,1],w 1,K+1] = ˆXij 35 with E d j X j, ˆX ] ij D ij for each i, j M ad i j. The proof of this theorem is relegated to Appedix C ad relies o the auxiliary results preseted i Appedix A ad the theorem o the cooperative Berger-Tug problem with side iformatio preseted i Appedix B. Remark 4: It is worth metioig here that our codig scheme is costraied to use successive decodig, i.e., by recoverig first the codig layer of commo descriptios ad the codig layer of private descriptios at each codig layer each ode employ joit-decodig. Obviously, this is a sub-optimum procedure sice the best scheme would be to use joit decodig where both commo ad private iformatios ca be joitly recovered. However, the aalysis of this scheme is much more ivolved. The associated achievable rate regio ivolves a large umber of equatios that combie rates belogig to private ad commo messages from differet odes. Also, several mutual iformatio terms i each of these rate equatios caot be combied, leadig to a proliferatio of may equatios that offer little isight to the problem. Remark 5: The idea behid our derivatio of the achievable regio ca be exteded to ay umber M > 3 of odes i the etwork. This ca be accomplished by geeratig a greater umber of superimposed codig layers. First a layer of codes that geerates descriptios destied

14 14 to be decoded by all odes. The ext layer correspodig to all subsets of size M 1, etc, util we reach the fial layer composed by codes that geerate private descriptios for each of odes. Agai, successive decodig is used at the odes to recover the descriptios i these layers destied to them. Of course, the umber of required descriptios will icrease with the umber of odes as well as the obtaied rate-distortio regio. Remark 6: It is iterestig to compare the mai ideas of our scheme with those of Kaspi 13]. The mai idea i 13] is to have a sigle codig tree shared by the two odes. Each leaf i the codig tree is codeword geerated either at ode 1 or 2. At a give roud each ode kows assumig o errors at the ecodig ad decodig procedures the path followed i the tree. For example, at roud l, ode 1, usig the kowledge of the path util roud l ad its source realizatio geerate a leaf from a set of possible oes usig joit typicality ecodig ad biig. Node 2, usig the same path kow at ode 1 ad its source realizatio, uses joit typicality decodig to estimate the leaf geerate at ode 1. If there is o error at these ecodig ad decodig steps, the previous path is updated with the ew leaf ad both -ode 1 ad 2- kow the updated path. Node 2 repeats the procedure. This is doe util roud K where the fial path is kow at both odes ad used to recostruct the desired sources. I the case of three odes the situatio is more ivolved. At a give roud, the ecoder at a arbitrary ode is seeig two decoders with differet side iformatio 8. I order to simplify the explaatio cosider that we are at roud l i the ecoder 1, ad that the listeig odes are odes 2 ad 3. This situatio forces ode 1 to ecode two sets of descriptios: oe commo for the other two odes ad a set of private oes associated with each of the listeig odes 2 ad 3. Followig the ideas of Kaspi, it is the atural to cosider three differet codig trees followed by ode 1. Oe codig tree has leaves that are the commo descriptios geerated ad shared by all the odes i the etwork. The secod tree is composed by leaves that are the private descriptios geerated ad shared with ode 2. The third tree is composed by leaves that are the private descriptios geerated ad shared with ode 3. As the private descriptios refie the commo oes, depedig o the quality of the side iformatio of the ode that is the iteded recipiet, it is clear that descriptios are correlated. For example, the private descriptio destied to ode 2, should deped ot oly o the past private descriptios geerated ad shared 8 Because at each ode the source realizatios are differet, ad the recovered previous descriptios ca also be differet.

15 15 by odes 1 ad 2, but also o the commo descriptios geerated at all previous rouds i all the odes ad o the commo descriptio geerated at the preset roud i ode 1. Somethig similar happes for the private descriptio destied to ode 3. It is clear that as the commo descriptios are to be recovered by all the odes i the etwork, they ca oly be coditioed with respect to the past commo descriptios geerated at previous rouds ad with respect to the commo descriptios geerated at the preset roud by a ode who acted before i.e. at roud l ode 1 acts before ode 2. The private descriptios, as they are oly required to be recovered at some set of odes, ca be geerated coditioed o the past exchaged commo descriptios ad the past private descriptios geerated ad recovered i the correspodig set of odes i.e., the private descriptios exchaged betwee odes 1 ad 2 at roud l, ca oly be geerated coditioed o the past commo descriptios geerated at odes 1, 2 ad 3 ad o the past private descriptios exchaged oly betwee 1 ad 2. We ca see clearly that there are basically four paths to be cooperatively followed i the etwork: Oe path of commo descriptios shared by odes 1, 2 ad 3. Oe path of private descriptios shared by odes 1 ad 2. Oe path of private descriptios shared by odes 1 ad 3. Oe path of private descriptios shared by odes 2 ad 3. It is also clear that each ode oly follows three of these paths simultaeously. The exchage of commo descriptios deserves special metio. Cosider the case at roud l i ode 3. This ode eeds to recover the commo descriptios geerated at odes 1 ad 2. But at the momet ode 2 geerated its ow commo descriptio, it also recovered the commo oe geerated at ode 1. This allows for a atural explicit cooperatio betwee odes 1 ad 2 i order to help ode 3 to recover both descriptios. Clearly, this is ot the case for private descriptios from odes 1 ad 2 to be recovered at ode 3. Node 2 does ot recover the private descriptio from ode 1 to 3 ad caot geerate a explicit collaboratio to help ode 3 to recover both private descriptios. Note, however, that as both private descriptios will be depedet o previous commo descriptios a implicit collaboratio itrisic to the code geeratio is also i force. I appedix B we cosider the problem ot i the iteractive settig of geeratig the explicit cooperatio for the commo descriptios through the use of what we call a super-biig procedure, i order

16 16 to use the results for our iteractive three-ode problem. IV. KNOWN CASES AND RELATED WORK Several ier bouds ad rate-distortio regios o multitermial source codig problems ca be derived by specializig the ier boud 34. Below we summarize oly a few of them. 1 Distributed source codig with side iformatio 4], 19]: Cosider the distributed source codig problem where two odes ecode separately sources X1 ad X2 to rates R 1, R 2 ad a decoder by usig side iformatio X3 must recostruct both sources with average distortio less tha D 1 ad D 2, respectively. By cosiderig oly oe-roud/oe-way iformatio exchage from odes 1 ad 2 the ecoders to ode 3 the decoder, the results i 4], 19] ca be recovered as a special case of the ier boud 34. Specifically, we set: U 1 23,l =U 2 13,l = U 3 12,l = U 1 2,l = U 2 1,l = U 3 1,l = U 3 2,l =, l U 1 3,l =U 2 3,l =, l > 1. I this case, the Markov chais of Theorem 1 reduce to: U 1 3,1 X 1 X 2, X 3, U 2 3,1, 36 U 2 3,1 X 2 X 1, X 3, U 1 3,1, 37 ad thus the ier boud from Theorem 1 recovers the results i 19] R 1 >IX 1 ; U 1 3,1 X 3 U 2 3,1, 38 R 2 >IX 2 ; U 2 3,1 X 3 U 1 3,1, 39 R 1 + R 2 >IX 1 X 2 ; U 1 3,1 U 2 3,1 X Source codig with side iformatio at 2-decoders 10], 11]: Cosider the settig where oe ecoder X1 trasmits descriptios to two decoders with differet side iformatios X2, X3 ad distortio requiremets D 2 ad D 3. Agai we cosider oly oe way/roud iformatio exchage from ode 1 the ecoder to odes 2 ad 3 the decoders. I this case, we set: U 2 13,l =U 3 12,l = U 2 1,l = U 3 1,l = U 3 2,l = U 2 3,l =, l U 1 23,l =U 1 23,l = U 1 2,l = U 1 3,l =, l > 1.

17 17 The above Markov chais imply U 1 23,1, U 1 2,1, U 1 3,1 X 1 X 2, X 3 41 ad thus the ier boud from Theorem 1 reduces to the results i 10], 11] R 1 >max { IX 1 ; U 1 23,1 X 2, IX 1 ; U 1 23,1 X 3 +IX 1 ; U 1 2,1 X 2 U 1 23,1 + IX 1 ; U 1 3,1 X 3 U 1 23, Two termial iteractive source codig 13]: Our ier boud 34 is basically the geeralizatio of the two termial problem to the three-termial settig. Assume oly two ecodersdecoders X1 ad X2 which must recostruct the other termial source 3 with distortio costraits D 1 ad D 2, ad after K rouds of iformatio exchage. Let us set: U 1 23,l =U 2 13,l = U 3 12,l = U 1 3,l = U 3 1,l = U 2 3,l = U 3 2,l =, l X 3 =. The Markov chais become U 1 2,l X 1, V 12,l,1] X 2, 43 U 2 1,l X 2, V 12,l,2] X 2, 44 for l 1 : K] ad thus the ier boud from Theorem 1 permit us to obtai the results i 13] R 1 >IX 1 ; V 12,K+1,1] X 2, 45 R 2 >IX 2 ; V 12,K+1,2] X Two termial iteractive source codig with a helper 15]: Cosider ow two ecoders/decoders, amely X2 ad X3, that must recostruct the other termial source with distortio costraits D 2 ad D 3, respectively, usig K commuicatio rouds. Assume also that aother ecoder X 1 provides both odes 2, 3 with a commo descriptio before begiig the iformatio exchage ad the remais silet. Such commo descriptio ca be exploited as coded side iformatio. Let us set: U 2 13,l =U 3 12,l = U 1 3,l = U 1 2,l = U 1 3,l = U 2 1,l = U 3 1,l =, l U 1 23,l =, l > 1.

18 18 The Markov chais reduce to: U 1 23,1 X 1 X 2, X 3, 47 U 2 3,l X 2, U 1 23,1, V 23,l,2] X 1, X 3, 48 U 3 2,l X 3, U 1 23,1, V 23,l,3] X 1, X A ier boud to the rate-distortio regio for this problem reduces to usig the rate equatios i our Theorem 1 R 1 >max { IX 1 ; U 1 23,1 X 2, IX 1 ; U 1 23,1 X 3, 50 R 2 >IX 2 ; V 23,K+1,2] X 3 U 1 23,1, 51 R 3 >IX 3 ; V 23,K+1,2] X 2 U 1 23,1. 52 This regio cotais as a special case the regio i 15]. I that paper it is further assumed i order to have a coverse result that X 1 X 3 X 2. The, the value of R 1 satisfies R 1 > IX 1 ; U 1 23,1 X 2. Obviously, with the same extra Markov chai we obtai the same limitig value for R 1 ad the above regio is the rate-distortio regio. V. NEW RESULTS ON INTERACTIVE AND COOPERATIVE SOURCE CODING A. Two ecoders ad oe decoder subject to lossy/lossless recostructio costraits without side iformatio Cosider ow the problem described i Fig. 2 where ecoder 1 wishes to commuicate the source X 1 of the source X 2 to ode 3 i a lossless maer while ecoder 2 wishes to sed a lossy descriptio to ode 3 with distortio costrait D 31. To achieve this, the ecoders use K commuicatio rouds. This problem ca be see as the cooperatig ecoders versio of the well-kow Berger-Yeug 5] problem. Theorem 2: The rate-distortio regio of the settig described i Fig. 8 is give by the uio over all joit probability measures p X1 X 2 U 2 13 such that there exists a recostructio mappig: g 32 X 1, U 2 13 = ˆX 32 with E dx 2, ˆX ] 32 D 32, 53

19 19 X 1 Node 1 R 1 R 1 R 2 Node 3 ˆX 31 X 1 ˆX 32,D 32 X 2 Node 2 R 2 Figure 2: Two ecoders ad oe decoder subject to lossy/lossless recostructio costraits without side iformatio. of the set of all tuples satisfyig: R 1 HX 1 X 2, 54 R 2 IX 2 ; U 2 13 X 1, 55 R 1 + R 2 HX 1 + IX 2 ; U 2 13 X The auxiliary radom variable U 2 13 has a cardiality boud of U 2 13 X 1 X Remark 7: It is worth emphasizig that the rate-distortio regio i Theorem 2 outperforms the o-cooperative rate-distortio regio first derived i 5]. This is due to two facts: the coditioal etropy give i the rate costrait 54 which is strictly smaller tha the etropy HX 1 preset i the rate-regio i 5], ad the fact that the radom descriptio U 2 13 may be arbitrarily depedet o both sources X 1, X 2 which is ot the case without cooperatio 5]. Therefore, cooperatio betwee ecoders 1 ad 2 reduces the rate eeded to commuicate the source X 1 while icreasig the optimizatio set of all admissible source descriptios. Remark 8: Notice that the rate-distortio regio i Theorem 2 is achievable with a sigle roud of iteractios K = 1, which implies that multiple rouds do ot improve the ratedistortio regio i this case. This holds because of the fact that ode 3 recostruct X 1 i a lossless fashio. Remark 9: Although i the cosidered settig of Fig. 8 ode 1 is ot supposed to decode either a lossy descriptio or the complete source X2, if odes 1 ad 3 wish to recover the same

20 20 descriptios the optimal rate-regio remais the same as give i Theorem 2. The oly differece relies o the fact that ode 1 is ow able to fid a fuctio g 12 X 1, U 2 13 = ˆX 12 which must satisfy a additioal distortio costrait E dx 2, ˆX ] 12 D 12. I order to show this, it is eough to check that i the coverse proof give below the specific choice of the auxiliary radom variable already allows ode 1 to recover a geeral fuctio ˆX 12t] = g 12 X1t], U 2 13t] for each time t {1,...,. Proof: The direct part of the proof simply follows by choosig: U 3 12,l =U 1 3,l = U 1 2,l = U 2 1,l = U 2 3,l = U 3 1,l = U 3 2,l =, l U 1 23,1 =X 1, U 1 23,l = U 2 13,l = l > 1, ad thus the rate-distortio regio 34 reduces to the desired regio i Theorem 2 where for simplicity we dropped the roud idex. We ow proceed to the proof of the coverse. If a pair of rates R 1, R 2 ad distortio D 32 are admissible for the K-steps iteractive cooperative distributed source codig settig described i Fig. 8, the for all ε > 0 there exists 0 ε, K, such that > 0 ε, K there exists a K-steps iteractive source code, K, F, G with itermediate rates satisfyig: 1 K log Ji l R i + ε, i {1, 2 57 l=1 ad with average per-letter distortios with respect to the source 2 ad perfect recostructio with respect to the source 1 at ode 3: E dx2, ˆX ] 32 D 32 + ε, 58 Pr X1 ˆX 31 ε, 59 where ˆX 32 g 32 J 1:K] 1, J 1:K] 2, ˆX 31 g 3 1, J 1:K] For each t {1,...,, defie radom variables U 2 13t] as follows: U 2 13t] J 1:K] 1, J 1:K] 2, X 11:t 1], X 1t+1:]. 61 By the coditio 59 which says that Pr X1 ˆX 31 ε ad Fao s iequality 21], we have HX1 ˆX 31 Pr X1 ˆX 31 log 2 X1 1 + H 2 Pr X1 ˆX 31 ɛ, 62 where ɛ ε 0 provided that ε 0 ad.

21 21 1 Rate at ode 1: For the first rate, we have R 1 + ε H J 1:K] 1 I J 1:K] 1 ; X1 X2 = HX 1 X 2 H a = HX 1 X 2 H X 1 X 2 J 1:K] 1 X1 X2 J 1:K] where b HX 1 X 2 HX 1 ˆX b HX 1 X 2 ɛ, 68 step a follows from the fact that by defiitio of the code the sequece J 1:K] 2 is a fuctio of the source X 2 ad the vector of messages J 1:K] 1, step b follows from the code assumptio that guaratees the existece of a recostructio fuctio ˆX 31 g 3 1, J 1:K] 2, step c follows from Fao s iequality i 62.

22 22 2 Rate at ode 2: For the secod rate, we have R 2 + ε H J 1:K] 2 I J 1:K] 2 ; X1 X2 = I J 1:K] 2 ; X1 + I J 1:K] 2 ; X2 X1 a I J 1:K] 2 ; X1 + I J 1:K] 2 ; X 2t] X 1t] X 1t+1:] X 11:t 1] X 21:t 1] where b = I J 1:K] 2 ; X1 + c I J 1:K] 2 ; X1 + d = I J 1:K] 2 ; X1 + I J 1:K] 2 X 1t+1:] X 11:t 1] X 21:t 1] ; X 2t] X 1t] I U 2 13t] ; X 2t] X 1t] I U 2 13Q] ; X 2Q] X 1Q], Q = t 75 e = I J 1:K] 2 ; X1 + I U 2 13Q] ; X 2Q] X 1Q], Q 76 f I J 1:K] 2 ; X1 + I Ũ2 13 ; X 2 X 1 77 g I Ũ2 13 ; X 2 X 1, 78 step a follows from the chai rule for coditioal mutual iformatio ad o-egativity of mutual iformatio, step b follows from the memoryless property across time of the sources X 1, X 2, step c follows from the o-egativity of mutual iformatio ad defiitios 61, step d follows from the use of a time sharig radom variable Q uiformly distributed over the set {1,...,, step e follows from the defiitio of the coditioal mutual iformatio, step f follows by lettig a ew radom variable Ũ2 13 U 2 13Q], Q, step g follows from the o-egativity of mutual iformatio.

23 23 3 Sum-rate of odes 1 ad 2: For the sum-rate, we have R 1 + R 2 + 2ε H + R 2 + ε 79 where J 1:K] 1 step a follows from iequality 77, a H J 1:K] 1 + I J 1:K] 2 ; X1 + I Ũ2 13 ; X 2 X 1 80 = H J 1:K] 1 J 1:K] 2 + I J 1:K] 1 ; J 1:K] 2 + I J 1:K] 2 ; X1 + I Ũ2 13 ; X 2 X 1 81 I J 1:K] 1 ; X1 J 1:K] 2 + I J 1:K] 1 ; J 1:K] 2 + I J 1:K] 2 ; X1 + I Ũ2 13 ; X 2 X 1 82 = I J 1:K] 1 ; X1 + I X1 J 1:K] 1 ; J 1:K] 2 + I Ũ2 13 ; X 2 X 1 83 ] b = HX 1 + I Ũ2 13 ; X 2 X 1 H X1 J 1:K] 1 + I X1 J 1:K] 1 ; J 1:K] 2 84 c ] HX 1 + I Ũ2 13 ; X 2 X 1 H X1 J 1:K] 2 85 d ] HX 1 + I Ũ2 13 ; X 2 X 1 HX1 ˆX e ] HX 1 + I Ũ2 13 ; X 2 X 1 ɛ, 87 step b follows from the memoryless property across time of the source X 1, step c follows from o-egativity of mutual iformatio, step d follows from the code assumptio that guaratees the existece of recostructio fuctio ˆX 31 g 3 1, J 1:K] 2 ad from the fact that ucoditioig icreases etropy, step e from Fao s iequality i Distortio at ode 3: Node 3 recostructs lossless ˆX 31 g 3 1, J 1:K] 2 ad lossy ˆX 32 g 32 J 1:K] 1, J 1:K] 2. For each t {1,...,, defie a fuctio ˆX 32t] as begig the t-th coordiate of this estimate: ˆX 32t] U2 13t] g32t] J 1:K] 1, J 1:K] 2. 88

24 24 The compoet-wise mea distortio thus verifies 1:K] D 32 + ε E d X 2, g 31 J 1, J 1:K] ] 2 89 = 1 E d X 2t], ] 32t] U2 13t] 90 where we defied fuctio X 32 by = 1 E d X 2Q], ˆX ] 32Q] U2 13Q] Q = t 91 = E d X 2Q], ˆX ] 32Q] U2 13Q] 92 = E d X 2, X ] 32 Ũ2 13, 93 X 32 Ũ2 13 = X 32 Q, U2 13Q] ˆX32Q] U2 13Q]. 94 This cocludes the proof of the coverse ad thus that of the theorem. B. Two ecoders ad three decoders subject to lossless/lossy recostructio costraits with side iformatio Cosider ow the problem described i Fig. 3 where ecoder 1 wishes to commuicate the lossless the source X 1 to odes 2 ad 3 while ecoder 2 wishes to sed a lossy descriptio of its source X 2 to odes 1 ad 3 with distortio costraits D 12 ad D 32, respectively. I additio to this, the ecoders overhead the commuicatio usig K commuicatio rouds. This problem ca be see as a geeralizatio of the settigs previously ivestigated i 3], 5]. Theorem 3: The rate-distortio regio of the settig described i Fig. 3 is give by the uio over all joit probability measures p X1 X 2 X 3 U 2 13 U 2 3 satisfyig the Markov chai U 2 13, U 2 3 X 1, X 2 X 3 95 ad such that there exists recostructio mappigs: g 32 X 1, X 3, U 2 13, U 2 3 = ˆX 32 with E dx 2, ˆX ] 32 D 32, 96 g 12 X 1, U 2 13 = ˆX 12 with E dx 2, ˆX ] 12 D 12, 97

25 25 ˆX 12,D 12 X 1 Node 1 R 1 R 1 R 2 Node 3 ˆX 31 X 1 ˆX 32,D 32 X 2 Node 2 R 2 X 3 ˆX 21 X 1 Figure 3: Two ecoders ad three decoders subject to lossless/lossy recostructio costraits with side iformatio. of the set of all tuples satisfyig: R 1 HX 1 X 2, 98 R 2 IU 2 13 ; X 2 X 1 + IU 2 3 ; X 2 U 2 13 X 1 X 3, 99 R 1 + R 2 HX 1 X 3 + IU 2 13 U 2 3 ; X 2 X 1 X The auxiliary radom variables have cardiality bouds: U 2 13 X 1 X 2 + 2, U 2 3 X 1 X 2 U Remark 10: Notice that the rate-distortio regio i Theorem 3 is achievable with a sigle roud of iteractios K = 1, which implies that multiple rouds do ot improve the ratedistortio regio i this case. Remark 11: It is worth metioig that cooperatio betwee ecoders reduces the rate eeded to commuicate the source X 2 while icreasig the optimizatio set of all admissible source descriptios. Proof: The direct part of the proof follows by choosig: U 3 12,l =U 1 3,l = U 1 2,l = U 2 1,l = U 3 1,l = U 3 2,l =, l U 1 23 U 1 23,1 = X 1, U 1 23,l = U 2 13,l = U 2 3,l = l > 1.

26 26 ad U 2 13,1 U 2 13 ad U 2 3,1 U 2 3 are auxiliary radom variables that accordig to Theorem 1 should satisfy: U 2 13 X 1, X 2 X 3, U 2 3, U 2 13, X 1, X 2 X Notice, however that these Markov chais are equivalet to 95. From the rate equatios i Theorem 1, ad the above choices for the auxiliary radom variables we obtai: R 1 23 >HX 1 X 2, 102 R 2 13 >max {IX 2 ; U 2 13 X 1, IX 2 ; U 2 13 X 1 X =IX 2 ; U 2 13 X 1, 104 R R 2 13 >HX 1 X 3 + IX 2 ; U 2 13 X 1 X 3, 105 R 2 3 >IX 2 ; U 2 3 U 2 13 X 1 X Noticig that R 1 R 1 23 ad R 2 R R 2 3 the rate-distortio regio 34 reduces to the desired regio i Theorem 3, where for simplicity we dropped the roud idex. We ow proceed to the proof of the coverse. If a pair of rates R 1, R 2 ad distortios D 12, D 32 are admissible for the K-steps iteractive cooperative distributed source codig settig described i Fig. 3, the for all ε > 0 there exists 0 ε, K, such that > 0 ε, K there is a K-steps iteractive source code, K, F, G with itermediate rates satisfyig: 1 K l=1 log J l i R i + ε, i {1, ad with average per-letter distortios with respect to the source 2 ad perfect recostructio with respect to the source 1 at all odes: E dx2, ˆX ] 32 D 32 + ε, 108 Pr X1 ˆX 21 ε, 109 E dx2, ˆX ] 12 D 12 + ε, 110 Pr X1 ˆX 31 ε, 111

27 27 where ˆX 32 g 32 J 1:K] 1, J 1:K] 2, X3 ˆX 31 g 31 J 1:K] 1, J 1:K] 2, X3, ˆX 12 g 12 J 1:K] 2, X1, ˆX 21 g 2 1, X2, For each t {1,...,, defie radom variables U 2 13t] ad U 2 3t] as follows: U 2 13t] J 1:K] 1, J 1:K] 2, X 11:t 1], X 1t+1:], X 31:t 1], 114 U 2 3t] U 2 13t], X 3t+1:], X 21:t 1]. 115 The fact that these choices of the auxiliary radom variables satisfy the Markov chai 95 ca be obtaied from poit 6 i Lemma 10. By the coditios 111 ad 109, ad Fao s iequality, we have HX1 ˆX 31 Pr X1 ˆX 31 log 2 X1 1 + H 2 Pr X1 ˆX 31 ɛ, 116 HX1 ˆX 21 Pr X1 ˆX 21 log 2 X1 1 + H 2 Pr X1 ˆX 21 ɛ, 117 where ɛ ε 0 provided that ε 0 ad. 1 Rate at ode 1: For the first rate, we have R 1 + ε H J 1:K] 1 H J 1:K] 1 X2 a = I J 1:K] 1 ; X 1 X 2 = HX 1 X 2 H X 1 X 2 J 1:K] where b HX 1 X 2 HX 1 ˆX c HX 1 X 2 ɛ ], 123 step a follows from the fact that by defiitio of the code the sequece J 1:K] 1 is a fuctio of the both sources X 1, X 2, step b follows from the code assumptio i 113 that guaratees the existece of a recostructio fuctio ˆX 21 g 2 1, X2, step c follows from Fao s iequality i 117.

28 28 2 Rate at ode 2: For the secod rate, we have R 2 + ε H J 1:K] 2 a = I J 1:K] 2 ; X1 X2 X3 b I J 1:K] 2 ; X2 X3 X1 where c = I = I d = e = + I f = = g = h = J 1:K] 2 ; X2 X3 X1 J 1:K] 2 ; X3 X1 + I I J 1:K] 2 ; X 3t] X1, X 31:t 1] J 1:K] 2 ; X2 X1 X3 ] +I J 1:K] 2 ; X 2t] X1 X3 X 21:t 1] I J 1:K] 2 X 11:t 1] X 1t+1:] X 31:t 1] ; X 3t] X 1t] J 1:K] 2 X 11:t 1] X 1t+1:] X 31:t 1] X 3t+1:] X 21:t 1] ; X 2t] X 1t] X 3t] ]130 I U 2 13t] ; X 3t] X 1t] + I U2 13t] ; X 2t] X 1t] X 3t] +I U 2 3t] ; X 2t] X 1t] X 3t] U 2 13t] ] 131 I U 2 13t] ; X 2t] X 3t] X 1t] + I U2 3t] ; X 2t] X 1t] X 3t] U 2 13t] ] 132 I U 2 13t] ; X 2t] X 1t] + I U2 3t] ; X 2t] X 1t] X 3t] U 2 13t] ] 133 I U 2 13Q] ; X 2Q] X 1Q], Q = t +I U 2 3Q] ; X 2Q] X 1Q] X 3Q] U 2 13Q], Q = t ] 134 i ] I Ũ2 13 ; X 2 X 1 + I Ũ2 3 ; X 2 X 1 X 3 Ũ 2 13, 135 step a follows from the fact that J 1:K] 2 is a fuctio of the sources X 1, X 2, step b follows from the o-egativity of mutual iformatio, step c follows from the fact that J 2:K] 1 is a fuctio of J 1:K] 2 ad the source X 1,

29 29 step d follows from the chai rule for coditioal mutual iformatio, step e follows from the memoryless property across time of the sources X 1, X 2, X 3, step f follows from the chai rule for coditioal mutual iformatio ad the defiitios 114 ad 115, step g follows from the Markov chai U 2 13t] X 1t], X 2t] X 3t], for all t {1,...,, step h follows from the use of a time sharig radom variable Q uiformly distributed over the set {1,...,, step i follows by lettig ew radom variables Ũ2 13 U 2 13Q], Q ad Ũ2 3 U 2 3Q], Q. 3 Sum-rate of odes 1 ad 2: For the sum-rate, we have R 1 + R 2 + 2ε H J 1:K] 1 + H J 1:K] 2 = H J 1:K] 2 + I J 1:K] 1 ; J 1:K] 2 a = I J 1:K] 2 ; X1 X3 X2 + I J 1:K] 1 ; J 1:K] 2 b I J 1:K] 2 ; X1 X2 X3 where = I J 1:K] 2 ; X1 X3 = H X 1 X 3 H +I + I X 1 J 1:K] 2 X 3 J 1:K] 2 ; X 2 X 1 X 3 c H X 1 X 3 HX 1 ˆX 31 + I J 1:K] 2 ; X2 X1 X3 J 1:K] 2 ; X2 X1 X3 d H X 1 X 3 ɛ ] + I J 1:K] 2 ; X2 X1 X3 e = I J 1:K] 2 X 11:t 1] X 1t+1:] X 31:t 1] X 3t+1:] X 21:t 1] ; X 2t] X 1t] X 3t] + H X 1 X 3 ɛ ] 144 f = H X 1 X 3 ɛ ] + I U 2 13t] U 2 3t] ; X 2t] X 1t] X 3t] 145 g = H X 1 X 3 ɛ + I U 2 13Q] U 2 3Q] ; X 2Q] X 1Q] X 3Q], Q ] 146 ] h = H X 1 X 3 ɛ + I Ũ2 13 Ũ 2 3 ; X 2 X 1 X 3, 147

30 30 step a follows from the fact that J 1:K] 1 ad J 1:K] 2 are fuctios of the sources X 1, X 2, X 3, to emphasize step b follows o-egativity of mutual iformatio, step c follows from the code assumptio i 113 that guaratees the existece of recostructio fuctio ˆX 31 g 3 1, J 1:K] 2, X3, step d follows from Fao s iequality i 111, step e follows from the chai rule of coditioal mutual iformatio ad the memoryless property across time of the source X 1, X 2, X 3, step f from follows from the defiitios 114 ad 115, step g follows from the use of a time sharig radom variable Q uiformly distributed over the set {1,...,, step h follows by lettig ew radom variables Ũ2 13 U 2 13Q], Q ad Ũ2 3 U 2 3Q], Q. 4 Distortio at ode 1: Node 1 recostructs a lossy ˆX 12 g 12 J 1:K] 2, X1. It is clear that we write without loss of geerality ˆX 12 g 12 J 1:K] 1, J 1:K] 2, X1. For each t {1,...,, defie a fuctio ˆX 12t] as begig the t-th coordiate of this estimate: ˆX 12t] U2 13t], X 1t] g12t] J 1:K] 1, J 1:K] 2, X The compoet-wise mea distortio thus verifies 1:K] D 12 + ε E d X 2, g 12 J 1, J 1:K] ] 2, X1 149 = 1 E d X 2t], ˆX ] 12t] U2 13t], X 1t] 150 = 1 E d X 2Q], ˆX ] 12Q] U2 13Q], X 1Q] Q = t 151 = E d X 2Q], ˆX ] 12Q] U2 13Q], X 1Q] 152 = E d X 2, X ] 12 Ũ2 13, X 1, 153 where we defied fuctio X 12 by X 12 Ũ2 13, X 1 = X 12 Q, U2 13Q], X 1Q] ˆX12Q] U2 13Q], X 1Q]. 154

31 5 Distortio at ode 3: Node 3 recostructs a lossy descriptio ˆX 32 g 32 J 1:K] 1, J 1:K] 2, X3. For each t {1,...,, defie a fuctio ˆX 32t] as begig the t-th coordiate of this estimate: ˆX 32t] U2 13t], U 2 3t], X 3t] g32t] J 1:K] 1, J 1:K] 2, X The compoet-wise mea distortio thus verifies 1:K] D 32 + ε E d X 2, g 32 J 1, J 1:K] ] 2, X3 156 = 1 E d X 2t], ˆX ] 32t] U2 13t], U 2 3t], X 3t] 157 = 1 E d X 2Q], ˆX ] 32Q] U2 13Q], U 2 3Q], X 3Q] Q = t 158 = E d X 2Q], ˆX ] 32Q] U2 13Q], U 2 3Q], X 3Q] 159 = E d X 2, X ] 32 Ũ2 13, Ũ2 3, X 3, 160 where we defied fuctio X 32 by X 32 Ũ2 13, Ũ2 3, X 3 = X 32 Q, U2 13Q], U 2 3Q], X 3Q] ˆX 32Q] U2 13Q], U 2 3Q], X 3Q]. 161 This cocludes the proof of the coverse ad thus that of the theorem. 31 C. Two ecoders ad three decoders subject to lossless/lossy recostructio costraits, reversal delivery ad side iformatio Cosider ow the problem described i Fig. 4 where ecoder 1 wishes to commuicate the lossless the source X1 to ode 2 ad a lossy descriptio to ode 3. Ecoder 2 wishes to sed a lossy descriptio of its source X2 to ode 1 ad a lossless oe to ode 3. The correspodig distortio at ode 1 ad 3 are D 12 ad D 31, respectively. I additio to this, the ecoders accomplish the commuicatio usig K commuicatio rouds. This problem is very similar to the problem described i Fig. 3, with the differece that the decodig at ode 3 is iverted. Theorem 4: The rate-distortio regio of the settig described i Fig. 4 is give by the uio over all joit probability measures p X1 X 2 X 3 U 2 13 satisfyig the Markov chai U 2 13 X 1, X 2 X 3 162