1458 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 Heegard Berger Cascade Source Coding Problems With Common Reconstruction Constraints Behzad Ahmadi, Student Member, IEEE, Ravi Ton, Member, IEEE, Osvaldo Simeone, Member, IEEE, H. Vincent Poor, Fellow, IEEE Abstract In lossy source coding with side inmation at the decoder (i.e., the Wyner Ziv problem), the estimate of the source obtained at the decoder cannot be generally reproduced at the encoder, due to its dependence on the side inmation. In some applications, this may be undesirable, a common reconstruction (CR) requirement, whereby one imposes that the encoder decoderbeabletoagree on the decoder s estimate, may be instead in order. The rate-distortion function under the CR constraint has been derived recently a point-to-point (Wyner Ziv) problem. In this paper, this result is extended to three multiterminal settings with three nodes, namely the Heegard Berger (HB) problem, its variant with cooperating decoders, the cascade source coding problem. The HB problem consists of an encoder broadcasting to two decoders with respective side inmation. The cascade source coding problem is characterized by a two-hop system with side inmation available at the intermediate final nodes. For the HB problem with the CR constraint, the rate-distortion function is derived under the assumption that the side inmation sequences are (stochastically) degraded. The rate-distortion function is also calculated explicitly three examples, namely Gaussian source side inmation with quadratic distortion metric, binary source side inmation with erasure Hamming distortion metrics. The rate-distortion function is then characterized the HB problem with cooperating decoders (physically) degraded side inmation. For the cascade problem with the CR constraint, the rate-distortion region is obtained under the assumption that side inmation at the final node is physically degraded with respect to that at the intermediate node.forthelattertwocases,it is worth emphasizing that the corresponding problem without the CR constraint is still open. Outer inner bounds on the rate-distortion region are also obtained the cascade problem under the assumption that the side inmation at the intermediate node is physically degraded with respect to that at the final node. For the three examples mentioned above, the bounds are shown to coincide. Finally, the HB problem, the rate-distortion function is obtained under the more general requirement of constrained reconstruction, whereby the decoder s estimate must be recovered at the encoder only within some distortion. Index Terms Cascade source coding, common reconstruction (CR), Heegard Berger (HB) problem, source coding with side inmation. Manuscript received December 07, 2011; revised August 25, 2012; accepted September 17, 2012. Date of publication November 15, 2012; date of current version February 12, 2013. O. Simeone was supported by the U.S. National Science Foundation under Grant CCF-0914899. H. V. Poor R. Ton were supported in part by the U.S. Air Force Office of Scientific Research under MURI Grant FA9550-09-1-0643 in part by the U.S. National Science Foundation under Grant CNS-09-05398. This paper was presented in part at the 2012 IEEE International Symposium on Inmation Theory. B. Ahmadi O. Simeone are with the Center of Wireless Communication Signal Processing Research, New Jersey Institute of Technology, Newark, NJ 07102 USA (e-mail: behzad.ahmadi@njit.edu; osvaldo.simeone@njit.edu). R. Ton was with Princeton University, Princeton, NJ 08544 USA. He is now with Virginia Tech, Blacksburg, VA 24061 USA (e-mail: tonr@vt.edu; rton@princeton.edu). H. V. Poor is with Princeton University, Princeton, NJ 08544 USA (e-mail: poor@princeton.edu). Communicated by M. Skoglund, Associate Editor Communications. Digital Object Identifier 10.1109/TIT.2012.2227679 0018-9448/$31.00 2012 IEEE I. INTRODUCTION SOURCE coding problems with side inmation at the decoder(s) model a large number of scenarios of practical interest, including video streaming [1] wireless sensor networks [2]. From an inmation-theoretic perspective, the baseline setting this class of problems is one in which a memoryless source is to be communicated by an encoder at a rate bits per source symbol to a decoder that has available a correlated sequence that is related to via a memoryless channel (see Fig. 1 1 ). Under the requirement of asymptotically lossless reconstruction of the source at the decoder, the minimum required rate was obtained by Slepian Wolf [3]. Later, the more general optimal tradeoff between rate the distortion between the source reconstruction was obtained by Wyner Ziv [4] any given distortion metric. It was shown to be given by the rate-distortion function where the minimum is taken over all probability mass functions (pmfs) deterministic function such that. A. Heegard Berger Cascade Source Coding Problems In applications such as the ones discussed above, the point-topoint setting of Fig. 1 does not fully capture the main features of the source coding problem. For instance, in video streaming, a transmitter typically broadcasts inmation to a number of decoders. As another example, in sensor networks, data are typically routed over multiple hops toward the destination. A model that accounts the aspect of broadcasting to multiple decoders is the Heegard Berger (HB) setup shown in Fig. 2. In this model, the link of rate bits per source symbol is used to communicate to two receivers having different side inmation sequences,, which are related to the source via a memoryless channel. The set of the achievable triples ( ) this model, where are the distortion levels at Decoders 1 2, respectively, was derived in [6] [7] under the assumption that the side inmation sequences are (stochastically) degraded versions of the source.inavariation of this model shown in Fig. 3, decoder cooperation is enabled by a limited capacity link from one decoder (Decoder 1) to the other (Decoder 2). Inner outer bounds on the rate-distortion region this problem are obtained in [8] under the assumption that the side inmation of Decoder 2 is (physically) degraded with respect to that of Decoder 1. As multihopping, a basic model that captures some of the key designissuesisshowninfig.4.inthiscascade setup, an 1 The presence of the function at the encoder will be explained later. (1)
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1459 Fig. 1. Point-to-point source coding with common reconstruction [5]. Fig. 4. Cascade source coding problem with common reconstruction. Fig. 2. HB source coding problem with common reconstruction. Fig. 3. HB source coding problem with common reconstruction decoder cooperation. encoder (Node 1) communicates with rate to an intermediate node (Node 2), which has side inmation, in turn communicates with rate to a final node (Node 3) with side inmation. Both Nodes 2 3 act as decoders, similar to the HB problem of Fig. 2, in the sense that they reconstruct a local estimate of the source. The rate-distortion function this problem has been derived various special cases in [9] [11] [12] (see [12, Table I] an overview). Chia et al. [11] derive the set of all achievable quadruples, i.e., the rate-distortion region, the case in which is also available at the encoder is a physically degraded version of with respect to. Instead, Vasudevan et al. [10] derive the rate-distortion region under the assumptions that the source the side inmation sequences are jointly Gaussian, that the distortion metric is quadratic, that the sequence is a physically degraded version of with respect to.the corresponding result binary source side inmation Hamming distortion metric was derived in [12]. B. Common Reconstruction Constraint A key aspect of the optimal strategies identified in [4], [6], [7], [10], [11] is that the side inmation sequences are, in general, used in two different ways: (i) as a means to reduce the rate required communication between encoder decoders via binning; (ii) as an additional observation that the decoder can leverage, along with the bits received from the encoder, in order to improve its local estimate. For instance, the point-to-point system of Fig. 1, the Wyner Ziv result (1) reflects point (i) of the discussion above in the conditioning on side inmation, which reduces the rate, point (ii)inthe fact that the reconstruction is a function of the signal received from the encoder the side inmation. Leveraging the side inmation as per point (ii), while advantageous in terms of rate-distortion tradeoff, may have unacceptable consequences some applications. In fact, this use of side inmation entails that the reconstruction of the decoder cannot be reproduced at the encoder. In other words, the encoder decoder cannot agree on the specific reconstruction obtained at the receiver side, but only on the average distortion level. In applications such as transmission of sensitive medical, military or financial data, this may not be desirable. Instead, one may want to add the constraint that the reconstruction at the decoder be reproducible by the encoder [5]. This idea, referred to as the common reconstruction (CR) constraint, was first proposed in [5], where it is shown the point-to-point setting of Fig. 1 2 that the rate-distortion function under the CR constraint is given by where the minimum is taken over all pmfs such that. Comparing (2) with the Wyner Ziv rate-distortion (1), it can be seen that the additional CR constraint prevents the decoder from using the side inmation as a means to improve its estimate (see point (ii) above). The original work of Steinberg [5] has been recently extended in [13], where a relaxed CR constraint is imposed in which only a distortion constraint is imposed between the decoder s reconstruction its reproduction at the encoder. We refer to this setting as imposing a constrained reconstruction (ConR) requirement. C. Main Contributions In this paper, we study the HB source coding problem (see Fig. 2) the cascade source coding problem (see Fig. 4) under the CR requirement. The considered models are thus relevant the transmission of sensitive inmation, which is constrained by CR, via broadcast or multihop links a common occurrence 2 The function at the encoder calculates the estimate of the encoder regarding the decoder s reconstruction. (2)
1460 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 in, e.g., medical, military, or financial applications (e.g., intranets of hospitals or financial institutions). Specifically, our main contributions are the following. 1) For the HB problem with the CR constraint (see Fig. 2), we derive the rate-distortion function under the assumption that the side inmation sequences are (stochastically) degraded. We also calculate this function explicitly three examples, namely Gaussian source side inmation with quadratic distortion metric, binary source erasure side inmation with erasure Hamming distortion metrics (see Section II). 2) For the HB problem with the CR constraint decoder cooperation (see Fig. 3), we derive the rate-distortion region under the assumption that the side inmation sequences are (physically) degraded in either direction (see Sections III-A III-B). We emphasize that the corresponding problem without the CR constraint is still open as per the discussion above. 3) For the cascade problem with the CR constraint (see Fig. 4), we obtain the rate-distortion region under the assumption that side inmation is physically degraded with respect to (see Section IV-B). We emphasize that the corresponding problem without the CR constraint is still open as per the discussion above. 4) For the cascade problem with CR constraint (see Fig. 4), we obtain outer inner bounds on the rate-distortion region under the assumption that the side inmation is physically degraded with respect to. Moreover, the three examples mentioned above in the context of the HB problem, we show that the bounds coincide we evaluate the corresponding rate-distortion region explicitly (see Section IV-C). 5) For the HB problem, we finally derive the rate-distortion function under the more general requirement of ConR under the assumption that side inmation is physically degraded with respect to (Sec.V). Notation: For integer with,wedefine as the interval we use to denote the sequence. We will also write simplicity. Upper case, lower case, calligraphic letters denote rom variables, specific values of rom variables their alphabets, respectively. Given discrete rom variables, or more generally vectors,, we will use the notation or, or, where the latter notations are used when the meaning is clear from the context. Given a set, we denoted by the -fold Cartesian product of. For rom variables,wedenoteby the (average) conditional variance of given,i.e.,. We adopt the notation convention in [14], in which represents any function such that as.wedefine the binary entropy function. Finally, we define. II. HB PROBLEM WITH CR In this section, we first detail the system model the HB source coding problem in Fig. 2 with CR in Section II-A. Next, the characterization of the corresponding rate-distortion permance is derived under the assumption that one of the two side inmation sequences is a stochastically degraded version of the other in the sense of [6] [see (10)]. Finally, three specific examples are worked out, namely Gaussian sources under quadratic distortion (see Section II-C), binary sources with side inmation sequences subject to erasures under Hamming or erasure distortion (see Section II-D). A. System Model In this section, the system model the HB problem with CR is detailed. The system is definedbythepmf discrete alphabets, as follows. The source sequence side inmation sequences,with,,, are such that the tuples are independent identically distributed (i.i.d.) with joint pmf.the encoder measures a sequence encodes it into a message of bits, which is delivered to the decoders. Decoders 1 2 wish to reconstruct the source sequence within given distortion requirements, to be discussed below, as, respectively. The estimated sequence is obtained as a function of the message the side inmation sequence. The estimates are constrained to satisfy distortion constraints defined by per-symbol distortion metrics with. Based on the given distortion metrics, the overall distortion the estimated sequences is defined as The reconstructions are also required to satisfy the CR constraints, as malized below. Definition 1: An code the HB problem with CR consists of an encoding function which maps the source sequence into a message ;adecoding function Decoder 1 which maps the message the side inmation into the estimated sequence ; a decoding function Decoder 2 which maps message the side inmation into the estimated sequence ; two reconstruction functions (3) (4) (5) (6) (7a) (7b) which map the source sequence into the estimated sequences at the encoder, namely, respectively, such that the distortion constraints are satisfied, i.e., (8)
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1461 the CR requirements hold, namely (9) Remark 1: Under the physical degradedness assumption that the Markov chain condition holds, (11) can be rewritten as Given distortion pairs, a rate pair is said to be achievable if, any sufficiently large, there exists an code. The rate-distortion function is defined as. B. Rate-Distortion Function In this section, a single-letter characterization of the rate-distortion function the HB problem with CR is derived, under the assumption that the joint pmf is such that there exists a conditional pmf which (10) (14) with the minimization defined as in (11). This expression quantifies by the additional rate that is required with respect to the ideal case in which both decoders have the better side inmation. Remark 2: If we remove the CR constraint, then the rate-distortion function under the assumption of Proposition 1 is given by [6] (15) where the mutual inmation terms are evaluated with respect to the joint pmf is a stochastically de- In other words, the side inmation graded version of. Proposition 1: If the side inmation is stochastically degraded with respect to, the rate-distortion function the HB problem with CR is given by (11) where the mutual inmation terms are evaluated with respect to the joint pmf (12) minimization is permed with respect to the conditional pmf under the constraints (13) The proof of the converse can be found in Appendix A. Achievability follows as a special case of [6, Th. 3] can be easily shown using stard arguments. In particular, the encoder romly generates a stard lossy source code the source with rate bits per source symbol. Rom binning is used to reduce the rate to. By the Wyner Ziv theorem [14, p. 280], this guarantees that both Decoders 1 2 are able to recover (since is a degraded version of ). The encoder then maps the source into the reconstruction sequence using a codebook that is generated conditional on with rate bits per source symbol. Rom binning is again used to reduce the rate to. From the Wyner Ziv theorem, the fact that Decoder 2 knows the sequence, it follows that Decoder 2 can recover the reconstruction as well. Note that, since the reconstruction sequences are generated by the encoder, functions that guarantee the CR constraints (9) exist by construction. (16) minimization is permed with respect to the conditional pmf the deterministic functions,, such that distortion constraints (13) are satisfied. Comparison of (11) with (15) reveals that, similar to the discussion around (1) (2), the CR constraint permits the use of side inmation only to reduce the rate via binning, but not to improve the decoder s estimates via the use of the auxiliary codebooks represented by variables, functions,,in(16). Remark 3: Consider the case in which the side inmation sequences are available in a causal fashion in the sense of [16], that is, the decoding functions (5) (6) are modified as,, respectively. Following similar steps as in the proof of Proposition 2 in [16], it can be concluded that, under the CR constraint, the rate-distortion function in this case is the same as if the two side inmation sequences were not available at the decoders is thus given by (11) upon removing the conditioning on the side inmation. Note that this is true irrespective of the joint pmf, hence, it holds also nondegraded side inmation. This result can be explained by noting that, as explained in [16], causal side inmation prevents the possibility of reducing the rate via binning. Since the CR constraint also prevents the side inmation from being used to improve the decoders estimates, it follows that the side inmation is useless in terms of rate-distortion permance, if used causally under the CR constraint. On a similar note, if only side inmation is causally available, while can still be used in the conventional noncausal fashion, then it can be proved that can be neglected without loss of optimality. Theree, the rate-distortion function follows from (11) by removing the conditioning on. Remark 4: In [19], a related model is studied in which the source is given as each decoder is interested
1462 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 in reconstructing a lossy version of the side inmation available at the other decoder. The CR constraint is imposed in a different way by requiring that each decoder be able to reproduce the estimate reconstructed at the other decoder. C. Gaussian Sources Quadratic Distortion In this section, we highlight the result of Proposition 1 by considering a zero-mean Gaussian source,with side inmation variables (17a) (17b) where are independent of each other of. Note that the joint distribution of satisfies the stochastic degradedness condition. We focus on the quadratic distortion. By leveraging stard arguments that allow us to apply Proposition 1 to Gaussian sources under mean-square-error constraint (see [14, pp. 50 51] [15]), we obtain a characterization of the rate-distortion function the given distortion metrics. We first recall that the point-to-point setup in Fig. 1 with side inmation,with independent of, the rate-distortion function with CR under quadratic distortion is given by [5] (18) where we have made explicit dependence on of the function convenience. The rate-distortion function (18) is obtained from (2) by choosing the distribution such that,where is independent of. Proposition 2: The rate-distortion function the HB problem with CR Gaussian sources (17) quadratic distortion is given by where if if if if is definedin(18) (19) (20) Remark 5: The rate-distortion function the HB problem Gaussian sources (17) without the CR constraint can be Fig. 5. Illustration of the distortion regions in the rate-distortion function (19) Gaussian sources quadratic distortion. found in [6]. Comparison with (19) confirms the permance loss discussed in Remark 2. Definition of the rate-distortion function (19) requires different consideration the four subregions of the planesketchedinfig.5.infact,, the required rate is zero, since the distortion constraints are trivially met by setting in the achievable rate (11). For the case,itissufficient to cater only to Decoder 2 by setting,with independent of, in the achievable rate (11). That this rate cannot be improved upon follows from the trivial converse (21) which follows by cut-set arguments. The same converse suffices also the regime.for this case, achievability follows by setting in (11), where is independent of. In the remaining case, namely,theratedistortion function does not follow from the point-to-point result (18) as the regimes discussed thus far. The analysis of this case requires use of the entropy-power inequality (EPI) can be found in Appendix B Fig. 6 depicts the rate in (19) versus different values of with,,. As discussed above,, which is larger than, becomes zero values of larger than, while this is not the case values. D. Binary Source With Erased Side Inmation Hamming or Erasure Distortion In this section, we consider a binary source with erased side inmation sequences. The source is an erased version of the source with erasure probability is an erased version of with erasure probability.thismeansthat,where represents an erasure, with probability with probability. Note that, with these assumptions, the side inmation is stochastically degraded with respect to. In fact, we have the factorization (10), where additional distributions are illustrated in Fig. 7. As seen in Fig. 7,
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1463 The rate-distortion function (24) is obtained from (2) by choosing the distribution such that with probability with probability. Remark 6: The rate-distortion function with erased side inmation Hamming distortion without the CR constraint is derived in [17] (see also [18]). Comparison with (23) shows again the limitation imposed by the CR constraint on the use of side inmation (see Remark 2). Proposition 3: The rate-distortion function the HB problem with CR the binary source with the stochastically degraded erased side inmation sequences illustrated in Fig. 7 under Hamming distortion is given by Fig. 6. Rate-distortion function in (19) versus distortion different values of distortion,,. if if if if (25) where is definedin(23) Fig. 7. Illustration of the pmfs in the factorization (10) of the joint distribution a binary source erased side inmation sequences. the pmf is characterized by the probability that satisfies the equality. We focus on Hamming erasure distortions. For Hamming distortion, the reconstruction alphabets are binary,,we have if otherwise. Instead, the erasure distortion, the reconstruction alphabets are, we have : otherwise. (22) In Appendix C, we prove that the point-to-point setup in Fig. 1 with erased side inmation,with erasure probability, the rate-distortion function with CR under Hamming distortion is given by (23) wherewehavemadeexplicitthedependenceon of the function convenience. The rate-distortion function (23) is obtained from (2) by choosing the distribution such that,where is independent of. Following the same steps as in Appendix C, it can be also proved that the point-to-point setup in Fig. 1 with erased side inmation,witherasure probability, the rate-distortion function with CR under erasure distortion is given by (24) (26) Moreover, the same source under erasure distortion, the ratedistortion function is given by (25) by substituting with as defined in (24) bysubstituting (26) with (27) Similar to the Gaussian example, the characterization of the rate-distortion function (25) requires different considerations the four subregions of the plane sketched in Fig. 8. In fact,, the required rate is zero, since the distortion constraints are trivially met by setting in the achievable rate (11). For the case,itissufficient to cater only to Decoder 2 by setting, with independent of, in the achievable rate (11). That this rate cannot be improved upon is a consequence from the trivial converse (28) which follows by cut-set arguments. The same converse suffices also the regime.for this case, achievability follows by setting in (11), where is independent of. In the remaining case, namely,theratedistortion function does not follow from the point-to-point result (23) as the regimes discussed thus far. The analysis of this case can be found in Appendix D. Similar arguments apply also the erasure distortion metric. We now compare the rate-distortion function the binary source with erased side inmation under Hamming distortion three settings. In the first setting, known as the Kaspi model [7], the encoder knows the side inmation
1464 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 permance, i.e.,.moreover, if is sufficiently smaller than, the operation of the encoder is limited by the distortion requirements of Decoder 1. In this case, Decoder 2 can in fact reconstruct as while still satisfying its distortion constraints. Theree, we obtain the same permance in all of the three settings, i.e.,.wealso note the general permance loss due to the CR constraint, unless, as discussed above, distortion is sufficiently smaller than. Fig. 8. Illustration of the distortion regions in the rate-distortion function (25) a binary source with degraded erased side inmation Hamming distortion. III. HB PROBLEM WITH COOPERATIVE DECODERS The system model the HB problem with CR decoder cooperation is similar to the one provided in Section II-A with the following differences. Here, in addition to the encoding function given in (4) which maps the source sequence into a message of bits, there is an encoder at Decoder 1 given by (30) which maps message the source sequence into a message. Moreover, instead of the decoding function given in (5), we have the decoding function Decoder 2 (31) which maps the messages the side inmation into the estimated sequence. Fig. 9. Rate-distortion functions [17], [12] (25) a binary source under erased side inmation versus distortion (,,, ). thus the position of the erasures. For this case, the rate-distortion function the example at h was calculated in [17]. Note that in the Kaspi model, the CR constraint does not affect the rate-distortion permance since the encoder has all the inmation available at the decoders. The second model of interest is the stard HB setting with no CR constraint, whose rate-distortion function the example at h was derived in [12]. The third model is the HB setup with CR studied here. We clearly have the inequalities (29) where the first inequality in (29) accounts the impact of the availability of the side inmation at the encoder, while the second reflects the potential permance loss due to the CR constraint. Fig. 9 shows the aementioned rate-distortion functions with, which corresponds to the case where Decoder 1 has no side inmation, two values of the distortion versus the distortion.for, the given settings reduce to one in which the encoder needs to communicate inmation only to Decoder 1. Since Decoder 1 has no side inmation, the Kaspi HB settings yield equal A. Rate-Distortion Region In this section, a single-letter characterization of the rate-distortion region is derived under the assumption that the joint pmf is such that the Markov chain condition holds. 3 Proposition 4: The rate-distortion region the HB source coding problem with CR cooperative decoders under the assumption is given by the union of all rate pairs that satisfy the conditions (32a) (32b) where the mutual inmation terms are evaluated with respect to the joint pmf (33) some pmf such that the constraints (13) are satisfied. The proof of the converse can be easily established following cut-set arguments bound (32a), while the bound (32b) on the sum-rate can be proved following the same step as in Appendix A substituting with. As the achiev- 3 Note that, unlike the conventional HB problem studied in Section II, the ratedistortion region with cooperative decoders depends on the joint distribution of the variables ( ), thus, stochastic physical degradedness of the side inmation sequences lead to different results.
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1465 ability, it follows as a straightward extension of [8, Sec. III] to the setup at h where Decoder 2 has side inmation as well. It is worth emphasizing that the reconstruction the Decoder 2, which has degraded side inmation, is conveyed by using both the direct link from the Encoder, of rate, the path Encoder-Decoder 1-Decoder 2. The latter path leverages the better side inmation at Decoder 1 the cooperative link of rate. Remark 7: If we remove the CR constraint, the problem of determining the rate-distortion region the setting of Fig. 3 under the Markov assumption is still open. In [8], inner outer bounds are obtained to the rate-distortion region, the case in which the side inmation is absent. The bounds were shown to coincide the case in which Decoder 1 wishes to recover losslessly (i.e., )also certain distortion regimes in the quadratic Gaussian case. Moreover, the rate-distortion tradeoff is completely characterized in [8] the case in which the encoder also has access to the side inmation. We note that, as per the discussion in Section II-D, these latter results immediately carry over to the case with the CR constraint since the encoder is inmed about the side inmation. Remark 8: To underst why imposing the CR constraint simplifies the problem of obtaining a single-letter characterization of the rate-distortion function, let us consider the degrees of freedom available at Decoder 1 in Fig. 3 the use of the link of rate. In general, Decoder 1 can follow two possible strategies: the first is warding, whereby Decoder 1 simply wards some of the bits received from the encoder to Decoder 2; while the second is recompression, whereby the data received from the encoder are combined with the available side inmation, compressedtoatmost bits per symbol, then sent to Decoder 2. It is the interplay contrast between these two strategies that makes the general problem hard to solve. In particular, while the strategies of warding/recompression combinations thereof appear to be natural cidates the problem, finding a matching converse when both such degrees of freedom are permissible at the decoder is difficult (see, e.g., [20]). However, under the CR constraint, the strategy of recompression becomes irrelevant, since any inmation about the side inmation that is not also available at the encoder cannot be leveraged by Decoder 2 without violating the CR constraint. This restriction in the set of available strategies Decoder 1 makes the problem easier to address under the CR constraint. B. Rate-Distortion Region In this section, a single-letter characterization of the rate-distortion region is derived under the assumption that the joint pmf is such that the Markov chain relationship holds. Proposition 5: The rate-distortion region the HB source coding problem with CR cooperative decoders under the assumption of the Markov chain relationship is given by the union of all rate pairs satisfy the conditions that (34a) (34b) where the mutual inmation terms are evaluated with respect to the joint pmf (35) some pmf such that the constraints (13) are satisfied. The proof of achievability follows immediately by neglecting the link of rate using rate as per the HB scheme of Proposition 1. The converse follows by considering an enhanced system in which Decoder 2 is provided with the side inmation of Decoder 1. In this system, link becomes useless since Decoder 2 possesses all the inmation available at Decoder 1. It follows that the system reduces to the HB problem with degraded sources studied in the previous section the bound (34a) follows immediately from Proposition 1. Remark 9: In the case without CR, the rate-distortion function is given similarly to (34), but with the HB rate-distortion function (15) in lieu of the rate-distortion function of the HB problem with CR in (34a). IV. CASCADE SOURCE CODING WITH CR In this section, we first detail the system model in Fig. 4 of cascade source coding with CR. As mentioned in Section I, the motivation studying this class of models comes from multihop applications. Next, the characterization of the corresponding rate-distortion permance is presented under the assumption that one of the two side inmation sequences is a degraded version of the other. Finally, following the previous section, three specific examples are worked out, namely Gaussian sources under quadratic distortion (see Section IV-C1), binary sources with side inmation subject to erasures under Hamming or erasure distortion (see Section IV-C2). A. System Model In this section, the system model the cascade source coding problem with CR is detailed similar to Section II-A. The problem is defined by the pmf discrete alphabets, as follows. The source sequence side inmation sequences,with,,,aresuch that the tuples are i.i.d. with joint pmf. Node 1 measures a sequence encodes it into a message of bits, which is delivered to Node 2. Node 2 estimates a sequence within given distortion requirements. Node 2 also encodes the message received from Node 1 the sequence into a message of bits, which is delivered to Node 3. Node 3 estimates a sequence within given distortion requirements. Distortion CR requirements are definedasinsectionii-a, leading to the following definition.
1466 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 Definition 2: An code the cascade source coding problem with CR consists an encoding function Node 1 (36) which maps the source sequence into a message ;anencoding function Node 2 (37) which maps the source sequence message into a message ; a decoding function Node 2 (38) which maps message the side inmation into the estimated sequence ; a decoding function Node 3 (39) which maps message the side inmation into the estimated sequence ; two encoder reconstruction functions as in (7), which map the source sequence into estimated sequences at Node 1; such that the distortion constraints (8) (9) are satisfied. Given a distortion pair,aratepair is said to be achievable if, any sufficiently large,there exists an code. The rate-distortion region is defined as the closure of all rate pairs that are achievable given the distortion pair. B. Rate-Distortion Region In this section, a single-letter characterization of the rate-distortion region is derived under the assumption that the joint pmf is such that the Markov chain relationship holds. 4 Proposition 6: The rate-distortion region the cascade source coding problem with CR is given by the union of all rate pairs that satisfy the conditions (40a) (40b) where the mutual inmation terms are evaluated with respect to the joint pmf (41) some pmf such that the constraints (13) are satisfied. The proof of the converse is easily established following cut-set arguments. To prove achievability, it is sufficient to consider a scheme based on binning at Node 1 decode 4 As the HB problem with cooperative decoders studied in Section III, the rate-distortion region of the cascade source coding problem depends on the joint distribution of the variables ( ), thus, stochastic physical degradedness of the side inmation sequences lead to different results. rebin at Node 2 (see [11]). Specifically, Node 1 romly generates a stard lossy source code the source with rate bits per source symbol. Rom binning is used to reduce the rate to. Node 1 then maps the source into the reconstruction sequence using a codebook that is generated conditional on with rate bits per source symbol. Using the side inmation available at Node 2, rom binning is again used to reduce the rate to. The codebook of is also romly binned to the rate. Node 2, having recovered, wards the corresponding bin index to Node 3. The latter, by choice of the binning rate, is able to obtain. Note that, since the reconstruction sequences are generated by the encoder, functions that guarantee the CR constraints (9) exist by construction. Remark 10: If we remove the CR constraint, the problem of determining the rate-distortion region the setting of Fig. 4 under the Markov condition is still open. In the special case in which the problem has been solved in [10] Gaussian sources under quadratic distortion in [12] binary sources with erased side inmation under Hamming distortion. Remark 11: Following Remark 3, if both side inmation sequences are causal, it can be shown that they have no impact on the rate-distortion function (40). Theree, the rate-distortion region follows immediately from the results in (40) by removing both of the side inmation terms. Note that with causal side inmation sequences, the rate-distortion function holds any joint pmf with no degradedness requirements. Moreover, if only the side inmation is causal, while is still observed noncausally, then the side inmation can be neglected without loss of optimality, the rate-distortion region follows from (40) by removing the conditioning on. C. Bounds on the Rate-Distortion Region In this section, outer inner bounds are derived the rate-distortion region under the assumption that the joint pmf is such that the Markov chain relationship holds. The bounds are then shown to coincide in Section IV-C1 Gaussian sources in Section IV-C2 binary sources with erased side inmation. Proposition 7 (Outer Bound): The rate-distortion region the cascade source coding problem with CR is contained in the region, which is given by the set of all rate pairs that satisfy the conditions (42a) (42b) where is defined in (11) we have, where the minimization is permed with respect to the conditional pmf under the distortion constraints (13).
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1467 Proposition 8 (Inner Bound): The rate-distortion region the cascade source coding problem with CR contains the region, which is given by the union of all rate pairs that satisfy the conditions (43a) (43b) (43c) where the mutual inmation terms are evaluated with respect to the joint pmf (44) some pmf such that the distortion constraints (13) are satisfied. The outer bound in Proposition 7 follows immediately from cut-set arguments similar to those in [10] [12]. As the inner bound of Proposition 19, the strategy works as follows. Node 1 sends the description to Node 2 using binning with rate. It also maps the sequence into the sequence using a conditional codebook with respect to, which is binned in order to leverage the side inmation at Node 3 with rate. Node 2 recovers,whose codebook is then binned to rate. Then, it wards the so obtained bin index the bin index the codebook of produced by Node 1 to Node 3. By the choice of the rates, the latter can recover both. Since both descriptions are produced by Node 1, the CR constraint is automatically satisfied. The inner outer bounds defined above do not coincide in general. However, in the next sections, we provide two examples in which they coincide thus characterize the rate-distortion region of the corresponding settings. Remark 12: Without the CR constraint, the problem of deriving the rate-distortion region the setting at h under the Markov chain condition is open. The problem has been solved in [10] Gaussian sources under quadratic distortion in [12] binary sources with erased side inmation under Hamming distortion. 1) Gaussian Sources Quadratic Distortion: In this section, we assume the Gaussian sources in (17) the quadratic distortionasinsectionii-cderive the rate-distortion region the cascade source coding problem with CR. Proposition 9: The rate-distortion region the cascade source coding problem with CR the Gaussian sources in (17) quadratic distortion is given by (42) with in (19) [see (18)]. The proof is given in Appendix E. 2) Binary Sources With Erased Side Inmation Hamming Distortion: In this section, we assume the binary sources in Fig. 7 Hamming distortion, as in Section II-D, derive the rate-distortion region the cascade source coding problem with CR. Proposition 10: The rate-distortion region the cascade source coding problem with CR the binary sources in Fig. 7 Hamming distortion is given by (42) with in (25) [see (23)]. The proof is given in Appendix F. V. HB PROBLEM WITH CONR In this section, we revisit the HB problem relax the CR constraint to the ConR constraint of [13]. This implies that we stilladoptthecodeasperdefinition 1, but we substitute (9) with the less stringent constraint (45) where is a per-symbol distortion metric we have used,, to denote the th letter of the vector. Definition 3: Given a distortion tuple, arate is said to be achievable if, any sufficiently large,thereaexistsan code. The rate-distortion function is defined as : the tuple is achievable. Note that, by setting,letting be the Hamming distortion metric (i.e., if if ), we obtain a relaxed CR constraint in which the average per-symbol, rather than per-block, error probability criterion is adopted. Remark 13: The problem at h reduces to the one studied in [13] by setting. Proposition 11: If the side inmation is stochastically degraded with respect to, the rate-distortion function the HB problem with ConR is given by (46a) (46b) where the mutual inmation terms are evaluated with respect to the joint pmf (47) minimization is permed with respect to the conditional pmf the deterministic functions, such that the distortion constraints, the ConR requirements (48)
1468 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 are satisfied. Finally, are auxiliary rom variables whose alphabet cardinalities can be constrained as. The proof is given in Appendix G. Remark 14: Proposition 11 reduces to [13, Th. 2] when setting. Remark 15: Similar to [13, Th. 2], it can be proved that, by setting letting be the Hamming distortion, the rate-distortion function (46),, reduces to the rate-distortion function with CR (11). Remark 16: Similar to Remark 15, if, the rate-distortion function (46) is given by (49) where the mutual inmation terms are evaluated with respect to the joint pmf (50) minimization is permed with respect to the conditional pmf the deterministic functions, such that the distortion constraints the ConR requirement are satisfied. It can be proved that this is also the rate-distortion function under the partial CR requirement that there exists a function such that (9) holds only. Similar conclusions apply symmetrically to the case where CR ConR requirements are imposed only on the reconstruction of Decoder 2. Remark 17: If both side inmation sequences are causally available at the decoders, it can be proved that they have no impact on the rate-distortion function (46). In this case, the ratedistortion function follows immediately from the results in (46) by removing conditioning on both side inmation sequences. Moreover, the result can be simplified by introducing a single auxiliary rom variable. Similarly, if only side inmation is causal, then it can be neglected with no loss of optimality, the results follow from (46) by removing the conditioning on. Remark 18: We note that the ConR mulation studied in this section is more general than the conventional mulation with distortion constraints the decoders only. Theree, problems that are open with the conventional mulation, such as HB with cooperative decoders (see Section III) cascade source coding (see Section IV), are a tiori also open in the ConR setup. VI. CONCLUDING REMARKS The CR requirement [5] its generalization in [13] substantially modify the problem of source coding in the presence of side inmation at the decoders. From a practical stpoint, in various applications, such as transmission of medical records, CR is a design constraint. In these cases, evaluation of the ratedistortion permance under CR thus reveals the cost, in terms of transmission resources, associated with this additional requirement. From a theoretical perspective, adding the CR constraint to stard source coding problems with decoder side inmation proves instrumental in concluding about the optimality of various known strategies in settings in which the more general problem, without the CR constraint, is open [5]. This paper has extended these considerations from a point-to-point setting to three baseline multiterminal settings, namely the HB problem, the HB problem with cooperating decoders the cascade problems. The optimal rate-distortion tradeoff has been derived in a number of cases explicitly evaluated in various examples. A general subject of theoretical interest is identifying those models which the CR requirements enable a solution of problems that have otherwise resisted solutions. Examples include the HB cascade source coding problems with no assumptions on side inmation degradedness the one-helper lossy source coding problem. APPENDIX A PROOF OF PROPOSITION 1 We first observe that from Definition 1, distortion CR constraints (8) (9) depend only on the marginal pmfs, so does the rate-distortion function. Theree, in the proof, we can assume, without loss of generality, that the joint pmf satisfies the Markov chain condition so that it factors as [cf., (10)] (51) Consider an code, whose existence is required achievability by Definition1.BytheCRrequirements (9), we first observe that we have the Fano inequalities (52) sufficiently large, where log.moreover, we can write (53a) (53b) where ( ) follows by the definition of mutual inmation. From now on, to simplify notation, we do not make explicit the dependence of,, on, respectively. We also define as the th symbol of the sequence so that.
AHMADI et al.: HEEGARD BERGER AND CASCADE SOURCE CODING PROBLEMS WITH COMMON RECONSTRUCTION CONSTRAINTS 1469 The first term in (53b),, can be treated as in [5, Sec. V.A.], or, more simply, we can proceed as follows: where ( ) follows because of the Markov chain relationship,.bydefining with, the proof is concluded as in [5]. (54a) (54b) (54c) (54d) (54e) (54f) (54g) where ( ) follows because is a function of ;() follows since are functions of, respectively; ( ) follows by using the Markov chain relationship ;( ) follows by the chain rule of mutual inmation since mutual inmation is nonnegative; ( ) follows by (52) since entropy is nonnegative; ( ) follows by the chain rule entropy, since are i.i.d., due to the fact that conditioning decreases entropy. Similarly, the second term in (53b), namely,, leads to APPENDIX B PROOF OF PROPOSITION 2 As explained in the text, we need to focus only on the case in which. As per the discussion in Appendix A, we can assume, without loss of generality, that the Markov chain relationship holds, so that (57a) (57b) where is independent of. We first prove a converse. Calculating the rate-distortion function in (14) requires minimization over the pmf under the constraint (13). A minimizing exists by the Weierstrass theorem due to the continuity of the mutual inmation the compactness of the set of pmfs defined by the constraint (13) [21]. Fixing one such optimizing, the rate-distortion function (14) can be written as (58) The first term in (58), i.e.,, can be easily bounded using the approach in [5, p. 5007]. Specifically, we have (55a) (55b) (55c) (55d) (55e) where ( ) follows because is a function of ;() follows by the chain rule of mutual inmation since mutual inmation is nonnegative; ( ) follows by (52) since entropy is nonnegative; ( ) follows by the chain rule entropy, since are i.i.d., due to the fact that conditioning decreases entropy. From (53b), (54g), (55e), we then have (56a) (59) where ( ) follows because conditioning decreases entropy; ( ) follows from the maximum conditional entropy lemma [14, p. 21], which implies that with. In fact, we have that, since the conditional variance is upper bounded by the linear minimum mean square error of the estimate of given. This mean square error is given by,sincewe have since is independent of due to the factorization (12) to the independence of.forthe second term in (58), we instead have the following: (56b) (60)
1470 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 59, NO. 3, MARCH 2013 Moreover, we can evaluate For achievability, we calculate (14) with,where are independent of each other of. This leads to the upper bound (61) where follows because is independent of of, due to the factorization (12) due to the independence of. Next, we obtain a lower bound on the term in (61) as a function of by using the EPI [14, p. 22]. Specifically, by using the conditional version of the EPI [14, p. 22], we have (62) where follows because is independent of as explained above. The first two terms in (61) can thus be bounded as (66) (63) where (a) follows because is an increasing function of, as can be proved by using the same approach used the bounds in (59). By substituting (63) into (61), using the result in (60), we obtain where ( ) follows using, being independent Gaussian sources with. By comparing (80) with (66), we complete the proof. APPENDIX C PROOF OF (23) Here, we prove that (2) equals (23) the given sources. For the converse, we have that (64) Finally, by substituting (59) (64) into (58), we obtain the lower bound (67) (65) where ( ) follows because conditioning decreases entropy. Achievability follows by calculating (2) with where.