Multiterminal Source Coding with an Entropy-Based Distortion Measure

Transcription

1 Multiterminal Source Coding with an Entropy-Based Distortion Measure Thomas Courtade and Rick Wesel Department of Electrical Engineering University of California, Los Angeles 4 August, 2011 IEEE International Symposium on Information Theory Saint Petersburg, Russia Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

2 Motivation The lossless one-helper problem X Decoder ˆX Y Question What is the achievable rate region for a lossless one-helper network with a single source? Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

3 Motivation The lossless one-helper problem X Decoder ˆX Y Question What is the achievable rate region for a lossless one-helper network with a single source? The answer to this question appears to be out of reach for now. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

4 Motivation The lossless one-helper problem X R x Decoder ˆX Y R y Theorem (Ahlswede-Körner-Wyner 1975) R = (R x, R y ) : R x H(X U) R y I(Y ; U) for some distribution p(x, y, u) = p(x, y)p(u y), where U Y + 2 Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

5 Motivation The lossless one-helper problem X R x Decoder ˆX R 0 Y R y Theorem (Kaspi-Berger 1982) R = (R 0, R x, R y ) : p(x, y, v, u) = p(x, y)p(u x)p(v y, u) such that R 0 I(X; U Y ), R x H(X V, U), R x + R y H(X) + I(Y ; V U, X) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

6 Motivation The lossless one-helper problem X R x Decoder ˆX R 0 Y R y R 1 Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

7 Motivation The lossless one-helper problem X R x Decoder ˆX R 0 Y R y R 1 Achievable rate region appears to be unknown. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

8 Motivation The lossless one-helper problem X R x Decoder ˆX R 0 Y R y R 1 Achievable rate region appears to be unknown. The encoder without a source is problematic. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

9 Motivation The lossless one-helper problem X R x Decoder ˆX R 0 Y R y R 1 Achievable rate region appears to be unknown. The encoder without a source is problematic. Intuitively, it should send some lossy estimate of X. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

10 Multiterminal Source Coding with an entropy-based distortion measure X R x Decoder Ẑ Y R y We study two cases of the MTSC problem: ] 1 Joint distortion constraint: E [d(x, Y, Ẑ) D, ] 2 Distortion constraint only on X: E [d(x, Ẑ) D. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

11 Multiterminal Source Coding with an entropy-based distortion measure X R x Decoder Ẑ Y R y We study two cases of the MTSC problem: ] 1 Joint distortion constraint: E [d(x, Y, Ẑ) D, ] 2 Distortion constraint only on X: E [d(x, Ẑ) D. We consider a particular choice of Ẑ and d( ) (Case 1): X Y Ẑ = R + (i.e., ( the set ) of functions from X Y to R + ). d(x, y, ẑ) = log. 1 ẑ(x,y) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

12 Multiterminal Source Coding with an entropy-based distortion measure X R x Decoder Ẑ Y R y We study two cases of the MTSC problem: ] 1 Joint distortion constraint: E [d(x, Y, Ẑ) D, ] 2 Distortion constraint only on X: E [d(x, Ẑ) D. We consider a particular choice of Ẑ and d( ) (Case 2): Ẑ = R X + (i.e., ( the ) set of functions from X to R + ). d(x, ẑ) = log. 1 ẑ(x) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

13 Examples An entropy-based distortion measure To make things interesting, we assume (x,y) X Y ẑ(x, y) M. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

14 Examples An entropy-based distortion measure To make things interesting, we assume (x,y) X Y ẑ(x, y) M. Without loss of generality: ẑ(x, y) = 1 d(x, y, ẑ) = D ( 1 {(x,y )=(x,y)} ẑ(x, y ) ). (x,y) X Y Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

15 Examples An entropy-based distortion measure To make things interesting, we assume (x,y) X Y ẑ(x, y) M. Without loss of generality: ẑ(x, y) = 1 d(x, y, ẑ) = D ( 1 {(x,y )=(x,y)} ẑ(x, y ) ). (x,y) X Y We obtain the erasure distortion measure if we restrict ẑ(x, y) to functions of the form: { 1 if (x, y) = (x ẑ(x, y) =, y ) 1 and ẑ(x, y) = x, y. 0 otherwise X Y Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

16 Examples An entropy-based distortion measure To make things interesting, we assume (x,y) X Y ẑ(x, y) M. Without loss of generality: ẑ(x, y) = 1 d(x, y, ẑ) = D ( 1 {(x,y )=(x,y)} ẑ(x, y ) ). (x,y) X Y We obtain the erasure distortion measure if we restrict ẑ(x, y) to functions of the form: { 1 if (x, y) = (x ẑ(x, y) =, y ) 1 and ẑ(x, y) = x, y. 0 otherwise X Y If (R x, R y ) = (0, 0), setting ẑ(x, y) = p(x, y) results in distortion H(X, Y ). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

17 Examples An entropy-based distortion measure To make things interesting, we assume (x,y) X Y ẑ(x, y) M. Without loss of generality: ẑ(x, y) = 1 d(x, y, ẑ) = D ( 1 {(x,y )=(x,y)} ẑ(x, y ) ). (x,y) X Y We obtain the erasure distortion measure if we restrict ẑ(x, y) to functions of the form: { 1 if (x, y) = (x ẑ(x, y) =, y ) 1 and ẑ(x, y) = x, y. 0 otherwise X Y If (R x, R y ) = (0, 0), setting ẑ(x, y) = p(x, y) results in distortion H(X, Y ). If (R x, R y ) = (0, H(Y )), setting ẑ(x, y) = p(x y) results in distortion H(X Y ). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

18 Examples Rate distortion with side information Theorem (Wyner-Ziv 1976) Let (X, Y ) be drawn i.i.d. and let d(x, ẑ) be given. The rate distortion function with side information is R Y (D) = min min p(w x) f I(X; W Y ) where the minimization is over all functions f : Y W Ẑ and conditional distributions p(w x) such that E [d(x, f(y, W ))] D. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

19 Examples Rate distortion with side information Theorem (Wyner-Ziv 1976) Let (X, Y ) be drawn i.i.d. and let d(x, ẑ) be given. The rate distortion function with side information is R Y (D) = min min p(w x) f I(X; W Y ) where the minimization is over all functions f : Y W Ẑ and conditional distributions p(w x) such that E [d(x, f(y, W ))] D. Corollary For our choice of Ẑ and d( ), the rate distortion function with side information is: R Y (D) = H(X Y ) D. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

20 Examples Rate distortion with side information Proof of Corollary: Therefore: D E [d(x, f(y, W ))] = E [ ( )] 1 log f(y, W )[X] = D(p(x y, w) f(y, w)[x]) + H(X Y, W ) H(X Y, W ). R Y (D) = min min p(w x) f Taking f(y, w)[x] = p(x y, w) and W = {H(X Y ) H(X Y, W )} H(X Y ) D. achieves equality throughout R Y (D) = H(X Y ) D. { X with probability 1 D H(X Y ) with probability D H(X Y ) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

21 Examples Rate distortion with side information X H(X Y ) D Decoder Ẑ Ed(X, Ẑ) D Y Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

22 Examples Rate distortion with side information X H(X Y ) D Decoder Ẑ Ed(X, Ẑ) D Y Intuition: every bit of distortion we tolerate saves one bit of rate. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

23 Examples Rate distortion with side information X H(X Y ) D Decoder Ẑ Ed(X, Ẑ) D Y Intuition: every bit of distortion we tolerate saves one bit of rate. What if Y is rate limited? Does a similar theme prevail? Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

24 d-lossy Coding of Correlated Sources Joint distortion criterion X R x Decoder Ẑ Ed(X, Y, Ẑ) D Y R y Theorem R = (R x, R y, D) : R x H(X Y ) D R y H(Y X) D R x + R y H(X, Y ) D Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

25 d-lossy Coding of Correlated Sources Joint distortion criterion We obtain a D-bit enlargement of the achievable rate region in all directions. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

26 d-lossy Coding of Correlated Sources Joint distortion criterion We obtain a D-bit enlargement of the achievable rate region in all directions. The proof relies on the WZ corollary and lemmas of the form: Lemma (Distortion Preimage) Define A(ẑ n ) = {(x n, y n ) : d(x n, y n, ẑ n ) D + ɛ}. Then A(ẑ n ) 2 n(d+2ɛ). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

27 d-lossy Coding of Correlated Sources Joint distortion criterion We obtain a D-bit enlargement of the achievable rate region in all directions. The proof relies on the WZ corollary and lemmas of the form: Lemma (Distortion Preimage) Define A(ẑ n ) = {(x n, y n ) : d(x n, y n, ẑ n ) D + ɛ}. Then A(ẑ n ) 2 n(d+2ɛ). Intuitively, if we have a reconstruction ẑ n then nd additional bits of information about (x n, y n ) are required to determine (x n, y n ) completely. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

28 d-lossy Coding of Correlated Sources Joint distortion criterion We obtain a D-bit enlargement of the achievable rate region in all directions. The proof relies on the WZ corollary and lemmas of the form: Lemma (Distortion Preimage) Define A(ẑ n ) = {(x n, y n ) : d(x n, y n, ẑ n ) D + ɛ}. Then A(ẑ n ) 2 n(d+2ɛ). Intuitively, if we have a reconstruction ẑ n then nd additional bits of information about (x n, y n ) are required to determine (x n, y n ) completely. This hints at a relationship with the Slepian-Wolf region. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

29 d-lossy Coding of Correlated Sources Joint distortion criterion: Proof sketch Claim 1 The basic idea is to show a correspondence with the Slepian-Wolf region as follows: If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Proved using the distortion preimage lemmas combined with a decomposition of the distortion measure and a random binning argument. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

30 d-lossy Coding of Correlated Sources Joint distortion criterion: Proof sketch Claim 1 The basic idea is to show a correspondence with the Slepian-Wolf region as follows: If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. ClaimProved 2 using the distortion preimage lemmas combined with a If (Rdecomposition x + θd, R y + (1 of the θ)d) distortion an achievable measure and Slepian-Wolf a random rate binning pair for someargument. θ [0, 1], then (R x, R y, D) is an achievable RD point. Proved via the WZ corollary and timesharing. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

31 d-lossy Coding of Correlated Sources Joint distortion criterion: Proof sketch Claim 1 The basic idea is to show a correspondence with the Slepian-Wolf region as follows: If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. ClaimProved 2 using the distortion preimage lemmas combined with a If (Rdecomposition x + θd, R y + (1 of the θ)d) distortion an achievable measure and Slepian-Wolf a random rate binning pair for someargument. θ [0, 1], then (R x, R y, D) is an achievable RD point. Proved via the WZ corollary and timesharing. Combining the two claims with the known form of the Slepian-Wolf region gives the expression for the achievable rate distortion region. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

32 Decomposing the Entropy Measure d( ) Joint distortion criterion: Proof sketch Given a rate distortion code, ẑ can be thought of as a probability mass function on X Y, we can decompose it uniquely as: ẑ(x, y) = ẑ(x)ẑ(y x). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

33 Decomposing the Entropy Measure d( ) Joint distortion criterion: Proof sketch Given a rate distortion code, ẑ can be thought of as a probability mass function on X Y, we can decompose it uniquely as: ẑ(x, y) = ẑ(x)ẑ(y x). This allows the definition [ of the marginal ] and conditional distortions D x and D y x : D E d(x n, Y n, Ẑn ) [ ] [ ] = E d x (X n, Ẑn ) + E d y x (X n, Y n, Ẑn ). }{{}}{{} D x D y x Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

34 Decomposing the Entropy Measure d( ) Joint distortion criterion: Proof sketch Given a rate distortion code, ẑ can be thought of as a probability mass function on X Y, we can decompose it uniquely as: ẑ(x, y) = ẑ(x)ẑ(y x). This allows the definition [ of the marginal ] and conditional distortions D x and D y x : D E d(x n, Y n, Ẑn ) [ ] [ ] = E d x (X n, Ẑn ) + E d y x (X n, Y n, Ẑn ). }{{}}{{} D x D y x This prompts the following lemma: Lemma (Marginal and Conditional Distortion Preimages) Define A x (ẑ n ) = {(x n ) : d x (x n, ẑ n ) D x + ɛ} and A y x (x n, ẑ n ) = { (y n ) : d y x (x n, y n, ẑ n ) D y x + ɛ }. Then A x (ẑ n ) 2 n(dx+2ɛ) and A y x (x n, ẑ n ) 2 n(d y x+2ɛ). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

35 Proving Claim 1 Joint distortion criterion: Proof sketch Claim 1 If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

36 Proving Claim 1 Joint distortion criterion: Proof sketch Claim 1 If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Suppose we have a sequence of (2 nrx, 2 nry, n) rate-distortion codes achieving average distortion D. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

37 Proving Claim 1 Joint distortion criterion: Proof sketch Claim 1 If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Suppose we have a sequence of (2 nrx, 2 nry, n) rate-distortion codes achieving average distortion [ D. ] With high probability, E d x (X n, Ẑn ) D x + ɛ and [ ] E d y x (X n, Y n, Ẑn ) D y x + ɛ. Also, D x + D y x D + ɛ. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

38 Proving Claim 1 Joint distortion criterion: Proof sketch Claim 1 If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Suppose we have a sequence of (2 nrx, 2 nry, n) rate-distortion codes achieving average distortion [ D. ] With high probability, E d x (X n, Ẑn ) D x + ɛ and [ ] E d y x (X n, Y n, Ẑn ) D y x + ɛ. Also, D x + D y x D + ɛ. Bin the x n s into 2 n(dx+3ɛ) bins and send bin index along with rate-distortion codeword. This requires rate R x + D x + 3ɛ and allows the decoder to recover X n w.h.p. by the MDPI lemma. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

39 Proving Claim 1 Joint distortion criterion: Proof sketch Claim 1 If (R x, R y, D) is an achievable RD point, then (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1]. Suppose we have a sequence of (2 nrx, 2 nry, n) rate-distortion codes achieving average distortion [ D. ] With high probability, E d x (X n, Ẑn ) D x + ɛ and [ ] E d y x (X n, Y n, Ẑn ) D y x + ɛ. Also, D x + D y x D + ɛ. Bin the x n s into 2 n(dx+3ɛ) bins and send bin index along with rate-distortion codeword. This requires rate R x + D x + 3ɛ and allows the decoder to recover X n w.h.p. by the MDPI lemma. Bin the y n s into 2 n(d y x+3ɛ) bins and send bin index along with rate-distortion codeword. This requires rate R y + D y x + 3ɛ and allows the decoder to recover Y n w.h.p. by the CDPI lemma. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

40 Proving Claim 2 Joint distortion criterion: Proof sketch Claim 2 If (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1], then (R x, R y, D) is an achievable RD point. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

41 Proving Claim 2 Joint distortion criterion: Proof sketch Claim 2 If (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1], then (R x, R y, D) is an achievable RD point. Suppose ( R x, R y ) is an achievable Slepian-Wolf rate pair. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

42 Proving Claim 2 Joint distortion criterion: Proof sketch Claim 2 If (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1], then (R x, R y, D) is an achievable RD point. Suppose ( R x, R y ) is an achievable Slepian-Wolf rate pair. Can assume ( R x, R y ) = (1 θ) (H(X), H(Y X)) + θ (H(X Y ), H(Y )). Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

43 Proving Claim 2 Joint distortion criterion: Proof sketch Claim 2 If (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1], then (R x, R y, D) is an achievable RD point. Suppose ( R x, R y ) is an achievable Slepian-Wolf rate pair. Can assume ( R x, R y ) = (1 θ) (H(X), H(Y X)) + θ (H(X Y ), H(Y )). By the WZ corollary, (R x, R y, D) = ( R x θd, R y (1 θ)d, D) is an achievable RD point. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

44 Proving Claim 2 Joint distortion criterion: Proof sketch Claim 2 If (R x + θd, R y + (1 θ)d) is an achievable Slepian-Wolf rate pair for some θ [0, 1], then (R x, R y, D) is an achievable RD point. Suppose ( R x, R y ) is an achievable Slepian-Wolf rate pair. Can assume ( R x, R y ) = (1 θ) (H(X), H(Y X)) + θ (H(X Y ), H(Y )). By the WZ corollary, (R x, R y, D) = ( R x θd, R y (1 θ)d, D) is an achievable RD point. Making the substitution ( R x, R y ) = (R x + θd, R y + (1 θ)d) completes the proof. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

45 d-lossy Coding with Coded Side Information Distortion constraint on X only X R x Decoder Ẑ Ed(X, Ẑ) D Y Theorem R y R = (R x, R y, D) : R x H(X U) D R y I(Y ; U) for some distribution p(x, y, u) = p(x, y)p(u y), where U Y + 2 Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

46 d-lossy Coding with Coded Side Information Distortion constraint on X only In this case, the achievable rate region is enlarged by D bits in the R x -direction only. The idea is that Y is already at D max, and thus any increase in D doesn t necessarily help decrease R y. Theorem R = (R x, R y, D) : R x H(X U) D R y I(Y ; U) for some distribution p(x, y, u) = p(x, y)p(u y), where U Y + 2. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

47 d-lossy Coding with Coded Side Information Distortion constraint on X only In this case, the achievable rate region is enlarged by D bits in the R x -direction only. The idea is that Y is already at D max, and thus any increase in D doesn t necessarily help decrease R y. The proof is similar in spirit to the previous case. In particular, we show a correspondence with the Ahlswede-Körner-Wyner region. Theorem R = (R x, R y, D) : R x H(X U) D R y I(Y ; U) for some distribution p(x, y, u) = p(x, y)p(u y), where U Y + 2. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

48 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

49 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Our insider friend is only able to provide rate-limited side information f(y n ) at rate R y. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

50 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Our insider friend is only able to provide rate-limited side information f(y n ) at rate R y. Starting with 1 ruble, in the i th race we bet a fraction ẑ i (x) of our wealth on horse x. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

51 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Our insider friend is only able to provide rate-limited side information f(y n ) at rate R y. Starting with 1 ruble, in the i th race we bet a fraction ẑ i (x) of our wealth on horse x. The expected doubling rate is: E 1 n log ẑ i (x i )o(x i ) = E log o(x) Ed(X n, n Ẑn ). i=1 Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

52 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Our insider friend is only able to provide rate-limited side information f(y n ) at rate R y. Starting with 1 ruble, in the i th race we bet a fraction ẑ i (x) of our wealth on horse x. The expected doubling rate is: E 1 n log ẑ i (x i )o(x i ) = E log o(x) Ed(X n, n Ẑn ). i=1 Theorem (Erkip 1996) The optimum doubling rate is: W = E log o(x) D, D = where inf {D : H(X U) D, I(Y ; U) R y}. p(x,y,u)=p(u y)p(x,y) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

53 An Application Gambling with Rate-Limited Side Information Suppose we bet on n i.i.d. horse races with outcomes X n and correlated side information Y n. Our insider friend is only able to provide rate-limited side information f(y n ) at rate R y, and an oracle provides side info g(x n ) at rate R x. Starting with 1 ruble, in the i th race we bet a fraction ẑ i (x) of our wealth on horse x. The expected doubling rate is: E 1 n log ẑ i (x i )o(x i ) = E log o(x) Ed(X n, n Ẑn ). i=1 Theorem (One oracle bit adds more than one bit to doubling rate) The optimum doubling rate is: W = E log o(x) D, where D = inf {D : H(X U) R x D, I(Y ; U) R y }. p(x,y,u)=p(u y)p(x,y) Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

54 Concluding Remarks Two lossy source coding problems which are open in general can be solved for our choice of d( ): 1 MTSC with a joint distortion constraint. 2 MTSC with a single distortion constraint: D x = D, D y = D max. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

55 Concluding Remarks Two lossy source coding problems which are open in general can be solved for our choice of d( ): 1 MTSC with a joint distortion constraint. 2 MTSC with a single distortion constraint: D x = D, D y = D max. It is possible that more cases can be solved. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

56 Concluding Remarks Two lossy source coding problems which are open in general can be solved for our choice of d( ): 1 MTSC with a joint distortion constraint. 2 MTSC with a single distortion constraint: D x = D, D y = D max. It is possible that more cases can be solved. The entropy-based distortion measure d( ) has shown up in the literature before (cf. image compression, the information bottleneck problem). This may point to some potential applications of the results presented today. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

57 Concluding Remarks Two lossy source coding problems which are open in general can be solved for our choice of d( ): 1 MTSC with a joint distortion constraint. 2 MTSC with a single distortion constraint: D x = D, D y = D max. It is possible that more cases can be solved. The entropy-based distortion measure d( ) has shown up in the literature before (cf. image compression, the information bottleneck problem). This may point to some potential applications of the results presented today. Due to the relationship between d( ) and relative entropy, there may be some additional applications in estimation/portfolio/gambling theory. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

58 Concluding Remarks Two lossy source coding problems which are open in general can be solved for our choice of d( ): 1 MTSC with a joint distortion constraint. 2 MTSC with a single distortion constraint: D x = D, D y = D max. It is possible that more cases can be solved. The entropy-based distortion measure d( ) has shown up in the literature before (cf. image compression, the information bottleneck problem). This may point to some potential applications of the results presented today. Due to the relationship between d( ) and relative entropy, there may be some additional applications in estimation/portfolio/gambling theory. Algorithmically satisfying from an engineering perspective. Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23

59 Thank You! Courtade and Wesel (UCLA) MTSC: Entropy-Based Distortion ISIT / 23