SHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe

Transcription

1 SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe

2 2/40 Acknowledgement Praneeth Boda Himanshu Tyagi Shun Watanabe

3 3/40 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel capacity Lossy source coding: rate distortion function Binary hypothesis testing: Stein s lemma Interactive communication and common randomness Two-terminal model: Mutual information Multiterminal model: Shared information Applications

4 4/40 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel capacity Lossy source coding: rate distortion function Binary hypothesis testing: Stein s lemma Interactive communication and common randomness Applications

5 5/40 Mutual Information Mutual information is a measure of mutual dependence between two rvs.

6 5/40 Mutual Information Mutual information is a measure of mutual dependence between two rvs. Let X 1 and X 2 be R-valued rvs with joint probability distribution P X1X 2. The mutual information between X 1 and X 2 is { [ ] EPX1X2 log dp X 1 X 2 dp I X 1 X 2 = X1 P X2 X 1, X 2, if P X1X 2 P X1 P X2, if P X1X 2 P X1 P X2 = D P X1X 2 P X1 P X2. Kullback Leibler divergence When X 1 and X 2 are finite-valued, I X 1 X 2 = H X 1 + H X 2 H X 1, X 2 = H X 1 H X 1 X 2 = H X 2 H X 2 X 1 [ ] = H X 1, X 2 H X 1 X 2 + H X 2 X 1.

7 6/40 Channel Coding Let X 1 and X 2 be finite alphabets, and W : X 1 X 2 be a stochastic matrix. message m {1,..., M} fm encoder f DMC decoder φ x 11,..., x 1n x 21,..., x 2n m Discrete memoryless channel DMC: n W n x 21,..., x 2n x 11,..., x 1n = W x 2i x 1i. i=1

8 7/40 Channel Capacity message m {1,..., M} fm encoder f DMC decoder φ x 11,..., x 1n x 21,..., x 2n m Goal: Make code rate 1 n log M as large as possible while keeping max P φ X 21,..., X 2n m fm m to be small, in the asymptotic sense as n. [C.E. Shannon, 1948] Channel capacity C = max I X 1 X 2. P X1 :P X2 X 1 =W

9 8/40 Lossy Source Coding Let {X 1t } t=1 be an X 1-valued i.i.d. source. source fx 11,..., x 1n φj encoder f decoder φ x 11,..., x 1n j {1,..., J} x 21,..., x 2n Distortion measure: d x 11,..., x 1n, x 21,..., x 2n = 1 n d x 1i, x 2i. n i=1

10 9/40 Rate Distortion Function source fx 11,..., x 1n φj encoder f decoder φ x 11,..., x 1n j {1,..., J} x 21,..., x 2n Goal: Make compression code rate 1 n log J as small as possible while keeping 1 P n n i=1 d X 1i, X 2i to be large, in the asymptotic sense as n. [Shannon, 1948, 1959] Rate distortion function R = min I X 1 X 2. P X2 X 1 : E[dX 1,X 2]

11 10/40 Simple Binary Hypothesis Testing Let {X 1t, X 2t } t=1 be an X 1 X 2 -valued i.i.d. process generated according to H 0 : P X1X 2 or H 1 : P X1 P X2. Test: Decides H 0 w.p. T 0 x 11,..., x 1n, x 21,..., x 2n, H 1 w.p. T 1 x 11,..., x 1n, x 21,..., x 2n = 1 T Stein s lemma [H. Chernoff, 1956]: For every 0 < ɛ < 1, lim n 1 n log inf T : P H0 T says H 0 1 ɛ P H 1 T says H 0 = D P X1X 2 P X1 P X2 = I X 1 X 2.

12 11/40 Outline Two-terminal model: Mutual information Interactive communication and common randomness Two-terminal model: Mutual information Multiterminal model: Shared information Applications

13 12/40 Multiterminal Model COMMUNICATION NETWORK F X 1 X 2 X m Set of terminals = M = {1,..., m}. X 1,..., X m are finite-valued rvs with known joint distribution P X1...X m on X 1 X m. Terminal i M observes data X i. Multiple rounds of interactive communication on a noiseless channel of unlimited capacity; all terminals hear all communication.

14 13/40 Interactive Communication Interactive communication Assume: Communication occurs in consecutive time slots in r rounds. Communication is described in terms of the mappings f 11,..., f 1m, f 21,..., f 2m,..., f r1,..., f rm f ji : message in round j from terminal i, 1 j r, 1 i m f ji is any function of X i and of all previous communication.

15 13/40 Interactive Communication Interactive communication Assume: Communication occurs in consecutive time slots in r rounds. Communication is described in terms of the mappings f 11,..., f 1m, f 21,..., f 2m,..., f r1,..., f rm f ji : message in round j from terminal i, 1 j r, 1 i m f ji is any function of X i and of all previous communication. The corresponding rvs representing the communication are F = F X 1,..., X m = F 11,..., F 1m, F 21,..., F 2m,..., F r1,..., F rm F 11 = f 11 X1, F12 = f 12 X2, F 11,... F ji = f ji Xi; all previous communication. Simple communication: F = F 1,..., F m, Fi = f i Xi, 1 i m.

16 14/40 Applications COMMUNICATION NETWORK F X 1 X 2 X m Data exchange: Omniscience. Signal recovery: Data compression. Function computation. Cryptography: Secret key generation. Applications in control?

17 15/40 Example: Function Computation X 1 = X 11 X 12 F 1 F 2 X 2 = X 21 X 22 [S. Watanabe] X 11, X 12, X 21, X 22 are mutually independent 0.5, 0.5 bits. Terminals 1 and 2 wish to compute: G = gx 1, X 2 = 1 X 11, X 12 = X 21, X 22. Simple communication: F = Communication complexity: HF = 4 bits. F 1 = X 11, X 12, F 2 = X 21, X 22. No privacy: Terminal 1 or 2, or an observer of F, learns all the data X 1, X 2.

18 16/40 Example: Function Computation X 1 = X 11 X 12 F 11 F 12 F 21 F 22 X 2 = X 21 X 22 Interactive communication 1: F 11 = X 11 X 12, F 12 = X 21 X 22. If F 11 F 12, protocol over. If F 11 = F 12, then F 21 = X 11, F 22 = X 21. Complexity: HF = 3 bits. Some privacy: W.p. 0.5 both terminals, or an observer of F, learn that X 1 X 2; and w.p. 0.5 everyone learns X 1, X 2.

19 17/40 Example: Function Computation X 1 = X 11 X 12 F 11 F 12 X 2 = X 21 X 22 Interactive communication 2: F = F 11 = X 11, X 12, F 12 = G. Complexity: HF = 2.81 bits. Some privacy: Terminal 2, or an observer of F, learns X 1; terminal 1, or an observer of F, either learns X 2 w.p or w.p that X 2 differs from X 1.

20 18/40 Related Work Exact function computation Yao 79: Communication complexity. Gallager 88: Algorithm for parity computation in a network. Giridhar-Kumar 05: Algorithms for computing functions over sensor networks. Freris-Kowshik-Kumar 10: Survey: Connectivity, capacity, clocks, computation in large sensor networks. Orlitsky-El Gamal 84: Communication complexity with secrecy. Information theoretic function computation Körner-Marton 79: Minimum rate for computing parity. Orlitsky-Roche 01: Two terminal function computation. Nazer-Gastpar 07: Computation over noisy channels. Ma-Ishwar 08: Distributed source coding for interactive computing. Ma-Ishwar-Gupta 09: Multiround function computation in colocated networks. Tyagi-Gupta-Narayan 11: Secure function computation. Tyagi-Watanabe 13, 14 Secrecy generation, secure computing. Compressing interactive communication Schulman 92: Coding for interactive communication. Braverman-Rao 10: Information complexity of communication. Kol-Raz 13, Heupler 14: Interactive communication over noisy channels.

21 19/40 Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L For 0 ɛ < 1, given interactive communication F, an rv L = LX 1,..., X m is ɛ-cr for the terminals in M using F, if there exist local estimates L i = L i Xi, F, i M, of L satisfying P L i = L, i M 1 ɛ.

22 20/40 Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L Examples: Omniscience: L = X 1,..., X m. Single signal: L = X i, for some fixed i M. Function computation: L = g X 1,..., X m for a given g. Secret CR, i.e., secret key: L with IL F = 0.

23 21/40 A Basic Question COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L What is the maximal CR, as measured by H L F, that can be generated by a given interactive communication F for a distributed processing task?

24 21/40 A Basic Question COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L What is the maximal CR, as measured by H L F, that can be generated by a given interactive communication F for a distributed processing task? Answer in two steps: Fundamental property of interactive communication Upper bound on amount of CR achievable with interactive communication. Shall start with the case of m = 2 terminals.

25 22/40 Fundamental Property of Interactive Communication COMMUNICATION NETWORK F X 1 X 2 Lemma: [U. Maurer], [R. Ahlswede - I. Csiszár] For interactive communication F of the terminals i M = {1, 2}, with terminal i possessing initial data X i, I X 1 X 2 F I X1 X 2. In particular, independent rvs X 1, X 2 remain so upon conditioning on an interactive communication.

26 23/40 Fundamental Property of Interactive Communication Lemma: [U. Maurer], [R. Ahlswede - I. Csiszár] For interactive communication F of the terminals i M = {1, 2}, with terminal i possessing initial data X i, I X 1 X 2 F I X1 X 2. In particular, independent rvs X 1, X 2 remain so upon conditioning on an interactive communication. Proof: For interactive communication F = F 11, F 12,..., F r1, F r2, followed by iteration. I X 1 X 2 = I X1, F 11 X 2 I X 1 X 2 F 11 = I X 1 X 2, F 12 F 11 I X 1 X 2 F 11, F 12,

27 24/40 An Equivalent Form For interactive communication F of terminals 1 and 2: I X 1 X 2 F I X1 X 2 HF HF X 1 + HF X 2.

28 Upper Bound on CR for Two Terminals COMMUNICATION NETWORK F X 1 X 2 L1 L 2 = L Using L is ɛ-cr for M = {1, 2} with interactive F; and HF HF X 1 + HF X 2, we get H L F H X 1, X 2 [H X 1 X 2 + H X2 X 1 ] + 2νɛ, where lim ɛ 0 νɛ = 0. 25/40

29 26/40 Upper Bound on CR for Two Terminals COMMUNICATION NETWORK F X 1 X 2 L1 L 2 = L Lemma: [I. Csiszár - P. Narayan] Let L be any ɛ-cr for the terminals i M = {1, 2} with terminal i possessing initial data X i, achievable with interactive communication F. Then H L F I X 1 X 2 + 2ν, lim νɛ = 0. ɛ 0 Remark: When {X 1t, X 2t } t=1 is an X 1 X 2 -valued i.i.d. process, the upper bound is attained.

30 27/40 Interactive Communication for m 2 Terminals Theorem 1: [I. Csiszár-P. Narayan] For interactive communication F of the terminals i M = {1,..., m}, with terminal i possessing initial data X i, H F B B λ B H F X B c for every family B = {B M, B } and set of weights fractional partition { } λ 0 λ B 1, B B, satisfying λ B = 1 i M. B B:B i Equality holds if X 1,..., X m are mutually independent. Special case of: M. Madiman and P. Tetali, Information inequalities for joint distributions, with interpretations and applications, IEEE Trans. Inform. Theory, June 2010.

31 28/40 CR for m 2 Terminals: A Suggestive Analogy [S. Nitinawarat-P. Narayan] For interactive communication F of the terminals i M = {1,..., m}, with terminal i possessing initial data X i, m = 2 : HF HF X 1 + HF X 2 I X 1 X 2 F I X1 X 2 H F B B λ B H F X B c H X 1,..., X m F λ B H X B X B c, F B B H X 1,..., X m λ B H X B X B c. B B

32 29/40 [S. Nitinawarat-P. Narayan] An Analogy For interactive communication F of the terminals i M = {1,..., m}, with terminal i possessing initial data X i, H F B B λ B H F X B c H X 1,..., X m F λ B H X B X B c, F B B H X 1,..., X m λ B H X B X B c. B B? Does the RHS suggest a measure of mutual dependence among the rvs X 1,..., X m?

33 30/40 CR for m 2 Terminals Theorem 2: [I. Csiszár-P. Narayan] Given 0 ɛ < 1, for an ɛ-cr L for M achieved with interactive communication F, H L F H X 1,..., X m λ B H X B X B c + mν B B for every fractional partition λ of M, with ν = νɛ = ɛ log L + hɛ. Remarks: The proof of Theorem 2 relies on Theorem 1. When {X 1t,..., X mt } t=1 is an i.i.d. process, the upper bound is attained.

34 31/40 Shared Information Theorem 2: [I. Csiszár-P. Narayan] H L F H X 1,..., X m max λ λ B H X B X B c + mν B B = SI X 1,..., X m + mν

35 32/40 Extensions Theorems 1 and 2 extend to: random variables with densities [S. Nitinawarat-P. Narayan] a larger class of probability measures [H.Tyagi-P. Narayan].

36 33/40 Shared Information and Kullback-Leibler Divergence [I. Csiszár-P. Narayan, C. Chan-L. Zheng] SI X 1,..., X m = H X1,..., X m max λ λ B H X B X B c B B m = 2 = H X 1, X 2 [ H X 1 X 2 + H X2 X 1 ] = I X 1 X 2 m = 2 = D P X1X 2 P X1 P X2

37 33/40 Shared Information and Kullback-Leibler Divergence [I. Csiszár-P. Narayan, C. Chan-L. Zheng] SI X 1,..., X m = H X1,..., X m max λ λ B H X B X B c B B m = 2 = H X 1, X 2 [ H X 1 X 2 + H X2 X 1 ] = I X 1 X 2 m = 2 = D P X1X 2 P X1 P X2 m 2 = min 2 k m min A k =A 1,...,A k 1 k 1 D k P X1...X m P XAi i=1 and equals 0 iff P X1...X m = P XA P XA c for some A M. Does shared information have an operational significance as a measure of the mutual dependence among the rvs X 1,..., X m?

38 34/40 Outline Two-terminal model: Mutual information Interactive communication and common randomness Applications

39 35/40 Omniscience COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L [I. Csiszár-P. Narayan] For L = X 1,..., X m, Theorem 2 gives H F H X 1,..., X m SI X 1,..., X m mν, which, for m = 2, is H F H X 1 X 2 + H X2 X 1 2ν. [Slepian Wolf]

40 36/40 Recovery of a Single Signal COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L [S. Nitinawarat-P. Narayan] With L = X 1, by Theorem 2 H F H X 1 SI X 1,..., X m mν, which, for m = 2, gives H F I X 1 F H X 1 X 2 2ν. [Slepian-Wolf]

41 37/40 Secret Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L Terminals 1,..., m generate CR L satisfying the secrecy condition I L F = 0. By Theorem 2, H L = H L F SI X1,..., X m + mν. Secret key generation [I. Csiszár-P. Narayan] Secure function computation [H. Tyagi-P. Narayan]

42 38/40 Shared information and a Hypothesis Testing Problem SI X 1,..., X m = min 2 k m min A k =A 1,...,A k 1 k 1 D k P X1...X m P XAi i=1 Related to exponent of P e -second kind for an appropriate binary composite hypothesis testing problem, involving restricted CR L and communication F. H. Tyagi and S. Watanabe, Converses for secret key agreement and secure computing, IEEE Trans. Information Theory, September 2015.

43 39/40 In Closing... How useful is the concept of shared information? A: Operational meaning in specific cases of distributed processing...

44 39/40 In Closing... How useful is the concept of shared information? A: Operational meaning in specific cases of distributed processing... For instance Consider n i.i.d. repetitions say, in time of the rvs X 1,..., X m. Data at time instant t is X 1t,..., X mt, t = 1,..., n. Terminal i observes the i.i.d. data X i1,..., X in, i M. Shared information-based results are asymptotically tight in n: Minimum rate of communication for omniscience. Maximum rate of a secret key. Necessary condition for secure function computation. Several problems in information theoretic cryptography.

45 40/40 Shared Information: Many Many Open Questions... Significance in network source and channel coding? Interactive communication over noisy channels? Communication links described by an undirected graph? Continuous-time models?..