SHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe

Transcription

1 SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe

2 2/41 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel capacity Lossy source coding: rate distortion function Binary hypothesis testing: Stein s lemma Interactive communication and common randomness Two-terminal model: Mutual information Multiterminal model: Shared information Applications

3 3/41 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel capacity Lossy source coding: rate distortion function Binary hypothesis testing: Stein s lemma Interactive communication and common randomness Applications

4 4/41 Mutual Information Mutual information is a measure of mutual dependence between two rvs.

5 4/41 Mutual Information Mutual information is a measure of mutual dependence between two rvs. Let X 1 and X 2 be R-valued rvs with joint probability distribution P X1X 2. The mutual information between X 1 and X 2 is { [ ] EPX1X2 log dp X 1 X 2 dp I X 1 X 2 = X1 P X2 X 1, X 2, if P X1X 2 P X1 P X2, if P X1X 2 P X1 P X2 = D P X1X 2 P X1 P X2. Kullback Leibler divergence When X 1 and X 2 are finite-valued, I X 1 X 2 = H X 1 + H X 2 H X 1, X 2 = H X 1 H X 1 X 2 = H X 2 H X 2 X 1 [ ] = H X 1, X 2 H X 1 X 2 + H X 2 X 1.

6 5/41 Channel Coding Let X 1 and X 2 be finite alphabets, and W : X 1 X 2 be a stochastic matrix. message m {1,..., M} fm encoder f DMC decoder φ x 11,..., x 1n x 21,..., x 2n m Discrete memoryless channel DMC: n W n x 21,..., x 2n x 11,..., x 1n = W x 2i x 1i. i=1

7 6/41 Channel Capacity message m {1,..., M} fm encoder f DMC decoder φ x 11,..., x 1n x 21,..., x 2n m Goal: Make code rate 1 n log M as large as possible while keeping max P φ X 21,..., X 2n m fm m to be small, in the asymptotic sense as n. [C.E. Shannon, 1948] Channel capacity C = max I X 1 X 2. P X1 :P X2 X 1 =W

8 7/41 Lossy Source Coding Let {X 1t } t=1 be an X 1-valued i.i.d. source. source fx 11,..., x 1n φj encoder f decoder φ x 11,..., x 1n j {1,..., J} x 21,..., x 2n Distortion measure: d x 11,..., x 1n, x 21,..., x 2n = 1 n d x 1i, x 2i. n i=1

9 8/41 Rate Distortion Function source fx 11,..., x 1n φj encoder f decoder φ x 11,..., x 1n j {1,..., J} x 21,..., x 2n Goal: Make compression code rate 1 n log J as small as possible while keeping 1 P n n i=1 d X 1i, X 2i to be large, in the asymptotic sense as n. [Shannon, 1948, 1959] Rate distortion function R = min I X 1 X 2. P X2 X 1 : E[dX 1,X 2]

10 9/41 Simple Binary Hypothesis Testing Let {X 1t, X 2t } t=1 be an X 1 X 2 -valued i.i.d. process generated according to H 0 : P X1X 2 or H 1 : P X1 P X2. Test: Decides H 0 w.p. T 0 x 11,..., x 1n, x 21,..., x 2n, H 1 w.p. T 1 x 11,..., x 1n, x 21,..., x 2n = 1 T Stein s lemma [H. Chernoff, 1956]: For every 0 < ɛ < 1, lim n 1 n log inf T : P H0 T says H 0 1 ɛ P H 1 T says H 0 = D P X1X 2 P X1 P X2 = I X 1 X 2.

11 10/41 Outline Two-terminal model: Mutual information Interactive communication and common randomness Two-terminal model: Mutual information Multiterminal model: Shared information Applications

12 11/41 Multiterminal Model COMMUNICATION NETWORK F X 1 X 2 X m Set of terminals = M = {1,..., m}. X 1,..., X m are finite-valued rvs with known joint distribution P X1...X m on X 1 X m. Terminal i M observes data X i. Multiple rounds of interactive communication on a noiseless channel of unlimited capacity; all terminals hear all communication.

13 12/41 Interactive communication Interactive Communication Assume: Communication occurs in consecutive time slots in r rounds. The corresponding rvs representing the communication are F = F X 1,..., X m = F 11,..., F 1m, F 21,..., F 2m,..., F r1,..., F rm F 11 = f 11 X1, F12 = f 12 X2, F 11,... F ji = f ji Xi; all previous communication. Simple communication: F = F 1,..., F m, Fi = f i Xi, 1 i m. A. Yao, Some complexity questions related to distributive computing, Proc. Annual Symposium on Theory of Computing, 1979.

14 13/41 Applications COMMUNICATION NETWORK F X 1 X 2 X m Data exchange: Omniscience Signal recovery: Data compression Function computation Cryptography: Secret key generation

15 14/41 WatanExample: Function Computation X 1 = X 11 X 12 F 1 F 2 X 2 = X 21 X 22 [S. Watanabe] X 11, X 12, X 21, X 22 are mutually independent 0.5, 0.5 bits. Terminals 1 and 2 wish to compute: G = gx 1, X 2 = 1 X 11, X 12 = X 21, X 22. Simple communication: F = Communication complexity: HF = 4 bits. F 1 = X 11, X 12, F 2 = X 21, X 22. No privacy: Terminal 1 or 2, or an observer of F, learns all the data X 1, X 2.

16 15/41 WatanExample: Function Computation X 1 = X 11 X 12 F 11 F 12 X 2 = X 21 X 22 An interactive communication protocol: F = F 11 = X 11, X 12, F 12 = G. Complexity: HF = 2.81 bits. Some privacy: Terminal 2, or an observer of F, learns X 1; Terminal 1, or an observer of F, either learns X 2 w.p or w.p that X 2 differs from X 1.

17 15/41 WatanExample: Function Computation X 1 = X 11 X 12 F 11 F 12 X 2 = X 21 X 22 An interactive communication protocol: F = F 11 = X 11, X 12, F 12 = G. Complexity: HF = 2.81 bits. Some privacy: Terminal 2, or an observer of F, learns X 1; Terminal 1, or an observer of F, either learns X 2 w.p or w.p that X 2 differs from X 1. Can a communication complexity of 2.81 bits be bettered?

18 16/41 Related Work Exact function computation Yao 79: Communication complexity. Gallager 88: Algorithm for parity computation in a network. Giridhar-Kumar 05: Algorithms for computing functions over sensor networks. Freris-Kowshik-Kumar 10: Survey: Connectivity, capacity, clocks, computation in large sensor networks. Orlitsky-El Gamal 84: Communication complexity with secrecy. Information theoretic function computation Körner-Marton 79: Minimum rate for computing parity. Orlitsky-Roche 01: Two terminal function computation. Nazer-Gastpar 07: Computation over noisy channels. Ma-Ishwar 08: Distributed source coding for interactive computing. Ma-Ishwar-Gupta 09: Multiround function computation in colocated networks. Tyagi-Gupta-Narayan 11: Secure function computation. Tyagi-Watanabe 13, 14 Secrecy generation, secure computing. Compressing interactive communication Schulman 92: Coding for interactive communication. Braverman-Rao 10: Information complexity of communication. Kol-Raz 13, Heupler 14: Interactive communication over noisy channels.

19 17/41 Mathematical Economics: Mechanism Design Thomas Marschak and Stefan Reichelstein, Communication requirements for individual agents in networks and hierarchies, in The Economics of Informational Decentralization: Complexity, Efficiency and Stability: Essays in Honor of Stanley Reiter, John O. Ledyard, Ed., Springer, Kenneth R. Mount and Stanley Reiter, Computation and Complexity in Economic Behavior and Organization, Cambridge U. Press, Courtesy: Demos Teneketzis

20 18/41 Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L For 0 ɛ < 1, given interactive communication F, a rv L = LX 1,..., X m is ɛ-cr for the terminals in M using F, if there exist local estimates L i = L i Xi, F, i M, of L satisfying P L i = L, i M 1 ɛ.

21 19/41 Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L Examples: Data exchange: Omniscience: L = X 1,..., X m. Signal recovery: Data compression: L X i, for some fixed i M. Function computation: L g X 1,..., X m for a given g. Cryptography: Secret CR, i.e., secret key: L with IL F = 0.

22 20/41 A Basic Operational Question COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L What is the maximal CR, as measured by H L F, that can be generated by a given interactive communication F for a distributed processing task?

23 20/41 A Basic Operational Question COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L What is the maximal CR, as measured by H L F, that can be generated by a given interactive communication F for a distributed processing task? Answer in two steps: Fundamental structural property of interactive communication Upper bound on amount of CR achievable with interactive communication. Shall start with the case of m = 2 terminals.

24 21/41 Fundamental Property of Interactive Communication COMMUNICATION NETWORK F X 1 X 2 Lemma: [U. Maurer], [R. Ahlswede - I. Csiszár] For interactive communication F of the Terminals 1 and 2 observing data X 1 and X 2, respectively, I X 1 X 2 F I X1 X 2. In particular, independent rvs X 1, X 2 remain so upon conditioning on an interactive communication.

25 22/41 Fundamental Property of Interactive Communication Lemma: [U. Maurer], [R. Ahlswede - I. Csiszár] For interactive communication F of the Terminals 1 and 2 observing data X 1 and X 2, respectively, I X 1 X 2 F I X1 X 2. In particular, independent rvs X 1, X 2 remain so upon conditioning on an interactive communication. Proof: For interactive communication F = F 11, F 12,..., F r1, F r2, followed by iteration. I X 1 X 2 = I X1, F 11 X 2 I X 1 X 2 F 11 = I X 1 X 2, F 12 F 11 I X 1 X 2 F 11, F 12,

26 23/41 An Equivalent Form For interactive communication F of Terminals 1 and 2: I X 1 X 2 F I X1 X 2 HF HF X 1 + HF X 2.

27 Upper Bound on CR for Two Terminals COMMUNICATION NETWORK F X 1 X 2 L1 L 2 = L Using L is ɛ-cr for Terminals 1 and 2 with interactive communication F; and HF HF X 1 + HF X 2, we get H L F H X 1, X 2 [H X 1 X 2 + H X2 X 1 ] + 2νɛ, where lim ɛ 0 νɛ = 0. 24/41

28 25/41 Maximum CR for Two Terminals: Mutual Information COMMUNICATION NETWORK F X 1 X 2 L1 L 2 = L Lemma: [I. Csiszár - P. Narayan] Let L be any ɛ-cr for Terminals 1 and 2 observing data X 1 and X 2, respectively, achievable with interactive F. Then H L F I X 1 X 2 = D PX1X 2 P X1 P X2. Remark: When {X 1t, X 2t } t=1 is an X 1 X 2 -valued i.i.d. process, the upper bound is attained.

29 26/41 Interactive Communication for m 2 Terminals Theorem 1: [I. Csiszár-P. Narayan] For interactive communication F of the terminals i M = {1,..., m}, with Terminal i oberving data X i, H F B B λ B H F X B c for every family B = {B M, B } and set of weights fractional partition { } λ 0 λ B 1, B B, satisfying λ B = 1 i M. B B:B i Equality holds if X 1,..., X m are mutually independent. Special case of: M. Madiman and P. Tetali, Information inequalities for joint distributions, with interpretations and applications, IEEE Trans. Inform. Theory, June 2010.

30 27/41 CR for m 2 Terminals: A Suggestive Analogy [S. Nitinawarat-P. Narayan] For interactive communication F of the terminals i M = {1,..., m}, with Terminal i observing data X i, m = 2 : HF HF X 1 + HF X 2 I X 1 X 2 F I X1 X 2 H F B B λ B H F X B c H X 1,..., X m F λ B H X B X B c, F B B H X 1,..., X m λ B H X B X B c. B B

31 28/41 [S. Nitinawarat-P. Narayan] An Analogy For interactive communication F of the terminals i M = {1,..., m}, with Terminal i observing data X i, H F B B λ B H F X B c H X 1,..., X m F λ B H X B X B c, F B B H X 1,..., X m λ B H X B X B c. B B Does the RHS suggest a measure of mutual dependence among the rvs X 1,..., X m?

32 29/41 Maximum CR for m 2 Terminals: Shared Information Theorem 2: [I. Csiszár-P. Narayan] Given 0 ɛ < 1, for an ɛ-cr L for M achieved with interactive communication F, H L F H X 1,..., X m λ B H X B X B c + mν B B for every fractional partition λ of M, with ν = νɛ = ɛ log L + hɛ. Remarks: The proof of Theorem 2 relies on Theorem 1. When {X 1t,..., X mt } t=1 is an i.i.d. process, the upper bound is attained.

33 30/41 Shared Information Theorem 2: [I. Csiszár-P. Narayan] H L F H X 1,..., X m max λ = SI X 1,..., X m λ B H X B X B c B B

34 31/41 Extensions Theorems 1 and 2 extend to: random variables with densities [S. Nitinawarat-P. Narayan] a larger class of probability measures [H.Tyagi-P. Narayan].

35 32/41 Shared Information and Kullback-Leibler Divergence [I. Csiszár-P. Narayan, C. Chan-L. Zheng] SI X 1,..., X m = H X1,..., X m max λ λ B H X B X B c B B m = 2 = H X 1, X 2 [ H X 1 X 2 + H X2 X 1 ] = I X 1 X 2 m = 2 = D P X1X 2 P X1 P X2

36 32/41 Shared Information and Kullback-Leibler Divergence [I. Csiszár-P. Narayan, C. Chan-L. Zheng] SI X 1,..., X m = H X1,..., X m max λ λ B H X B X B c B B m = 2 = H X 1, X 2 [ H X 1 X 2 + H X2 X 1 ] = I X 1 X 2 m = 2 = D P X1X 2 P X1 P X2 m 2 = min 2 k m min A k =A 1,...,A k 1 k 1 D k P X1...X m P XAi i=1 and equals 0 iff P X1...X m = P XA P XA c for some A M. Does shared information have an operational significance as a measure of the mutual dependence among the rvs X 1,..., X m?

37 33/41 Outline Two-terminal model: Mutual information Interactive communication and common randomness Applications

38 34/41 Omniscience COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L [I. Csiszár-P. Narayan] For L = X 1,..., X m, Theorem 2 gives which, for m = 2, is H F H X 1,..., X m SI X 1,..., X m, H F H X 1 X 2 + H X2 X 1. [Slepian Wolf]

39 35/41 Signal Recovery: Data Compression COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L [S. Nitinawarat-P. Narayan] With L = X 1, by Theorem 2 H F H X 1 SI X 1,..., X m, which, for m = 2, gives H F H X 1 X 2. [Slepian-Wolf]

40 36/41 Secret Common Randomness COMMUNICATION NETWORK F X 1 X 2 X m L 1 L 2 L m = L Terminals 1,..., m generate CR L satisfying the secrecy condition I L F = 0. By Theorem 2, H L = H L F SI X1,..., X m. Secret key generation [I. Csiszár-P. Narayan] Secure function computation [H. Tyagi-P. Narayan]

41 37/41 Querying Common Randomness COMMUNICATION NETWORK F X n 1 X n 2 X n m L 1 L 2 L m = L [H. Tyagi-P. Narayan] A querier observes communication F and seeks to resolve the value of CR L by asking questions: Is L = l? with yes-no answers. The terminals in M seek to generate L using F so as to make the querier s burden as onerous as possible. What is the largest query exponent?

42 38/41 Largest Query Exponent COMMUNICATION NETWORK F X n 1 X n 2 X n m L 1 L 2 L m = L E arg sup E [ inf q P q L F 2 ne ] 1 as n E = SI X 1,..., X m

43 39/41 Shared information and a Hypothesis Testing Problem SI X 1,..., X m = min 2 k m min A k =A 1,...,A k 1 k 1 D k P X1...X m P XAi i=1 Related to exponent of P e -second kind for an appropriate binary composite hypothesis testing problem, involving restricted CR L and communication F. H. Tyagi and S. Watanabe, Converses for secret key agreement and secure computing, IEEE Trans. Information Theory, September 2015.

44 40/41 In Closing... How useful is the concept of shared information? A: Operational meaning in specific cases of distributed processing...

45 40/41 In Closing... How useful is the concept of shared information? A: Operational meaning in specific cases of distributed processing... For instance Consider n i.i.d. repetitions say, in time of the rvs X 1,..., X m. Data at time instant t is X 1t,..., X mt, t = 1,..., n. Terminal i observes the i.i.d. data X i1,..., X in, i M. Shared information-based results are asymptotically tight in n: Minimum rate of communication for omniscience Maximum rate of a secret key Largest query exponent Necessary condition for secure function computation Several problems in information theoretic cryptography.

46 41/41 Shared Information: Many Open Questions... Significance in network source and channel coding? Interactive communication over noisy channels? Data-clustering applications? [C. Chan-A. Al-Bashabsheh-Q. Zhou-T. Kaced-T.Liu, 2016]..