Asymptotic Coupling and Its Applications in Information Theory

Transcription

1 Asymptotic Couplig ad Its Applicatios i Iformatio Theory Vicet Y. F. Ta Joit Work with Lei Yu Departmet of Electrical ad Computer Egieerig, Departmet of Mathematics, Natioal Uiversity of Sigapore IMS-APRM 2018 Vicet Y. F. Ta (NUS) Asymptotic Couplig 1 / 26

2 Outlie 1 Problem Formulatio 2 Mai Results 3 Applicatios i Iformatio Theory 4 Coclusio ad Future Work Vicet Y. F. Ta (NUS) Asymptotic Couplig 2 / 26

3 Geeral Couplig Problem A joit distributio Q XY P (X Y) such that Q X = P X, Q Y = P Y is called a couplig of P X, P Y. Vicet Y. F. Ta (NUS) Asymptotic Couplig 3 / 26

4 Geeral Couplig Problem A joit distributio Q XY P (X Y) such that Q X = P X, Q Y = P Y is called a couplig of P X, P Y. The set of coupligs of P X, P Y is defied as C(P X, P Y ) := {Q XY P (X Y) : Q X = P X, Q Y = P Y } Vicet Y. F. Ta (NUS) Asymptotic Couplig 3 / 26

5 Geeral Couplig Problem A joit distributio Q XY P (X Y) such that Q X = P X, Q Y = P Y is called a couplig of P X, P Y. The set of coupligs of P X, P Y is defied as C(P X, P Y ) := {Q XY P (X Y) : Q X = P X, Q Y = P Y } Couplig Problem: Give margials P X ad P Y ad a real-valued fuctio g(p XY ), what is the value of max g(p XY )? P XY C(P X,P Y ) Vicet Y. F. Ta (NUS) Asymptotic Couplig 3 / 26

6 Several Couplig Problems Geeral couplig problem max g(p XY ) P XY C(P X,P Y ) Vicet Y. F. Ta (NUS) Asymptotic Couplig 4 / 26

7 Several Couplig Problems Geeral couplig problem max g(p XY ) P XY C(P X,P Y ) Geeral Couplig Problem g(p XY ) Maximal Couplig P(Y = X) Maximal Guessig Couplig max f P(Y = f(x)) Miimum Distace Couplig E[d(X, Y )] Miimum Excess-Distortio Couplig P{d(X, Y ) > D} Vicet Y. F. Ta (NUS) Asymptotic Couplig 4 / 26

8 Several Couplig Problems Geeral couplig problem max g(p XY ) P XY C(P X,P Y ) Geeral Couplig Problem g(p XY ) Maximal Couplig P(Y = X) Maximal Guessig Couplig max f P(Y = f(x)) Miimum Distace Couplig E[d(X, Y )] Miimum Excess-Distortio Couplig P{d(X, Y ) > D} I our work we cosider a large umber of radom variables, i.e., X X ad Y Y as. Vicet Y. F. Ta (NUS) Asymptotic Couplig 4 / 26

9 Maximal Couplig Problem The Maximal Couplig Problem is defied as M(P X, P Y ) := max P {Y = X}. P XY C(P X,P Y ) Vicet Y. F. Ta (NUS) Asymptotic Couplig 5 / 26

10 Maximal Couplig Problem The Maximal Couplig Problem is defied as M(P X, P Y ) := max P {Y = X}. P XY C(P X,P Y ) The Total Variatio Distace betwee P ad Q is P Q T V := 1 P (x) Q(x). 2 x Vicet Y. F. Ta (NUS) Asymptotic Couplig 5 / 26

11 Useful Lemma: Maximal Couplig Equality Lemma Give P X ad P Y, we have M(P X, P Y ) := max P {Y = X} = 1 P X P Y T V. P XY C(P X,P Y ) Vicet Y. F. Ta (NUS) Asymptotic Couplig 6 / 26

12 Useful Lemma: Maximal Couplig Equality Lemma Give P X ad P Y, we have M(P X, P Y ) := max P {Y = X} = 1 P X P Y T V. P XY C(P X,P Y ) Furthermore, the (optimal) maximal couplig is { mi {P X (x), P Y (y)}, x = y; P XY (x, y) = q x,y, x y where q x,y (for x y) ca take o ay value as log as P XY forms a valid distributio. Vicet Y. F. Ta (NUS) Asymptotic Couplig 6 / 26

13 Useful Lemma: Maximal Couplig Equality Proof. P {Y = X} = x P XY (x, x) x mi {P X (x), P Y (x)} = 1 P X P Y T V. Moreover, the = holds for P XY defied for the maximal couplig equality. Vicet Y. F. Ta (NUS) Asymptotic Couplig 7 / 26

14 Useful Lemma: Maximal Couplig Equality Proof. P {Y = X} = x P XY (x, x) x mi {P X (x), P Y (x)} = 1 P X P Y T V. Moreover, the = holds for P XY defied for the maximal couplig equality. M(P X, P Y ) correspods to Regio III= x mi {P X(x), P Y (x)} Vicet Y. F. Ta (NUS) Asymptotic Couplig 7 / 26

15 Maximal Couplig M(P X, P Y ) Theorem If P X P Y, the give P X ad P Y, we have M(P X, P Y ) 0 expoetially fast as. More explicitly, the expoet is lim 1 log M(P X, PY ) = mi max {D(Q P X), D(Q P Y )}. Q Note that Q is the mid-poit of the e-geodesic coectig P X ad P Y. Vicet Y. F. Ta (NUS) Asymptotic Couplig 8 / 26

16 Maximal Couplig M(P X, P Y ) Theorem If P X P Y, the give P X ad P Y, we have M(P X, P Y ) 0 expoetially fast as. More explicitly, the expoet is lim 1 log M(P X, PY ) = mi max {D(Q P X), D(Q P Y )}. Q Note that Q is the mid-poit of the e-geodesic coectig P X ad P Y. A optimal product couplig P X Y = P XY with P XY achievig M(P X, P Y ) oly achieves the smaller expoet log M(P X, P Y ) = log (1 P X P Y ), which is suboptimal i geeral. Vicet Y. F. Ta (NUS) Asymptotic Couplig 8 / 26

17 Maximal Guessig Couplig Problem The Maximal Guessig Couplig Problem is defied as G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f Vicet Y. F. Ta (NUS) Asymptotic Couplig 9 / 26

18 Maximal Guessig Couplig Problem The Maximal Guessig Couplig Problem is defied as G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f Tryig to guess the value of Y usig X by desigig a fuctio f : X Y. Vicet Y. F. Ta (NUS) Asymptotic Couplig 9 / 26

19 Maximal Guessig Couplig Problem The Maximal Guessig Couplig Problem is defied as G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f Tryig to guess the value of Y usig X by desigig a fuctio f : X Y. Our questio: What is the value of lim G(P X, P Y )? Vicet Y. F. Ta (NUS) Asymptotic Couplig 9 / 26

20 Maximal Guessig Couplig Problem The Maximal Guessig Couplig Problem is defied as G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f Tryig to guess the value of Y usig X by desigig a fuctio f : X Y. Our questio: What is the value of lim G(P X, P Y )? How does the limit deped o P X ad P Y? Vicet Y. F. Ta (NUS) Asymptotic Couplig 9 / 26

21 Outlie 1 Problem Formulatio 2 Mai Results 3 Applicatios i Iformatio Theory 4 Coclusio ad Future Work Vicet Y. F. Ta (NUS) Asymptotic Couplig 10 / 26

22 Maximal Guessig Couplig Equality Lemma The maximal guessig couplig problem is equivalet to the distributio approximatio problem. That is, G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f = 1 mi f P Y P f(x) T V. Vicet Y. F. Ta (NUS) Asymptotic Couplig 11 / 26

23 Maximal Guessig Couplig Equality Lemma The maximal guessig couplig problem is equivalet to the distributio approximatio problem. That is, I the problem G(P X, P Y ) := max max P {Y = f(x)} P XY C(P X,P Y ) f = 1 mi f P Y P f(x) T V. mi P Y P f(x) T V f we try to approximate the distributio of Y by give a radom variable X ad we desig a fuctio f : X Y. Vicet Y. F. Ta (NUS) Asymptotic Couplig 11 / 26

24 Maximal Guessig Couplig Equality Proof. G(P X, P Y ) = max max P {Y = f(x)} P XY C(P X,P Y ) f (Deitio) Vicet Y. F. Ta (NUS) Asymptotic Couplig 12 / 26

25 Maximal Guessig Couplig Equality Proof. G(P X, P Y ) = max max P {Y = f(x)} P XY C(P X,P Y ) f = max f max P XY C(P X,P Y ) P {Y = f(x)} (Deitio) (Exchagig maximizatios) Vicet Y. F. Ta (NUS) Asymptotic Couplig 12 / 26

26 Maximal Guessig Couplig Equality Proof. G(P X, P Y ) = max max P {Y = f(x)} P XY C(P X,P Y ) f = max f = max f max P XY C(P X,P Y ) max P f(x),y C(P f(x),p Y ) P {Y = f(x)} (Deitio) (Exchagig maximizatios) P {Y = f(x)} (P {Y = f(x)} depeds o P XY oly through P f(x),y ) Vicet Y. F. Ta (NUS) Asymptotic Couplig 12 / 26

27 Maximal Guessig Couplig Equality Proof. G(P X, P Y ) = max max P {Y = f(x)} P XY C(P X,P Y ) f = max f = max f max P XY C(P X,P Y ) max P f(x),y C(P f(x),p Y ) P {Y = f(x)} ( ) = max 1 PY P f(x) T V f (Deitio) (Exchagig maximizatios) P {Y = f(x)} (P {Y = f(x)} depeds o P XY oly through P f(x),y ) (Maximal couplig equality) Vicet Y. F. Ta (NUS) Asymptotic Couplig 12 / 26

28 Maximal Guessig Couplig Equality Proof. G(P X, P Y ) = max max P {Y = f(x)} P XY C(P X,P Y ) f = max f = max f max P XY C(P X,P Y ) max P f(x),y C(P f(x),p Y ) P {Y = f(x)} ( ) = max 1 PY P f(x) T V f = 1 mi P Y P f(x) T V f (Deitio) (Exchagig maximizatios) P {Y = f(x)} (P {Y = f(x)} depeds o P XY oly through P f(x),y ) (Maximal couplig equality) Vicet Y. F. Ta (NUS) Asymptotic Couplig 12 / 26

29 Mai Result The distributio approximatio problem was studied by T. S. Ha usig the iformatio spectrum method. Vicet Y. F. Ta (NUS) Asymptotic Couplig 13 / 26

30 Mai Result The distributio approximatio problem was studied by T. S. Ha usig the iformatio spectrum method. It was proved If H(X) > H(Y ), the mi f P Y P f(x ) T V 0 at least expoetially fast as. If H(X) < H(Y ), the mi f P Y P f(x ) T V 1 at least expoetially fast as. Vicet Y. F. Ta (NUS) Asymptotic Couplig 13 / 26

31 Mai Result The distributio approximatio problem was studied by T. S. Ha usig the iformatio spectrum method. It was proved If H(X) > H(Y ), the mi f P Y P f(x ) T V 0 at least expoetially fast as. If H(X) < H(Y ), the mi f P Y P f(x ) T V 1 at least expoetially fast as. We obtai differet expoets for these two covergeces by usig the method of types. Vicet Y. F. Ta (NUS) Asymptotic Couplig 13 / 26

32 Mai Result The distributio approximatio problem was studied by T. S. Ha usig the iformatio spectrum method. It was proved If H(X) > H(Y ), the mi f P Y P f(x ) T V 0 at least expoetially fast as. If H(X) < H(Y ), the mi f P Y P f(x ) T V 1 at least expoetially fast as. We obtai differet expoets for these two covergeces by usig the method of types. We also show that if H(X) = H(Y ), the G(P X, P Y ) G(P X, P Y ), N Vicet Y. F. Ta (NUS) Asymptotic Couplig 13 / 26

33 Expoets Theorem 1 If H(X) > H(Y ), the G(PX, P Y ) 1 at least expoetially fast as. Moreover, the expoet is E (P X, P Y ) := lim if 1 log (1 G(P X, P Y )) E iid (P X, P Y ). with E iid (P X, P Y ) := 1 2 max t [0,1] t (H 1+t(X) H 1 t (Y )). Vicet Y. F. Ta (NUS) Asymptotic Couplig 14 / 26

34 Expoets Theorem 1 If H(X) > H(Y ), the G(PX, P Y ) 1 at least expoetially fast as. Moreover, the expoet is E (P X, P Y ) := lim if 1 log (1 G(P X, P Y )) E iid (P X, P Y ). with E iid (P X, P Y ) := 1 2 max t [0,1] t (H 1+t(X) H 1 t (Y )). 2 If H(X) < H(Y ), the G(PX, P Y ) 0 at least expoetially fast as. Moreover, the expoet is E (P X, P Y ) : = lim if 1 log G(P X, PY ) { 1 sup mi ɛ (0,1) 3 ɛ2 P (mi) X, 1 } 3 ɛ2 P (mi) Y, (1 ɛ)h(y ) (1 + ɛ)h(x). Vicet Y. F. Ta (NUS) Asymptotic Couplig 14 / 26

35 Outlie 1 Problem Formulatio 2 Mai Results 3 Applicatios i Iformatio Theory 4 Coclusio ad Future Work Vicet Y. F. Ta (NUS) Asymptotic Couplig 15 / 26

36 Applicatios Chael Capacity With Iput Distributio Costrait Commuicatio with Perfect Stealth/Covert Commuicatios Vicet Y. F. Ta (NUS) Asymptotic Couplig 16 / 26

37 Chael Capacity With Iput Distributio Costrait R M [1: e ] Ecoder X P Y X Y Decoder ˆM X ~ P P M Mˆ 0 X Vicet Y. F. Ta (NUS) Asymptotic Couplig 17 / 26

38 Chael Capacity With Iput Distributio Costrait R M [1: e ] Ecoder X P Y X Y Decoder ˆM X ~ P P M Mˆ 0 Chael Capacity With Iput Distributio Costrait is defied as { C (P X ) := sup R : (P X M, ) P M Y =1 s.t. X P X = PX, { lim P M M } } = 0 Vicet Y. F. Ta (NUS) Asymptotic Couplig 17 / 26

39 Chael Capacity With Iput Distributio Costrait R M [1: e ] Ecoder X P Y X Y Decoder ˆM X ~ P P M Mˆ 0 Chael Capacity With Iput Distributio Costrait is defied as { C (P X ) := sup R : (P X M, ) P M Y =1 s.t. C (P X ) also depeds o P Y X. X P X = PX, { lim P M M } } = 0 Vicet Y. F. Ta (NUS) Asymptotic Couplig 17 / 26

40 Chael Capacity With Iput Distributio Costrait R M [1: e ] Ecoder X P Y X Y Decoder ˆM X ~ P P M Mˆ 0 Chael Capacity With Iput Distributio Costrait is defied as { C (P X ) := sup R : (P X M, ) P M Y =1 s.t. C (P X ) also depeds o P Y X. X P X = PX, { lim P M M } } = 0 Without the costrait P X = PX, the Shao capacity is C(P Y X ) = max P X I(X; Y ). Vicet Y. F. Ta (NUS) Asymptotic Couplig 17 / 26

41 Mai Result Theorem We have where C (P X ) = C GK (X; Y ), C GK (X; Y ) := sup H(f (X)) f,g:f(x)=g(y ) = sup H(V ) V :V X Y V deotes the Gäcs-Körer (GK) commo iformatio betwee X ad Y (uder the distributio P X P Y X ). Vicet Y. F. Ta (NUS) Asymptotic Couplig 18 / 26

42 Mai Result Theorem We have where C (P X ) = C GK (X; Y ), C GK (X; Y ) := sup H(f (X)) f,g:f(x)=g(y ) = sup H(V ) V :V X Y V deotes the Gäcs-Körer (GK) commo iformatio betwee X ad Y (uder the distributio P X P Y X ). Note that C GK (X; Y ) I(X; Y ), ad max C GK (X; Y ) max I(X; Y ) P X P X Vicet Y. F. Ta (NUS) Asymptotic Couplig 18 / 26

43 Proof of Achievability: GK Mappig C GK (X; Y ) := sup H(V ) V :V X Y V The copy versio of the sigle-shot Markov chai V X Y V is: V X P P Y X V Y X P V Y X ~ P X V Vicet Y. F. Ta (NUS) Asymptotic Couplig 19 / 26

44 Proof of Achievability: GK Mappig C GK (X; Y ) := sup H(V ) V :V X Y V The copy versio of the sigle-shot Markov chai V X Y V is: V X P P Y X V Y X P V Y X ~ P X V Ay GK commo iformatio V ca be trasmitted losslessly However, V is ot a uiform radom variable! Vicet Y. F. Ta (NUS) Asymptotic Couplig 19 / 26

45 Proof of Achievability: Maximal Guessig + GK Mappig Ay rate R < sup V :V X Y V H(V ) = C GK (X; Y ) ca be achieved Vicet Y. F. Ta (NUS) Asymptotic Couplig 20 / 26

46 Proof of Coverse C GK (X; Y ) := sup H(V ) V :V X Y V V X P Y P V X XY P V ˆ Y Vˆ Theorem (Gäcs-Körer (1973)) For ay ɛ (0, 1), lim sup (P V X,P V Y ) { 1 H(V ) : P (V ˆV ) } ɛ = C GK (X; Y ) Our coverse is a special case: V is uiform Vicet Y. F. Ta (NUS) Asymptotic Couplig 21 / 26

47 Applicatio to Perfect Stealth Commuicatio R M [1: e ] Ecoder X P Y X Y Decoder ˆM Eve Mˆ 0 P M Beig i trasmissio or ot? Vicet Y. F. Ta (NUS) Asymptotic Couplig 22 / 26

48 Applicatio to Perfect Stealth Commuicatio R M [1: e ] Ecoder X P Y X Y Decoder ˆM Eve Mˆ 0 P M Beig i trasmissio or ot? Whe M is ot trasmitted over the chael, Eve observes X PX, which ca be regarded as pure oise Vicet Y. F. Ta (NUS) Asymptotic Couplig 22 / 26

49 Applicatio to Perfect Stealth Commuicatio R M [1: e ] Ecoder X P Y X Y Decoder ˆM Eve Mˆ 0 P M Beig i trasmissio or ot? Whe M is ot trasmitted over the chael, Eve observes X PX, which ca be regarded as pure oise To prevet Eve to detect the trasmissio, ecoder ad decoder should satisfy { X PX, ad lim P M M } = 0 Vicet Y. F. Ta (NUS) Asymptotic Couplig 22 / 26

50 Applicatio to Perfect Stealth Commuicatio R M [1: e ] Ecoder X P Y X Y Decoder ˆM Eve Mˆ 0 P M Beig i trasmissio or ot? Whe M is ot trasmitted over the chael, Eve observes X PX, which ca be regarded as pure oise To prevet Eve to detect the trasmissio, ecoder ad decoder should satisfy { X PX, ad lim P M M } = 0 Perfect stealth commuicatio problem is equivalet to chael codig problem with iput distributio costrait Vicet Y. F. Ta (NUS) Asymptotic Couplig 22 / 26

51 Applicatio to Perfect Stealth Commuicatio R M [1: e ] Ecoder X P Y X Y Decoder ˆM Eve Mˆ 0 P M Beig i trasmissio or ot? Theorem Perfect stealth capacity is C GK (X; Y ). Vicet Y. F. Ta (NUS) Asymptotic Couplig 23 / 26

52 Outlie 1 Problem Formulatio 2 Mai Results 3 Applicatios i Iformatio Theory 4 Coclusio ad Future Work Vicet Y. F. Ta (NUS) Asymptotic Couplig 24 / 26

53 Summary I this work, we studied the maximal guessig couplig problem: showed that it typically coverge at least expoetially fast to 0 or 1. applied this result to two ew iformatio-theoretic problems chael capacity with iput distributio costrait, ad perfect stealth commuicatio. Vicet Y. F. Ta (NUS) Asymptotic Couplig 25 / 26

54 Summary I this work, we studied the maximal guessig couplig problem: showed that it typically coverge at least expoetially fast to 0 or 1. applied this result to two ew iformatio-theoretic problems chael capacity with iput distributio costrait, ad perfect stealth commuicatio. A iterestig observatio (Maximal Guessig Couplig Equality): Maximal guessig couplig problem is equivalet to the distributio approximatio problem. Vicet Y. F. Ta (NUS) Asymptotic Couplig 25 / 26

55 Summary I this work, we studied the maximal guessig couplig problem: showed that it typically coverge at least expoetially fast to 0 or 1. applied this result to two ew iformatio-theoretic problems chael capacity with iput distributio costrait, ad perfect stealth commuicatio. A iterestig observatio (Maximal Guessig Couplig Equality): Maximal guessig couplig problem is equivalet to the distributio approximatio problem. Ope problem: maximal guessig couplig problem for H(X) = H(Y ) but P X P Y. Vicet Y. F. Ta (NUS) Asymptotic Couplig 25 / 26

56 Summary I this work, we studied the maximal guessig couplig problem: showed that it typically coverge at least expoetially fast to 0 or 1. applied this result to two ew iformatio-theoretic problems chael capacity with iput distributio costrait, ad perfect stealth commuicatio. A iterestig observatio (Maximal Guessig Couplig Equality): Maximal guessig couplig problem is equivalet to the distributio approximatio problem. Ope problem: maximal guessig couplig problem for H(X) = H(Y ) but P X P Y. Some other couplig problems ca also be foud the exteded versio of our paper Lei Yu ad Vicet Y. F. Ta, Asymptotic couplig ad its applicatios i iformatio theory, submitted to IEEE Tras. If. Theory, Dec Available at arxiv: Vicet Y. F. Ta (NUS) Asymptotic Couplig 25 / 26

57 Thak you for your attetio! Vicet Y. F. Ta (NUS) Asymptotic Couplig 26 / 26