On the Applications of the Minimum Mean p th Error to Information Theoretic Quantities

Size: px

Start display at page:

Download "On the Applications of the Minimum Mean p th Error to Information Theoretic Quantities"

Elvin Snow
5 years ago
Views:

1 On the Applications of the Minimum Mean p th Error to Information Theoretic Quantities Alex Dytso, Ronit Bustin, Daniela Tuninetti, Natasha Devroye, H. Vincent Poor, and Shlomo Shamai (Shitz)

2 Outline Notation and Past Work Minimum Mean p th Error (MMPE) Properties of the MMPE Conditional MMPE Applications to Information Theory Problems

3 Notation X 2 R n, Y snr = p n Tr E[XXT ] apple, snrx + Z, I n (X, snr) = n I(X; Y snr), mmse(x Y snr )=mmse(x, snr) := n Tr (E [cov(x Y snr)]) I-MMSE relationship d dsnr I n(x, snr) = mmse(x, snr) 2 I n (X, snr) = 2 Z snr 0 mmse(x,t)dt D. Guo, S. Shamai, and S. Verdu, Mutual information and minimum mean-square error in Gaussian channels, IEEE Trans. Inf. Theory, vol. 5, no. 4, pp , April 2005.

4 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990., Take X D to be PAM with N = p + snr then 2 log( + snr) I(X D; Y ) = gap 0.75 bit

5 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j, L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990. Can this be improved Take X D to be PAM with N = p + snr then 2 log( + snr) I(X D; Y ) = gap 0.75 bit

6 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990., Improvement gap = e 2 log log + 2 mmse(x D Y ) d 2 min (X D) mmse(x D Y )=E (X D E[X D Y ]) 2. A. Dytso, D. Tuninetti, and N. Devroye, Interference as noise: Friend or foe? IEEE Trans. Inf. Theory, vol. 62, no. 6, pp , 206.,

7 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j, L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990. Improvement gap = e 2 log log + 2 mmse(x D Y ) d 2 min (X D) mmse(x D Y )=E (X D E[X D Y ]) 2., mmse(x D Y ) apple lmmse(x D Y ), lmmse(x D Y )=O, snr mmse(x D Y )=O e snrd 2 min 4. A. Dytso, D. Tuninetti, and N. Devroye, Interference as noise: Friend or foe? IEEE Trans. Inf. Theory, vol. 62, no. 6, pp , 206.

8 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j, L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990. Improvement gap = e 2 log log + 2 mmse(x D Y ) d 2 min (X D) mmse(x D Y )=E (X D E[X D Y ]) 2., 2 log( + snr) I(X D; Y ) 0.75 bit. 2 log( + snr) I(X D; Y ) 0.65 bit. A. Dytso, D. Tuninetti, and N. Devroye, Interference as noise: Friend or foe? IEEE Trans. Inf. Theory, vol. 62, no. 6, pp , 206.

9 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j, L. Ozarow and A. Wyner, On the capacity of the Gaussian channel with a finite number of input levels, IEEE Trans. Inf. Theory, vol. 36, no. 6, pp , Nov 990. Improvement gap = e 2 log log + 2 mmse(x D Y ) d 2 min (X D) mmse(x D Y )=E (X D E[X D Y ]) 2. A. Dytso, D. Tuninetti, and N. Devroye, Interference as noise: Friend or foe? IEEE Trans. Inf. Theory, vol. 62, no. 6, pp , 206., 2 log( + snr) I(X D; Y ) 0.75 bit. 2 log( + snr) I(X D; Y ) 0.65 bit. Can we do better?

10 MMPE The p-norm of a random vector U 2 R n is given by kuk p p = htr n E p 2 UU T i. For n =, kuk p =E[ U p ]. Example: U N (0, I) nkuk p p =2 p 2 (x) = Z 0 t x n+p 2 n 2, e t dt.

11 MMPE We define the Minimum Mean p-th Error (MMPE) of estimating X from Y as h i mmpe(x Y; p) :=inf kx f f(y)kp p =inf E Err p 2 (X,f(Y)), f q Err = (X f(y)) T (X f(y)). We denote the optimal estimator as f p (X Y). For n = : Example: for p =2 mmpe(x Y ; p) =inf f E [ X f(y ) p ]. mmpe(x Y; p = 2) = mmse(x Y), f p=2 (X Y) =E[X Y]. We shall denote Study Properties mmpe(x Y; p) =mmpe(x, snr,p).

12 Properties of the MMPE Optimal Estimator Theorem. For mmpe(x, snr, p) and p by the following point-wise relationship: 0 the optimal estimator is given h i f p (X Y = y) = arg min E Err p 2 (X, v) Y = y. v2r n Orthogonality-like Property Theorem. For any apple p< optimal estimator f p (X Y) satisfies apple E W T W p 2 2 W T g(y) =0, where W = X f p (X Y) for any deterministic function g( ).

13 Properties of Optimal Estimators E[X Y]. if 0 apple X 2 R then 0 apple E[X Y ], 2. (Linearity) E[aX + b Y] = ae[x Y] + b, 3. (Stability) E[g(Y) Y] = g(y) for any function g( ), 4. (Idempotent) E[E[X Y] Y] =E[X Y], 5. (Degradedness)0 E [X Y snr, Y snr0 ]=E[X Y snr0 ], for X! Y snr0! Y snr, 6. (Shift Invariance) mmse(x + a,snr) = mmse(x, snr), 7. (Scaling) mmse(ax, snr) = a 2 mmse(x,a 2 snr). f p (X Y). if 0 apple X 2 R then 0 apple f p (X Y ), 2. (Linearity) f p (ax + b Y) =af p (X Y)+b, 3. (Stability) f p (g(y) Y) =g(y) for any function g( ), 4. (Idempotent) f p (f p (X Y) Y) =f p (X Y), 5. (Degradedness) f p (X Y snr0, Y snr )=f p (X Y snr0 ), for X! Y snr0! Y snr, 6. (Shift) mmpe(x + a, snr,p)=mmpe(x, snr,p), 7. (Scaling) mmpe(ax, snr,p)=a p mmpe(x,a 2 snr,p). E[E[X Y ]] = E [X] E[f p (X Y )] = E [X]

14 Examples of Optimal Estimators if X N (0, I) then for p f p (X Y = y) = p snr +snr if X = {±} equally likely (BPSK) then for p p=2 p=.5 p= p=3 y (i.e., linear ); f p (X Y ) = tanh psnr p y. Function of p f p (X Y=y) S. Sherman, Non-mean-square error criteria, IRE Transactions on Information Theory, vol. 3, no. 4, pp , y

15 Bounds on the MMPE mmse(x, snr) apple min snr, kxk2 2 Theorem. For snr 0, 0 <qapplep, and input X kzk p p mmpe(x, snr,p) apple min, kxk p snr p p, 2 n p q mme p q (X, snr,q) apple mme(x, snr,p) applekx E[X Y]k p p.

16 Bounds on the MMPE Interpolation Bounds Theorem. For any 0 <p<q<rappleand 2 (0, ) such that where =, In particular, q = p + r () = q r p r, (a) mmpe q (X, snr,q) apple inf f kx f(y)k p kx f(y)k r. (b) mmpe q (X, snr,q) applekx fr (X Y)k p mmpe r (X, snr,r), (c) mmpe q (X, snr,q) apple mmpe p (X, snr,p)kx fp (X Y)k r. (d)

17 MMPE with Gaussian Input (Estimation of Gaussian Input.) For X G N (0, 2 I) and p mope(x G, snr,p)= with the optimal estimator given by f p (X G Y = y) = p kzk p p ( + snr 2 ) p 2 2 p snr + snr 2 y., (Asymptotic Optimality of Gaussian Input.) For every p, and a random variable X such that kxk p p apple p kzk p p,wehave mmpe(x, snr,p) apple k p, 2 snr p kzk p p ( + snr 2 ) p 2. where 8 >< k p, 2 snr = p =2, >: apple k p p,snr = +p p snr +snr apple + p +snr p 6= 2.

18 Conditional MMPE We define the conditional MMPE as mmpe(x, snr,p U) :=kx f p (X Y snr, U)k p p. Additional Information (Conditioning Reduces MMPE.)For every snr 0, p 0, and random variable X, wehave mmpe(x, snr,p) mmpe(x, snr,p U).

19 Bounds on the MMPE (Extra Independent Noisy Observation.) Let U = p X + Z where Z N (0, I) and where (X, Z, Z ) are mutually independent. Then mmpe(x, snr,p U) =mmpe(x, snr +,p). Consequences: MMPE is decreasing function of SNR New Proof of the Single-Crossing-Point Property

20 MMSE Bounds: Single Crossing Point Property (SCPP.) For any fixed X, suppose that mmse(x, snr 0 )= + snr0, for some fixed 0. Then mmse(x, snr) apple + snr, snr 2 [snr 0, ) mmse(x, snr) + snr, snr 2 [0, snr 0). SCPP+I-MMSE important tool for derive converses: Version of EPI BC Wiretap MIMO channels D. Guo, Y. Wu, S. Shamai, and S. Verdu, Estimation in Gaussian noise: Properties of the minimum mean-square error, IEEE Trans. Inf. Theory, vol. 57, no. 4, pp , April 20. R. Bustin, R. Schaefer, H. Poor, and S. Shamai, On MMSE properties of optimal codes for the Gaussian wiretap channel, in Proc. IEEE Inf. Theory Workshop, April 205, pp. 5. R. Bustin, M. Payaro, D. Palomar, and S. Shamai, On MMSE crossing properties and implications in parallel vector Gaussian channels, IEEE Trans. Inf. Theory, vol. 59, no. 2, pp , Feb 203.

21 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p)

22 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z

23 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z = kx f p (X Y, Y snr0 )k p

24 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z = kx f p (X Y, Y snr0 )k p applekx ˆXk p, where ˆX = ( ) p Y + f p (X Y snr0 ), 2 [0, ]

25 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z = kx f p (X Y, Y snr0 )k p applekx ˆXk p, where ˆX = ( ) p Y + f p (X Y snr0 ), 2 [0, ] = (X f p (X Y snr0 )) ( ) p Z p

26 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z = kx f p (X Y, Y snr0 )k p applekx ˆXk p, where ˆX = ( ) p Y + f p (X Y snr0 ), 2 [0, ] = (X f p (X Y snr0 )) = ( ) p Z p kzk 2 p(x f p (X Y snr0 )) m Z p kzk 2, choose = p + m p kzk 2 p kzk 2 p + m

27 New Proof of the SCPP m := mmpe 2 p (X, snr0,p)= kzk2 p + snr 0 for some 0,,, = m kzk 2 p snr 0m. Proof: mmpe p (X, snr,p)=mmpe p (X, snr0 +,p) =mmpe p (X, snr0,p Y ), where Y = p X + Z = kx f p (X Y, Y snr0 )k p applekx ˆXk p, where ˆX = ( ) p Y + f p (X Y snr0 ), 2 [0, ] = (X f p (X Y snr0 )) ( ) p Z p kzk 2 p(x f p (X Y snr0 )) m Z p kzk 2 p = kzk 2, choose = p + m kzk 2 p + p ( mkzkp 2 p, triangle inequality apple c p q,c p = kzk 2 p + m p =2, expand p m = c p kzk 2 p + snr

28 New Proof of the SCPP (SCPP Bound for MMPE.) Suppose mmpe 2 p (X, snr0,p)= 0. Then kzk2 p + snr 0 for some where c p = mmpe 2 p (X, snr,p) apple cp ( 2 p p =2. kzk 2 p + snr, for snr snr 0, Observations: Previous proof relied on the knowing the derivative of the MMSE New proof relies on simple estimation observations Extends to the MMPE for all p>

29 Connections to IT Bounds on Conditional Entropy h(u V) apple n 2 log(2 e mmse(u V)), Theorem. If kuk p < for some p 2 (0, ) then for any V 2 R n,wehave h(u V) apple n 2 log k 2 n,2p n p mmpe p (U V; p), where k n,p := p ( p n) p e p n ( n n ( n 2 + ) p + ) = p 2 e n 2 ( p 2 ) 2n + o n p. Sharper version of Ozarow-Wyner bound Bounds on the derivative of the MMSE

30 Ozarow-Wyner Bound (Ozarow-Wyner Bound.) Let X D be a discrete random variable then H(X D ) gap apple I(X D ; Y ) apple H(X D ), lmmse(x D Y )= d min (X D )= gap = 2 log e 6 E[X 2 D ] + snr E[X 2 D ] + 2 log + 2 lmmse(x D Y ) d 2 min (X D) inf x i x j. x i,x j 2supp(X D ):i6=j,

31 Generalized Ozarow-Wyner Bound Let X D be a discrete random vector, then for any p where [H(X D ) gap p ] + apple I(X D ; Y) apple H(X D ), gap p = inf gap(u), U2K ku + n XD f p (X D Y)k p gap(u) =log kuk p + log! k n,p n p kukp. e n h e(u) where K = {U : diam(supp(u)) apple 2 d min (X D )}. Moreover,! ku + XD f p (X D Y)k for p p log apple log + mmpe p (X, snr,p). kuk p kuk p

32 Take X D to be PAM with N = p + snr then 0.8 Generalized Ozarow-Wyner Bound 2 log( + snr) I(X D; Y ) = gap p=2 p=4 p=6 Original OW Shaping Loss gap SNRdB

33 Concluding Remarks Properties of the MMPE functional Conditional MMPE New Proof of the SCPP and its extension Bounds on differential entropy Generalized Ozarow-Wyner bound

34 A. Dytso, R. Bustin, D. Tuninetti, N. Devroye, S. Shamai, and H. V. Poor, On the Minimum Mean p-th Error in Gaussian Noise Channels and its Applications, Submitted to IEEE Trans. Inf. Theory, Thank you arxiv:

35 Back Up Slides arxiv:

36 Problem W Encoder X n p snr Y n Decoder Ŵ p snr0 Z n Y n snr 0 Estimator ˆX Y n = p snrx n + Z n Z n 0 Y n snr 0 = p snr 0 X n + Z n 0 where Z n N (0, I),Z n 0 N (0, I) mmse(x n Y n snr 0 ) apple c max-i problem C n (snr, snr 0, s.t.tr E ):=sup X n n I(Xn,Ysnr), n h X n (X n ) Ti apple n, and mmse(x n, snr 0 ) apple + snr0.

37 max-mmse problem max-mmse problem M n (snr, snr 0, ):=supmmse(x, snr), X s.t. n Tr E[XXT ] apple, and mmse(x, snr 0 ) apple + snr0. Relationship to max-i problem snr 2 M n(snr, snr 0, ) apple C n (snr, snr 0, ) apple 2 Z snr 0 M n (t, snr 0, )dt.

On the Minimum Mean p-th Error in Gaussian Noise Channels and its Applications

On the Minimum Mean p-th Error in Gaussian Noise Channels and its Applications Alex Dytso, Ronit Bustin, Daniela Tuninetti, Natasha Devroye, H. Vincent Poor, and Shlomo Shamai (Shitz) Notation X 2 R n,