much more on minimax (order bounds) cf. lecture by Iain Johnstone

Transcription

1 much more on minimax (order bounds) cf. lecture by Iain Johnstone

2 today s lecture parametric estimation, Fisher information, Cramer-Rao lower bound: Ch. 4, Sec. 9.3 information and estimation: Ch. 7 universal denoising: Ch. 8 (chapters and sections from new version of notes)

3

4 mean squared error estimation

5 bias-variance

6 Fisher Information exercise:

7 exercise

8 note

9

10 note: r.h.s. depends on estimator far from tight: consider estimator identically 0

11 multi-parameter case

12 Fisher information for a location family

13 Fisher Information and MMSE

14

15 recall D(P Q) = log dp dq dp I(U; V )=D(P U,V P U P V ),

16 mutual information and MMSE Y = γ X + W W is a standard Gaussian, independent of X I(γ) =I(X; Y ) mmse(γ) =E (X E[X Y ]) 2

17 [Guo, Shamaiand Verdú 2005]: d dγ I(γ) =1 2 mmse(γ) (follows from J-MMSE and De-Bruijn)

18 continuous time dy t = γx t dt + dw t, 0 t T I(γ) =I(X T ; Y T ) T mmse(γ) =E 0 (X t E[X t Y T ]) 2 dt

19 [Guo, Shamai and Verdú 2005]:, [Zakai 2005]: d dγ I(γ) =1 2 mmse(γ) or in its integral version snr I(snr) = mmse(γ)dγ

20 Duncan dy t = X t dt + dw t, 0 t T W is standard white Gaussian noise, independent of X [Duncan 1970]: I(X T ; Y T )= 1 2 E T 0 (X t E[X t Y t ]) 2 dt

21 SNR in Duncan dy t = γx t dt + dw t, 0 t T I(γ) =I(X T ; Y T ) cmmse(γ) =E T 0 [Duncan 1970]: (X t E[X t Y t ]) 2 dt I(γ) = γ 2 cmmse(γ)

22 Recap [Duncan 1970]: I(γ) = γ 2 cmmse(γ) [Guo, Shamai and Verdú 2005]:, [Zakai 2005]: snr I(snr) = mmse(γ)dγ?

23 Relationship between cmmse and mmse? cmmse(snr) = 1 snr snr 0 mmse(γ)dγ

24 Mismatch Y = γ X + W W is a standard Gaussian, independent of X What if X P but the estimator thinks X Q? mse P,Q (γ) =E P (X EQ [X Y ]) 2

25 A new representation of relative entropy [Verdu 2010]: D(P Q) = 0 [mse P,Q (γ) mse P,P (γ)]dγ D(P Ysnr Q Ysnr )= snr 0 [mse P,Q (γ) mse P,P (γ)]dγ

26 Causal vs. Non-causal Mismatched Estimation dy t = γx t dt + dw t, 0 t T W is standard white Gaussian noise, independent of X T cmse P,Q (γ) =E P (X t E Q [X t Y t ]) 2 dt T mse P,Q (γ) =E P 0 0 (X t E Q [X t Y T ]) 2 dt Relationship between cmse P,Q and mse P,Q?

27 Relationship between cmse P,Q and mse P,Q cmse P,Q (snr) = 1 snr snr 0 mse P,Q (γ)dγ

28 Relationship between cmse P,Q and mse P,Q cmse P,Q (snr) = 1 snr snr 0 mse P,Q (γ)dγ = 2 0 snr [I(snr)+D (P Y T Q Y T )]

29 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr)

30 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr) classical

31 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr) classical ours

32 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr) classical ours Redundancy-Capacity theory

33 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr) classical ours Redundancy-Capacity theory Shannon

34 minimax estimation { } P minimax(p, snr) =min ˆX( ) max P P cmse P, ˆX(snr) cmse P,P (snr) classical minimax(p, snr) = min Q ours Redundancy-Capacity theory = 2 snr min Q Shannon max [cmse P,Q(snr) cmse P,P (snr)] P P max D P Y T QY T P P snr snr = 2 snr max I Θ; Ysnr T = 2 PY snr C T snr P P : Θ is a P-valued RV

35 Strong Converse ε > 0 and any ˆX( ) for all P P with the possible exception of sources in a subset B P where cmse P, ˆX(snr) cmse P,P (snr) (1 ε) minimax(p, snr) B P B P w (B) e 2 ε C(P,snr), w being the capacity achieving prior

36 ISIT 2013 IEEE International Symposium on Information Theory July 7-12, 2013 Istanbul, Turkey Minimax Filtering via Relations Between Information and Estimation Albert No and T. Weissman

37 lookahead

38 question can I( ) determinelmmse(d, snr)?

39 question can I( ) determinelmmse(d, snr)? how about I( ) and S x ( )?

40 a time irreversible process

41 ?

42 Poisson Channel Scalar Channel: X 0 Y γ X Poisson(γ X) Continuous-time Channel: X T a non-negative stochastic process Y T γ X T non-homogenous Poisson of intensity γ X T

43 quest for :[0, ) [0, ) [0, ]

44 AWGN Channels, IEEE Trans. Information Theory, [34] M. Zakai, On mutual information, likelihood ratios, IEEE Trans. Information Theory, vol. 51, no. 9, pp. 3 (x, ˆx) =x log(x/ˆx) x +ˆx, 2.5 (1, ˆx) (a) (1, ˆx) Figure 1: The 0.3

45 An observation (and hint) D (Poisson(λ 1 ) Poisson(λ 2 )) = (λ 1, λ 2 )

46 An observation (and hint) D (Poisson(λ 1 ) Poisson(λ 2 )) = (λ 1, λ 2 ) D (N (µ 1, 1)N (µ 2, 1)) = 1 2 (µ 1 µ 2 ) 2

47 Punch Line [Rami Atar and T.W. 2012]: under the above (x, ˆx)

48 and i mean everything i-mmse Duncan causal - non-causal mismatch minimax

49 the universal picture

50 universal denoising

51 universal probability assignments: Q is universal if lim n 1 n D(P X n Q X n)=0 for every stationary P and pointwise universal

52 universal compressors (e.g.: Lempel-Ziv 78, CTW) universal probability assignment univ. sequential prob. assignment univ. prediction, filtering, denoising, lossy compression (much more in ee376c)