information estimation feedback

Transcription

1 relations between information and estimation in the presence of feedback talk at workshop on: Tsachy Weissman Information and Control in Networks LCCC - Lund Center for Control of Complex Engineering Systems

2 information, control, networks real-time and limited delay communication feedback communications action in information theory relations between information and estimation (w. feedback + networks)

3 Outline relations between information and estimation the presence of feedback implications for networks

4 Haves and Have-Nots (in this talk) we ll have: some theorems cute (and meaningful) relations an algorithmic framework we won t have: account of related literature stipulations proofs algorithms data

5 de Bruijn s identity [A. J. Stam 1959]: X independent of Z N (, 1) d dt h X + tz = 1 2 J(X + tz)

6 GuoShamaiVerdu setting Y = X + W W is a standard Gaussian, independent of X I( )=I(X; Y ) mmse( )=E (X E[X Y ]) 2

7 [Guo, Shamai and Verdú 25]: d d I( )=1 2 mmse( )

8 GSV in continuous time dy t = X t dt + dw t, t T I( )=I(X T ; Y T ) T mmse( )=E (X t E[X t Y T ]) 2 dt

9 [Guo, Shamai and Verdú 25]:, [Zakai 25]: d d I( )=1 2 mmse( ) or in its integral version snr I(snr) = 1 2 mmse( )d

10 Duncan dy t = X t dt + dw t, t T W is standard white Gaussian noise, independent of X [Duncan 197]: I(X T ; Y T )= 1 2 E T (X t E[X t Y t ]) 2 dt

11 SNR in Duncan dy t = X t dt + dw t, t T I( )=I(X T ; Y T ) cmmse( )=E T [Duncan 197]: (X t E[X t Y t ]) 2 dt I( )= 2 cmmse( )

12 Recap [Duncan 197]: I( )= cmmse( ) 2 [Guo, Shamai and Verdú 25]:, [Zakai 25]: snr I(snr) = 1 2 mmse( )d?

13 Relationship between cmmse and mmse? [Guo, Shamai and Verdú 25]: cmmse(snr) = 1 snr snr mmse( )d

14 Mismatch Y = X + W W is a standard Gaussian, independent of X What if X P but the estimator thinks X Q? mse P,Q ( )=E P (X E Q [X Y ]) 2

15 A representation of relative entropy [Verdu 21]: D(P Q) = [mse P,Q ( ) mse P,P ( )]d D(P Ysnr Q Ysnr )= snr [mse P,Q ( ) mse P,P ( )]d

16 Causal vs. Non-causal Mismatched Estimation dy t = X t dt + dw t, t T W is standard white Gaussian noise, independent of X T cmse P,Q ( )=E P (X t E Q [X t Y t ]) 2 dt T mse P,Q ( )=E P (X t E Q [X t Y T ]) 2 dt

17 Causal vs. Non-causal Mismatched Estimation dy t = X t dt + dw t, t T W is standard white Gaussian noise, independent of X T cmse P,Q ( )=E P (X t E Q [X t Y t ]) 2 dt T mse P,Q ( )=E P (X t E Q [X t Y T ]) 2 dt Relationship between cmse P,Q and mse P,Q?

18 Relationship between cmse P,Q and mse P,Q [Weissman 21]: cmse P,Q (snr) = 1 snr snr mse P,Q ( )d

19 Relationship between cmse P,Q and mse P,Q [Weissman 21]: cmse P,Q (snr) = 1 snr snr mse P,Q ( )d = 2 snr [I(snr)+D (P Y T Q Y T )]

20 Implications and Applications many

21 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt # cmse P,P (snr) )

22 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt # cmse P,P (snr) ) classical

23 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt # cmse P,P (snr) ) classical ours

24 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt # cmse P,P (snr) ) classical ours Redundancy-Capacity theory

25 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt # cmse P,P (snr) ) classical ours Redundancy-Capacity theory Shannon

26 Z Minimax (causal) Estimation minimax(p, snr) 4 = min max { ˆX t ( )} appletapplet P 2P ( E P " Z T `(X t, ˆX t (Y t ))dt ( " Z # # ) cmse P,P (snr) classical minimax(p, snr) = min Q ours Redundancy-Capacity theory = 2 snr min Q = 2 snr max = 2 snr C Shannon max [cmse P,Q(snr) P 2P max P 2P D P Y T snr I ; Y T snr P Y T snr P 2P Q Y T snr cmse P,P (snr)] : is a P-valued RV

27 e strong d cap snr Q P 2P = ma2 2 snr redundancy-capacity 2 results are =directly applicable and im T here = : is a P-v max I ; Y T = : is a P-valuedsnrRV max I ; Ysnr snr 2 sn snr = C(P 2 2 snr = C(P, snr) T = C P Ysnr P strong Furthermore, the 2P snr redundancy-capacity snr Strong Converse Furthermore, the strong redundancy-capacity resu r {X, Furthermore, the strong strongredundancy-capacity redundancy-capacity results directly applicabl t ( )} t T of arehere ore, the strong redundancy-capacity results are directlyresult applicable and imply: [Merhavstrong and Feder 1995] applied here implies: 6.5 red cap T 6.5capstrong 6.5 red cap ng red t strong red cap (Xt, X t (Y ))dt cmsep,p (snr) (1 ) minimax(p, snr) EP >>{X and any filter { X ( )}, t t T > and t T any ( )}, ny filter {X t( )},filter t t T and any filter {X t ( )} t T, # "Z T T he possible exception of sources in a subset B P where T t EP t cmset,p,p (X X t(snr) (Y ))dt(1 `(Xt, X t (YE))dt P E T ") minimax(p, cmle (1 t ) min P,P (snr) tsnr) (X, X (Y ))dtcm nep Po «t, X tt (Y t))dt (X C PY T snr P P w (B) e 2 with for theall possible exception of sources in a subset B P where P P with the possible exception of sources in a, subset B P where exception of sources in npossible for all P P with the o for all P P with the possible exception C(P,snr) of sourc w (B) e 2 (B) e 2, minimax(p,snr) " C w (B) e 2 w being the capacity achieving prior capacity achieving prior ty achieving prior PY T w snr P 2P, w being the capacity achieving prior w being the capacity achieving prior , w (B) w (

28 Example Given: orthonormal signal set { i (t), t T } n i=1 X t = n B i i (t) i=1 P = laws P on X T : E P B 2 n and E P B n max I(X T ; Y T ) =?

29 Example (cont.) = = Y i = T i (t)dy t 1 i n are su cient statistics for Y T, I(X T ; Y T )=I(B n ; Y n ) max I(X T ; Y T ) = max I(B n ; Y n ) = max{i(b; Y ):B 2,P(B = ) (1 )} latter considered and numerically solved in: Lei Zhang and Dongning Guo, Capacity of Gaussian Channels with Duty Cycle and Power Constraints, IEEE Int. Symposium on Information Theory 211

30 Example (cont.) thus the minimax filter here is the Bayes filter assuming: X t = n i=1 B i i (t) where B i are iid according to the capacity achieving distribution of [Zhang and Guo, 211] cf. [Albert No + T.W., ISIT 213]...

31 (well) beyond Gaussian noise Poisson channel Lévy-type channels: Input-Output relationship expressed via Lévytype stochastic integral can obtain formulae via Lévy-Khintchine-type decompositions

32 ) X information control networks

33 The presence of Feedback

34 The presence of Feedback what of what we ve seen carries over to presence of feedback?

35 Duncan dy t = X t dt + dw t, t T W is standard white Gaussian noise, independent of X [Duncan 197]: I(X T ; Y T )= 1 2 E T (X t E[X t Y t ]) 2 dt Breaks down in presence of feedback!

36 cont time directed info [W., Permuter, Kim 212] I X T! Y T := inf I t X T! Y T t where I(X n! Y n ), nx i=1 I(X i ; Y i Y i 1 )

37 Duncan with feedback Theorem R { Let {(X t,b t )} T t= be adapted to the filtration {F t } T t=, where X T is a signal of finite average power R T E[X2 t ]dt < 1 and B T is a standard Brownian motion. Let Y T be the output of the AWGN channel whose input is X T and whose noise is driven by B T, i.e., dy t = X t dt + db t. Suppose that the regularity assumptions of Proposition 2 are satisfied for all <t<t. Then Z 1 2 Z T [W., Permuter, Kim 212] E (X t E[X t Y t ]) 2 dt = I(X T! Y T ) compare with [Kadota, Zakai, Ziv 1971]

38 GSV in continuous time dy t = X t dt + dw t, t T d d I( )=1 2 mmse( ) or in its integral version snr I(snr) = 1 2 mmse( )d

39 GSV in continuous time dy t = X t dt + dw t, t T d d I( )=1 2 mmse( ) or in its integral version snr I(snr) = 1 2 mmse( )d Breaks down in presence of feedback

40 GSV in continuous time with DI? I(X T Y T ) =? 1 2 snr No. In general I(X T Y T ) = 1 2 snr and so mmse( )d mmse( )d cmmse(snr) = 1 snr snr mmse( )d I.e., breakdown in presence of feedback

41 Mismatched setting a fortiori, in presence of feedback, in general cmse P,Q (snr) = 1 snr 2 snr mse P,Q ( )d

42 Mismatched setting a fortiori, in presence of feedback, in general cmse P,Q (snr) = 1 snr 2 snr mse P,Q ( )d end of story?

43 Mismatched setting (cont.) cmse P,Q cmse P,P = D(P Y T Q Y T ) holds with or without FB, appears in TW21 implicitly and explicitly in workshop book chapter [Asnani, Venkat, W. 212] (why?)

44 implications and apps minimax estimation setting carries over directed info maximization instead of mutual info but same idea similar extensions to the more general channels

45 ) X ) information X control networks

46 Distributed estimation (known source) Y 1, ˆX 1 (Y 1 ).. X P X known source network noise. Y i, ˆX i (Y i ). Y n, ˆX n (Y n )

47 Distributed estimation (known source) Y 1, ˆX 1 (Y 1 ).. X P X known source network noise. Y i, ˆX i (Y i ). Y n, ˆX n (Y n ) can (and should) be greedy!

48 Distributed estimation (source uncertainty) Y 1, ˆX 1 (Y 1 ).. X P X 2 P network Y i, ˆX i (Y i ) source uncertainty noise.. Y n, ˆX n (Y n )

49 Distributed estimation (source uncertainty) Y 1, ˆX 1 (Y 1 ).. X P X 2 P network Y i, ˆX i (Y i ) source uncertainty noise.. Y n, ˆX n (Y n ) should we be greedy?

50 Distributed estimation (source uncertainty) Y 1, ˆX 1 (Y 1 ).. X P X 2 P network Y i, ˆX i (Y i ) source uncertainty noise.. Y n, ˆX n (Y n ) should we be greedy? no! (in general)

51 Distributed estimation (source uncertainty) Y 1, ˆX 1 (Y 1 ).. X P X 2 P network Y i, ˆX i (Y i ) source uncertainty noise.. Y n, ˆX n (Y n ) should we be greedy? no! (in general) yes! (in causal estimation over Gaussian, Poisson, or general Levy-type noise) minimax estimation for each observation separately would be essentially optimal

52 ) X) information X) control X networks

53 conclusion relations between mutual information, relative entropy, and estimation findings of pure estimation theoretic significance allow the transfer of tools much carries over to presence of feedback implications for networks