Mismatched Estimation in Large Linear Systems

Transcription

1 Mismatched Estimation in Large Linear Systems Yanting Ma, Dror Baron, and Ahmad Beirami North Carolina State University Massachusetts Institute of Technology & Duke University Supported by NSF & ARO

2 Motivation True prior minimum mean square error (MMSE) True prior unknown in practical problems use postulated prior different from true prior mismatch in prior Mismatched estimation mean square error (MSE) bigger than MMSE excess mean square error (EMSE)

3 Motivation Mismatched estimation in scalar channels [Verdú 010] Many practical problems are modeled by linear systems (e.g., medical imaging, compressed sensing) Extension to large linear systems y x σ z z y A x σ z z = + = + scalar channel linear system

4 Simplification y x σ z z = + x, z i.i.d. Y X σ z Z = +

5 Mismatch in Scalar Channels [Verdú 010] Additive white Gaussian noise channel: Y = X + σ z Z R; X~P, Z~N(0,1) = + Y X σ z Z

6 Mismatch in Scalar Channels [Verdú 010] Additive white Gaussian noise channel: Y = X + σ z Z R; X~P, Z~N(0,1) = + Y X σ z Z True prior P: Conditional expectation: X P Y = E P X Y Mean square error (MSE): MSE P (σ z ) = E X P Y X minimum MSE (MMSE): MMSE(σ z ) = MSE P (σ z )

7 Mismatch in Scalar Channels [Verdú 010] Additive white Gaussian noise channel: Y = X + σ z Z R; X~P, Z~N(0,1) Y X σ z Z = + True prior P: Conditional expectation: X P Y = E P X Y Mean square error (MSE): MSE P (σ z ) = E X P Y X minimum MSE (MMSE): MMSE(σ z ) = MSE P (σ z ) Mismatched prior Q: MSE: MSE Q (σ z ) = E X Q Y X Bigger than MMSE

8 Mismatch in Scalar Channels [Verdú 010] Excess MSE of scalar channel (S-EMSE): S-EMSE(σ z ) = MSE Q (σ z ) MMSE(σ z ) D(P Q) = 1 0 S EMSE 1/γ dγ, γ = 1/σz relative entropy

9 Linear System Linear system: y = Ax + σ z z R M Input signal: x R N, x j ~P Noise: z i ~N(0,1) Measurement matrix: A R M N, E A ij = 0, Var A ij = 1 M σ z z

10 Decoupling [Tanaka 00, Guo & Verdú 005, ] Linear system: y = Ax + σ z z decoupling N, M N δ large system limit Scalar channel: y i = x i + σ P z i x i ~P, z i ~N(0,1) Fixed point equation: δ σ P σ z = MSE P σ p

11 Problem Statement

12 MMSE in Large Linear Systems Focusing on large linear systems True prior P: Conditional expectation: x P y, A = E P x y, A MSE: LMSE P σ z 1 = lim E x N N P y, A x = MMSE(σ z ) linear system

13 LMSE via Decoupling [Bernoulli-Gaussian input] LMSE P σ z (MMSE) Fixed point equation: δ σ σ z = MSE P σ with solution σ P

14 Mismatch in Large Linear Systems Mismatched prior Q: MSE: LMSE Q σ z 1 = lim E x N N Q y, A x Excess MSE of linear system: L-EMSE(σ z )=LMSE Q σ z MMSE(σ z )

15 Mismatch in Large Linear Systems LMSE Q σ z LMSE P σ z (MMSE) Goal: relate L-EMSE(σ z ) to S-EMSE(σ P )

16 Main Results

17 Main Result (derivation in paper) L-EMSE σ z = S-EMSE σ P + σp σ P + 1 δ L EMSE(σ z ) d dσ MSE Q σ dσ

18 Main Result (derivation in paper) L-EMSE σ z = S-EMSE σ P + σp σ P + 1 δ L EMSE(σ z ) Taylor expansion d dσ MSE Q σ dσ

19 Taylor Approximations First order approximation: L-EMSE σ z = δ δ α S-EMSE σ p + o(δ) Second order approximation: L EMSE σ z = δ δ α S EMSE σ p β δ α S EMSE σ p + o(δ ) Δ = σ Q σ P α: first derivative of MSE Q (σ ) at σ P β: second derivative of MSE Q (σ ) at σ P

20 Taylor Approximations First order approximation: L-EMSE σ z = δ δ α S-EMSE σ p + o(δ) Second order approximation: L EMSE σ z = δ δ α S EMSE σ p β δ α S EMSE σ p + o(δ ) Δ = σ Q σ P α: first derivative of MSE Q (σ ) at σ P β: second derivative of MSE Q (σ ) at σ P

21 Numerical Examples

22 Example 1: Bernoulli-Gaussian input f X x = θ N 0,1 + 1 θ δ 0 (x)

23 Example : Markov Source Two state Markov source with state space {0,1} For non-i.i.d. input (e.g., Markov source) decoupling principal not well-understood Alternative tool: state evolution (SE) of approximate message passing (AMP) [Bayati & Montanari 011]

24 Approximate Message Passing [Donoho et al. 009] AMP solves linear system by iterative decoupling Decoupled scalar channel characterized by state evolution (SE) t SE for AMP with E x i y i coincides with fixed point equation for information theoretic decoupling For non-i.i.d. input E x i window i better than E x i y i t SE for AMP with E x i window i not rigorous but numerically verified [Ma et al. 014] y x σ z z window = +

25 Example : Markov Source Two state Markov source with state space {0,1} W(3): E x i window i with window-size (3) δ σ W3 σ z = MSE W3 σ W3 Y = X + σ W3 Z

26 Example : Markov Source Two state Markov source with state space {0,1} W(3): E x i window i with window-size (3) δ σ W3 σ z = MSE W3 σ W3 Apply W to Y = X + σ W3 Z

27 Example : Markov Source Two state Markov source with state space {0,1} W(3): E x i window i with window-size (3) δ σ W σ z = MSE W σ W δ σ W3 σ z = MSE W3 σ W3 Y = X + σ W3 Z

28 Example : Markov Source Two state Markov source with state space {0,1} W(3): E x i window i with window-size (3) δ σ W3 σ z = MSE W3 σ W3 Y = X + σ W3 Z

29 Discussion

30 Summary Provided expression relating L-EMSE to S-EMSE Two Taylor approximations derived and accuracy verified numerically Next step How does parameter estimation error affect EMSE in linear systems?

31 Thank you!