Structured signal recovery from non-linear and heavy-tailed measurements

Transcription

1 Structured signal recovery from non-linear and heavy-tailed measurements Larry Goldstein* Stanislav Minsker* Xiaohan Wei # *Department of Mathematics # Department of Electrical Engineering UniversityofSouthern California

2 Problem Formulation Single index model: y = f x, θ + δ

3 Problem Formulation Single index model: y = f x, θ + δ x R - random measurement vector;

4 Problem Formulation Single index model: y = f x, θ + δ x R - random measurement vector; δ R random unknown noise independent of x.

5 Problem Formulation Single index model: y = f x, θ + δ x R - random measurement vector; δ R random unknown noise independent of x. θ R - fixed unit unknown vector, d is potentially very large and θ may have certain low dimensional structure.

6 Problem Formulation Single index model: y = f x, θ + δ x R - random measurement vector; δ R random unknown noise independent of x. θ R - fixed unit unknown vector, d is potentially very large and θ may have certain low dimensional structure. f: R R unknown link function.

7 Problem Formulation Single index model: y = f x, θ + δ x R - random measurement vector; δ R random unknown noise independent of x. θ R - fixed unit unknown vector, d is potentially very large and θ may have certain low dimensional structure. f: R R unknown link function. Goal: Estimate θ from i.i.d. sample x J, y J JKL M.

8 Previous Approach If we are only interested in estimating θ, then, solving Least-squares!

9 Previous Approach If we are only interested in estimating θ, then, solving Least-squares! M θn = argmin O P Q y R x J, θ JKL T

10 Previous Approach If we are only interested in estimating θ, then, solving Least-squares! θn = argmin O P Q y R x J, θ x J is Gaussian and Θ R -. M JKL T

11 Previous Approach If we are only interested in estimating θ, then, solving Least-squares! θn = argmin O P Q y R x J, θ x J is Gaussian and Θ R -. M JKL [Brillinger, 1982]; [Plan and Vershynin, 2016]. T

12 Previous Approach Suppose x J are Gaussian and m d, then, with high probability, θn θ T d m

13 Previous Approach Suppose x J are Gaussian and m d, then, with high probability, θn θ T d m Suppose θ is k-sparse, m k log d k, then, with high probability, θn θ T k log (d k) m

14 An example in signal processing:

15 An example in signal processing: One-bit compressed sensing: y J = sign x J, θ, i = 1,2,, m θ is a k-sparse vector, θ T = 1.

16 An example in signal processing: One-bit compressed sensing: y J = sign x J, θ, i = 1,2,, m θ is a k-sparse vector, θ T = 1. Solve: min y R x J, θ T M JKL s.t. θ L θ L

17 An example in signal processing: Geometric intuition: (Picture from

18 An example in signal processing: Geometric intuition: (Picture from

19 Can we always recover θ?

20 Can we always recover θ? The distribution of the measurement vector x is important!

21 Can we always recover θ? The distribution of the measurement vector x is important! Breaks down when x J are Bernoulli 1,1.

22 Can we always recover θ? The distribution of the measurement vector x is important! Breaks down when x J are Bernoulli 1,1. θ L = 1 L 0 0 T θ T = 1 L 0 0 T

23 Can we always recover θ? The distribution of the measurement vector x is important! Breaks down when x J are Bernoulli 1,1. θ L = 1 L 0 0 T θ T = 1 L 0 0 T sign x J, θ L = sign( x J, θ T ) for any x J!

24 Beyond Gaussian distribution

25 Beyond Gaussian distribution Let x N(0, Σ).

26 Beyond Gaussian distribution Let x N(0, Σ). Another way of writing Gaussian: (Σ = BB k ) x = χ T BU.

27 Beyond Gaussian distribution Let x N(0, Σ). Another way of writing Gaussian: (Σ = BB k ) x = χ T BU.

28 Beyond Gaussian distribution Let x N(0, Σ). Another way of writing Gaussian: (Σ = BB k ) x = χ T BU. x = μbu

29 Beyond Gaussian distribution Let x N(0, Σ). Another way of writing Gaussian: (Σ = BB k ) x = χ T BU. x = μbu (centered) Elliptical distribution

30 Why elliptical distribution works? Lemma 1: Let x be elliptical, [Li and Duan, 1991] θ argmin O P E[ y x, θ T ]

31 Finite sample guarantee

32 Finite sample guarantee Difficulty:

33 Finite sample guarantee Difficulty: y = f x, θ + δ

34 Finite sample guarantee Difficulty: y = f x, θ + δ 1. The elliptical distribution of x = μbu can be heavy-tailed.

35 Finite sample guarantee Difficulty: y = f x, θ + δ 1. The elliptical distribution of x = μbu can be heavy-tailed. 2. The noise δ can also be heavy-tailed.

36 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

37 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

38 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

39 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

40 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

41 Goal: Find argmin E[ y x, θ T ] θ T T 2E[y x, θ ] (assuming B = I) θ T T T M y M JKL J x J, θ θ T T T M y M JKL J μ J U J, θ θ T T T M U M JKL J, θ ( μ J y J T) sign(μ J y J )

42 Let q J = ( μ J y J T) sign(μ J y J ) The robust (constrained) estimator: M θn = argmin O P θ T T 2 m Q U J, θ q J The robust (unconstrained) estimator: M JKL θn ƒ = argmin θ T T 2 m Q U J, θ q J JKL + λ θ

43 Theorem 1. Suppose for some κ > 0, Set T = m Œ Ž, E x Tˆ, E δ Tˆ <. Pr θn θ T C L βω Θ m where ω Θ =E š(,œ) sup g, θ. O P,žO Œ Corollary 1: When θ is k-sparse, C T e /T θn θ T (-/ ) M

44 Theorem 1. Suppose for some κ > 0, Set T = m Œ Ž, E x Tˆ, E δ Tˆ <. Pr θn θ T C L βω Θ m where ω Θ =E š(,œ) sup g, θ. O P,žO Œ C T e /T Corollary 1: When θ is k-sparse, with high probability, θn θ T (-/ ) M

45 Theorem 2. Set T = m Œ Ž and λ M, with high probability θn ƒ θ T λ Ψ(S T (ηθ, K)) where G = θ R - : θ 1, and Ψ(S T (ηθ, K)) is a restricted compatibility constant for. Corollary 2: When θ is k-sparse, θn θ T (-) M

46 Theorem 2. Set T = m Œ Ž and λ M, with high probability θn ƒ θ T λ Ψ(S T (ηθ, K)) where G = θ R - : θ 1, and Ψ(S T (ηθ, K)) is a restricted compatibility constant for. Corollary 2: When θ is k-sparse, with high probability, θn ƒ θ T (-) M

47 Corollary 3: When d = d L d T, θ R - Œ -, and θ has rank r, with high probability, and θn θ T r d L + d T m θn ƒ θ T r d L + d T m,,

48 Low computational complexity When θ is k-sparse, the estimator is θ = argmin θ T T 2 b, θ + λ θ L, b = 2 m Q U J q J. A closed form solution: (Soft-thresholding) b J λ 2, b J λ 2, M JKL θ J = 0, λ 2 b J λ 2, b J + λ 2, b J λ 2.

49 Low computational complexity When θ is k-sparse, the estimator is θ± ƒ = argmin θ T T 2 b, θ + λ θ L, b = 1 m Q U J q J. A closed form solution: (Soft-thresholding) b J λ 2, b J λ 2, M JKL θ J = 0, λ 2 b J λ 2, b J + λ 2, b J λ 2.

50 Low computational complexity When θ is k-sparse, the estimator is θ± ƒ = argmin θ T T 2 b, θ + λ θ L, b = 1 m Q U J q J. A closed form solution: (Soft-thresholding) b J λ 2, b J λ 2, M JKL θ J = 0, λ 2 b J λ 2, b J + λ 2, b J λ 2.

51 Simulation: Noiseless One-bit d = 512, m = 128, k = 5 E μ T.L <

52 Simulation: Noisy One-bit (SNR = 10dB) d = 512, m = 128, k = 5 E μ T.L <

53 LS estimator: θ T T 2E[y x, θ ] Markov inequality Truncated LS estimator: θ T T E[ q U, θ ] Generic chaining with customized bounds Empirical estimator: θ T T T M q M JKL J U J, θ

54 LS estimator: θ T T 2E[y x, θ ] Truncated LS estimator: θ T T 2E[ q U, θ ] Empirical estimator: θ T T T M q J U J, θ M JKL

55 LS estimator: θ T T 2E[y x, θ ] Truncated LS estimator: θ T T 2E[ q U, θ ] Empirical estimator: θ T T T M q J U J, θ M JKL

56 LS estimator: θ T T 2E[y x, θ ] Markov inequality Truncated LS estimator: θ T T 2E[ q U, θ ] Generic chaining with customized bounds Empirical estimator: θ T T T M q M JKL J U J, θ

57 Thank you!