Solving Corrupted Quadratic Equations, Provably

Transcription

1 Solving Corrupted Quadratic Equations, Provably Yuejie Chi London Workshop on Sparse Signal Processing September 206

2 Acknowledgement Joint work with Yuanxin Li (OSU), Huishuai Zhuang (Syracuse) and Yingbin Liang (Syracuse). Research supported by NSF, AFOSR and ONR.

3 Estimation of Low-rank PSD Matrices Consider estimation of a low-rank positive-semidefinite (PSD) matrix X R n n from symmetric rank-one measurements: y i = a i a T i, X = a T i Xa i, i =,..., m. The measurements are nonnegative since X 0. If rank(x) = r, decompose X as X = UU T, where U R n r, then the measurements are quadratic in U: The rank r may be unknown. y i = a T i UU T a i = U T a i 2 2. Goal: recover X or U from as a small number of measurements. Related to low-rank matrix recovery but more structured/restricted. 2

4 Application - phase retrieval Quadratic measurements arise in optical applications such as phase retrieval, namely, recover x R n /C n from y i = a i, x 2 = a i (xx )a i, i =,..., m. Figure : A typical setup for structured illuminations in diffraction imaging using a phase mask. E. J. Candès, Y. C. Eldar, T. Strohmer and V. Voroninski, Phase retrieval via matrix completion, SIAM J. on Imaging Sciences. 3

5 Application - projection retrieval Quadratic measurements arise in the problem of projection (or subspace) retrieval via energy measurements, namely, recover a subspace U R n r from y i = U T a i 2 2 = a T i (UU T )a i, i =,..., m. a j a i ku T a j k ku T a i k U Useful in SAR imaging. M. Fickus and D. Mixon, Projection Retrieval: Theory and Algorithms, SAMPTA

6 Application - covariance sketching Question: how to sketch a high-dimensional data stream in order to recover its covariance matrix? Consider a data stream possible distributively observed at m sensors: Quadratic Sketching: For each sketching vector a i R n with i.i.d. sub-gaussian entries, i =,..., m: Sketch a substream indexed by {l i t} T t= with a i, x l i t 2 and compute the average: ) y i,t = T ( T 2 a i, x l i t = a T i T t= T x l i t x T l i t t= where X = E[xx T ] is approximately low-rank. a i T a T i Xa i, Y. Chen, Y. Chi, and A. J. Goldsmith. Exact and stable covariance estimation from quadratic sampling via convex programming. IEEE Trans. on IT,

7 Near-optimal recovery via convex programming When a i s are composed of i.i.d. Gaussian entries, we aim to recover X using the following trace minimization algorithm: ˆX = argmin Tr(M) subject to y i = a T i Ma i, i =,..., m. M 0 Theorem (Chen, Chi and Goldsmith, 203) With probability exceeding c exp( c 2 m), the solution ˆX exactly recovers all rank-r matrices X, provided that m > c 0 nr, where c 0, c, c 2 are universal constants. Exact recovery with m = O(nr); Robust against approximate low-rankness and bounded noise. r/n information theoretic limit similar guarantees also hold for the sub-gaussian case m / (n*n) 0 6

8 What about outliers? Outliers happen with sensor failures, malicious attacks, and missing data; For covariance sketching, insufficiently aggregated sketches can be regarded as an outlier; In this talk, we re interested when the measurements are corrupted by both sparse outliers and bounded noise: or equivalently y i = a T i Xa i + η i + w i, i =,..., m, y = A (X) + η + w, where η is a sparse vector with η 0 sm and w is a dense bounded noise. 7

9 Pursuit of outlier-robust algorithms Previous approaches are sensitive to outliers. Goal: algorithms that are oblivious to outliers, i.e. perform equally well with or without outliers, and without any special treatments of outliers. And also statistically and computationally efficient. small sample size: hopefully m is linear in n; large fraction of outliers: hopefully s is a small constant; low computational complexity and easy to implement. We will outline two approaches, based on convex and non-convex optimization respectively. 8

10 Outlier-robust recovery by convex programming To motivate, ideally one would like to look for low-rank matrices that maintain outlier sparsity: ˆX = argmin y A(M) 0, s.t. rank(m) = r M 0 By relaxing the objective function to the l -norm minimization, and dropping the rank constraint, ˆX = argmin y A(M) M 0 We call this algorithm l -regularized Phaselift, or Phaselift-l. Parameter-free formulation without trace minimization or tuning parameters; No prior information is required for the matrix rank, corruption level or bounded noise level. 9

11 Performance guarantee of Phaselift-l Theorem (Li, Sun and Chi, 206) Suppose that w ɛ. Assume the support of η is selected uniformly at random with the signs of η are generated from a symmetric Bernoulli distribution. Then for a fixed rank-r PSD matrix X R n n, there exist some absolute constants C > 0 and 0 < s 0 < such that as long as m > C nr 2, s s 0 /r, the solution to the proposed algorithm satisfies ˆX rɛ X C 2 F m, with probability exceeding e γm/r2 for some constants C 2 and γ. Proof by dual certificate construction. Exact recovery when w = 0 as long as m nr 2 and s /r. When r = we obtain near-optimal guarantee, which recovers the result by Hand for the phase retrieval case ; P. Hand, Phaselift is robust to a constant fraction of arbitrary errors. 0

12 Numerical Performance: Outlier robustness Rank (r) Rank (r) Number of measurements (m) Percent of outliers (s) 0 (a) Fix s, r vs m (b) Fix m, r vs s Figure: Phase transitions of PSD matrix recovery with respect to (a) the number of measurements and the rank, with 5% of measurements corrupted by arbitrary standard Gaussian variables; (b) the percent of outliers and the rank, when the number of measurements is m = 400, where n = 40.

13 2 Robust recovery of Toeplitz PSD Matrices If X is additionally Toeplitz, this can be incorporated: ˆX = argmin y A(M). M 0, Toeplitz Rank (r) Number of measurements (m) Figure: Phase transitions of low-rank Toeplitz PSD matrix recovery w.r.t. the number of measurements and the rank with 5% of measurements corrupted by standard Gaussian variables, when n = 64. 0

14 3 Non-convex approach based on factored model Can we reduce the computational complexity? If rank(x) is known a priori as r, using the Cholesky factorization X = UU T where U R n r, one can directly recover U: Û = argmin l(u) := argmin U R n r U R m n r m l(y i ; U) i= for some loss function l(y i, U): ( quadratic loss of power: l(u; y i) = y i ) U T a i quadratic loss of amplitude: l(u; y i) = ( yi ) U T 2 a 2 i Poisson loss: l(u; y i) = U T a i 2 2 y i log U T a i 2 2 What are the challenges? l(u) can be non-convex and non-smooth. With outliers, we want the loss to sum over only clean samples.

15 4 Non-convex phase retrieval Rank- case (phase retrieval): y i = a i, x 2 + η i + w i, i =,..., m where η 0 = s m is the outliers, w is the additive noise. Exciting developments (without outliers) all following the same recipe: Initialize z (0) via the (truncated) spectral method to land in the neighborhood of the ground truth; Iterative update using (truncated) gradient descent; Provable near-optimal performance for Gaussian measurement model: Statistically: m = O(n) near-optimal sample complexity Computationally: geometric convergence with near-linear run time Examples: Wirtinger Flow (WF) (Candès et.al. 204), Truncated Wirtinger Flow (TWF) (Chen and Candès 205), Reshaped Wirtinger Flow (Zhang and Liang 206), Truncated Amplitude Flow (Wang, Giannakis and Eldar, 206)

16 Non-convex phase retrieval with outliers In the presence of arbitrary outliers, existing approaches fail: Spectral initialization would fail: the eigenvector of Y can be arbitrarily perturbed Y = m y i a i a T i or Y = m y i a i a T i { yi α m m y mean({y i})}. i= i= }{{}}{{} WF TWF Gradient descent would fail: the search direction can be arbitrarily perturbed z (t+) = z (t) µ l(z (t) ; y z (0) 2 i ) i T t where T t = {,..., m} for WF and { } T t = i : y i a T i z (t) 2 α h mean({ y i a T i z (t) 2 }) for TWF. with some details hiding 5

17 6 Robust phase retrieval via median-truncation Need better strategy to eliminate outliers! Key approach: median-truncation well-known in robust statistics to be outlier-resilient; little appearance in high-dimensional estimation; Median is more stable than mean and top-k truncation (which truncates a fixed amount of samples) for various levels of outliers. 2 =0, s=0. 2 = x 2, s=0. 2 =20 x 2, s= mean threshold median threshold mean threshold median threshold mean threshold median threshold top-k threshold top-k threshold top-k threshold 0 Measurements 0 Measurements 0 Measurements no outliers small outlier magnitudes large outlier magnitudes

18 7 Median-Truncated Wirtinger Flow (median-twf) We adopt the Poisson loss function (other loss functions work too) and the Gaussian measurement model. Median-truncated spectral initialization: Set z (0) := λ 0 z where Direction estimation: z is the leading eigenvector of Y = m m y ia ia T i { yi 9/0.455 median({y i })}. i= Norm estimation: λ 0 = median({y i})/0.455 y i = a T i x 2 χ 2 and E[median(χ 2 )] = As long as m = O(n log n) and s = O(), the initialization is provably close to the ground truth: dist(z (0), x) 0 x, where dist(z (0), x) = min{ z (0) + x, z (0) x }.

19 8 Median-Truncated Wirtinger Flow (median-twf) Median-truncated gradient descent: z (t+) = z (t) 2µ m where { } E = i : 0.3 at i z (t) z (t) 5, E 2 = i z(t) 2 y i at a T i E E 2 i z(t) a i } {{ } l tr(z) { } i : r (t) i 2 at i z (t) z (t) median({r(t) i }), with r (t) i = y i (a T i z(t) ) 2. As long as m = O(n log n) and s = O(), l tr (z) satisfies the Regularity Condition RC(µ, λ) for all z, h = z x: m l tr(z), h µ m l tr(z) 2 + λ h 2, h 0 z. which guarantees dist(z (t+), x) ( µλ)dist(z (t), x).,

20 9 Performance guarantee of median-twf Theorem (Zhang, Chi and Liang, 206) Consider the model y i = a i, x 2 + w i + η i, where w c x 2 and η 0 sm. If m c 0 n log n and s < s 0, then with prob. c exp( c 2m), median-twf yields dist(z (t), x) w x + ( ρ)t x, t N simultaneously for all x R n \{0} and some constants c 0, c, c 2 > 0 and 0 < ρ <. Exact recovery when w = 0 with slight more samples (m = O(n log n)) but a constant fraction of outliers s = O(). Stable recovery with additional bounded noise; Resist outliers obliviously: no prior knowledge of outliers. First non-asymptotic robust recovery guarantee using median: much more involved due to the nonlinearity of median.

21 20 Proof sketch Definition (Generalized quantile function) Let 0 < p <. If F is a CDF, the generalized quantile function is F (p) = inf{x R : F (x) p}. Denote θ p (F ) := F (p) and θ p ({X i }) := θ p ( ˆF ), where ˆF is the empirical distribution of the samples {X i } m i=. Concentration of sample quantile: Assume {X i } m i= are i.i.d. drawn from some distribution F. Under some minor assumptions, w.h.p. θ p ({X i } m i=) θ p (F ) < ɛ Bound median by quantiles of clean samples: Consider clean samples { X i } m i= and contaminated samples {X i} m i=. Then θ 2 s ({ X i }) θ 2 ({X i}) θ 2 +s ({ X i }).

22 2 Proof sketch Lemma (Concentration of median) If m > c 0 n log n, then with probability at least c exp( c 2 m), there exist constants β and β such that β z h median( { a T i x 2 a T i z 2 } m i= ) β z h, holds for all z, h := z x satisfying h < / z. A similar property is established for the mean when m = O(n); here we lose a factor of log n due to working with the median.

23 Success rate Success rate Success rate Success rate Numerical experiments with median-twf median-twf median-twf 0.8 trimean-twf TWF 0.8 trimean-twf TWF Outliers fraction s (a) η = 0. x Outliers fraction s (b) η = x median TWF trimean TWF TWF median-twf trimean-twf TWF Outliers fraction s (c) η = 0 x Outliers fraction s (d) η = 00 x 2 Figure: Success rate of exact recovery with outliers for median-twf, trimean-twf, and TWF at different levels of outlier magnitudes. 22

24 Relative error 23 Numerical experiments with median-twf Recovery with both dense noise and sparse outliers: With outliers, median-twf achieve better accuracy than TWF. Moreover, median-twf with outliers achieves almost the same accuracy of TWF without outliers median-twf with outliers TWF with outliers TWF without outliers Iterations Figure: Relative error of median-twf vs. TWF w.r.t. iteration when s = 0., w = 0.0 x 2, and η = w.

25 24 Conclusion We have discussed two approaches for combating outliers: Convex optimization based on PhaseLift-l : ˆX = argmin y A(X) X 0 Non-convex optimization based on median-twf: z (t+) = z (t) µ m l(y i, z (t) ) Ei m No prior knowledge of outliers are required: we can run these algorithms as if outliers do not exist; Exact recovery guarantees for Gaussian measurement model are obtained, even with a constant proportion of arbitrary outliers; Stability against additional bounded noise. Work in progress: extending median-twf to the low-rank setting. i=

26 25 References. Outlier-Robust Recovery of Low-rank Positive Semidefinite Matrices From Magnitude Measurements, ICASSP Provable Non-convex Phase Retrieval with Outliers: Median Truncated Wirtinger Flow, ICML 206