Sparse and Low Rank Recovery via Null Space Properties

Transcription

1 Sparse and Low Rank Recovery via Null Space Properties Holger Rauhut Lehrstuhl C für Mathematik (Analysis), RWTH Aachen Convexity, probability and discrete structures, a geometric viewpoint Marne-la-Vallée, October 29, 2015 Based on joint works with Sjoerd Dirksen, Guillaume Lecué; Maryia Kabanava, Richard Kueng, Ulrich Terstiege 1 / 27

2 Overview Sparse and Low Rank Recovery Null space properties Restricted isometry property and its limitations Implication for quantized compressive sensing Null space property for random matrices with finite number of finite moments Low rank matrix recovery from rank-one measurements related to 4-designs 2 / 27

3 Recovery problem Basic problem: Recovery a vector x R N from measurements y = Ax, A R m N, m N. 3 / 27

4 Recovery problem Basic problem: Recovery a vector x R N from measurements y = Ax, A R m N, m N. Sparse recovery: assume that x 0 = #{l : x l 0} s N, i.e., x is s-sparse 3 / 27

5 Recovery problem Basic problem: Recovery a vector x R N from measurements y = Ax, A R m N, m N. Sparse recovery: assume that x 0 = #{l : x l 0} s N, i.e., x is s-sparse Low rank recovery: assume that x = X R n 1 n 2 (N = n 1 n 2 ) with rank(x ) = r min{n 1, n 2 }. In this case, we also write y = A(X ) for A : R n 1 n 2 R m. 3 / 27

6 Approximate sparsity and noise In practice: approximate sparsity / approximately low rank: σ s (x) q := σ r (X ) Sq := inf z: z 0 s x z q inf Z:rank(Z) r X Z S q is small / decays rapidly in s is small / decays rapidly in r Here, X Sp := ( j σ j(x ) p ) 1/p denotes the Schatten p-norm (with σ j (X ) being the singular values of X ); below q = 1 4 / 27

7 Approximate sparsity and noise In practice: approximate sparsity / approximately low rank: σ s (x) q := σ r (X ) Sq := inf z: z 0 s x z q inf Z:rank(Z) r X Z S q is small / decays rapidly in s is small / decays rapidly in r Here, X Sp := ( j σ j(x ) p ) 1/p denotes the Schatten p-norm (with σ j (X ) being the singular values of X ); below q = 1 Noisy measurements y = Ax + e, y = A(X ) + e, with e R m. Assumption e p η for some η 0 and some p [1, ]. 4 / 27

8 Approximate sparsity and noise In practice: approximate sparsity / approximately low rank: σ s (x) q := σ r (X ) Sq := inf z: z 0 s x z q inf Z:rank(Z) r X Z S q is small / decays rapidly in s is small / decays rapidly in r Here, X Sp := ( j σ j(x ) p ) 1/p denotes the Schatten p-norm (with σ j (X ) being the singular values of X ); below q = 1 Noisy measurements y = Ax + e, y = A(X ) + e, with e R m. Assumption e p η for some η 0 and some p [1, ]. p = 2: standard, Gaussian noise p = 1: Poisson noise, doubly exponential noisy p = : quantized compressive sensing 4 / 27

9 Recovery via convex optimization Naive approaches of l 0 -minimization and rank-minimization, min z 0, z:az=y min rank(z) Z:A(Z)=y are NP-hard. 5 / 27

10 Recovery via convex optimization Naive approaches of l 0 -minimization and rank-minimization, min z 0, z:az=y min rank(z) Z:A(Z)=y are NP-hard. Convex relaxations: l p -constrained l 1 -minimization min z R N z 1 subject to Az y p η l p -constrained nuclear norm minimization min Z Z R n 1 n 2 subject to A(Z) y p η Here, Z = X S1 = min{n 1,n 2 } j=1 σ j (X ) Noiseless case: η = 0 5 / 27

11 Null space properties for sparse recovery A R m n satisfies the (stable) null space property (NSP) of order s with constant ρ (0, 1) if v S 1 ρ v S c 1 for all v ker(a)\{0} and all S [n], #S = s. Characterizes exact reconstruction of all s-sparse vectors x R n from y = Ax via l 1 -minimization. 6 / 27

12 Null space properties for sparse recovery A R m n satisfies the (stable) null space property (NSP) of order s with constant ρ (0, 1) if v S 1 ρ v S c 1 for all v ker(a)\{0} and all S [n], #S = s. Characterizes exact reconstruction of all s-sparse vectors x R n from y = Ax via l 1 -minimization. Stability: for y = Ax with x R n and x = arg min z:az=y z 1 : x x 1 2(1 + ρ) 1 ρ σ s(x) 1. 6 / 27

13 Null space properties for sparse recovery A R m n satisfies the (stable) null space property (NSP) of order s with constant ρ (0, 1) if v S 1 ρ v S c 1 for all v ker(a)\{0} and all S [n], #S = s. Characterizes exact reconstruction of all s-sparse vectors x R n from y = Ax via l 1 -minimization. Stability: for y = Ax with x R n and x = arg min z:az=y z 1 : x x 1 2(1 + ρ) 1 ρ σ s(x) 1. Noisy measurements: A R m n satisfies the l p -robust stable NSP (1 p ) of order s with constants ρ (0, 1) and τ > 0 if x S 2 ρ s x S c 1 + τ Ax p for all x R n and all S [n], #S = s. If y = Ax + e with e p η and x = arg min z: Az y p η x 1 then x x 2(1 + ρ)2 σ s (x) 1 τ(3 + ρ) ρ s 1 ρ η. 6 / 27

14 Null space properties for low rank recovery A linear map A : R n 1 n 2 R m satisfies the l p -robust stable rank NSP with constants ρ (0, 1) and τ > 0 if X r F ρ r X X r + τ A(X ) p for all X R n 1 n 2. Here, X r denotes the best rank-r approximation to X, so that X r 2 F = r j=1 σ j(x ) 2 and X X r = min{n 1,n 2 } j=r+1 σ j (X ). 7 / 27

15 Null space properties for low rank recovery A linear map A : R n 1 n 2 R m satisfies the l p -robust stable rank NSP with constants ρ (0, 1) and τ > 0 if X r F ρ r X X r + τ A(X ) p for all X R n 1 n 2. Here, X r denotes the best rank-r approximation to X, so that X r 2 F = r j=1 σ j(x ) 2 and X X r = min{n 1,n 2 } j=r+1 σ j (X ). If y = A(X ) + e for e R m with e p η and X = arg min Z: A(Z) y p η Z, then X X F 2(1 + ρ)2 1 ρ σ r (X ) F τ(3 + ρ) + r 1 ρ η. 7 / 27

16 NSP via Restricted Isometry Property The restricted isometry constant δ s of a matrix A R m n is the smallest number δ such that (1 δ) x 2 2 Ax 2 2 (1 + δ) x 2 2 for all s-sparse x R n. Theorem Suppose A satisfies δ 2s < 1/ 2. Then A satisfies the l p -robust stable NSP of order s for p = 2 and some constants ρ (0, 1) and τ > 0. 8 / 27

17 NSP via Restricted Isometry Property The restricted isometry constant δ s of a matrix A R m n is the smallest number δ such that (1 δ) x 2 2 Ax 2 2 (1 + δ) x 2 2 for all s-sparse x R n. Theorem Suppose A satisfies δ 2s < 1/ 2. Then A satisfies the l p -robust stable NSP of order s for p = 2 and some constants ρ (0, 1) and τ > 0. The (rank) restricted isometry constant δ r of A : R n 1 n 2 R m is the smallest number δ s.t. for all X R n 1 n 2 with rank(x ) r (1 δ) X 2 F A(X ) 2 2 (1 + δ) X 2 F. Theorem Suppose A satisfies δ 2r < 1/ 2. Then A satisfies the l p -robust stable rank NSP of order r for p = 2 and some constants ρ (0, 1) and τ > 0. 8 / 27

18 RIP for subgaussian random measurements A is called subgaussian random matrix if its entries are independent, mean zero, variance one subgaussian random variables. Examples: Gaussian, Bernoulli matrices The map A : R n 1 n 2 R m is called subgaussian if in A(X ) j = k,l A j,k,lx k,l the entries A l,j,k are independent, mean, zero, variance one subgaussian random variables. Theorem Let A R m n be a random draw of a subgaussian matrix and A : R n 1 n 2 R m be a random draw of a subgaussian random measurement map. Further, let δ, ε (0, 1). (a) If m Cδ 2 (s log(en/s) + log(ε 1 )) then A satisfies δ s δ with probability at least 1 ε. (b) (Fazel, Parrilo, Recht 09; Candès, Plan 11) If m Cδ 2 (r(n 1 + n 2 ) + log(ε 1 )) then A satisfies δ r δ with probability at least 1 ε. 9 / 27

19 RIP for subgaussian random measurements A is called subgaussian random matrix if its entries are independent, mean zero, variance one subgaussian random variables. Examples: Gaussian, Bernoulli matrices The map A : R n 1 n 2 R m is called subgaussian if in A(X ) j = k,l A j,k,lx k,l the entries A l,j,k are independent, mean, zero, variance one subgaussian random variables. Theorem Let A R m n be a random draw of a subgaussian matrix and A : R n 1 n 2 R m be a random draw of a subgaussian random measurement map. Further, let δ, ε (0, 1). (a) If m Cδ 2 (s log(en/s) + log(ε 1 )) then A satisfies δ s δ with probability at least 1 ε. (b) (Fazel, Parrilo, Recht 09; Candès, Plan 11) If m Cδ 2 (r(n 1 + n 2 ) + log(ε 1 )) then A satisfies δ r δ with probability at least 1 ε. Sparse recovery from m s log(en/s) random measurements Low rank recovery from m r(n 1 + n 2 ) random measurements 9 / 27

20 Questions What about noisy measurements with error bound in l p for p 2? Guarantees for reconstruction via min z z 1 subject to Az y p η? Can we relax the subgaussian assumption on the distribution of the entries of A, A? 10 / 27

21 Questions What about noisy measurements with error bound in l p for p 2? Guarantees for reconstruction via min z z 1 subject to Az y p η? Can we relax the subgaussian assumption on the distribution of the entries of A, A? Analyze the l p -robust NSP directly! 10 / 27

22 l p -constrained l 1 -minimization via RIP Previous approaches: RIP p,2 : (Jacques, Hammond, Fadili 2011) A satisfies RIP p,2 (p 2) of order s if (1 δ) 1/2 x 2 Ax p (1 + δ) 1/2 x 2 for all s-sparse x. RIP p,2 (p 2) of order 3s for sufficiently small δ implies error bound for x = arg min z: Az Ax p η x 1 : x x 2 C σ s(x) 1 s + D p η. 11 / 27

23 l p -constrained l 1 -minimization via RIP Previous approaches: RIP p,2 : (Jacques, Hammond, Fadili 2011) A satisfies RIP p,2 (p 2) of order s if (1 δ) 1/2 x 2 Ax p (1 + δ) 1/2 x 2 for all s-sparse x. RIP p,2 (p 2) of order 3s for sufficiently small δ implies error bound for x = arg min z: Az Ax p η x 1 : x x 2 C σ s(x) 1 s + D p η. (Rescaled) Gaussian matrix A R m n satisfies RIP p,2 of order s if m C ( δ 2 s log(en/(δs)) + δ 2 log(η 1 ) ) p/2 + (p 1)2 p 11 / 27

24 l p -constrained l 1 -minimization via RIP Previous approaches: RIP p,2 : (Jacques, Hammond, Fadili 2011) A satisfies RIP p,2 (p 2) of order s if (1 δ) 1/2 x 2 Ax p (1 + δ) 1/2 x 2 for all s-sparse x. RIP p,2 (p 2) of order 3s for sufficiently small δ implies error bound for x = arg min z: Az Ax p η x 1 : x x 2 C σ s(x) 1 s + D p η. (Rescaled) Gaussian matrix A R m n satisfies RIP p,2 of order s if m C ( δ 2 s log(en/(δs)) + δ 2 log(η 1 ) ) p/2 + (p 1)2 p Lower bound for RIP p,2 (p 2): m s p/2 11 / 27

25 RIP p,p (Allen-Zhu, Gelashvili, Razenshteyn 2014): A satisfies RIP p,p (p 2) of order s if (1 δ) x p p Ax p p (1 + δ) x p p for all s-sparse x. Implies recovery guarantee for x = arg min z: Az y p η x 1 of the form x x p C σ s(x) 1 + Dη. s1 1/p Lower bound: m C p s p for p 2 Upper bounds for adjacency matrices of random bipartite graphs 12 / 27

26 Towards analyzing the NSP Recall NSP x S 2 ρ s x S c 1 + Ax p for all x R n 13 / 27

27 Towards analyzing the NSP Recall NSP x S 2 ρ x S c 1 + Ax p s for all x R n Introduce T ρ,s = {x R n : S [n], #S = s : x S 2 ρ x S c 1 } s Lemma If A satisfies inf x T ρ,s, x 2 =1 Ax p 1 τ > 0 then A satisfies the l p -robust NSP of order s with constants ρ (0, 1) and τ > / 27

28 Low rank recovery case T ρ,r = {X R n 1 n 2 : X r F r ρ } X X r Lemma If A : R n 1 n 2 R m satisfies inf A(X ) p 1 X T ρ,s, X F =1 τ > 0 then A satisfies the l p -robust NSP of order s with constants ρ (0, 1) and τ > / 27

29 B n 2 := {x R n : x 2 1}, B n 1 n 2 F := {X R n 1 n 2 : X F 1} Lemma It holds (a) (b) T ρ,s B n 2 (2 + ρ 1 ) conv{x R n : x 2 = 1, x 0 s}, T ρ,r B n 1 n 2 F (2 + ρ 1 ) conv{x R n 1 n 2 : X F = 1, rank(x ) r} 15 / 27

30 B n 2 := {x R n : x 2 1}, B n 1 n 2 F := {X R n 1 n 2 : X F 1} Lemma It holds (a) (b) T ρ,s B n 2 (2 + ρ 1 ) conv{x R n : x 2 = 1, x 0 s}, T ρ,r B n 1 n 2 F (2 + ρ 1 ) conv{x R n 1 n 2 : X F = 1, rank(x ) r} Consequence: Need to show a lower bound over a slightly larger set than for the lower bound in the restricted isometry property. Advantage: No need to show an upper bound 15 / 27

31 Main tool: Mendelson s small ball method Lemma (Koltchinskii, Mendelson 13) For 1 p < and a class F of functions from R n into R introduce Q F (u) = inf P( f (X ) u) f F and R m (F) = E sup 1 f F m m ε i f (X i ), where (ε i ) i 1 is a Rademacher sequence. Let u > 0 and t > 0, then, with probability at least 1 2e 2t2, inf f F 1 m m i=1 i=1 f (X i ) p u p( Q F (2u) 4 u R m(f) t m ). (Koltchinskii, Mendelson 13 proved this for p = 2, but extension to arbitrary p is immediate.) 16 / 27

32 Main result: sparse recovery Theorem (Dirksen, Lecué, R 15) Fix 1 p and let A be an m n random matrix with independent, mean-zero, variance one, i.i.d. entries A ij which satisfy (E A i,j r ) 1/r λr α for all 2 r log(n) for some λ > 0 and α 1/2 and P( A i,, x u ) β for all x S n 1 for some u, β > 0. Let x R n, y = Ax + e, e p η and x = arg min z: Az y p η x 1. If { λ 2 e 4α 2 m C max u β 2 2 s log(en/s), log(ε 1 ) β 2, (log(n)) 2α 1}, then with probability at least 1 η, for 1 q 2, x x σ s (x) 1 C 2 C 1 2 η + s β 1/p u m. 1/p Lecué, Mendelson 14: p = 2 (and worse exponent at log(n)-term) 17 / 27

33 Distributions satisfying assumptions of the theorem Gaussian and more generally subgaussian random variables 18 / 27

34 Distributions satisfying assumptions of the theorem Gaussian and more generally subgaussian random variables Exponential random variables (see also Adamczak, Lata la, Litvak, Pajor, Tomczak-Jaegermann 11; Foucart 14) 18 / 27

35 Distributions satisfying assumptions of the theorem Gaussian and more generally subgaussian random variables Exponential random variables (see also Adamczak, Lata la, Litvak, Pajor, Tomczak-Jaegermann 11; Foucart 14) Random variables with density p(x) = γ 1 2γ min{1, x γ }, x R, for γ max{log(n) + 2, 6}. 18 / 27

36 Distributions satisfying assumptions of the theorem Gaussian and more generally subgaussian random variables Exponential random variables (see also Adamczak, Lata la, Litvak, Pajor, Tomczak-Jaegermann 11; Foucart 14) Random variables with density p(x) = γ 1 2γ min{1, x γ }, x R, for γ max{log(n) + 2, 6}. Independence of entries can be relaxed to independence of rows Theorem holds true if rows are samples of an isotropic, unconditional, log-concave vector on R n 18 / 27

37 Application to quantized compressive sensing In practice, measurements are often subject to quantization. Uniform quantizer with bin width θ (rounding): Q θ (v) = θ v/θ + θ/2, v R 19 / 27

38 Application to quantized compressive sensing In practice, measurements are often subject to quantization. Uniform quantizer with bin width θ (rounding): Q θ (v) = θ v/θ + θ/2, v R Observations y = Q θ (Ax) = (Q θ ((Ax) i )) i = Ax + e with e = Ax Q θ (Ax) so that e θ/2. 19 / 27

39 Application to quantized compressive sensing In practice, measurements are often subject to quantization. Uniform quantizer with bin width θ (rounding): Q θ (v) = θ v/θ + θ/2, v R Observations y = Q θ (Ax) = (Q θ ((Ax) i )) i = Ax + e with e = Ax Q θ (Ax) so that e θ/2. Desired: Quantization consistency for a reconstruction x from y: Q θ (Ax) = Q θ (Ax ) 19 / 27

40 Reconstruction Reconstruction of x from y = Q θ (Ax) via l -constrained l 1 -minimization, min z 1 subject to Az y θ/2. is quantization consistent! 20 / 27

41 Reconstruction Reconstruction of x from y = Q θ (Ax) via l -constrained l 1 -minimization, min z 1 subject to Az y θ/2. is quantization consistent! For Gaussian random matrices (for instance), the reconstruction satisfies x x 2 Cs 1/2 σ s (x) 1 + Dθ with high probability if m Cs log(en/s). (Extends to other random matrices.) 20 / 27

42 Reconstruction Reconstruction of x from y = Q θ (Ax) via l -constrained l 1 -minimization, min z 1 subject to Az y θ/2. is quantization consistent! For Gaussian random matrices (for instance), the reconstruction satisfies x x 2 Cs 1/2 σ s (x) 1 + Dθ with high probability if m Cs log(en/s). (Extends to other random matrices.) In contrast, l p -constrained l 1 -minimization is not quantization consistent for p <. 20 / 27

43 Main result: low rank recovery Theorem (Kabanva, Kueng, R, Terstiege 15) Fix 1 p. Let A : R n1 n2 R m with independent mean zero, variance one entries satisfying E A j,k,l 4 D for all j, k, l. For 0 < ρ < 1 and r min{n 1, n 2 } assume that m c 1 ρ 2 r(n 1 + n 2 ). Then with probability at least 1 e c2m, for any X R n1 n2 y = A(X ) + e with e p η the solution X of and min Z Z R n 1 n 2 subject to A(Z) y p η satisfies X X F 2(1 + ρ)2 (1 ρ) σ r (X ) r ρ η 1 ρ m. 1/p Error bound also available for S q -norms, 1 q / 27

44 Quantum state tomography from rank one measurements The state of a (finite-dimensional) quantum system is described by symmetric positive semidefinite matrix A C n n with tr A = 1. Quantum measurements are often with respect to rank one matrices y j = A(X ) j := a j Xa j = tr(xa j a j ), j = 1,..., m Pure states: rank(x ) = 1 mixed states: rank(x ) = r n 22 / 27

45 Quantum state tomography from rank one measurements The state of a (finite-dimensional) quantum system is described by symmetric positive semidefinite matrix A C n n with tr A = 1. Quantum measurements are often with respect to rank one matrices y j = A(X ) j := a j Xa j = tr(xa j a j ), j = 1,..., m Pure states: rank(x ) = 1 mixed states: rank(x ) = r n Special case: Phase retrieval via PhaseLift: Phaseless measurements of x C n y i = a j, x 2 22 / 27

46 Quantum state tomography from rank one measurements The state of a (finite-dimensional) quantum system is described by symmetric positive semidefinite matrix A C n n with tr A = 1. Quantum measurements are often with respect to rank one matrices y j = A(X ) j := a j Xa j = tr(xa j a j ), j = 1,..., m Pure states: rank(x ) = 1 mixed states: rank(x ) = r n Special case: Phase retrieval via PhaseLift: Phaseless measurements of x C n y i = a j, x 2 = a j xx a j = tr(xx a j a j ) can be interpreted as rank-one measurements of the rank one matrix X = xx C n n. 22 / 27

47 Recovery from Gaussian rank one measurements Theorem (Kueng, R, Terstiege 14; Kabanava, Kueng, R, Terstiege 15) Let A : C n n C m be generated by a sequence a j C n, j = 1,..., m of independent standard Gaussian random vectors and assume that m Cρ 2 rn. Then with probability at least 1 e cm, for any Hermitian matrix X C n n, y = A(X ) + e, e p η, and X = min Z: A(Z) y p η Z, it holds X X F C ρ σ r (X ) r + D ρ η m 1/p. Proof via Mendelson s small ball method 23 / 27

48 Towards real quantum experiments: t-designs Definition A weighted set {p i, w i } N i=1 of vectors with w i 2 = 1 is called an approximate t-design of p-norm accuracy θ p, if N ( ) p i (w i wi ) t (ww ) t n + t 1 1 dw θ p. w l2 =1 t p i=1 If θ p = 0 then {p i, w i } N i=1 is called exact design. Certain constructions of approximate t-designs can be implemented efficiently in quantum experiments (in principle?). 24 / 27

49 Recovery with 4-designs Theorem (Kueng, R, Terstiege 14; Kabanava, Kueng, R, Terstiege 15) Let {p i, w i } N i=1 be a an approximate 4-design with either θ 1/(16r 2 ), or θ 1 1/4 that furthermore obeys N i=1 p iw i wi 1 n I 1 n. Suppose that the measurement operator A is generated by m C 4 ρ 2 nr log n measurement matrices A j = n(n + 1)a j a j, where each a j is drawn independently from {p i, w i } N i=1. Then, with probability at least 1 e C5m, for every Hermitian matrix X C n n, measurements y = A(X ) + e = (tr(a j X )) j + e with e p η and X = arg min Z: A(Z) y p η Z it holds X X F C ρ σ r (X ) r + D ρ η m 1/p. Improves and generalizes previous result for phase retrieval by Gross, Krahmer, Kueng / 27

50 Positive semidefinite case In the positive semidefinite case and rank one measurements, we can replace nuclear norm minimization min tr(z) Z 0 subject to A(Z) y 2 η by positive semidefinite least squares min A(Z) y 2. Z 0 Advantage: No estimate of noise level η is required. 26 / 27

51 Thank you! Questions? 27 / 27