Invertibility of random matrices

Transcription

1 University of Michigan February 2011, Princeton University

2 Origins of Random Matrix Theory Statistics (Wishart matrices) PCA of a multivariate Gaussian distribution. [Gaël Varoquaux s blog gael-varoquaux.info] Physics (Wigner matrices) Slow neutron resonance on thorium 232 and uranium 238 nuclei [Rahn et al, Phys. Rev. C 6 (1972), 1854]

3 Statistics: covariance estimation Basic problem in statistics: estimate the covariance matrix Σ = EXX T of a high-dimensional distribution; X is the random vector. What for? Principal Component Analysis (PCA): detect the principal axes along which most dependence occurs: PCA of a multivariate Gaussian distribution. [Gaël Varoquaux s blog gael-varoquaux.info]

4 Statistics: covariance estimation Unbiased estimator of Σ is the sample covariance matrix Σ N = 1 N N X k X T k k=1 obtained from N independent samples X k. Σ N is a random matrix, called Wishart matrix after John Wishart (1928). The origin of random matrix theory.

5 Statistics: covariance estimation Covariance Estimation Problem. Determine the minimal sample size N = N(n) that guarantees with high probability (say, 0.99) that the sample covariance matrix Σ N estimates the actual covariance matrix Σ of an n-dimensional distribution with fixed accuracy (say, ε = 0.01) in the operator norm: Σ n Σ ε Σ. PCA of a multivariate Gaussian distribution. [Gaël Varoquaux s blog gael-varoquaux.info]

6 Statistics: covariance estimation Theorem (follows from Rudelson 99) For general distributions supported in a ball of radius O( n) in R n, the optimal sample size for covariance estimation is N n log n. Theorem can be proved by noting that Σ n = 1 N N k=1 X kx T k is a sum of independent random matrices X k X T k, and applying non-commutative Khinchine or Bernstein inequalities. Problem. Describe the distributions for which log n is needed. Not needed for log-concave distributions (Adamczak, Litvak, Pajor, Tomczak 10). Conjecture: not needed under even mild moment conditions, e.g. (2 + ε)th moment. True for (4 + ε)th moment with log log n oversampling (V 10).

7 Statistics: structured covariance estimation In modern applications, smaller sample sizes are desirable, N n (cf. compressed sensing). This is possible in presence of structure. Low Rank Theorem (still follows from Rudelson 99) Suppose a distribution is supported in a ball of radius O( n) in R n and Σ is approximately of low rank k. Then the optimal sample size for covariance estimation is Sparse Theorem (Levina-V. 10) N k log n. Consider a Gaussian distribution in R n, whose covariance matrix Σ is sparse, having k nonzero entries per row, whose locations are known. Then the optimal sample size for covariance estimation is N k log 6 n. Optimal result should be N k log(n/k). General distributions?

8 Statistics: structured covariance estimation Sparse Theorem (Levina-V. 10) Consider a Gaussian distribution in R n, whose covariance matrix Σ is sparse, having k nonzero entries per row, whose locations are known. Then the optimal sample size for covariance estimation is N k log 6 n. Planted Clique Problem. What if sparsity pattern is not known? For example, an adversary puts entries 1/k in some k k minor of Σ ( clique ); all other entries are zero. Note that Σ = 1. What is the sample size needed to determine the location of the clique? Conjectured: N = O (k); existing technique gives N = O (k 2 ).

9 Connection with random matrix theory The sample covariance matrix of an n-dimensional distribution Σ N = 1 N N X k X T k is a random matrix an n n Wishart matrix. Suppose for simplicity that the actual (population) covariance matrix Σ = EXX T equals identity ( isotropic distribution). Then the desired estimation Σ N I ε is equivalent to saying that all eigenvalues of Σ N are concentrated around 1: i=1 Marchenko-Pastur density General problem: describe the distribution of the extreme eigenvalues of of random matrices (hard edge and soft edge).

10 Extreme eigenvalues of random matrices Model: H is an N n matrix with iid entries, zero mean, unit variance, sub-gaussian moments. The singular values λ k (H) are the eigenvalues of H T H. Bai-Yin law: as N, n with N/n const, λ min (H) N n, λ max (H) N + n. Moreover, the limiting distribution of λ min (H), λ max (H) properly normalized is the Tracy-Widom distribution. (Tracy-Widom 94, Soshnikov 02, Feldheim-Sodin 10.) Non-asymptotic versions (Rudelson-V 09): { P λ max (H) t( N + } n) 2e ct2 N (standard); { P λ min (H) ε( N } n) (Cε) N n+1 + c N. Also (Feldheim-Sodin 10).

11 Invertibility problem Determining the hard edge is most difficult for square random matrices H. They correspond to the phase transition from underdetermined to overdetermined linear systems. The edges determine the spectral norm of H and the inverse: 1/λ min (H) = H 1, λ max (H) = H. Invertibility Problem for square random matrices H: (a) What is the singularity probability for H? (b) What is the typical value of H 1? Applications (von Neumann-Goldstine 47): test numerical linear solvers on random inputs H. Average-case analysis. One needs to know: (a) how often H is singular; (b) what is the typical condition number κ(h) = λ max (H)/λ min (H).

12 Invertibility problem: iid entries Invertibility Problem for n n random matrices H: (a) What is the singularity probability p n for H? (b) What is the typical value of H 1? Matrices H with iid entries ( Wigner matrices ). Examples: Gaussian matrices (GUE) with N(0, 1) entries, Bernoulli matrix with ±1 entries. (a) for Bernoulli: p n 0 as n (Komlos 68). Exponentially small: p n c n (Kahn-Komlos-Szemeredi 95). Conjecture: p n = ( o(1))n. Best known: p n = ( o(1)) n (Bourgain, Vu, Wood 10). (b): H 1 n with high probability (Edelman 88, Szarek 90 for Gaussian, Rudelson-V. 08 for general): { P λ min (H) ε/ } n Cε + c n.

13 Invertibility problem: symmetric matrices Invertibility Problem for n n random matrices H: (a) What is the singularity probability p n for H? (b) What is the typical value of H 1? Symmetric matrices H with iid above-diagonal entries; diagonal arbitrary fixed. (n n Wigner matrices). (a) for symmetric Bernoulli: p n n 1/8 (Costello-Tao-Vu 06). Conjecture same as for iid entries: p n = ( o(1))n. (b): H 1 n with high probability, for continuous distributions (Erdös-Schlein-Tau 10). Universality: if the first four moments of H ij are the same as for Gaussian (Tao-Vu 10). Not readily applied for Bernoulli. New results: p n exp( n const ) and H 1 n for general symmetric matrices (V 11):

14 Invertibility problem: symmetric matrices Theorem (Invertibility of symmetric random matrices, V 11) Let H be a symmetric random matrix, whose above-diagonal entries are iid with mean zero, unit variance, and finite subgaussian moments. Then for every z R, { P min λ k (H) z ε/ } n Cε 1/9 + exp( n c ). k This implies delocalization of the spectrum. Eigenvalues miss intervals of length 1/ n (average spacing); do not stick to any point with high probability 1 exp( n c ): Controls Green s function (H zi ) 1 = 1/ min λ k (H) z : (H zi ) 1 n with high probability.

15 Invertibility problem: symmetric matrices Theorem (Invertibility of symmetric random matrices, V 11) Let H be a symmetric random matrix, whose above-diagonal entries are iid with mean zero, unit variance, and finite subgaussian moments. Then for every z R, { P min λ k (H) z ε/ } n Cε 1/9 + exp( n c ). k For continuous distributions, the singularity probability is 0: { P min λ k (H) z ε/ } n Cε. k (Erdös-Schlein-Tau 10). Independent simultaneous result (Nguyen 11): for every B > 0 there exists A > 0: { P min k λ k (H) z n A} n B.

16 Proof of Invertibility Theorem Theorem (Invertibility of symmetric random matrices, V 11) Let H be a symmetric random matrix, whose above-diagonal entries are iid with mean zero, unit variance, and finite subgaussian moments. Then for every z R, { P min λ k (H) z ε/ } n Cε 1/9 + exp( n c ). k For simplicity, assume z = 0. Variational characterization: min λ k (H) = min Hx. k n 1 x S So we need, with high probability, a uniform lower bound Hx n 1/2 for all vectors x on the sphere S n 1. This is a geometric problem.

17 Proof. Step 1: Decomposition of the sphere Hx n 1/2 for all vectors x on the sphere S n 1? General architecture of proof: (Rudelson-V 08). Decompose S n 1 into two classes of compressible and incompressible vectors: S n 1 = Comp Incomp. Compressible vectors are those within distance 0.1 from sparse vectors (of support 0.1n). Incompressible are the rest. Prove the lower bound (invertibility) for each class separately.

18 Proof. Step 2: Compressible vectors Hx n 1/2 for all vectors x on the sphere S n 1? Compressible vectors are simple to control. There are not too many of them the metric entropy of Comp is small. A covering argument reduces the problem to a lower bound for a single vector x. Replace H by its above-diagonal minor G by conditioning; then G has independent entries, and independent rows G k. Gx 2 2 = k G k, x 2 is a sum of independent random variables. Finish by standard concentration technique: Hx Gx n 1/2 with high probability 1 e cn. This is even better than we need.

19 Proof. Step 3: Incopressible vectors Incompressible vectors are difficult there are many of them. The problem reduces to: Distance problem. Estimate the distance between a random vector X and a random hyperplane E in R n. Show that dist(x, E) 1 with high probability, where X = a column of H and E = span of the other columns. This is a quantitative form of saying H is non-singular. For matrices with iid entries, a solution in (Rudelson-V 08). Difficulty here: X and E are not independent.

20 Proof. Step 4: Distance problem Distance Theorem. Let X = first column of a symmetric random matrix H, and E = span of the other columns. Then dist(x, E) 1 w.h.p. To prove this result, decompose Linear algebra allows to express B 1 Z, Z a 11 dist(x, E) = 1 + B 1 Z. 2 Here B is a symmetric random matrix (similar to H); Z is an independent random vector with iid coordinates. The problem reduces to concentration of quadratic forms:

21 Proof. Step 5: Concentration for quadratic forms Theorem (concentration of quadratic forms, V 11) Let H be a symmetric random matrix, X = independent random vector with iid coordinates. Then the distribution of the quadratic form H 1 X, X is spread on R. Specifically, for every u R, H 1 X, X u (E H 1 X, X 2 ) 1/2 = H 1 HS w.h.p. Proof uses decoupling: replace by a bilinear form H 1 Y, X for an independent Y. The problem reduces to showing that a, X u 1 w.h.p. where a = H 1 Y H 1 Y 2.

22 Proof. Step 5: Concentration for quadratic forms H = symmetric random matrix, X, Y = indep. random vectors. a, X u 1 w.h.p.? where a = H 1 Y H 1 Y 2. a and X are independent, so condition on a and express n S := a, X = a k X k, k=1 sum of independent random variables. We need to show that the distribution of S is spread. This is a Littlewood-Offord Problem. The spread of S depends on the additive structure of the coefficient vector a (crucial). The less structure in a, the more S is spread:

23 Proof. Step 6: Littlewood-Offord Problem Littlewood-Offord Problem. Consider a sum of ind. rand. variables S := a, X = n k=1 a kx k. If a has little additive structure, then the distribution of S is spread. How to quantify spread? With Lévy concentration function L(S, ε) = sup P { S u ε }, ε 0. u R How to quantify additive structure? With Diophantine approximation: the least common denominator (LCD) D(a) = inf { θ > 0 : dist(θx, Z n ) 10 log + θ }.

24 Proof. Step 6: Littlewood-Offord Problem Littlewood-Offord Theorem. (Rudelson-V 08) A sum of ind. rand. variables S := a, X = a k X k satisfies L(S, ε) ε + 1 D(a), ε 0. The less structure (the larger D(a)), the more S is spread (the smaller L(S, ε)). To successfully use this theorem, we need to know that the coefficient vector a has little structure. In our problem, H 1 Y a = H 1 Y 2 where H = symmetric random matrix, Y = ind. rand. vector. In other words, we need to show that the random matrix H 1 destroys additive structure.

25 Proof. Step 7: Action of H 1 is unstructured Theorem (Structure of inverse, V 11) Let H be an n n symmetric random matrix. Consider a = H 1 y H 1 y 2, where y is an arbitrary fixed vector. Then, with high probability 1 e cn, (i) a is an incompressible vector; (ii) a is unstructured. Conjecture: D(a) e cn. Proved: D(â) n c/λ, where â is a restriction of a onto some carefully chosen set of λn coefficients. λ (0, 1) is arbitrary. Most difficult step, currently not optimal. From this theorem, everything follows (apply together with Littlewood-Offord theory concentration of bilinear and quadratic forms distance theorem invertibility for incompressible vectors invertibility theorem.)

26 Action of H 1 is unstructured Theorem (Structure of inverse, V 11) Let H be an n n symmetric random matrix. Consider a = H 1 y H 1 y 2, where y is an arbitrary fixed vector. Then, w.h.p., the vector a is incompressible and unstructured. Corollary. Columns of H 1 are incompressible, unstructured. Note that the action of H does not destroy structure: the columns of Bernoulli matrix H are structured (±1 entries). Corollary. Same theorem holds for Green s function (H z) 1. Corollary (Delocalization of eigenvectors) All eigenvectors of H are incompressible, unstructured. Proof. If Hv = λv then v (H λ) 1 (0). Apply Structure Theorem for H λ and y = 0. (+ approximation argument). (Erdös-Schlein-Tau 09): a different version of delocalization.

27 References Survey: M. Rudelson, R. Vershynin, Non-asymptotic theory of random matrices: extreme singular values, Tutorial: R. Vershynin, Introduction to the non-asymptotic analysis of random matrices, General Covariance Estimation: R. Vershynin, How close is the sample covariance matrix to the actual covariance matrix? Sparse Covariance Estimation: L. Levina, R. Vershynin, Partial estimation of covariance matrices, Invertibility of Symmetric Matrices: R. Vershynin, Invertibility of symmetric random matrices, 2011.