Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices 1

Upper Bound for Intermediate Singular Values of Random Sub-Gaussian Matrices 1 Feng Wei 2 University of Michigan July 29, 2016 1 This presentation is based a project under the supervision of M. Rudelson. 2 Partially supported by M. Rudelson s NSF Grant DMS-1464514, and USAF Grant FA9550-14-1-0009. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 1 / 16

Motivation and Backgound Asymptotic Distribution of Singular Values Let A be an n n random matrix with i.i.d. entries of mean 0 and variance 1. s 1 (A) s 2 (A) s n (A) denote the singular values of A. Consider µ(j) = 1 { ( ) } A n n # i : s i J, J R. By Quarter Circular Law, dµ(x) 1 π 4 x 2 1 [0,2] (x)dx as n. A simple computation using the limiting distribution shows the lth l smallest singular value s n+1 l (A) is in the order of n for l = 1, 2,, n. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 2 / 16

Motivation and Backgound Asymptotic Distribution of Singular Values Let A be an n n random matrix with i.i.d. entries of mean 0 and variance 1. s 1 (A) s 2 (A) s n (A) denote the singular values of A. Consider µ(j) = 1 { ( ) } A n n # i : s i J, J R. By Quarter Circular Law, dµ(x) 1 π 4 x 2 1 [0,2] (x)dx as n. A simple computation using the limiting distribution shows the lth l smallest singular value s n+1 l (A) is in the order of n for l = 1, 2,, n. Question What is the distribution of the singular values for a fixed large n? Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 2 / 16

Motivation and Backgound Sub-gaussian Random Variables Definition Let θ > 0. Let Z be a random variable. Then the ψ θ -norm of Z is defined as { ( ) Z θ Z ψθ := inf λ > 0 : E exp 2} λ If Z ψθ <, then Z is called a ψ θ random variable. This condition is satisfied for broad classes of random variables. In particular, a bounded random variable is ψ θ for any θ > 0, a normal random variable is ψ 2, and a Poisson variable is ψ 1. A ψ 2 random variable is also called sub-gaussian. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 3 / 16

Motivation and Backgound Sub-gaussian Random Variables Definition Let θ > 0. Let Z be a random variable. Then the ψ θ -norm of Z is defined as { ( ) Z θ Z ψθ := inf λ > 0 : E exp 2} λ If Z ψθ <, then Z is called a ψ θ random variable. This condition is satisfied for broad classes of random variables. In particular, a bounded random variable is ψ θ for any θ > 0, a normal random variable is ψ 2, and a Poisson variable is ψ 1. A ψ 2 random variable is also called sub-gaussian. Moreover, X ψ2 = K P ( X > t) exp(1 ct2 K 2 ). Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 3 / 16

Motivation and Backgound Non-asymptotic Distribution of Singular Values The extreme singular values are better studied: It is easy to show that the operator norm of N n i.i.d. sub-gaussian matrix is in the order of N with high probability. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 4 / 16

Motivation and Backgound Non-asymptotic Distribution of Singular Values The extreme singular values are better studied: It is easy to show that the operator norm of N n i.i.d. sub-gaussian matrix is in the order of N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n 3/2 with high probability. This result was later extended and improved by A. Basak and M. Rudelson, M. Rudelson and R. Vershynin, T. Tao and V. Vu in square matrices case. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 4 / 16

Motivation and Backgound Non-asymptotic Distribution of Singular Values The extreme singular values are better studied: It is easy to show that the operator norm of N n i.i.d. sub-gaussian matrix is in the order of N with high probability. In 2008, M. Rudelson proved that the smallest singular value of a square i.i.d. sub-gaussian matrix is lower bounded by n 3/2 with high probability. This result was later extended and improved by A. Basak and M. Rudelson, M. Rudelson and R. Vershynin, T. Tao and V. Vu in square matrices case. In 2009, M. Rudelson and R. Vershynin proved a sharp bound for smallest singular value of all rectangular sub-gaussian matrices. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 4 / 16

Motivation and Backgound Lower Bound for Singular Values Theorem.(M. Rudelson and R. Vershynin, 2009) Let G be an N n random matrix, N n, whose elements are independent copies of a centered sub-gaussian random variable with unit variance. Then for every ε > 0, we have ( ( )) P s n (G) ε N n 1 (Cε) N n+1 + e C N where C, C > 0 depend (polynomially) only on the sub-gaussian moment K. Consider an n n i.i.d. sub-gaussian matrix A and let B be the first n + 1 l columns of A. Then with high probability, ( n ) s n+1 l (A) s n+1 l (B) c n l c l n. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 5 / 16

Motivation and Backgound Upper Bound for Smallest Singular Values Theorem.(M. Rudelson and R. Vershynin, 2008) Let A be an n n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t 2, ( ) P s n (A) tn 1 2 C log t + c n t where C > 0 and c (0, 1) depend only on K. Theorem.(H. Nguyen and V. Vu., 2016) Let A be an n n i.i.d. sub-gaussian matrix whose entries have mean 0, variance 1 and sub-gaussian moment K. Then for any t > 0, ( ) P s n (A) tn 1 2 C 1 exp( C 2 t) where C 1, C 2 > 0 depend only on K. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 6 / 16

Motivation and Backgound Two-side Bound for Singular Values Theorem.(S. Szarek, 1990) Let G be an n n i.i.d. standard Gaussian matrix. Then there exist universal constants c 1, c 2, c, C, s.t. ( c1 l P s n+1 l (G) c ) 2l 1 C exp( cl 2 ). n n From a paper of T. Tao and V. Vu. in 2010, together with the result of Szarek, one can deduce some non-asymptotic bounds for random i.i.d. square matrix for l n c where c is a small constant. However, the tail bound is not exponential type. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 7 / 16

Main Results Levy Concentration Function Definition Let Z be a random vector that takes values in R n. The concentration function of Z is defined as L(Z, t) = sup u R n P{ Z u 2 t}, t 0. Assumption 1 Let p > 0. Let A be an n m random matrix whose entries are i.i.d. random variables, with mean 0, variance 1 and ψ 2 -norm K. Assume also that there exists 0 < s s 0 (p, K) such that L(A i,j, s) ps. Here, s 0 (p, K) is a given function depending only on p and K. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 8 / 16

Main Results Main Theorem Theorem Let A be an n n random matrix that satisfies Assumption 1. Then there exist constants C 1, C 2 > 0 such that for all t > 1 and l = 1, 2,, n, ( ) tl P s n+1 l (A) C 1 1 exp( C 2 tl) n where C 1, C 2 are constants that depend only on K, p. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices 9 / 16

Main Results where C i s are constants that depends only on K, p. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices10 / 16 Corollaries Corollary Let A be an n (n k) random matrix that satisfies Assumption 1. Then there exist constants C 1, C 2 > 0 such that for all t > 1 and l = 1,, n, ( ) tl P s n+1 l (A) C 1 1 exp( C 2 tl) n where C 1, C 2 are constants that depend only on K, p. Corollary Let A be an n n random matrix that satisfies Assumption 1. Then there exist 0 < C 1 < C 2 and C 3 > 0, such that for all l = 1, 2,, n, ( C1 l P s n+1 l (A) C ) 2l 1 exp( C 3 l) n n

Outline of Proof Basic Idea of the Proof We want to prove s n+1 l (A) cl n with high probability. Instead of proving upper bound for s n+1 l (A), we prove lower bound for s l (A 1 ). So we only need to find an l-dimensional random subspace E such that with high probability, for all y E, A 1 y 2 y 2 C n l Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices11 / 16

Outline of Proof Basic Idea of the Proof We want to prove s n+1 l (A) cl n with high probability. Instead of proving upper bound for s n+1 l (A), we prove lower bound for s l (A 1 ). So we only need to find an l-dimensional random subspace E such that with high probability, for all y E, A 1 y 2 y 2 C n l Let X i, i = 1,, n denote the columns of A and H l = span{x l+1,, X n }. Then our aimed subspace is Hl to prove it using union bound argument. and we want Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices11 / 16

Outline of Proof Union Bound Argument on Random Subspace Two obstacles if we argue over the sphere on H l : H l is random and it depends on A. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices12 / 16

Outline of Proof Union Bound Argument on Random Subspace Two obstacles if we argue over the sphere on H l : H l is random and it depends on A. Estimating A 1 y 2 may be as hard as the orginal problem. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices12 / 16

Outline of Proof Union Bound Argument on Random Subspace Two obstacles if we argue over the sphere on H l : H l is random and it depends on A. Estimating A 1 y 2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on Pl A ls l 1, where A l is the first l columns of A and Pl is the orthogonal projection onto Hl. Let s construct a net N on S l 1, if we have control Pl A l, then Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices12 / 16

Outline of Proof Union Bound Argument on Random Subspace Two obstacles if we argue over the sphere on H l : H l is random and it depends on A. Estimating A 1 y 2 may be as hard as the orginal problem. Instead of applying union bound on unit sphere, we apply it on Pl A ls l 1, where A l is the first l columns of A and Pl is the orthogonal projection onto Hl. Let s construct a net N on S l 1, if we have control Pl A l, then For any y N, P l A ly 2 is bounded from above. P l A ln is still a net on P l A ls l 1 but with a different scale. A 1 Pl A ly 2 2 BA ly 2 2, where B is an (n l) n random matrix independent to A l. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices12 / 16

Outline of Proof Quantities to be Estimated There are several Quantities need to be estimated: Large deviation of Pl A l. Large deviation of A 1. H l Property of the matrix B. Small ball probability of BX 2 given B is a fixed matrix and X is a sub-gaussian random vector. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices13 / 16

Remarks Measure Concentration Tools Major measure concentration results we applied: Corollaries of Hanson-Wright inequality. Small ball probability for linear image of high dimension distribution. Small ball probability of distance from a sub-gaussian vector to a random subspace. Lower bound for smallest singular value of rectangular i.i.d. sub-gaussian matrix. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices14 / 16

Remarks The Concentration Function Condition Theorem.(M. Rudelson and R. Vershynin, 2014) Consider a random vector Z where Z i are real-valued independent random variables. Let t, p 0 be such that L(Z i, t) p for all i = 1,, n Let D be an m n matrix and ε (0, 1). Then L(DZ, t D HS ) (c ε p) (1 ε)r(d) where r(d) = D 2 HS / D 2 2 and c ε depend only on ε. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices15 / 16

Remarks Thanks. Feng Wei (University of Michigan) Upper Bound for Intermediate Singular Values of Random Sub-Gaussian July 29, 2016 Matrices16 / 16