Recovering any low-rank matrix, provably

Transcription

1 Recovering any low-rank matrix, provably Rachel Ward University of Texas at Austin October, 2014 Joint work with Yudong Chen (U.C. Berkeley), Srinadh Bhojanapalli and Sujay Sanghavi (U.T. Austin)

2 Matrix completion problem Given some entries of a matrix M, exactly recover remaining entries Motivation: collaborative filtering, etc. Ill-posed problem in general

3 Low rank assumption 1. Assume underlying matrix M has low rank 2. n n matrix of rank r has O(nr) degrees of freedom SVD of M = UΣV T. U, V R n r.

4 Incoherence - avoid bad cases Bad case 1. SVD of M = UΣV T. U, V R n r. 2. Incoherence: left and right singular spaces form small angles with Euclidean coordinates: µi := U i 2 = e i U 2 r µ 0 n, i = 1, 2,..., n νj := V j 2 = e j V r 2 µ 0 n, i = 1, 2,..., n

5 Nuclear norm minimization (NNM) min X rank (X ), s.t. X ij = M ij for (i, j) Ω 1. The nuclear norm X = j σ j is convex relaxation of the rank of a matrix (Fazel 2002; Fazel Recht Parrilo 2007; Candès and Recht; 2009,... ). 2. Nuclear norm minimization (NNM) problem is an SDP; can be solved efficiently. min X X, s.t. X ij = M ij for (i, j) Ω

6 Previous results [Candès and Recht, 2009; Candès and Tao, 2009; Recht 2011; Gross ] NNM algorithm exactly recovers matrix M if: M is a rank-r, µ0 -incoherent matrix Uniform sampling with number of samples Ω Cµ 0 nr log 2 (n) Other algorithms with similar guarantees: OptSpace (Keshavan et al., 2010) Alternating minimization (Jain et al., 2013)...

7 Previous results [Candès and Recht, 2009; Candès and Tao, 2009; Recht 2011; Gross ] NNM algorithm exactly recovers matrix M if: M is a rank-r, µ0 -incoherent matrix Uniform sampling with number of samples Ω Cµ 0 nr log 2 (n) Other algorithms with similar guarantees: OptSpace (Keshavan et al., 2010) Alternating minimization (Jain et al., 2013)... Can we relax these two assumptions?

8 What is the relation between sampling and structure of the matrix that guarantees exact recovery in O(nr log(n)) samples?

9 Matrix local coherences Uniform sampling will fail for coherent matrices SVD of rank-r matrix M = UΣV Row coherences of M are µ i = U i 2, i = 1, 2,..., n Column coherences of M are ν j = V j 2, j = 1, 2,..., n Sometimes called leverage scores or effective resistances Popular in statistics, matrix sketching and graph sparsification Mahoney and Drineas, 2009; Mahoney 2011; Spielman and Srivastava 2011; Drineas et. al. 2012

10 Local coherence sampling - example Sample entry (i, j) with probability p ij min{c 0 (µ i + ν j ) log 2 (n), 1} For this example, p ij c log2 (n) n (higher probability) in green region p ij c log2 (n) n (lower probability), else

11 Local coherence sampling - main result Theorem Consider samples generated according to p ij where p ij min{c 0 ( U i 2 + V j 2 ) log 2 (n), 1} With high probability, nuclear norm minimization will exactly recover the matrix M = UΣV if M is rank-r and samples are generated according to p ij. 1 Krishnamurthy, Singh: O(µnr log 2 r) samples to recover a rank-r matrix with only left- or right- incoherence µ

12 Local coherence sampling - main result Theorem Consider samples generated according to p ij where p ij min{c 0 ( U i 2 + V j 2 ) log 2 (n), 1} With high probability, nuclear norm minimization will exactly recover the matrix M = UΣV if M is rank-r and samples are generated according to p ij. Consequence: O(nr log 2 (n)) samples are sufficient to exactly recover a given rank-r matrix. First such sufficient conditions for exact recovery of low-rank matrices (no incoherence assumptions) 1 Similar to local coherence sampling which give sampling strategies in compressive sensing Rauhut, W. 11, Krahmer, W Krishnamurthy, Singh: O(µnr log 2 r) samples to recover a rank-r matrix with only left- or right- incoherence µ

13 Proof outline: By convex analysis, suffices to construct a dual certificate matrix Y satisfying certain optimality conditions Crucial optimality condition: spectral norm Y should be small Previous work requiring incoherence of M bounds Y using matrix l estimates: Z = max i,j Z ij. Our result uses estimates based on mixed weighted l / weighted l,2 norms: n Z µ(,2) := max max n Z 2 i µ i r ib, max j ν j r b a Z 2 aj, and n n Z µ( ) := max Z i,j i,j µ i r ν j r.

14 Weighted l /l,2 norm is closely related to the spectral norm of a random matrix: Theorem (Bandeira, Van Handel 14) For X an n n symmetric matrix with X ij N (0, b 2 ij ), E X max i j bij 2 + max b ij log n i,j

15 Two ways to apply the main result Adapt sampling to coherence structure A two-phase algorithm Adapt local coherence to sampling Weighted nuclear norm minimization

16 Adapt sampling to coherence structure In general, local coherences are unknown, and sampled data may not be distributed according to local coherences Try a two-phase procedure: First sample uniformly to estimate local coherences, then generate new samples accordingly.

17 Two-phase procedure Rank-r matrix M is unknown, but we can try to learn it through entry wise sampling. Let M 0 = P Ω (M) be initial uniformly-sampled matrix Learn local coherences: Let Ũ ΣṼ be best rank-r approximation to M 0 Generate new samples using estimates of local coherences: p ij min{c 0 ( Ũi 2 + Ṽj 2 ) log 2 (n), 1} Complete to full matrix (say, using nuclear norm minimization), using both sets of samples (uniform and biased)

18 Simulation results Consider power law matrices of form M = DUV D; U, V are Gaussian. D is diagonal with D jj = j α and 0 α 1.

19 Two ways to apply the main result Adapt sampling to local coherence A two-phase algorithm Adapt local coherences to sampling Weighted nuclear norm minimization

20 Weighted Nuclear Norm minimization Samples are generated according to p ij not necessarily aligned with coherence structure Given non-uniform samples Ω, solve min X s.t. RXC X ij = M ij, for (i, j) Ω R and C are diagonal matrices with positive weights. Equivalent to changing the coherence structure of X Previous empirical studies: shown to have better performance for non-uniform sampling. Salakhutdinov and Srebro, 2010; Foygel et al., 2011; Negahban and Wainwright, 2012.

21 Theoretical result for weighted nuclear norm minimization Theorem Weighted nuclear norm minimization algorithm exactly recovers the matrix M if: M is rank-r M is incoherent: µ i, ν j µ 0 Samples are generated according to p ij p ij c 0 ( R 2 i n/(µ0 r) I =1 RI 2 + ) Cj 2 n/(µ0 log 2 (n) r) J=1 CJ 2 Rows with higher probability of sampling have bigger values of R i

22 Extension: low-rank matrix approximation Given: (Entire) matrix M of size n n such that M M r, where M r is best rank-r approximation. n is very large, computing M r is costly Want rank-r matrix M such that M M M M r + ε M M r F using minimal computation Any algorithm needs at least O(nr) computations

23 Extension: low-rank matrix approximation Consider the leveraged element sampling p ij 1 ( Mi 2 + M j 2 2 2n M 2 + m ) ij F M 1,1 p ij is a function of row/column norms, which are easily and quickly computable given the matrix M. p ij is equivalent to local coherence sampling, up to the condition number κ = σ 1 /σ r. Idea: Draw m Cnr of sampled elements according to p, then complete sampled matrix to a rank-r matrix M using fast alternative to nuclear norm minimization (weighted alternating minimization).

24 Extension: low-rank matrix approximation Theorem (Bhojanapalli, Jain, Sanghavi ( 14) ) For a given n n matrix M, using O nnz(m) + nκ2 r 5 ε 2 computations suffice to generate a rank-r matrix M satisfying M M M M r + ε M M r F 2 Clarkson, Woodruff 2013, Boutsidis and Gittens 2013, Sarlos 2006, Woolfe, Liberty, Rokhlin, Tygert 2008

25 Extension: low-rank matrix approximation Theorem (Bhojanapalli, Jain, Sanghavi ( 14) ) For a given n n matrix M, using O nnz(m) + nκ2 r 5 ε 2 computations suffice to generate a rank-r matrix M satisfying M M M M r + ε M M r F Improves on existing bounds 2 when condition number κ = σ 1 /σ r is small. 2 Clarkson, Woodruff 2013, Boutsidis and Gittens 2013, Sarlos 2006, Woolfe, Liberty, Rokhlin, Tygert 2008

26 Summary Sufficient conditions for recovery of any matrix from O(nr log 2 (n)) samples Local coherence sampling also necessary to achieve such sample complexity Two-phase sampling using estimates of local coherences (theoretical guarantees?) Exact recovery guarantees from non-uniform sampling via weighted NNM

27 Summary Sufficient conditions for recovery of any matrix from O(nr log 2 (n)) samples Local coherence sampling also necessary to achieve such sample complexity Two-phase sampling using estimates of local coherences (theoretical guarantees?) Exact recovery guarantees from non-uniform sampling via weighted NNM Thanks!