Matrix completion: Fundamental limits and efficient algorithms. Sewoong Oh Stanford University

Transcription

1 Matrix completion: Fundamental limits and efficient algorithms Sewoong Oh Stanford University 1 / 35

2 Low-rank matrix completion Low-rank Data Matrix Sparse Sampled Matrix Complete the matrix from small number of sampled entries 2 / 35

3 Example 1. Recommendation system 1 1? ? ? 4? ? movies users Better recommendation improves customer experience Less than 1% known entries Goal: Predict the (9.9 billion) unknown ratings 10 6 queries 3 / 35

4 Example 2. Positioning Distance Matrix Location information needed to route packets, habitat monitoring, etc. Only distances between close-by sensors are measured Q : How can we find the sensor positions up to a rigid motion? 4 / 35

5 Low-rank matrix completion More applications: Ultrasound tomography: Calibration Computer vision: Structure-from-motion Molecular biology: Microarray Theoretical computer science: Fast low-rank approximations etc. 5 / 35

6 Outline 1 Background 2 Algorithm and main results 3 Applications 6 / 35

7 Background 7 / 35

8 The model Σ V T r αn Low-rank M = U n r Rank-r matrix M Random sample set E (Uniformly random for fixed E ) Sample noise matrix Z Sample matrix N E = (M + Z) E N E ij = { Mij + Z ij if (i, j) E 0 otherwise 8 / 35

9 The model αn Sample N E Rank-r matrix M Random sample set E (Uniformly random for fixed E ) Sample noise matrix Z Sample matrix N E = (M + Z) E N E ij = n { Mij + Z ij if (i, j) E 0 otherwise 8 / 35

10 Pathological example M = = 0. [1 ] [1 0 0 ] P(sampling M 11 ) = E αn 2 9 / 35

11 Pathological example M = = 0. [1 ] [1 0 0 ] P(sampling M 11 ) = E αn 2 We focus on incoherent matrices M = UΣV T which have well balanced singular vectors U and V A0. A1. r k=1 r k=1 U 2 ik µ r αn, r Vjk 2 µ r n k=1 U ik V jk µ r n 9 / 35

12 Previous work [Candès, Recht 08] Semidefinite Programming(SDP) for matrix completion [Fazel 02] In the noiseless case, SDP reconstructs M exactly with high probability, if E C(α, µ) r n 6/5 log n Open problems 1. Complexity: SDP is computationally complex 2. Optimality: n 1/5 gap from the lower bound Ω(n log n) 3. Noise: Cannot deal with noise 10 / 35

13 Main contributions OptSpace 1 Complexity: Low complexity 2 Optimality: Order-optimal 3 Noise: Robust against noise 11 / 35

14 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

15 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

16 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

17 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

18 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

19 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

20 Example: rank-8 random matrix (noiseless) low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 35

21 Algorithmic aspect 13 / 35

22 Naïve approach Singular Value Decomposition (SVD) Compute rank-r approximation MSVD using SVD SVD is the optimal thing to do if we have complete matrix N E MSVD n N E = x k σ k yk T k=1 M SVD αn2 E r x k σ k yk T k=1 14 / 35

23 Naïve approach fails Singular Value Decomposition (SVD) Compute rank-r approximation MSVD using SVD SVD is the optimal thing to do if we have complete matrix N E MSVD n N E = x k σ k yk T k=1 M SVD αn2 E r x k σ k yk T k=1 14 / 35

24 Naïve approach fails Singular Value Decomposition (SVD) Compute rank-r approximation MSVD using SVD SVD is the optimal thing to do if we have complete matrix N E MSVD n N E = x k σ k yk T k=1 M SVD αn2 E r x k σ k yk T k=1 14 / 35

25 Naïve approach fails Define : deg(row i ) # of samples in row i deg(row i ) i.i.d. Binom(n, p), where p = E /αn 2 For E = O(n), maximum degree is Ω(log n/(log log n)) N E has spurious singular values of Ω( log n/(log log n)) 15 / 35

26 Trimming N E = Ñ E ij = N E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 16 / 35

27 Trimming N E = Ñ E ij = N E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 16 / 35

28 Trimming Ñ E = Ñ E ij = N E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 16 / 35

29 The algorithm OptSpace Input : sample indices E, sample values N E, rank r Output : estimation M 1: Trimming 2: SVD 3: Greedy minimization 17 / 35

30 The algorithm OptSpace Input : sample indices E, sample values N E, rank r Output : estimation M 1: Trimming 2: SVD M SVD can be computed efficiently for sparse matrices 17 / 35

31 Main results Theorem 1. (Keshavan, Montanari, Oh 09) For any E, M SVD achieves, with high probability, RMSE CM max nr E }{{} missing entries ( 1 RMSE = αn i,j (M M 2 SVD ) 2 i,j M max max i,j M i,j 2 is the spectral norm ) 1/2 + C n r E ZE 2 }{{} sample noise Keshavan, Montanari, Oh, NIPS 09 Keshavan, Montanari, Oh, Journ. Machine Learning Research (submitted) 18 / 35

32 Noiseless case Theorem 1. (Keshavan, Montanari, Oh 09) For any E, M SVD achieves, with high probability, RMSE CM max nr E }{{} missing entries ( 1 RMSE = αn i,j (M M 2 SVD ) 2 i,j M max max i,j M i,j 2 is the spectral norm ) 1/2 Keshavan, Montanari, Oh, NIPS 09 Keshavan, Montanari, Oh, Journ. Machine Learning Research (submitted) 18 / 35

33 Noiseless case Theorem 1. (Keshavan, Montanari, Oh 09) For any E, M SVD achieves, with high probability, RMSE CM max nr E }{{} missing entries SVD [Achlioptas, McSherry 07] If E n(8 log n) 4, with high probability, RMSE 4M max nr E For n = 10 5, (8 log n) Keshavan, Montanari, Oh, NIPS 09 Keshavan, Montanari, Oh, Journ. Machine Learning Research (submitted) 18 / 35

34 Noiseless case Theorem 1. (Keshavan, Montanari, Oh 09) For any E, M SVD achieves, with high probability, RMSE CM max nr E }{{} missing entries SVD [Achlioptas, McSherry 07] If E n(8 log n) 4, with high probability, RMSE 4M max nr E Netflix dataset A single user rated 17,000 movies. Miss Congeniality : 200,000 ratings. For n = 10 5, (8 log n) Keshavan, Montanari, Oh, NIPS 09 Keshavan, Montanari, Oh, Journ. Machine Learning Research (submitted) 18 / 35

35 Can we do better? 19 / 35

36 Greedy minimization minimize F (X, Y) subject to X T X = I, Y T Y = I F (X, Y) min S R r r X S Y T (i,j) E (N E ij (XSY T ) ij ) 2 rank F (X, Y) only depends on the column spaces of X and Y Perform gradient descent on Grassmann manifold 20 / 35

37 Greedy minimization minimize F (X, Y) subject to X T X = I, Y T Y = I F (X, Y) min S R r r X S Y T (i,j) E (N E ij (XSY T ) ij ) 2 rank F (X, Y) only depends on the column spaces of X and Y Perform gradient descent on Grassmann manifold Can be computed efficiently for sparse matrices 20 / 35

38 The algorithm OptSpace Input : sample indices E, sample values N E, rank r Output : estimation M 1: Trimming 2: SVD 3: Greedy minimization 21 / 35

39 Main results Theorem 1. (Trimming+SVD) For any E, M SVD achieves, with high probability, nr RMSE CM max + C n r E E ZE 2 }{{}}{{} missing entries sample noise Theorem 2. (Trimming+SVD+Greedy minimization) For E Cnr max{r, log n}, OptSpace achieves, with high probability, RMSE C n r E ZE 2, provided that the RHS is smaller than σ r (M). Keshavan, Montanari, Oh, NIPS 09 Keshavan, Montanari, Oh, Journ. Machine Learning Research (submitted) 22 / 35

40 Noiseless case 23 / 35

41 Noiseless case Theorem 2. (OptSpace) For E Cnr max{r, log n}, OptSpace achieves, with high probability, RMSE C n r E ZE 2 24 / 35

42 Noiseless case Theorem 2. (OptSpace) For E Cn log n, OptSpace achieves, with high probability, RMSE = 0 (Exact reconstruction) Assuming the rank of M does not depend on n Lower bound (coupon collector s problem): If E C n log n, then exact reconstruction is impossible OptSpace is order-optimal Keshavan, Montanari, Oh, IEEE Trans. Information Theory / 35

43 Noiseless case Theorem 2. (OptSpace) For E Cn log n, OptSpace achieves, with high probability, RMSE = 0 (Exact reconstruction) Assuming the rank of M does not depend on n Lower bound (coupon collector s problem): If E C n log n, then exact reconstruction is impossible Convex Relaxation: [Candès, Recht 08, Candès, Tao 09, Recht 09, Gross et al. 09] If E C n (log n) 2, then exact reconstruction by SDP OptSpace is order-optimal Keshavan, Montanari, Oh, IEEE Trans. Information Theory / 35

44 Noiseless case rank-10 matrix M 1 P success 0.5 Lower Bound OptSpace FPCA SVT ADMiRA Sampling rate Lower Bound [Singer, Cucuringu 09], FPCA [Ma, Goldfarb, Chen 09], SVT [Cai, Candès, Shen 08], ADMiRA [Lee, Bresler 09] 25 / 35

45 Noisy case 26 / 35

46 Gaussian noise case Theorem 2. (OptSpace) For E Cnr max{r, log n}, OptSpace achieves, with high probability, RMSE C n r E ZE 2 27 / 35

47 Gaussian noise case Theorem 2. (OptSpace) For E Cnr max{r, log n}, OptSpace achieves, with high probability, C RMSE σ r n z E Lower bound: [Candès, Plan 09] RMSE σ z 2 r n E OptSpace is order-optimal 27 / 35

48 Gaussian noise case rank-4 matrix M, Gaussian noise with σ z = 1 Example from [Candès, Plan 09] 1.5 Trim+SVD RMSE Lower Bound 0 OptSpace Sampling rate 28 / 35

49 Gaussian noise case rank-4 matrix M, Gaussian noise with σ z = 1 Example from [Candès, Plan 09] 1.5 Trim+SVD RMSE Lower Bound 0 OptSpace FPCA ADMiRA Sampling rate FPCA [Ma, Goldfarb, Chen 09], ADMiRA [Lee, Bresler 09] 28 / 35

50 Proof strategy (noiseless case) 29 / 35

51 Proof strategy Proving Theorem 1. RMSE(M, M SVD ) CM max nr/ E [Friedman,Kahn,Szemeredi 1989], [Feige, Ofek 2005] Erdős-Renýi Graph Adjacency Matrix Proving Theorem For E Cnr max{r, log n}, OptSpace reconstructs M exactly Prove d(m, MSVD ) δ In the δ neighborhood, F (X, Y) has unique local minimum at M = UΣV T 30 / 35

52 Applications 31 / 35

53 Ultrasound tomography 256 sensors Reconstructed image Time-of-flight measurements image reconstruction Accurate positions of the sensors to get more accurate images Images courtesy of Hormati, Jovanovic, Roy, Vetterli 32 / 35

54 Ultrasound tomography Ideal time-of-flight measurements Real time-of-flight measurements x x Measurements with water Ideal: Complete time-of-flight measurements Reality: Noisy measurements with missing entries T 0 : Transmission delay M i,j = T 0 + d i,j /v water + Z i,j 33 / 35

55 Ultrasound tomography Iterative algorithm for sensor positioning using OptSpace Estimated sensor positions 0.1 Water image (with calibration) Water image (without calibration) Parhizkar, Karbasi, Oh, Vetterli, Intern. Congress on Acoustics / 35

56 Ultrasound tomography Iterative algorithm for sensor positioning using OptSpace Estimated sensor positions 0.1 Water image (with calibration) Water image (without calibration) Parhizkar, Karbasi, Oh, Vetterli, Intern. Congress on Acoustics / 35

57 Conclusion Application Theory Matrix Completion Algorithm Open challenges Non-uniform sampling (e.g. Positioning) Noiseless: Sub-optimality in high rank regime (e.g. r = Ω( n) ) Noisy: Regularization in high noise regime 35 / 35

58 Thank you! 35 / 35