Matrix Completion: Fundamental Limits and Efficient Algorithms

Size: px

Start display at page:

Download "Matrix Completion: Fundamental Limits and Efficient Algorithms"

Jerome Parrish
6 years ago
Views:

1 Matrix Completion: Fundamental Limits and Efficient Algorithms Sewoong Oh PhD Defense Stanford University July 23, / 33

2 Matrix completion Find the missing entries in a huge data matrix 2 / 33

3 Example 1. Recommendation systems? ???? users movies 10 6 queries Given less than 1% of the movie ratings Goal: Predict missing ratings 3 / 33

4 Example 2. Positioning Distance Matrix Only distances between close-by sensors are measured Goal: Find the sensor positions up to a rigid motion 4 / 33

5 Matrix completion More applications: Computer vision: Structure-from-motion Molecular biology: Microarray Numerical linear algebra: Fast low-rank approximations etc. 5 / 33

6 Outline 1 Background 2 Algorithm and main results 3 Applications in positioning 6 / 33

7 Background 7 / 33

8 The model Σ V T r αn Low-rank M = U n r Rank-r matrix M Random uniform sample set E Sample matrix M E M E ij = { Mij if (i, j) E 0 otherwise 8 / 33

9 The model αn Sample M E n Rank-r matrix M Random uniform sample set E Sample matrix M E M E ij = { Mij if (i, j) E 0 otherwise 8 / 33

10 Which matrices? Pathological example M = = 0. [1 ] [1 0 0 ] / 33

11 Which matrices? Pathological example M = = 0. [1 ] [1 0 0 ] [Candès, Recht 08] M = UΣV T has coherence µ if A0. max A1. max i,j 1 i αn k=1 r U 2 ik µ r n, r k=1 U ik V jk µ r n max 1 j n k=1 Intuition µ is small if singular vectors are well balanced We need low-coherence for matrix completion r Vjk 2 µ r n 9 / 33

12 Previous work Rank minimization minimize subject to rank(x) X ij = M ij, (i, j) E NP-hard 10 / 33

13 Previous work Rank minimization Heuristic [Fazel 02] minimize subject to rank(x) X ij = M ij, (i, j) E minimize subject to X X ij = M ij, (i, j) E NP-hard Convex relaxation Nuclear norm n X = σ i (X ) i=1 Can be solved using Semidefinite Programming(SDP) 10 / 33

14 Previous work [Candès, Recht 08] Nuclear norm minimization reconstructs M exactly with high probability, if E C µ r n 6/5 log n Surprise? 11 / 33

15 Previous work [Candès, Recht 08] Nuclear norm minimization reconstructs M exactly with high probability, if E C µ r n 6/5 log n Degrees of freedom (1 + α)rn Open questions Optimality: Do we need n 6/5 log n samples? Complexity: SDP is computationally expensive Noise: Can not deal with noise 11 / 33

16 Previous work [Candès, Recht 08] Nuclear norm minimization reconstructs M exactly with high probability, if E C µ r n 6/5 log n Degrees of freedom (1 + α)rn Open questions Optimality: Do we need n 6/5 log n samples? Complexity: SDP is computationally expensive Noise: Can not deal with noise A new approach to Matrix Completion: OptSpace 11 / 33

Example: 2000 2000 rank-8 random matrix low-rank matrix M sampled

17 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

18 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

19 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

20 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

21 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

22 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

23 Example: rank-8 random matrix low-rank matrix M sampled matrix M E OptSpace output M squared error (M M) % sampled 12 / 33

24 Algorithm 13 / 33

25 Naïve approach Singular Value Decomposition (SVD) n M E = σ i x i yi T i=1 Compute rank-r approximation M SVD r M SVD αn2 σ i x i yi T E i=1 M E MSVD 14 / 33

26 Naïve approach fails Singular Value Decomposition (SVD) n M E = σ i x i yi T i=1 Compute rank-r approximation M SVD r M SVD αn2 σ i x i yi T E i=1 M E MSVD 14 / 33

27 Naïve approach fails Singular Value Decomposition (SVD) n M E = σ i x i yi T i=1 Compute rank-r approximation M SVD r M SVD αn2 σ i x i yi T E i=1 M E MSVD 14 / 33

28 Trimming M E = M E ij = M E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 15 / 33

29 Trimming M E = M E ij = M E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 15 / 33

30 Trimming M E = M E ij = M E ij 0 if deg(row i ) > 2 E /αn 0 if deg(col j ) > 2 E /n otherwise deg( ) is the number of samples in that row/column 15 / 33

31 Algorithm OptSpace Input : sample indices E, sample values M E, rank r Output : estimation M 1: Trimming 2: Compute M SVD using SVD 3: Greedy minimization of the residual error 16 / 33

32 Algorithm OptSpace Input : sample indices E, sample values M E, rank r Output : estimation M 1: Trimming 2: Compute M SVD using SVD M SVD can be computed efficiently for sparse matrices 16 / 33

33 Main results Theorem For any E, M SVD achieves, with high probability, RMSE CM max nr E ( 1 RMSE = αn i,j (M M 2 SVD ) 2 ij M max max i,j M ij ) 1/2 Keshavan, Montanari, Oh, IEEE Trans. Information Theory, / 33

34 Main results Theorem For any E, M SVD achieves, with high probability, RMSE CM max nr E [Achlioptas, McSherry 07] If E n(8 log n) 4, with high probability, RMSE 4M max nr E For n = 10 5, (8 log n) Keshavan, Montanari, Oh, IEEE Trans. Information Theory, / 33

35 Main results Theorem For any E, M SVD achieves, with high probability, RMSE CM max nr E [Achlioptas, McSherry 07] If E n(8 log n) 4, with high probability, RMSE 4M max nr E Netflix dataset A single user rated 17,000 movies. Miss Congeniality : 200,000 ratings. For n = 10 5, (8 log n) Keshavan, Montanari, Oh, IEEE Trans. Information Theory, / 33

36 Can we do better? 18 / 33

37 Greedy minimization of residual error Starting from (X 0, Y 0 ) for M SVD = X 0 S 0 Y0 T, use gradient descent methods to solve minimize F (X, Y) subject to X T X = I, Y T Y = I ) 2 F (X, Y) (M E ij (XSY T ) ij min S R r r (i,j) E X S Y T rank Can be computed efficiently for sparse matrices 19 / 33

38 Algorithm OptSpace Input : sample indices E, sample values M E, rank r Output : estimation M 1: Trimming 2: Compute M SVD using SVD 3: Greedy minimization of the residual error 20 / 33

39 Main results Theorem (Trimming+SVD) M SVD achieves, with high probability, RMSE CM max nr E Theorem (Trimming+SVD+Greedy minimization) OptSpace reconstructs M exactly, with high probability, if E C µ r n max{µr, log n} Keshavan, Montanari, Oh, IEEE Trans. Information Theory, / 33

40 OptSpace is order-optimal Theorem If µ and r are bounded, OptSpace reconstructs M exactly, with high probability, if E C n log n Lower bound (coupon collector s problem): If E C n log n, then exact reconstruction is impossible 22 / 33

41 OptSpace is order-optimal Theorem If µ and r are bounded, OptSpace reconstructs M exactly, with high probability, if E C n log n Lower bound (coupon collector s problem): If E C n log n, then exact reconstruction is impossible Nuclear norm minimization: [Candès, Recht 08, Candès, Tao 09, Recht 09, Gross et al. 09] If E C n (log n) 2, then exact reconstruction by SDP 22 / 33

42 Comparison rank-10 matrix M 1 P success 0.5 Fundamental Limit OptSpace FPCA SVT ADMiRA Sampling rate Fundamental Limit [Singer, Cucuringu 09], FPCA [Ma, Goldfarb, Chen 09], SVT [Cai, Candès, Shen 08], ADMiRA [Lee, Bresler 09] 23 / 33

43 Story so far OptSpace reconstructs M from a few sampled entries, when M is exactly low-rank and samples are exact In reality, M is only approximately low-rank samples are corrupted by noise 24 / 33

44 The model with noise Σ V T r αn Low-rank M = U n r Rank-r matrix M Random sample set E Sample noise Z E Sample matrix N E = M E + Z E 25 / 33

45 The model with noise αn Sample N E n Rank-r matrix M Random sample set E Sample noise Z E Sample matrix N E = M E + Z E 25 / 33

46 Main results Theorem For E Cµrn max{µr, log n}, OptSpace achieves, with high probability, RMSE C n r E ZE 2, provided that the RHS is smaller than σ r (M)/n. 2 is the spectral norm Keshavan, Montanari, Oh, Journ. Machine Learning Research, / 33

47 OptSpace is order-optimal when noise is i.i.d. Gaussian Theorem For E Cµrn max{µr, log n}, OptSpace achieves, with high probability, RMSE C σ z r n E provided that the RHS is smaller than σ r (M)/n. Lower bound: [Candès, Plan 09] RMSE σ 2 r n z E Trimming + SVD r n r n RMSE CM max + C σ z E E }{{}}{{} missing entries sample noise 27 / 33

48 Comparison rank-4 matrix M, Gaussian noise with σ z = 1 Example from [Candès, Plan 09] 1.5 Trim+SVD RMSE Lower Bound 0 OptSpace Sampling rate 28 / 33

49 Comparison rank-4 matrix M, Gaussian noise with σ z = 1 Example from [Candès, Plan 09] 1.5 Trim+SVD RMSE Lower Bound 0 OptSpace FPCA ADMiRA Sampling rate FPCA [Ma, Goldfarb, Chen 09], ADMiRA [Lee, Bresler 09] 28 / 33

50 Positioning 29 / 33

51 The model Distance Matrix D R n wireless devices uniformly distributed in a bounded convex region Distance measurements between devices within radio range R Find the locations up to a rigid motion 30 / 33

52 The model Distance Matrix D R How is it related to Matrix Completion? Need to find the missing entries rank(d) = 4 How is it different? Non-uniform sampling Rich information not used in Matrix Completion 30 / 33

53 The model Distance Matrix D R MDS-MAP [Shang et al. 03] 1. Fill in the missing entries with shortest paths 2. Compute rank-4 approximation 30 / 33

54 Main results Theorem log n For R > C n, with high probability, RMSE C log n R n ( RMSE = 1 n 2 i,j ( D D) 2 ij) 1/2 + o(1). Lower Bound: If R <, then the graph is disconnected log n πn Generalized to quantized measurements and distributed algorithms We can add Greedy Minimization step Oh, Karbasi, Montanari, Information Theory Workshop, 2010 Karbasi, Oh, ACM SIGMETRICS, / 33

55 Numerical simulation RMSE n=500 n=1,000 n=2, n=500 n=1,000 n=2, R/ log n πn R/ πn 32 / 33

56 Conclusion Matrix Completion is an important problem with many practical applications OptSpace is an efficient algorithm for Matrix Completion OptSpace achieves performance close to the fundamental limit 33 / 33

57 Special thanks to: PMCRQOAVNBETHRDAZOKNWXAOURHLIO 33 / 33

58 Special thanks to: PMCRQOAVNBETHRDAZOKNWXAOURHLIO 33 / 33

59 Special thanks to: PMCRQOAVNBETHRDAZOKNWXAOURHLIO 33 / 33

60 Special thanks to: PMCRQOAVNBETHRDAZOKNWXAOURHLIO 33 / 33

61 Special thanks to: PMCRQOAVNBETHRDAZOKNWXAOURHLIO 33 / 33

62 Special thanks to: Officemates: Morteza, Yash, Jose, Raghu, Satish Friends: Mohsen, Adel, Farshid, Fernando, Arian, Haim, Sachin, Ivana, Ahn, Cha, Choi, Rhee, Kang, Kim, Lee, Na, Park, Ra, Seok, Song 33 / 33

63 Special thanks to: My family and Kyung Eun 33 / 33

64 Thank you! 33 / 33

Matrix completion: Fundamental limits and efficient algorithms. Sewoong Oh Stanford University

Matrix completion: Fundamental limits and efficient algorithms Sewoong Oh Stanford University 1 / 35 Low-rank matrix completion Low-rank Data Matrix Sparse Sampled Matrix Complete the matrix from small