BEYOND MATRIX COMPLETION

Transcription

1 BEYOND MATRIX COMPLETION ANKUR MOITRA MASSACHUSETTS INSTITUTE OF TECHNOLOGY Based on joint work with Boaz Barak (Harvard)

2 Part I: RecommendaJon systems and parjally observed matrices

3 THE NETFLIX PROBLEM movies users

4 THE NETFLIX PROBLEM movies users 1-5 stars

5 THE NETFLIX PROBLEM movies users 480,189 17,770

6 THE NETFLIX PROBLEM movies users 1% observed 480,189 17,770

7 THE NETFLIX PROBLEM movies users 1% observed 480,189 17,770 Can we (approximately) fill- in the missing entries?

8 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix

9 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M + + +

10 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M drama

11 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M drama comedy

12 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M drama comedy sports

13 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M drama comedy sports Model: we are given random observajons M i,j for all i,j Ω

14 MATRIX COMPLETION Let M be an unknown, approximately low- rank matrix M drama comedy sports Model: we are given random observajons M i,j for all i,j Ω Is there an efficient algorithm to recover M?

15 MATRIX COMPLETION The natural formulajon is non- convex, and NP- hard min rank(x) s.t. 1 Ω (i,j) Ω X i,j M i,j η

16 MATRIX COMPLETION The natural formulajon is non- convex, and NP- hard min rank(x) s.t. 1 Ω (i,j) Ω There is a powerful, convex relaxajon X i,j M i,j η

17 THE NUCLEAR NORM Consider the singular value decomposi;on of X: X = U Σ V T orthogonal diagonal orthogonal

18 THE NUCLEAR NORM Consider the singular value decomposi;on of X: X = U Σ V T orthogonal diagonal orthogonal Let σ 1 σ 2 σ r > σ r+1 = σ m = 0 be the singular values

19 THE NUCLEAR NORM Consider the singular value decomposi;on of X: X = U Σ V T orthogonal diagonal orthogonal Let σ 1 σ 2 σ r > σ r+1 = σ m = 0 be the singular values Then rank(x) = r, and X * = σ 1 + σ σ r (nuclear norm)

20 This yields a convex relaxajon, that can be solved efficiently: min X * s.t. 1 Ω (i,j) Ω X i,j M i,j η [Fazel], [Srebro, Shraibman], [Recht, Fazel, Parrilo], [Candes, Recht], [Candes, Tao], [Candes, Plan], [Recht], (P)

21 This yields a convex relaxajon, that can be solved efficiently: min X * s.t. 1 Ω (i,j) Ω X i,j M i,j η [Fazel], [Srebro, Shraibman], [Recht, Fazel, Parrilo], [Candes, Recht], [Candes, Tao], [Candes, Plan], [Recht], (P) Theorem: If M is n x n and has rank r, and is C- incoherent then (P) recovers M exactly from C 6 nrlog 2 n observajons

22 This yields a convex relaxajon, that can be solved efficiently: min X * s.t. 1 Ω (i,j) Ω X i,j M i,j η [Fazel], [Srebro, Shraibman], [Recht, Fazel, Parrilo], [Candes, Recht], [Candes, Tao], [Candes, Plan], [Recht], (P) Theorem: If M is n x n and has rank r, and is C- incoherent then (P) recovers M exactly from C 6 nrlog 2 n observajons This is nearly opjmal, since there are 2nr parameters

23 This yields a convex relaxajon, that can be solved efficiently: min X * s.t. 1 Ω (i,j) Ω X i,j M i,j η [Fazel], [Srebro, Shraibman], [Recht, Fazel, Parrilo], [Candes, Recht], [Candes, Tao], [Candes, Plan], [Recht], (P) Theorem: If M is n x n and has rank r, and is C- incoherent then (P) recovers M exactly from C 6 nrlog 2 n observajons This is nearly opjmal, since there are 2nr parameters Many other approaches, e.g. alterna;ng minimiza;on: [Keshavan, Montanari, Oh], [Jain, Netrapalli, Sanghavi], [Hardt],

24 Part II: Higher order structure?

25 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons?

26 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? ac;vi;es users e.g. Groupon

27 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? ;me users e.g. Groupon ;me: season, Jme of day, weekday/weekend, etc

28 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? users ;me T = r i = 1 e.g. Groupon snow sports ;me: season, Jme of day, weekday/weekend, etc

29 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? users ;me r T = a i b i c i i = 1

30 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? users ;me r T = a i b i c i i = 1 Can we (approximately) fill- in the missing entries?

31 TENSOR PREDICTION Can using more than two asributes can lead to beser recommendajons? users ;me r T = a i b i c i i = 1 Can we (approximately) fill- in the missing entries? More asributes lead to beser recommendajons, but more complex objects

32 THE TROUBLE WITH TENSORS Natural approach (suggested by many authors): min X s.t. * 1 Ω (i,j,k) Ω X i,j,k T i,j,k η (P) tensor nuclear norm

33 THE TROUBLE WITH TENSORS Natural approach (suggested by many authors): min X s.t. * 1 Ω (i,j,k) Ω X i,j,k T i,j,k η (P) tensor nuclear norm The tensor nuclear norm is NP- hard to compute! [Gurvits], [Liu], [Harrow, Montanaro]

34 In fact, most of the linear algebra toolkit is ill- posed, or computa;onally hard for tensors

35 In fact, most of the linear algebra toolkit is ill- posed, or computa;onally hard for tensors e.g. [Hillar, Lim] Most Tensor Problems are NP- Hard

36 In fact, most of the linear algebra toolkit is ill- posed, or computa;onally hard for tensors e.g. [Hillar, Lim] Most Tensor Problems are NP- Hard Table I. Tractability of Tensor Problems Problem Complexity Bivariate Matrix Functions over R, C Undecidable (Proposition 12.2) Bilinear System over R, C NP-hard (Theorems 2.6, 3.7, 3.8) Eigenvalue over R NP-hard (Theorem 1.3) Approximating Eigenvector over R NP-hard (Theorem 1.5) Symmetric Eigenvalue over R NP-hard (Theorem 9.3) Approximating Symmetric Eigenvalue over R NP-hard (Theorem 9.6) Singular Value over R, C NP-hard (Theorem 1.7) Symmetric Singular Value over R NP-hard (Theorem 10.2) Approximating Singular Vector over R, C NP-hard (Theorem 6.3) Spectral Norm over R NP-hard (Theorem 1.10) Symmetric Spectral Norm over R NP-hard (Theorem 10.2) Approximating Spectral Norm over R NP-hard (Theorem 1.11) Nonnegative Definiteness NP-hard (Theorem 11.2) Best Rank-1 Approximation NP-hard (Theorem 1.13) Best Symmetric Rank-1 Approximation NP-hard (Theorem 10.2) Rank over R or C NP-hard (Theorem 8.2) Enumerating Eigenvectors over R #P-hard (Corollary 1.16) Combinatorial Hyperdeterminant NP-, #P-, VNP-hard (Theorems 4.1, 4.2, Corollary 4.3) Geometric Hyperdeterminant Conjectures 1.9, 13.1 Symmetric Rank Conjecture 13.2 Bilinear Programming Conjecture 13.4 Bilinear Least Squares Conjecture 13.5 Note: Except for positive definiteness and the combinatorial hyperdeterminant, which apply to -tensors,

37 FLATTENING A TENSOR Many tensor methods rely on flatening:

38 FLATTENING A TENSOR Many tensor methods rely on flatening: n 3 flat( ) = n 1 n 1 n 2 n 3

39 FLATTENING A TENSOR Many tensor methods rely on flatening: n 3 flat( ) = n 1 users ac;vi;es ;me

40 FLATTENING A TENSOR Many tensor methods rely on flatening: flat( ) = n 1 n 3 i (j,k) T i,j,k

41 FLATTENING A TENSOR Many tensor methods rely on flatening: flat( ) = n 1 n 3 i (j,k) T i,j,k This is a rearrangement of the entries, into a matrix, that does not increase its rank

42 FLATTENING A TENSOR Many tensor methods rely on flatening: flat( ) = n 1 n 3 i r flat( a i b i c i i = 1 r (j,k) ) = a i T vec(b i c i ) i = 1 T i,j,k n 2 n 3 - dimensional vector

43 Let n 1 = n 2 = n 3 = n We would need O(n 2 r) observajons to fill- in flat(t)

44 Let n 1 = n 2 = n 3 = n We would need O(n 2 r) observajons to fill- in flat(t) There are many other variants of flatening, but with comparable guarantees [Liu, Musialski, Wonka, Ye], [Gandy, Recht, Yamada], [SignoreTo, De Lathauwer, Suykens], [Tomioko, Hayashi, Kashima], [Mu, Huang, Wright, Goldfarb],

45 Let n 1 = n 2 = n 3 = n We would need O(n 2 r) observajons to fill- in flat(t) There are many other variants of flatening, but with comparable guarantees [Liu, Musialski, Wonka, Ye], [Gandy, Recht, Yamada], [SignoreTo, De Lathauwer, Suykens], [Tomioko, Hayashi, Kashima], [Mu, Huang, Wright, Goldfarb], Can we beat flasening?

46 Let n 1 = n 2 = n 3 = n We would need O(n 2 r) observajons to fill- in flat(t) There are many other variants of flatening, but with comparable guarantees [Liu, Musialski, Wonka, Ye], [Gandy, Recht, Yamada], [SignoreTo, De Lathauwer, Suykens], [Tomioko, Hayashi, Kashima], [Mu, Huang, Wright, Goldfarb], Can we beat flasening? Can we make beser predicjons than we do by treajng each ac;vity x ;me as unrelated?

47 Part III: Nearly opjmal algorithms for tensor predicjon

48 OUR RESULTS r T = a i b i c i i = 1 σ i + noise standard Gaussian r.v.

49 OUR RESULTS r T = a i b i c i i = 1 σ i + noise standard Gaussian r.v. Theorem: Suppose var(t i,j,k ) r. Then there is an efficient algorithm that outputs X which sajsfies: X i,j,k = (1±o(1))T i,j,k for a 1- o(1) fracjon of entries, provided m = Ω (n 3/2 r)

50 OUR RESULTS r T = a i b i c i i = 1 σ i + noise standard Gaussian r.v. Theorem: Suppose var(t i,j,k ) r. Then there is an efficient algorithm that outputs X which sajsfies: X i,j,k = (1±o(1))T i,j,k for a 1- o(1) fracjon of entries, provided m = Ω (n 3/2 r) This variance bound holds for random tensors, but also tensors where the factors (a i s, b i s, c i s) have large inner- product

51 OUR RESULTS r T = a i b i c i i = 1 σ i + noise standard Gaussian r.v. Theorem: Suppose var(t i,j,k ) r. Then there is an efficient algorithm that outputs X which sajsfies: X i,j,k = (1±o(1))T i,j,k for a 1- o(1) fracjon of entries, provided m = Ω (n 3/2 r) Even for r = n 3/2- δ (highly overcomplete), we only need to observe an o(1) fracjon of the entries to predict almost everything

52 LOWER BOUNDS Not only is the tensor nuclear norm hard to compute, but

53 LOWER BOUNDS Not only is the tensor nuclear norm hard to compute, but Tensor predicjon with m observajons Refute random 3- SAT with m clauses

54 LOWER BOUNDS Not only is the tensor nuclear norm hard to compute, but Tensor predicjon with m observajons Refute random 3- SAT with m clauses The best known algorithms require m = n 3/2, and even powerful SDP hierarchies fail with fewer clauses [Shor], [Nesterov], [Parrilo], [Lasserre], [Grigoriev], [Schoenebeck]

55 LOWER BOUNDS Not only is the tensor nuclear norm hard to compute, but Tensor predicjon with m observajons Refute random 3- SAT with m clauses The best known algorithms require m = n 3/2, and even powerful SDP hierarchies fail with fewer clauses [Shor], [Nesterov], [Parrilo], [Lasserre], [Grigoriev], [Schoenebeck] Corollary [informal]: Any algorithm for solving tensor predicjon, in the sum- of- squares hierarchy that uses m=n 3/2- δ r observajons must run in exponenjal Jme

56 Part IV: Our techniques: connecjons to random CSPs

57 Can we disjnguish between low- rank and random?

58 Can we disjnguish between low- rank and random? Case #1: Approximately low- rank a T a + noise

59 Can we disjnguish between low- rank and random? Case #1: Approximately low- rank a T a + noise For each (i,j) Ω M i,j = a i a j w/ probability ¾ random ±1 w/ probability ¼ where each a i = ±1

60 Can we disjnguish between low- rank and random?

61 Can we disjnguish between low- rank and random? Case #2: Random random For each (i,j) Ω, M i,j = random ±1

62 Can we disjnguish between low- rank and random? Case #2: Random random For each (i,j) Ω, M i,j = random ±1 In Case #1 the entries are (somewhat) predictable, but in Case #2 they are completely unpredictable

63 There are two very different communijes that (essenjally) asacked this same disjnguishing problem:

64 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on

65 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs

66 AN INTERPRETATION We can interpret: (i 1, j 1 ; σ 1 ), (i 2, j 2 ; σ 2 ),, (i m, j m ; σ m ) ±1 r.v. as a random 2- XOR formula ψ

67 AN INTERPRETATION We can interpret: (i 1, j 1 ; σ 1 ), (i 2, j 2 ; σ 2 ),, (i m, j m ; σ m ) ±1 r.v. as a random 2- XOR formula ψ In parjcular each observajon/fctn value maps to a clause: (i, j, σ) v i v j = σ variables constraint

68 AN INTERPRETATION We can interpret: (i 1, j 1 ; σ 1 ), (i 2, j 2 ; σ 2 ),, (i m, j m ; σ m ) ±1 r.v. as a random 2- XOR formula ψ (and vice- versa) In parjcular each observajon/fctn value maps to a clause: (i, j, σ) v i v j = σ variables constraint

69 STRONG REFUTATION We will say that an algorithm strongly refutes* random 2- XOR with m clauses if:

70 STRONG REFUTATION We will say that an algorithm strongly refutes* random 2- XOR with m clauses if: (1) On any 2- XOR formula ψ, it outputs val where: OPT(ψ) val(ψ)

71 STRONG REFUTATION We will say that an algorithm strongly refutes* random 2- XOR with m clauses if: (1) On any 2- XOR formula ψ, it outputs val where: OPT(ψ) val(ψ) largest fracjon of clauses of ψ that can be sajsfied

72 STRONG REFUTATION We will say that an algorithm strongly refutes* random 2- XOR with m clauses if: (1) On any 2- XOR formula ψ, it outputs val where: OPT(ψ) val(ψ)

73 STRONG REFUTATION We will say that an algorithm strongly refutes* random 2- XOR with m clauses if: (1) On any 2- XOR formula ψ, it outputs val where: OPT(ψ) val(ψ) (2) With high probability (for random ψ with m clauses): val(ψ) = o(1)

74 Lemma: If (i 1, j 1 ; σ 1 ),, (i m, j m ; σ m ) j ψ then σ i A 2 OPT(ψ) 1 n 1 A m 2

75 Lemma: If (i 1, j 1 ; σ 1 ),, (i m, j m ; σ m ) j ψ then σ i A 2 OPT(ψ) 1 n 1 A m 2 Proof: Map the assignment to a unit vector so that x i = ±1/ n and take the quadrajc form on A

76 Lemma: If (i 1, j 1 ; σ 1 ),, (i m, j m ; σ m ) j ψ then σ i A 2 OPT(ψ) 1 n 1 A m 2 Proof: Map the assignment to a unit vector so that x i = ±1/ n and take the quadrajc form on A 1 m A 1 mn

77 Lemma: If (i 1, j 1 ; σ 1 ),, (i m, j m ; σ m ) j ψ then σ i A 2 OPT(ψ) 1 n 1 A m 2 Proof: Map the assignment to a unit vector so that x i = ±1/ n and take the quadrajc form on A 1 m A 1 mn m = ω(n) OPT(ψ) o(1)

78 Lemma: If (i 1, j 1 ; σ 1 ),, (i m, j m ; σ m ) j ψ then σ i A 2 OPT(ψ) 1 n 1 A m 2 Proof: Map the assignment to a unit vector so that x i = ±1/ n and take the quadrajc form on A 1 m A 1 mn m = ω(n) OPT(ψ) o(1) This solves the strong refutajon problem

79 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs

80 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs Noisy matrix complejon with m observajons Rademacher Complexity Strongly refute* random 2- XOR/2- SAT with m clauses *Want an algorithm that cerjfies a formula is far from sajsfiable

81 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs Noisy tensor complejon with m observajons Rademacher Complexity Strongly refute* random 3- XOR/3- SAT with m clauses *Want an algorithm that cerjfies a formula is far from sajsfiable

82 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs Noisy tensor complejon with m observajons Rademacher Complexity Strongly refute* random 3- XOR/3- SAT with m clauses [Coja- Oghlan, Goerdt, Lanka] *Want an algorithm that cerjfies a formula is far from sajsfiable

83 There are two very different communijes that (essenjally) asacked this same disjnguishing problem: The community working on matrix comple;on The community working on refu;ng random CSPs Noisy tensor complejon with m observajons Embedding in SOS Strongly refute* random 3- XOR/3- SAT with m clauses [Coja- Oghlan, Goerdt, Lanka] We then embed this algorithm into the sixth level of the sum- of- squares hierarchy, to get a relaxajon for tensor predicjon *Want an algorithm that cerjfies a formula is far from sajsfiable

84 GENERALIZATION BOUNDS Suppose we are given Ω = m noisy observajons T i,j,k ± η, and the factors of T are C- incoherent: Theorem: There is an efficient algorithm that with prob 1- δ, outputs X with 1 n 3 i,j,k X i,j,k T i,j,k C 3 r n 3/2 log 4 n m + 2C 3 r ln(2/δ) m + 2η

85 GENERALIZATION BOUNDS Suppose we are given Ω = m noisy observajons T i,j,k ± η, and the factors of T are C- incoherent: Theorem: There is an efficient algorithm that with prob 1- δ, outputs X with 1 n 3 i,j,k X i,j,k T i,j,k C 3 r n 3/2 log 4 n m + 2C 3 r ln(2/δ) m + 2η This comes from giving an efficiently computable norm whose Rademacher complexity is asymptojcally smaller than the trivial bound whenever m=ω(n 3/2 log 4 n) K

86 SUMMARY New Algorithm: We gave an algorithm for 3 rd - order tensor predicjon that uses m = n 3/2 rlog 4 n observajons

87 SUMMARY New Algorithm: We gave an algorithm for 3 rd - order tensor predicjon that uses m = n 3/2 rlog 4 n observajons An Inefficient Algorithm: (via tensor nuclear norm) There is an inefficient algorithm that use m = nrlogn observajons

88 SUMMARY New Algorithm: We gave an algorithm for 3 rd - order tensor predicjon that uses m = n 3/2 rlog 4 n observajons An Inefficient Algorithm: (via tensor nuclear norm) There is an inefficient algorithm that use m = nrlogn observajons A Phase Transi;on: Even for n δ rounds of the powerful sum- of- squares hierarchy, no norm solves tensor predicjon with m = n 3/2- δ r observajons

89 Epilogue: New direcjons in linear inverse problems

90 Robust PCA [Candes et al.], [Chandrasekaran et al.], Can we recover a low rank matrix from sparse corrupjons?

91 Robust PCA [Candes et al.], [Chandrasekaran et al.], Can we recover a low rank matrix from sparse corrupjons? Superresolu;on, compressed sensing off- the- grid [Candes, Fernandez- Granda], [Tang et al.], Can we recover well- separated points from low- frequency measurements? f 1 f 2 f 3 f 4

92 DISCUSSION Convex programs are unreasonably effecjve for linear inverse problems!

93 DISCUSSION Convex programs are unreasonably effecjve for linear inverse problems! But we gave simple linear inverse problems that exhibit striking gaps between efficient and inefficient esjmators

94 DISCUSSION Convex programs are unreasonably effecjve for linear inverse problems! But we gave simple linear inverse problems that exhibit striking gaps between efficient and inefficient esjmators Where else are there computajonal vs stajsjcal tradeoffs?

95 DISCUSSION Convex programs are unreasonably effecjve for linear inverse problems! But we gave simple linear inverse problems that exhibit striking gaps between efficient and inefficient esjmators Where else are there computajonal vs stajsjcal tradeoffs? New Direc;on: Explore computajonal vs. stajsjcal tradeoffs through the powerful sum- of- squares hierarchy