Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization

Transcription

1 Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization Nicolas Gillis Department of Mathematics and Operational Research Université de Mons, Belgium Joint work with Stephen Vavasis (U. of Waterloo) itwist 14 itwist 14 SDP Preconditioning for Separable NMF 1

2 Outline 1. Why Nonnegative Matrix Factorization (NMF)? Definition, motivations & applications 2. How to Solve Nonnegative Matrix Factorization (NMF)? A subclass of efficiently solvable NMF problems (separable NMF) Projection methods SDP-based Preconditioning itwist 14 SDP Preconditioning for Separable NMF 2

3 Part 1 Why? itwist 14 SDP Preconditioning for Separable NMF 3

4 Nonnegative Matrix Factorization (NMF) Given a matrix M R p n + and a factorization rank r min(p, n), find U R p r and V R r n such that min M UV U 0,V 0 2 F = (M UV ) 2 ij. (NMF) i,j NMF is a linear dimensionality reduction technique for nonnegative data : M(:, i) }{{} 0 r k=1 U(:, k) }{{} 0 V (k, i) }{{} 0 for all i. Why nonnegativity? Interpretability: Nonnegativity constraints lead to easily interpretable factors (and a sparse and part-based representation). Many applications. image processing, text mining, hyperspectral unmixing, community detection, clustering, etc. itwist 14 SDP Preconditioning for Separable NMF 4

5 Nonnegative Matrix Factorization (NMF) Given a matrix M R p n + and a factorization rank r min(p, n), find U R p r and V R r n such that min M UV U 0,V 0 2 F = (M UV ) 2 ij. (NMF) i,j NMF is a linear dimensionality reduction technique for nonnegative data : M(:, i) }{{} 0 r k=1 U(:, k) }{{} 0 V (k, i) }{{} 0 for all i. Why nonnegativity? Interpretability: Nonnegativity constraints lead to easily interpretable factors (and a sparse and part-based representation). Many applications. image processing, text mining, hyperspectral unmixing, community detection, clustering, etc. itwist 14 SDP Preconditioning for Separable NMF 4

6 Nonnegative Matrix Factorization (NMF) Given a matrix M R p n + and a factorization rank r min(p, n), find U R p r and V R r n such that min M UV U 0,V 0 2 F = (M UV ) 2 ij. (NMF) i,j NMF is a linear dimensionality reduction technique for nonnegative data : M(:, i) }{{} 0 r k=1 U(:, k) }{{} 0 V (k, i) }{{} 0 for all i. Why nonnegativity? Interpretability: Nonnegativity constraints lead to easily interpretable factors (and a sparse and part-based representation). Many applications. image processing, text mining, hyperspectral unmixing, community detection, clustering, etc. itwist 14 SDP Preconditioning for Separable NMF 4

7 Example 1: topic recovery and document classification Basis elements allow to recover the different topics; Weights allow to assign each text to its corresponding topics. itwist 14 SDP Preconditioning for Separable NMF 5

8 Example 1: topic recovery and document classification Basis elements allow to recover the different topics; Weights allow to assign each text to its corresponding topics. itwist 14 SDP Preconditioning for Separable NMF 5

9 Example 1: topic recovery and document classification Basis elements allow to recover the different topics; Weights allow to assign each text to its corresponding topics. itwist 14 SDP Preconditioning for Separable NMF 5

10 Exemple 2: feature extraction and classification U 0 constraints the basis elements to be nonnegative. Moreover V 0 imposes an additive reconstruction. The basis elements extract facial features such as eyes, nose and lips. itwist 14 SDP Preconditioning for Separable NMF 6

11 Exemple 2: feature extraction and classification U 0 constraints the basis elements to be nonnegative. Moreover V 0 imposes an additive reconstruction. The basis elements extract facial features such as eyes, nose and lips. itwist 14 SDP Preconditioning for Separable NMF 6

12 Exemple 2: feature extraction and classification U 0 constraints the basis elements to be nonnegative. Moreover V 0 imposes an additive reconstruction. The basis elements extract facial features such as eyes, nose and lips. itwist 14 SDP Preconditioning for Separable NMF 6

13 Example 3: blind hyperspectral unmixing Figure : Urban hyperspectral image with 162 spectral bands and 307-by-307 pixels. itwist 14 SDP Preconditioning for Separable NMF 7

14 Example 3: blind hyperspectral unmixing Basis elements allow to recover the different materials; Weights allow to know which pixel contains which material. itwist 14 SDP Preconditioning for Separable NMF 8

15 Example 3: blind hyperspectral unmixing Basis elements allow to recover the different materials; Weights allow to know which pixel contains which material. itwist 14 SDP Preconditioning for Separable NMF 8

16 Example 3: blind hyperspectral unmixing Basis elements allow to recover the different materials; Weights allow to know which pixel contains which material. itwist 14 SDP Preconditioning for Separable NMF 8

17 Example 3: blind hyperspectral unmixing Figure : Decomposition of the Urban dataset. itwist 14 SDP Preconditioning for Separable NMF 9

18 Example 3: blind hyperspectral unmixing Figure : Decomposition of the Urban dataset. itwist 14 SDP Preconditioning for Separable NMF 9

19 Example 3: blind hyperspectral unmixing Figure : Decomposition of the Urban dataset. itwist 14 SDP Preconditioning for Separable NMF 9

20 Part 2 How? itwist 14 SDP Preconditioning for Separable NMF 10

21 Can we only solve NMF problems? Given a matrix M R p n + and a factorization rank r N, find U R p r and V R r n such that min M UV U 0,V 0 2 F = (M UV ) 2 ij. (NMF) i,j NMF is NP-hard [V09], and highly ill-posed [G12]. In practice, it is often satisfactory to use locally optimal solutions for further analysis of the data. In other words, heuristics often solve the problem efficiently with acceptable answers. Try to analyze this state of affairs by considering generative models and algorithms that can recover hidden data. [V09] Vavasis, On the Complexity of Nonnegative Matrix Factorization, SIAM J. Optim., [G12] G., Sparse and Unique Nonnegative Matrix Factorization Through Data Preprocessing, JMLR, itwist 14 SDP Preconditioning for Separable NMF 11

22 Can we only solve NMF problems? Given a matrix M R p n + and a factorization rank r N, find U R p r and V R r n such that min M UV U 0,V 0 2 F = (M UV ) 2 ij. (NMF) i,j NMF is NP-hard [V09], and highly ill-posed [G12]. In practice, it is often satisfactory to use locally optimal solutions for further analysis of the data. In other words, heuristics often solve the problem efficiently with acceptable answers. Try to analyze this state of affairs by considering generative models and algorithms that can recover hidden data. [V09] Vavasis, On the Complexity of Nonnegative Matrix Factorization, SIAM J. Optim., [G12] G., Sparse and Unique Nonnegative Matrix Factorization Through Data Preprocessing, JMLR, itwist 14 SDP Preconditioning for Separable NMF 11

23 Separability Assumption Heuristics often solve the problem efficiently with acceptable answers (usually one needs to use additional regularization on U and V ). Try to analyze this state of affairs by considering generative models and algorithms that can recover hidden data. NMF can be solved in polynomial time if M is separable (rather strong condition) [AGKM12]. Separability of M = there exists an NMF (U, V ) 0 with M = UV where each column of U is equal to a column of M. [AGKM12] Arora, Ge, Kannan, Moitra, Computing a Nonnegative Matrix Factorization Provably, STOC itwist 14 SDP Preconditioning for Separable NMF 12

24 Separability Assumption Heuristics often solve the problem efficiently with acceptable answers (usually one needs to use additional regularization on U and V ). Try to analyze this state of affairs by considering generative models and algorithms that can recover hidden data. NMF can be solved in polynomial time if M is separable (rather strong condition) [AGKM12]. Separability of M = there exists an NMF (U, V ) 0 with M = UV where each column of U is equal to a column of M. [AGKM12] Arora, Ge, Kannan, Moitra, Computing a Nonnegative Matrix Factorization Provably, STOC itwist 14 SDP Preconditioning for Separable NMF 12

25 Is separability a reasonable assumption? Text mining: for each topic, there is a pure document on that topic, or, for each topic, there is a anchor word used only by that topic. [KSK13] Kumar, Sindhwani, Kambadur, Fast Conical Hull Algorithms for Near-separable Non-Negative Matrix Factorization, ICML [A+13] Arora, Ge, Halpern, Mimno, Moitra, Sontag, Wu, Zhu, A Practical Algorithm for Topic Modeling with Provable Guarantees, ICML [DRIS13] Ding, Rohban, Ishwar, Saligrama, Topic Discovery through Data Dependent and Random Projections, ICML Hyperspectral unmixing: separability is particularly natural: for each constitutive material, there is a pure pixel containing only that material. This is the so called pure-pixel assumption. [Ma+13] Ma, Bioucas-Dias, Chan, G., Gader, Plaza, Ambikapathi, Chi, Signal Processing Perspective on Hyperspectral Unmixing, IEEE Signal Proc. Mag., General image processing: No. itwist 14 SDP Preconditioning for Separable NMF 13

26 Is separability a reasonable assumption? Text mining: for each topic, there is a pure document on that topic, or, for each topic, there is a anchor word used only by that topic. [KSK13] Kumar, Sindhwani, Kambadur, Fast Conical Hull Algorithms for Near-separable Non-Negative Matrix Factorization, ICML [A+13] Arora, Ge, Halpern, Mimno, Moitra, Sontag, Wu, Zhu, A Practical Algorithm for Topic Modeling with Provable Guarantees, ICML [DRIS13] Ding, Rohban, Ishwar, Saligrama, Topic Discovery through Data Dependent and Random Projections, ICML Hyperspectral unmixing: separability is particularly natural: for each constitutive material, there is a pure pixel containing only that material. This is the so called pure-pixel assumption. [Ma+13] Ma, Bioucas-Dias, Chan, G., Gader, Plaza, Ambikapathi, Chi, Signal Processing Perspective on Hyperspectral Unmixing, IEEE Signal Proc. Mag., General image processing: No. itwist 14 SDP Preconditioning for Separable NMF 13

27 Is separability a reasonable assumption? Text mining: for each topic, there is a pure document on that topic, or, for each topic, there is a anchor word used only by that topic. [KSK13] Kumar, Sindhwani, Kambadur, Fast Conical Hull Algorithms for Near-separable Non-Negative Matrix Factorization, ICML [A+13] Arora, Ge, Halpern, Mimno, Moitra, Sontag, Wu, Zhu, A Practical Algorithm for Topic Modeling with Provable Guarantees, ICML [DRIS13] Ding, Rohban, Ishwar, Saligrama, Topic Discovery through Data Dependent and Random Projections, ICML Hyperspectral unmixing: separability is particularly natural: for each constitutive material, there is a pure pixel containing only that material. This is the so called pure-pixel assumption. [Ma+13] Ma, Bioucas-Dias, Chan, G., Gader, Plaza, Ambikapathi, Chi, Signal Processing Perspective on Hyperspectral Unmixing, IEEE Signal Proc. Mag., General image processing: No. itwist 14 SDP Preconditioning for Separable NMF 13

28 Geometric Interpretation of Separability Using normalization, the entries of each column of M, U and V can be assumed to sum to one: for all j, M(:, j) = r U(:, k)v (k, j) where V (k, j) 0 and k=1 r V (k, j) = 1. The columns of U are the vertices of the convex hull of the columns of M. k=1 itwist 14 SDP Preconditioning for Separable NMF 14

29 Separable NMF M is r-separable M = U[I r, V ]Π, for some V 0, and some permutation matrix Π. itwist 14 SDP Preconditioning for Separable NMF 15

30 Near-Separable NMF M = U[I r, V ]Π + N, where N is noise. itwist 14 SDP Preconditioning for Separable NMF 15

31 Near-Separable NMF: Noise and Conditioning We will assume that the noise is bounded (but otherwise arbitrary): N(:, j) 2 ɛ, for all j, and some dependence on some condition number is unavoidable: itwist 14 SDP Preconditioning for Separable NMF 16

32 Near-Separable NMF: Noise and Conditioning We will assume that the noise is bounded (but otherwise arbitrary): N(:, j) 2 ɛ, for all j, and some dependence on some condition number is unavoidable: itwist 14 SDP Preconditioning for Separable NMF 16

33 Near-Separable NMF: Noise and Conditioning We will assume that the noise is bounded (but otherwise arbitrary): N(:, j) 2 ɛ, for all j, and some dependence on some condition number is unavoidable: itwist 14 SDP Preconditioning for Separable NMF 16

34 Successive Projection Algorithm (SPA) 0: Initially K =. For i = 1 : r 1: Find j such that M(:, j) is max. 2: K = K {j }. 3: M ( I uu T ) M where u = M(:,j ) M(:,j ) 2. end modified Gram-Schmidt with column pivoting. ) Theorem. If ɛ O, SPA satisfies ( σmin (U) rκ 2 (U) min M M(:, K)V O ( ɛκ 2 (U) ). V 0 Advantages. Extremely fast, no parameter. Drawbacks. Requires U to be full rank; bound is weak. [GV12] G., Vavasis, Fast and Robust Recursive Algorithms for Separable Nonnegative Matrix Factorization, IEEE Trans. Patt. Anal. Mach. Intell. 36 (4), pp , itwist 14 SDP Preconditioning for Separable NMF 17

35 Successive Projection Algorithm (SPA) 0: Initially K =. For i = 1 : r 1: Find j such that M(:, j) is max. 2: K = K {j }. 3: M ( I uu T ) M where u = M(:,j ) M(:,j ) 2. end modified Gram-Schmidt with column pivoting. ) Theorem. If ɛ O, SPA satisfies ( σmin (U) rκ 2 (U) min M M(:, K)V O ( ɛκ 2 (U) ). V 0 Advantages. Extremely fast, no parameter. Drawbacks. Requires U to be full rank; bound is weak. [GV12] G., Vavasis, Fast and Robust Recursive Algorithms for Separable Nonnegative Matrix Factorization, IEEE Trans. Patt. Anal. Mach. Intell. 36 (4), pp , itwist 14 SDP Preconditioning for Separable NMF 17

36 Pre-conditioning for More Robust SPA Observation. Pre-multiplying M preserves separability: P M = P (U[I r, V ] + N) = (P U) [I, V ] + P N. Ideally, P = U 1 so that κ(p U) = 1 (assuming p = r). Solving the minimum volume ellipsoid centered at the origin and containing all the columns of M (which is SDP representable) min A S r + log det(a) 1 s.t. m j T A m j 1 j, allows to approximate U 1 : in fact, A (UU T ) 1. Theorem. If ɛ O ( σmin (U) r r ), preconditioned SPA satisfies min M M(:, K)V O (ɛκ(u)). V 0 [GV13] G., Vavasis, SDP-based Preconditioning for More Robust Near-Separable NMF, arxiv: itwist 14 SDP Preconditioning for Separable NMF 18

37 Pre-conditioning for More Robust SPA Observation. Pre-multiplying M preserves separability: P M = P (U[I r, V ] + N) = (P U) [I, V ] + P N. Ideally, P = U 1 so that κ(p U) = 1 (assuming p = r). Solving the minimum volume ellipsoid centered at the origin and containing all the columns of M (which is SDP representable) min A S r + log det(a) 1 s.t. m j T A m j 1 j, allows to approximate U 1 : in fact, A (UU T ) 1. Theorem. If ɛ O ( σmin (U) r r ), preconditioned SPA satisfies min M M(:, K)V O (ɛκ(u)). V 0 [GV13] G., Vavasis, SDP-based Preconditioning for More Robust Near-Separable NMF, arxiv: itwist 14 SDP Preconditioning for Separable NMF 18

38 Pre-conditioning for More Robust SPA Observation. Pre-multiplying M preserves separability: P M = P (U[I r, V ] + N) = (P U) [I, V ] + P N. Ideally, P = U 1 so that κ(p U) = 1 (assuming p = r). Solving the minimum volume ellipsoid centered at the origin and containing all the columns of M (which is SDP representable) min A S r + log det(a) 1 s.t. m j T A m j 1 j, allows to approximate U 1 : in fact, A (UU T ) 1. Theorem. If ɛ O ( σmin (U) r r ), preconditioned SPA satisfies min M M(:, K)V O (ɛκ(u)). V 0 [GV13] G., Vavasis, SDP-based Preconditioning for More Robust Near-Separable NMF, arxiv: itwist 14 SDP Preconditioning for Separable NMF 18

39 Pre-conditioning for More Robust SPA Observation. Pre-multiplying M preserves separability: P M = P (U[I r, V ] + N) = (P U) [I, V ] + P N. Ideally, P = U 1 so that κ(p U) = 1 (assuming p = r). Solving the minimum volume ellipsoid centered at the origin and containing all the columns of M (which is SDP representable) min A S r + log det(a) 1 s.t. m j T A m j 1 j, allows to approximate U 1 : in fact, A (UU T ) 1. Theorem. If ɛ O ( σmin (U) r r ), preconditioned SPA satisfies min M M(:, K)V O (ɛκ(u)). V 0 [GV13] G., Vavasis, SDP-based Preconditioning for More Robust Near-Separable NMF, arxiv: itwist 14 SDP Preconditioning for Separable NMF 18

40 Geometric Interpretation Figure : Geometric Interpretation of the SDP-based Preconditioning. itwist 14 SDP Preconditioning for Separable NMF 19

41 Synthetic data sets Each entry of U R uniform in [0, 1]; each column normalized. The other columns of M are the middle points of the columns of U (hence there are ( 20 2 ) = 190). The noise moves the middle points toward the outside of the convex hull of the column of U. itwist 14 SDP Preconditioning for Separable NMF 20

42 Synthetic data sets Each entry of U R uniform in [0, 1]; each column normalized. The other columns of M are the middle points of the columns of U (hence there are ( 20 2 ) = 190). The noise moves the middle points toward the outside of the convex hull of the column of U. itwist 14 SDP Preconditioning for Separable NMF 20

43 Results for the synthetic data sets Figure : Average of the percentage of columns correctly extracted depending on the noise level (for each noise level, 10 matrices are generated). itwist 14 SDP Preconditioning for Separable NMF 21

44 Hubble telescope hyperspectral image Figure : Sample of images for the Hubble telescope hyperspectral image with 100 spectral bands and pixels. itwist 14 SDP Preconditioning for Separable NMF 22

45 Hubble telescope hyperspectral image Figure : Spectral signatures extracted by SPA, corresponding to constitutive materials (matrix U with κ(u) = 115). itwist 14 SDP Preconditioning for Separable NMF 22

46 Hubble telescope hyperspectral image Figure : Reconstructed abundance maps (matrix V ). itwist 14 SDP Preconditioning for Separable NMF 22

47 Hubble telescope with blur and noise Figure : Sample of images for the Hubble telescope. itwist 14 SDP Preconditioning for Separable NMF 23

48 Hubble telescope with blur and noise Figure : Spectral signatures extracted by SPA, corresponding to constitutive materials (matrix U). itwist 14 SDP Preconditioning for Separable NMF 23

49 Hubble telescope with blur and noise Figure : Reconstructed abundance maps (matrix V ). With the blur and noise, SPA fails to identify good columns (two correspond to the same material). itwist 14 SDP Preconditioning for Separable NMF 23

50 Hubble telescope with blur and noise Figure : Spectral signatures extracted by preconditioned SPA, corresponding to constitutive materials (matrix U). itwist 14 SDP Preconditioning for Separable NMF 24

51 Hubble telescope with blur and noise Figure : Reconstructed abundance maps (matrix V ). With the blur and noise, preconditioned SPA is able to identify the right columns. itwist 14 SDP Preconditioning for Separable NMF 24

52 Conclusion 1. Nonnegative matrix factorization (NMF) Easily interpretable linear dimensionality reduction technique for nonnegative data, with many applications 2. Separable NMF Separability makes NMF problems efficiently solvable Need for fast, practical and robust algorithms SPA, a fast and robust recursive algorithm for separable NMF (simple to implement, no parameter tuning, no need to recompute the solution from scratch when r is changed). Pre-conditioning allows to make it much more robust to noise. Other robust algorithms, e.g., based on linear optimization (G. and Luce, Robust Near-Separable Nonnegative Matrix Factorization Using Linear Optimization, JMLR, 2014). itwist 14 SDP Preconditioning for Separable NMF 25

53 Conclusion 1. Nonnegative matrix factorization (NMF) Easily interpretable linear dimensionality reduction technique for nonnegative data, with many applications 2. Separable NMF Separability makes NMF problems efficiently solvable Need for fast, practical and robust algorithms SPA, a fast and robust recursive algorithm for separable NMF (simple to implement, no parameter tuning, no need to recompute the solution from scratch when r is changed). Pre-conditioning allows to make it much more robust to noise. Other robust algorithms, e.g., based on linear optimization (G. and Luce, Robust Near-Separable Nonnegative Matrix Factorization Using Linear Optimization, JMLR, 2014). itwist 14 SDP Preconditioning for Separable NMF 25

54 Conclusion 1. Nonnegative matrix factorization (NMF) Easily interpretable linear dimensionality reduction technique for nonnegative data, with many applications 2. Separable NMF Separability makes NMF problems efficiently solvable Need for fast, practical and robust algorithms SPA, a fast and robust recursive algorithm for separable NMF (simple to implement, no parameter tuning, no need to recompute the solution from scratch when r is changed). Pre-conditioning allows to make it much more robust to noise. Other robust algorithms, e.g., based on linear optimization (G. and Luce, Robust Near-Separable Nonnegative Matrix Factorization Using Linear Optimization, JMLR, 2014). itwist 14 SDP Preconditioning for Separable NMF 25

55 Thank you for your attention! Code and papers available on [GV13] G., Vavasis, SDP-based Preconditioning for More Robust Near-Separable NMF, arxiv: itwist 14 SDP Preconditioning for Separable NMF 26