Robust Principal Component Analysis

Size: px

Start display at page:

Download "Robust Principal Component Analysis"

Corey Strickland
5 years ago
Views:

1 ELE 538B: Mathematics of High-Dimensional Data Robust Principal Component Analysis Yuxin Chen Princeton University, Fall 2018

2 Disentangling sparse and low-rank matrices Suppose we are given a matrix M = L }{{} low-rank + }{{} S R n n sparse Question: can we hope to recover both L and S from M? Robust PCA 12-2

3 Principal component analysis (PCA) N samples X = [x 1, x 2,..., x N ] R n N that are centered PCA: seeks r directions that explain most variance of data minimize L:rank(L)=r X L F best rank-r approximation of X Robust PCA 12-3

4 Sensitivity to corruptions / outliers What if some samples are corrupted (e.g. due to sensor errors / attacks)? Classical PCA fails even with a few outliers Robust PCA 12-4

5 Video surveillance Separation of background (low-rank) and foreground (sparse) Candès, Li, Ma, Wright 11 Robust PCA 12-5

6 Graph clustering / community recovery n nodes, 2 (or more) clusters A friendship graph G: for any pair (i, j), { 1, if (i, j) G M i,j = 0, else Edge density within clusters > edge density across clusters Goal: recover cluster structure Robust PCA 12-6

7 Graph clustering / community recovery M = }{{} L low-rank + M L }{{} sparse An equivalent goal: recover the ground truth matrix { 1, if i and j are in the same community L i,j = 0, else Clustering robust PCA Robust PCA 12-7

8 Gaussian graphical models Fact 12.1 Consider a Gaussian vector x N (0, Σ). For any u and v, x u x v x V\{u,v} iff Θ u,v = 0, where Θ = Σ 1 is the inverse covariance matrix conditional independence sparsity Robust PCA 12-8

Gaussian graphical models 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

9 Gaussian graphical models } {{ } Θ The inverse covariance matrix Θ is often sparse Robust PCA 12-9

10 Graphical models with latent factors What if one only observes a subset of variables? [ xo x h ] (observed variables) (hidden variables) x o = [x 1,, x 6], x h = [x 7, x 8] The covariance and precision matrices can be partitioned as observed part {}}{ Σ = Σ o Σ o,h [ ] 1 Θo Θ Σ o,h = o,h Θ o,h Θ Σ h h Robust PCA 12-10

11 Graphical models with latent factors What if one only observes a subset of variables? [ xo x h ] (observed variables) (hidden variables) x o = [x 1,, x 6], x h = [x 7, x 8] Σ 1 o }{{} observed = Θ }{{} o sparse h Θ h,o }{{} low-rank if # latent vars is small Θ o,h Θ 1 sparse + low-rank decomposition Robust PCA 12-10

12 When is decomposition possible? Identifiability issue: a matrix might be simultaneously low-rank and sparse! }{{} sparse and low-rank vs }{{} sparse but not low-rank Nonzero entries of sparse component need to be spread out This lecture: assume locations of the nonzero entries are random Robust PCA 12-11

13 When is decomposition possible? Identifiability issue: a matrix might be simultaneously low-rank and sparse! }{{} low-rank and dense vs }{{} low-rank but sparse The low-rank component needs to be incoherent Robust PCA 12-11

14 Low-rank component: coherence Definition 12.2 X Coherence parameter µ 1 of M = UΣV is the smallest quantity s.t. max i U e i 2 2 µ 1r 2n 6 4 h P U (e i ) and h e i max V e i 2 2 µ 1r i n i U Robust PCA 12-12

15 Low-rank component: joint coherence Definition 12.3 (Joint coherence) Joint coherence parameter µ 2 of M = UΣV is the smallest quantity s.t. UV µ2 r n 2 This prevents UV from being too peaky µ 1 µ 2 µ 2 1r, since (UV ) ij = e i UV e j e i U 2 V e j 2 µ1r n UV 2 UV e j 2 F n = V e j 2 2 n = µ1r n 2 (suppose V e j 2 2 = µ1r n ) Robust PCA 12-13

16 Convex relaxation minimize L,S rank(l) + λ S 0 s.t. M = L + S (12.1) minimize L,S L + λ S 1 s.t. M = L + S (12.2) : nuclear norm; 1 : entry-wise l 1 norm λ > 0: regularization parameter that balances two terms Robust PCA 12-14

17 Theoretical guarantee Theorem 12.4 (Candès, Li, Ma, Wright 11) rank(l) n max{µ 1,µ 2 } log 2 n ; Nonzero entries of S are randomly located, and S 0 ρ s n 2 for some constant ρ s > 0 (e.g. ρ s = 0.2). Then (12.2) with λ = 1/ n is exact with high prob. rank(l) can be quite high (up to n/polylog(n)) Parameter free: λ = 1/ n Ability to correct gross error: S 0 n 2 Sparse component S can have arbitrary magnitudes / signs! Robust PCA 12-15

18 Geometry Fig. credit: Candès 14 Robust PCA 12-16

19 Empirical success rate s rank(l)/n n = 400 Fig. credit: Candès, Li, Ma, Wright 11 Robust PCA 12-17

20 Dense error correction Theorem 12.5 (Ganesh, Wright, Li, Candès, Ma 10, Chen, Jalali, Sanghavi, Caramanis 13) rank(l) n max{µ 1,µ 2 } log 2 n ; Nonzero entries of S are randomly located, have random sign, and S 0 = ρ s n 2. Then (12.2) with λ 1 ρs ρ sn 1 ρ s }{{} non-corruption rate succeeds with high prob., provided that max{µ 1, µ 2 }r polylog(n) n When additive corruptions have random signs, (12.2) works even when a dominant fraction of the entries are corrupted Robust PCA 12-18

21 Is joint coherence needed? Matrix completion: does not need µ 2 Robust PCA: so far we need µ 2 Question: is µ 2 needed? can we recover L with rank up to n µ 1 polylog(n) (rather than n max{µ 1,µ 2 }polylog(n) )? Answer: no (example: planted clique) Robust PCA 12-19

22 Planted clique problem Setup: a graph G of n nodes generated as follows 1. connect each pair of nodes independently with prob pick n 0 nodes and make them a clique (fully connected) Goal: find the hidden clique from G Information theoretically, one can recover the clique if n 0 > 2 log 2 n Robust PCA 12-20

23 Conjecture on computational barrier Conjecture: constant ɛ > 0, if n 0 n 0.5 ɛ, then no tractable algorithm can find the clique from G with prob. 1 o(1) often used as a hardness assumption Lemma 12.6 If there is an algorithm that allows recovery of any L from M with rank(l), then the above conjecture is violated n µ 1 polylog(n) Robust PCA 12-21

24 Proof of Lemma 12.6 Suppose L is the true adjacency matrix, L i,j = { 1, if i, j are both in the clique 0, else Let A be the adjacency matrix of G, and generate M s.t. M i,j = Therefore, one can write M = L + { Ai,j, with prob. 2/3 0, else M L }{{} each entry is nonzero w.p. 1/3 Robust PCA 12-22

25 Proof of Lemma 12.6 Note that µ 1 = n n 0 and µ 2 = n2 n 2 0 If there is an algorithm that can recover any L of rank from M, then rank(l) = 1 n µ 1 polylog(n) n µ 1 polylog(n) n 0 polylog(n) But this contradicts the conjecture (which claims computational infeasibility to recover L unless n 0 n 0.5 o(1) ) Robust PCA 12-23

26 Matrix completion with corruptions What if we have missing data + corruptions? Observed entries M ij = L ij + S ij, (i, j) Ω for some observation set Ω, where S = (S ij ) is sparse A natural extension of RPCA minimize L,S L + λ S 1 s.t. P Ω (M) = P Ω (L + S) Theorems easily extend to this setting Robust PCA 12-24

27 Efficient algorithm In the presence of noise, one needs to solve minimize L,S L + λ S 1 + µ 2 M L S 2 F which can be solved efficiently Algorithm 12.1 Iterative soft-thresholding for t = 0, 1, : L t+1 ( = T 1/µ M S t) S t+1 ( = ψ λ/µ M L t+1) where T : singular-value thresholding operator; ψ: soft thresholding operator Robust PCA 12-25

28 Reference [1] Lecture notes, Advanced topics in signal processing (ECE 8201), Y. Chi, [2] Robust principal component analysis?, E. Candes, X. Li, Y. Ma, and J. Wright, Journal of ACM, [3] Rank-sparsity incoherence for matrix decomposition, V. Chandrasekaran, S. Sanghavi, P. Parrilo, and A. Willsky, SIAM Journal on Optimization, [4] Latent variable graphical model selection via convex optimization, V. Chandrasekaran, P. Parrilo, and A. Willsky, Annals of Statistics, [5] Incoherence-optimal matrix completion, Y. Chen, IEEE Transactions on Information Theory, [6] Dense error correction for low-rank matrices via principal component pursuit, A. Ganesh, J. Wright, X. Li, E. Candes, Y. Ma, ISIT, Robust PCA 12-26

29 Reference [7] Low-rank matrix recovery from errors and erasures, Y. Chen, A. Jalali, S. Sanghavi, C. Caramanis, IEEE Transactions on Information Theory, Robust PCA 12-27

Analysis of Robust PCA via Local Incoherence

Analysis of Robust PCA via Local Incoherence Huishuai Zhang Department of EECS Syracuse University Syracuse, NY 3244 hzhan23@syr.edu Yi Zhou Department of EECS Syracuse University Syracuse, NY 3244 yzhou35@syr.edu