Closed-Form Solutions in Low-Rank Subspace Recovery Models and Their Implications. Zhouchen Lin ( 林宙辰 ) 北京大学 Nov. 7, 2015

Transcription

1 Closed-Form Solutions in Low-Rank Subspace Recovery Models and Their Implications Zhouchen Lin ( 林宙辰 ) 北京大学 Nov. 7, 2015

2 Outline Sparsity vs. Low-rankness Closed-Form Solutions of Low-rank Models Applications of Closed-Form Solutions Conclusions

3 Challenges of High-dim Data Images > 1M dim Videos > 1B dim Web data > 10B+ dim??? Courtesy of Y. Ma.

4 Sparsity vs. Low-rankness

5 Sparse Models Sparse Representation min jjzjj 0 ; s:t: x = Az: (1) Sparse Subspace Clustering m in jjz i jj 0 ; s:t: x i = X^iz i ; 8i: (2) where X^i = [x 1 ; ; x i 1 ; x i+1 ; ; x n ]. m in jjz jj 0 ; s:t: X = X Z ; diag(z ) = 0: (3) m in jjz jj 1 ; s:t: X = X Z ; diag(z ) = 0: Elhamifar and Vidal. Sparse Subspace Clustering. CVPR2009. (4)

6 Low-rank Models Matrix Completion (MC) Robust PCA m in rank(a ); s:t: D = ¼ (A ): m in rank(a ) + ke k l0 ; s:t: D = A + E : Low-rank Representation (LRR) min rank(z) + jjejj 2;0 ; s:t: X = XZ + E: Filling in missing entries Denoising Clustering ke k l0 = # fe ij je ij 6= 0g jjejj 2;0 = #fijjje :;i jj 2 6= 0g. NP Hard!

7 Convex Program Formulation Matrix Completion (MC) Robust PCA m in ka k ; s:t: D = ¼ (A ): m in ka k + ke k l1 ; s:t: D = A + E : Low-rank Representation (LRR) min jjzjj + jjejj 2;1 ; s:t: X = XZ + E: ka k = P i ke k l1 = P i;j ¾ i (A ); je ij j: nuclear norm jjejj 2;1 = P i jje :;i jj 2.

8 Applications of Low-rank Models Background modeling Robust Alignment Image Rectification Motion Segmentation Image Segmentation Saliency Detection Image Tag Refinement Partial Duplicate Image Search 林宙辰 马毅, 信号与数据处理中的低秩模型, 中国计算机学会通讯,2015 年第 4 期

9 Closed-form Solution of LRR Closed form solution at noiseless case m in kz k ; Z s:t: X = X Z ; has a unique closed-form optim al solution: Z = V r V T r skinny SV D of X. Shape Interaction Matrix, w here U r r V T r when X is sampled from independent subspaces, Z* is block diagonal, each block corresponding to a subspace is th e min kzk = rank(x): X=XZ Wei and Lin. Analysis and Improvement of Low Rank Representation for Subspace segmentation, arxiv: , Liu et al. Robust Recovery of Subspace Structures by Low-Rank Representation, TPAMI 2013.

10 Liu et al., Robust Recovery of Subspace Structures by Low-Rank Representation, TPAMI Yao-Liang Yu and Dale Schuurmans, Rank/Norm Regularization with Closed-Form Solutions: Application to Subspace Clustering, UAI2011. Closed-form Solution of LRR Closed form solution at general case m in Z kz k ; s:t: X = A Z ; has a unique closed-form optim al solution: Z = A y X. Valid for any unitary invariant norm!

11 Hongyang Zhang, Zhouchen Lin, and Chao Zhang, A Counterexample for the Validity of Using Nuclear Norm as a Convex Surrogate of Rank, ECML/PKDD2013. Closed-form Solution of LRR Closed form solution of the original LRR min Z rank(z); s.t. A = XZ: (1) Theorem: Suppose U X X VX T and U A A VA T are the skinny SVD of X and A, respectively. The complete solutions to feasible generalized LRR problem (1) are given by Z = X y A + SVA T ; (2) where S is any matrix such that V T X S = 0.

12 Latent LRR Small sample issue min kzk ; s:t: X = XZ: Consider unobserved samples min kzk ; s:t: X O = [X O ; X H ]Z: Liu and Yan. Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction, ICCV 2011.

13 Latent LRR min kzk ; s:t: X O = [X O ; X H ]Z: Theorem: Suppose Z O;H = [Z OjH ; Z HjO ]. Then Z OjH = V OV T O ; Z HjO = V HV T O ; where V O and V H are calculated as follows. Compute the skinny SVD: [X O ; X H ] = U V T and partition V as V = [V O ; V H ]. Proof: That [X O ; X H ] = U [V O ; V H ] T implies So X O = [X O ; X H ]Z reduces to: So Z O;H = V V T O = [V OV T O ; V HV T O ]: X O = U V T O ; X H = U V T H : V T O = V T Z: Liu and Yan. Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction, ICCV 2011.

14 Latent LRR min kzk ; s:t: X O = [X O ; X H ]Z: Z OjH = V OV T O ; Z HjO = V HV T O : X O = [X O ; X H ]Z O;H = X O Z OjH + X HZ HjO = X O Z OjH + X HV H VO T = X O Z OjH + U V H T V HVO T = X O Z OjH + U V H T V H 1 U T X O X O Z OjH + L HjO X O: low rank! min rank(z) + rank(l); s:t: X = XZ + LX: min kzk + klk ; s:t: X = XZ + LX: Liu and Yan. Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction, ICCV 2011.

15 Latent LRR min kzk + klk + kek 1 ; s:t: X = XZ + LX + E: Liu and Yan. Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction, ICCV 2011.

16 Zhang et al, A Counterexample for the Validity of Using Nuclear Norm as a Convex Surrogate of Rank, ECML/PKDD Analysis on LatLRR Noiseless LatLRR has non-unique closed form solutions! Theorem: The complete solutions to are as follows min Z;L rank(z) + rank(l); s:t: X = XZ + LX min Z rank(z); s:t: 1 2 X = XZ: min L rank(l); s:t: Z = V X ~ W V T X + S 1 ~ W V T X and L = U X X (I ~ W ) 1 X U T X + U X X (I ~ W )S 2 ; where ~ W is any idempotent matrix and S 1 and S 2 are any matrices satisfying: 1. V T X S 1 = 0 and S 2 U X = 0; and 1 2 X = LX: min rank(z); s:t: X = XZ: Z min L rank(l); s:t: X = LX: 2. rank(s 1 ) rank( ~ W ) and rank(s 2 ) rank(i ~ W ). A 2 = A + = 1:

17 Zhang et al, A Counterexample for the Validity of Using Nuclear Norm as a Convex Surrogate of Rank, ECML/PKDD Analysis on LatLRR Theorem: The complete solutions to min Z;L kzk + klk ; s:t: X = XZ + LX are as follows Z = V X c W V T X and L = U X (I c W )U T X; where c W is any block diagonal matrix satisfying: 1. its blocks are compatible with X, i.e., if [ X ] ii 6= [ X ] jj then [ c W ] ij = 0; and 2. both c W and I c W are positive semi-de nite.

18 Robust LatLRR Comparison on the synthetic data as the percentage of corruptions increases. Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Robust Latent Low Rank Representation for Subspace Clustering, Neurocomputing, Vol. 145, pp , December 2014.

19 Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Robust Latent Low Rank Representation for Subspace Clustering, Neurocomputing, Vol. 145, pp , December Robust LatLRR Table 1: Segmentation errors (%) on the Hopkins 155 data set. For robust LatLRR, the parameter is set as 0:806= p n. The parameters of other methods are also tuned to be the best. SSC LRR RSI LRSC LatLRR Robust LatLRR MAX MEAN STD

20 Relationship Between LR Models Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

21 Relationship Between LR Models Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

22 Relationship Between LR Models Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

23 Implications We could obtain a globally optimal solution to other low rank models. We could have much faster algorithms for other low rank models. Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

24 Comparison of optimality Implications Comparison of accuracies of solutions to relaxed R-LRR computed by REDU-EXPR and partial ADM, where the parameter is adopted as 1/\sqrt(log n) and n is the input size. The program is run by 10 times and the average accuracies are reported. Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

25 Implications Comparison of speed Model Method Accuracy CPU Time (h) LRR ADM - >10 R-LRR ADM - did not converge R-LRR partial ADM - >10 R-LRR REDU-EXPR % Table 1: Unsupervised face image clustering results on the Extended YaleB database. REDU-EXPR means reducing to RPCA rst and then express the solution as that of RPCA. Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , 2015.

26 Hongyang Zhang, Zhouchen Lin, Chao Zhang, and Junbin Gao, Relation among Some Low Rank Subspace Recovery Models, Neural Computation, Vol. 27, No. 9, pp , Implications Comparison of optimality and speed

27 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Assumption: rank(a) = o(n). n r=o(n) D min kak + kek l1 ; s:t: D = A + E:

28 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering First, randomly sample D sub. K=cr n D sub D min kak + kek l1 ; s:t: D = A + E:

29 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Second, solve RPCA for D sub : D sub =A sub +E sub. n K=cr min ka sub k + 1kE sub k 1 subj D sub = A sub + E sub : D sub < O(n) complexity! D min kak + kek l1 ; s:t: D = A + E:

30 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Third, find the full rank submatrix B of A sub. K=cr r=o(n) B n A sub D min kak + kek l1 ; s:t: D = A + E:

31 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Fourth, correct the Kx(n-K) submatrix of D by C sub. n min kd c C sub p c k 1 K=cr p c r=o(n) d c O(n) complexity! Can be done in parallel!! B C sub A sub D min kak + kek l1 ; s:t: D = A + E:

32 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Fifth, correct the (n-k)xk submatrix of D by R sub. min q r kd r q r R sub k 1 O(n) complexity! Can be done in parallel!! K=cr r=o(n) B R sub A sub n Rows and columns can also be processed in parallel!! d r D min kak + kek l1 ; s:t: D = A + E:

33 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering Finally, the rest part of A is A c =Q r BP c. P c = (p 1 ; p 2 ; ; p n K ) 0 1 q 1 Q r = B q 2 A q n K K=cr r=o(n) B A sub n A compact representation of A! D A c min kak + kek l1 ; s:t: D = A + E:

34 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , O(n 1 ) RPCA by l 1 -Filtering What if the rank is unknown? 1. If rank(a sub ) > K/c, then increase K to c rank(a sub ). 2. Otherwise, resample another D sub for cross validation. K=cr r=o(n) B A sub D n B D sub min kak + kek l1 ; s:t: D = A + E:

35 Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , Experiments Size kl 0 L k F Synthetic Data Method kl 0 k F rank(l ) kl k ks k l0 ks k l1 Time(s) rank(l 0 ) = 20, kl 0 k = 39546, ks 0 k l0 = 40000, ks 0 k l1 = S-ADM L-ADM l = rank(l 0 ) = 50, kl 0 k = , ks 0 k l0 = , ks 0 k l1 = S-ADM L-ADM l = rank(l 0 ) = 100, kl 0 k = , ks 0 k l0 = , ks 0 k l1 = S-ADM L-ADM l = Structure form Motion Data rank(l 0 ) = 4, kl 0 k = 31160, ks 0 k l0 = , ks 0 k l1 = S-ADM L-ADM l =

36 Experiments Risheng Liu, Zhouchen Lin, Zhixun Su, and Junbin Gao, Linear time Principal Component Pursuit and its extensions using l1 Filtering, Neurocomputing, Vol. 142, pp , 2014.

37 Conclusions Low-rank models have much richer mathematical properties than sparse models. Closed-form solutions to low-rank models are useful in both theory and applications.

38 Thanks!