ROBUST LOW-RANK TENSOR MODELLING USING TUCKER AND CP DECOMPOSITION

Transcription

1 ROBUST LOW-RANK TENSOR MODELLING USING TUCKER AND CP DECOMPOSITION Niannan Xue, George Papamakarios, Mehdi Bahri, Yannis Panagakis, Stefanos Zafeiriou Imperial College London, UK University of Edinburgh, UK Middlesex University, UK ABSTRACT A framework for reliable seperation of a low-rank subspace from grossly corrupted multi-dimensional signals is pivotal in modern signal processing applications. Current methods fall short of this separation either due to the radical simplification or the drastic transformation of data. This has motivated us to propose two new robust low-rank tensor models: Tensor Orthonormal Robust PCA (TORCPA) and Tensor Robust CP Decomposition (TRCPD). They seek Tucker and CP decomposition of a tensor respectively with l p norm regularisation. We compare our methods with state-of-the-art low-rank models on both synthetic and real-world data. Experimental results indicate that the proposed methods are faster and more accurate than the methods they compared to. Index Terms Tensor Decomposition, Robust Principal Component Analysis, Tucker, CANDECOMP/PARAFAC. INTRODUCTION In many real-world applications, input data are naturally representd by tensors (i.e., multi-dimensional arrays). Traditionally, such data would require vectorising before processing and thus destroy the inherent higher-order interactions. As a result, novel models must be developed to preserve the multilinear structure when extracting the hidden and evolving trends in such data. Typical tensor data are video clips, color images, multi-channel EEG records, etc. In practice, important information usually lie in a (multilinear) low-dimensional space whose dimensionality is much lower dimensional space than observations.this is the essence of low-rank modelling. In this paper, we focus on the problem of recovering a low-dimensional multilinear structure from tensor data corrupted by gross corruptions. Given a tensor L R d d N, its tensor rank [] is defined by the smallest r such that L = r a() i a (N) i, where denotes outer products among some Nr vectors a () i,, a (N) i, i r. As such, robust low-rank tensor modelling seeks a decomposition X = L + S for an N th -order tensor X R d d N, where L has a low tensor rank and S is sparse. However, the tensor rank is Corresponding Author (s.zafeiriou@imperial.ac.uk). The work of Stefanos Zafeiriou was partially funded by EPSRC project EP/N007743/ (FACER2VM). usually intractable [2]. A common adjustment [3 5] is to use a convex combination of the n-ranks of L, that is γ = N α irank i (L), where α i 0, N α i = and rank i (L) is the the column rank of the mode-i matricisation [6] of L. It is, therefore, natural to obtain the decomposition by solving optimisation problem () γ+λ E 0 s.t. γ = α i rank i (L), X = L+S, () L,S where S 0 is the l 0 norm of the vectorisation of S and λ is a weighting parameter. Here we present two novel robust tensor methods based on Tucker and CP decomposition that recover the latent lowrank component from noisy observations by relaxing (), which is NP-hard. In section 2, we review relevant literature on matrix and tensor algorithms. In section 3, we explain our proposed tensor methods in detail. In section 4, we demonstrate the advantages of our models on both synthetic data and a real-world dataset. Finally, in section 5, we summarise our new models and point out possible future improvements. 2. RELATED WORK RSTD [7] is a direct multi-linear extension of matrix principal component pursuit (PCP) [8]. It approximates () by replacing rank i (L) and S 0 with convex surrogates L (i) and S respectively, where L (i) is the nuclear norm of the mode-i matricisation of L and S is the l norm of the vectorisation of S. As a result, it solves the following alternative objective L,S α i L (i) + λ S s.t. X = L + S. (2) An ALM solver can be found in [9]. It is also worth noting that under certain conditions RSTD is guaranteed to exactly recover the low-rank component [0]. Much recent research on subspace analysis for the matrix case has direct applicability to tensor data. The costly singular value decomposition step in classical PCP prohibits largescale analysis. A general approach to mitigate this issue is to look for a factorisation of the low rank component A. OR- PCA [] uses a linear combination of the active subspace, A = UV, U T U = I, where bilinear factors U R m k and V R k n are the principal components and the com- ISBN EURASIP

2 bination coefficients respectively and k is an upper bound of rank(a). 3. ROBUST LOW-RANK TENSOR MODELLING 3.. Notation We first introduce some notations used throughout the paper. Lowercase latin and greek letters denote scalars, e.g. r, γ. Bold lowercase latin letters denote vectors, e.g. a. Bold uppercase latin letters denote matrices, e.g. A. Bold uppercase calligraphic latin letters denote tensors, e.g. L. Bold uppercase greek letters denote operators on tensors and matrices, e.g. Θ(S), Φ(U). A, B represents tr(a T B). A F is the Frobenius norm. A B denotes the Khatri-Rao product between matrices A and B and X i U is the i-mode product [6] Soft and hard thresholding operators For fixed X R d d N, the optimal analytical solution for Y κ Y + 2 X Y 2 F is given by the soft thresholding operation Θ κ (X ), where Θ κ (X ) ι ι N = (X ι ι N κ) + ( X ι ι N κ) +. (3) And for fixed X R m n, the optimal analytical solution for Y κ X + 2 X Y 2 F is given by the hard thresholding operation Φ κ (X), where Φ κ (X) = UΘ κ (S)V T, (4) for singular value decomposition X = USV T Tensor Orthonormal Robust PCA Generalisation of ORPCA to tensors corresponds to the following factorisation of the low-rank component L: L = V U N U N V N U i, U T i U i = I, (5) which is exactly the HOSVD [2] of L and the following relationship holds L (i) = V (i). (6) Based on the above, (2) can be re-written as α i V (i) + λ S, V,S s.t. X = V N U i + S, U T i U i = I, i N. To separate variables, we make the substitution V (i) = J i, to arrive at an equivalent problem: α i J i + λ S, J i,s s.t. X = V N U i + S, U T i U i = I, V (i) = J i, i N. (7) (8) To apply ADMM, the augmented Lagrangian of (8) is constructed first: l(j i, V, S, U i, Y, Z i ) = α i J i + λ S + X V N U i S, Y + + µ 2 X V N U i S 2 F + V (i) J i, Z i where U T i U i = I has not been incorporated. J i is updated by the imiser of l(j i ): µ 2 V (i) J i 2 F, J i = arg α i µ J i + J i 2 J i (V (i) + µ Z i) 2 F = Φ αiµ (V (i) + µ Z i) V is updated by the imiser of l(v): (9) (0) V = arg V, (µ(x S) + Y) V N Ui T () + V J i, Z i + µ µ 2 V 2 F + 2 V J i 2 F, where we have used the fact that Ui T U i = I, the Frobenius norm is invariant under rotations and J i, Z i are the inverse of mode-i matricisations, J i, Z i respectively. To obtain V, setting the gradient of () to zero gives: V = N + ((X S + µ Y) N U T i + S is updated by the imiser of l(s): S =arg S λµ S (J i µ Z i)). + 2 S (X V N U i + µ Y) 2 F =Θ λµ (X V N U i + µ Y). (2) (3) U i is updated by the imiser of l(u i ) subject to Ui T U i = I: U i =arg U i 2 X (i) S (i) + µ Y (i) U i B i 2 F, (4) where B i = (V i j= U j N j=i+ U j ) (i). If we have the following SVD (X (i) S (i) + µ Y (i))b T i = C i D i V T i, (5) then according to the Reduced Rank Procrustes Theorem [3], the solution is given by U i = C i V T i. (6) The complete algorithm is presented in Algorithm. ISBN EURASIP

3 Algorithm ADMM solver for TORPCA Input: Observation X, parameter λ > 0, scaling κ >, weights α i, ranks k i : Initialise: J i = Z i = 0, S = Y = 0, V = 0, U i = first k i left singular vectors of X (i), µ > 0 2: while not converged do 3: for i {, 2,, N} do 4: J i = Φ αiµ (V (i) + µ Z i) 5: end for 6: V = N+ ((X S + µ Y) N U i T + N (J i µ Z i)) 7: S = Θ λµ (X V N U i + µ Y) 8: for i {, 2,, N} do 9: B i = (V i j= U j N j=i+ U j) (i) 0: C i D i Vi T = (X (i) S (i) + µ Y (i))bi T : U i = C i V T i 2: end for 3: Y = Y + µ(x V N U i S) 4: for i {, 2,, N} do 5: Z i = Z i + µ(v (i) J i ) 6: end for 7: µ = µ κ 8: end while Return: V, S, U i 3.4. Tensor robust CP decomposition Let U (i) = [a (i), a(i) 2,, a(i) r ], then we can express L compactly as L = U () U (2) U (N). In particular, it can be shown that rank i (L) rank(u i ), for i N. So, it is beneficial to solve the following objective U i,s α i U i + λ S, X = U U 2 U N + S. (7) Again, we make the substitution U i = J i before perforg ADMM, which leads to the following problem J i,s α i J i + λ S + µ 2 X U U N S 2 F + (8) s.t. X = U U N + S, U i = J i, i N. The corresponding augmented Lagrangian is l(j i, U i, S, Y, Z i ) = α i J i + λ S (9) + X U U N S, Y + U i J i, Z i µ 2 U i J i 2 F. Algorithm 2 ADMM solver for TRCPD Input: Observation X, parameter λ > 0, scaling κ >, weights α i, rank k : Initialise: J i = U i = rand, S = Y = 0, Z i = 0, µ > 0 2: while not converged do 3: for i {, 2,, N} do 4: J i = Φ αiµ (U i + µ Z i) 5: Ũ i = (U N U i+ U i U ) T 6: U i = ((X (i) S (i) + µ Y (i))ũ T i + J i µ Z i)(ũ i Ũi T + I) 7: end for 8: S = Θ λµ (X U U N + µ Y) 9: Y = Y + µ(x U U N S) 0: for i {, 2,, N} do : Z i = Z i + µ(u i J i ) 2: end for 3: µ = µ κ 4: end while Return: U i, S J i is updated by the imiser of l(j i ): J i = arg α i µ J i + J i 2 J i (U i + µ Z i) 2 F = Φ αiµ (U i + µ Z i). U i is updated by the imiser of l(u i ): (20) U i = arg U i X (i) U i Ũ i S (i), Y (i) + U i J i, Z i + µ 2 X (i) U i Ũ i S (i) 2 F + µ 2 U i J i 2 F, where Ũ i = (U N U i+ U i U ) T (2) Setting the derivative of (2) to zero gives: U i =((X (i) S (i) + µ Y (i))ũ T i + J i µ Z i)(ũ i Ũ T i + I). S is updated by the imiser of l(s): S = arg S λµ S + 2 S (X U U N ) + µ Y 2 F = Θ λµ (X U U N + µ Y). The complete algorithm is described in Algorithm Complexity and convergence (22) (23) For ease of exposition, we assume that d,, d N = ζ. For TORPCA, the most expansive calculation in each iteration is the i-mode product which has a time complexity of O(Nrζ N ). For TRCPD, the doant term is the chain of matrix outer products which costs O(Nrζ N ). Note that both ISBN EURASIP

4 methods have lower complexity than RSTD whose complexity is O(Nζ N+ ) due to SVD if r < ζ. Although both of our proposed tensor methods are nonconvex, we have empirically found that the warm initialisation of using the first k i left singular vectors of X (i) for U i works well for TORPCA and uniform initialisation of U i, i k from [0, ] suffices for TRCPD (see Section 4). 4. EXPERIMENTAL RESULTS 4.. Implementation details For stopping criteria, we use one of the KKT opimality conditions, X L S F X F < δ and we have set δ = 0 7. The initial value of µ is set to 0 3, which is geometrically increased by a factor of κ =.2 up to 0 9. The weights α i are assumed equal Simulation We first evaluate the performance of all algorithms on synthetic data. A low-rank tensor L R is generated via L = U U 2 U 3, where elements of U, U 2, U 3 R 00 8 are independently sampled from the standard Gaussian distribution. The variance of L is normalised to afterwards. A sparse tensor S R is constructed by uniform sampling from [ 0, 0]. Then only 20% of the elements are kept, with others set to zero. Relative error RSTD TORPCA TRCPD PCP ORPCA Fig.. Relative error from all algorithms for a range of λ. Each tensor algorithm takes X = L+S as input, whereas matrix algorithms take mode- matricisation of X as input. Since rank(l) 8, the rank k in TRCPD is set to 8 and the ranks k i in TORPCA are all set to 8 because rank i (L) 8. The relative error L L F L F averaged over 5 trials against λ is plotted for the optimal L in each algorithm in Fig. The total execution time for each algorithm versus λ is shown in Fig 2. It is clear that tensor methods are superior to matrix-based methods. Particularly, TRCPD performs the best and TOR- PCA is also better than RSTD. Both TRCPD and TORPCA are stable in terms of λ whereas RSTD depends on tuning heavily. The execution time confirms our complexity analysis. Both of TORCPA and TRCPD are significantly faster than RSTD. Execution time/s RSTD TORPCA TRCPD PCP ORPCA Fig. 2. Running time of all algorithms as λ varies. Senario Algorithm λ [0 4, 0 ] Salt & Pepper occlusion k {0, 20,, 200} α {0., 0.2,, 0.9} RSTD TORPCA TRCPD ORPCA RSTD TORPCA TRCPD ORPCA Table. Optimal parameter choices for all algorithms used in different experiments Facial image denoising It is well understood that a convex Lambertian surface, viz. faces, under distant and isotropic lighting has a low-rank underlying model. In light of this, we consider images of a fixed pose under different illuations from the extended Yale B database for benchmarking. All 64 images for one person were studied. For matrix-based methods, observation matrices were formed by vectorising each image. All images are also re-scaled such that every pixel lies in [0, ]. Salt & Pepper Noise Salt & pepper noise is observed in real images, commonly caused by data transmission errors. To apply salt & pepper noise, we randomly set pixels to black (0) or white () with equal probability. This is close to the Laplacian noise hypothesis, where noise is heavy, non- Gaussian and potentially wide-ranging. We test an extreme case, where 60% of all the pixels are affected. Partial Occlusion Partial occlusion is ubiquitous in visual information, which can usually be completed during human visual perception [4]. For the partial occlusion noise, we generate randomly sized patches at random locaions. The maximum dimension is 60 pixels and the occlusion is full of Salt & Pepper noise. The successful application of various algorithms requires careful tuning of the algorithmic parameters. These include the penalty parameter λ, an estimate of k = rank(l) and k i = rank i (L) = d i α. The ranges of interest and the optimal choices are summarised in Table. Reconstruction from salt & pepper noise is illustrated in the first row of Fig 3, where the first image in the sequence ISBN EURASIP

5 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) (m) (n) Fig. 3. Image Denoising Experiments: (a) & (h) are original images from the sequence. Salt & pepper is introduced as shown in (b) and occlusion is demonstrated in (i). (c) & (j) present recovery results for RSTD. (d) & (k) for TORPCA. (e) & (l) for TRCPD. (f) & (m) for PCP. And (g) & (n) for ORCPA. is shown. RSTD and matrix-based methods fail to remove the introduced noise, whereas TORPCA and TRCPD are extremely promising such that no trail of noise can be seen. Recovery from partial occlusion is displayed in the second row of Fig 3. ORPCA has little effect. The region where noise was introduced is severely distorted in the recovered image of RSTD. Both TORPCA and TRCPD mananged to denoise the occlusion though they have an additional smoothing effect. PCP achieves the highest quality of recovery but there is still unremoved noise left in the image. This may be attributed to the fact that the nature of the occlusion is inherently in a matrix form. 5. CONCLUSIONS In this paper, two novel methods, namely the TORPCA and TRCPD have been proposed in order to address the problem of robust low-rank tensor recovery. The proposed methods surpass existing tensor- and matrix-based methods in our experiments. We anticipate our work to lay foundations for more general signal processing problems. Future direction of research can extend our methods to hierarchical tensor decomposition. REFERENCES [] J.B. Kruskal, Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics, Linear Algebra and its Applications, vol. 8, no. 2, pp , 977. [2] J. Håstad, Tensor rank is np-complete, Journal of Algorithms, vol., no. 4, pp , 990. [3] S. Gandyand B. Recht and I. Yamada, Tensor completion and low-n-rank tensor recovery via convex optimization, Inverse Problems, vol. 27, no. 2, pp , 20. [4] J. Liu, P. Musialski, P. Wonka, and J. Ye, Tensor completion for estimating missing values in visual data, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no., pp , 203. [5] L. Yang, Z. Huang, and X. Shi, A fixed point iterative method for low n-rank tensor pursuit, IEEE Transactions on Signal Processing, vol. 6, no., pp , 203. [6] T.G. Kolda and B.W. Bader, Tensor decompositions and applications, SIAM Review, vol. 5, no. 3, pp , [7] Y. Li, J. Yan, Y. Zhou, and J. Yang, Optimum subspace learning and error correction for tensors, in European Conference on Computer Vision, 200. [8] E.J. Candès, X. Li, Y. Ma, and J. Wright, Robust principal component analysis?, Journal of the ACM, vol. 58, no. 3, 20. [9] D. Goldfarb and Z. Qin, Robust low-rank tensor recovery: Models and algorithms, SIAM Journal on Matrix Analysis and Applications, vol. 35, no., 204. [0] B. Huang, C. Mu, J. Wright, and D. Goldfarb, Provable models for robust low-rank tensor completion, Pacific Journal of Optimisation, vol., no. 2, pp , 205. [] G. Liu and S. Yan, Active subspace: Toward scalable low-rank learning, Neural Computation, vol. 24, no. 2, pp , 202. [2] L. De Lathauwer, B. De Moor, and J. Vandewalle, A multilinear singular value decomposition, SIAM Journal on Matrix Analysis and Applications, vol. 2, no. 4, [3] H. Zou, T. Hastie, and R. Tibshirani, Sparse principal component analysis, Journal of Computational and Graphical Statistics, vol. 5, no. 2, [4] A.B. Sekuler and S.E. Palmer, Perception of partly occluded objects: A microgenetic analysis, Journal of Experimental Psychology: General, vol. 2, no., 992. ISBN EURASIP