LOW RANK TENSOR DECONVOLUTION

Transcription

1 LOW RANK TENSOR DECONVOLUTION Anh-Huy Phan, Petr Tichavský, Andrze Cichocki Brain Science Institute, RIKEN, Wakoshi, apan Systems Research Institute PAS, Warsaw, Poland Institute of Information Theory and Automation, Prague, Czech Republic ABSTRACT In this paper, we propose a low-rank tensor deconvolution problem which seeks multiway replicative patterns and corresponding activating tensors of rank- An alternating least squares (ALS) algorithm has been derived for the model to sequentially update loading components and the patterns In addition, together with a good initialisation method using tensor diagonalization, the update rules have been implemented with a low cost using fast inversion of block Toeplitz matrices as well as an efficient update strategy Experiments show that the proposed model and the algorithm are promising in feature extraction and clustering Index Terms tensor decomposition, tensor deconvolution, tensor diagonalization, CANDECOMP/PARAFAC INTRODUCTION Tensor decomposition especially the CANDECOMP/PARAFAC tensor decomposition (CPD) has found a wide range of applications in variety of areas such as in chemometrics, telecommunication, data mining, neuroscience, blind source separation [ 4] One of important applications of CPD is to extract hidden loading components which can provide physical insight of the source data In order to deal with shifting factors in sequential data such as time series or spectra data, Harshman et al [5] proposed the shifted CPD The model has been applied to inspecting neural activity [6] FitzGerald et al extended the shift model to nonnegative tensor factorisation and applied it to music separation [7] Cemgil et al [8] introduced a probabilistic framework for non-negative factor deconvolution for audio modeling Some other existing shifting models have been considered in nonnegative matrix factorisation (NMF) such as the convolutive NMF [9 ], two way CNMF, or multichannel NMF [2,3] In this paper, we consider a novel tensor deconvolution problem whose maor aim is to represent multiway data by replicative patterns and activating maps following a convolutive model, that is Y H M + + H R M R where denotes the tensor convolution, Y is of size I I 2 I 3, H r and M r are tensors of size r r2 r3, and K r K r2 K r3 with rn + K rn = I n, rn I n, The work of P Tichavský was supported by the Czech Science Foundation through proect No 4 373S Y ++ (a) CANDECOMP/PARAFAC (CPD) * ++ * Y H M H R M R (b) Low rank tensor deconvolution Fig (a) Illustration of CANDECOMP/PARAFAC (CPD) as a tool to extract rank- patterns from multiway data Y,and (b) low rank tensor deconvolution which represents the data by small patches H r and maps M r, which are rank- tensors respectively Although the roles of H r and M r are interchangeable because of commutativity of the convolution, we often consider patterns of relatively small sizes and K n n More specifically, we consider a rank constrained model of in which M r are rank- tensors, for r =,,R, ie, M r = a r b r c r,wherea r R K r, b r R K r2 and c r R K r3 are vectors of unit length, and represents the outer product The low rank tensor deconvolution in is rewritten as Y H (a b c ) + + H R (a R b R c R ) (2) and illustrated in Fig When data is a matrix, ie, I 3 =, the tensor deconvolution becomes rank- blind matrix deconvolution proposed in [4] In a particular case, when patterns H r all are scalar, ie, r = r2 = r3 =, the tensor deconvolution with R patterns simplifies into the rank-r CP tensor decomposition In general cases, we will show that this tensor deconvolution can be expressed as the rank-(, 2, 3 ) block tensor decomposition [5] with Toeplitz factor matrices For simplicity, patterns H r are supposed to be the same size, ie, r =, r2 = 2 and r3 = 3 for r =,,R It follows that loading components are also of the same length, ie, K r = K, K r2 = K 2, K r3 = K 3 for r =,,R Inthe following section, we will derive an ALS algorithm which sequentially updates loading components a r, b r, c r and the tensors H r Application of the model is then demonstrated for feature extraction and clustering of the hand-written digits /5/$3 25 IEEE 269 ICASSP 25

2 2 ALGORITHM We define a linear operator X = τ (x) which maps a vector x of length K to a lower triangular Toeplitz matrix X of size I, I = K + whose first column is [x T,,,] T X = τ (x), vec(x) = T K, x (3) where T K, = [I K, K,,I K, K, I K ] T of size (K + ) K comprises only ones and zeros The tensor decomposition in (2) can be expressed as Y H r A r 2 B r 3 C r, (4) where n denotes product between a tensor and a matrix along mode-n, A r = τ (a r ), B r = τ 2 (b r ), C r = τ r3 (c r )are Toeplitz matrices of size I, I 2 2 and I 3 3, respectively 2 Update of loading components a r In order to solve the problem (4), we minimise the following cost function min D = 2 Y H r A r 2 B r 3 C r 2 F We denote by Y the mode- matricization of Y The approximation in (4) can be rewritten in matrix form as Y A r H (r) (C r B r ) T, (6) where denotes the Kronecker product, and H (r) are mode- matricizations of the tensors H r for r =,,R Thecost function in is therefore expressed in an equivalent form D = 2 vec( ) ( Y (C r B r ) ( ) H (r) T ) II T K, a r 2 F, whereas its gradients with respect to a r for r =, 2,,R are given by D = w r + Φ r,s a s, (7) a r s= where vectors w r of length K, and matrices Φ r,s of size K K are defined as ( w r = T T K, H (r) (C ) r B r ) T I I vec(y) = T T K, vec ( ) Y 2 B T r 3 C T r, H r, (8) Φ r,s = T T ( ) K, Zr,s I I TK,, (9) ( )( ) Z r,s = H (r) C T r C s B T r B s H (s) T = H r 2 B T s B r 3 C T s C r, H s () but mode- between two tensors H r 2 B T s B r 3 C T s C r and H s of size 2 3 We define and sums of all entries on the -th super diagonal or -th sub diagonal of the ( ) matrices Z r,s,for =,,, that is = z (r,s), + + z(r,s) 2, z(r,s),, () = z (r,s) +, + z(r,s) +2,2 + + z(r,s), (2) From definition of the matrices T K,, we obtain that Φ r,s in (9) are banded Toeplitz matrices of size K K given as Φ r,s = + φ (r,s) φ (r,s) + φ(r,s) By setting the gradients in (7) with respect to all a r to zeros, we derive the following update rule for a = [ ] a T T,,aT R a Φ w, (3) where w = [ w T T R],,wT is a vector of length K R,and Φ = [Φ r,s ]isanr R partitioned matrix of Toeplitz matrices Φ r,s for r, s R SinceZ r,s = Z T s,r, Φ r,s = Φ T s,r and Φ r,r are symmetric It follows that Φ is a symmetric matrix of size (RK RK ) We will show that Φ can be expressed as a block Toeplitz matrix after swapping its row and columns, and thereby its inversion in (3) can be efficiently performed using fast inversion methods, eg, the block Levinson recursion algorithm [6], or the algorithm in [7] We define a permutation matrix P K,R such that vec ( X T K,R) = P K,R vec ( X K,R) for any matrix XK,R of size K R It can be verified that permutation of columns and rows of Φ using P K,R yields a block Toeplitz matrix of (R R) matrices Φ and Φ = Φ T for =,,, given as Φ Φ Φ = P K,R Φ P T K,R = Φ +, Φ The notation X, Y = X Y T denotes contraction between two tensors along all modes but mode- Entries of vectors w r are simply sums of entries on the main diagonal or on the k-sub diagonal of the (I ) matrices W r = Y 2 B T r 3 C T r, H r for k =, K +, that is w r (k) = W r (k, ) + W r (k +, 2) + + W r (k +, ), for k =,,K In addition, the matrices Z r,s of size are efficiently computed through contraction along all modes where Φ = φ (,) φ (,2) φ (2,) φ (2,2) φ (R,) φ (R,2) φ (,R) φ (2,R) φ (R,R) Φ + Φ, (4) 27

3 and are defined in () and (2) Using the above expression, the update rule in (3) is rewritten as ( ) a P T K,R PK,R Φ P T ( K,R PK,R w ) ( = P T K,R Φ w ) This update rule requires a cost of O(K 2R2 ) and needs only matrices Φ of size R R, ie, R 2 coefficients [6, 8] In a particular case when =, the update rule can be simplified into a simple form given as [a,,a R ] [w,,w R ] Z, (6) where Z is a matrix of size (R R) whose entries Z(r, s) are given in () Loading components b r and c r are updated in the similar way 22 Update of core tensors H r In order to derive update rules for H r, we compute derivatives of the cost function in with respect to H r The derivatives are set to zero to obtain the following update rule vec(h ) Ψ, Ψ,R vec(v ), (7) vec(h R ) Ψ R, Ψ R,R vec(v R ) where V r = Y A T r 2B T r rc T r are tensors of size 2 3, and Ψ r,s = (C T r C s ) (B T r B s ) (A T r A s ) The partitioned matrix Ψ = [Ψ r,s ] in (7) is of size R( 2 3 ) R( 2 3 ) In practice, we often seek relatively small patterns H r, the inversion Ψ can be proceeded quickly 23 Initialization and efficient implementation For the noise-less case, we use the tensor diagonalization (TE- DIA) [9] to seek for matrices which transform the data tensor Y to be diagonal or block diagonal form Loading components a r, b r and c r are then estimated using the Toeplitz matrix factorization [2] For other cases, TEDIA is used to generate initial points for the deconvolution The procedure is explained in more detail in the next section In update rules in and (7), besides the cost due to solving linear systems, construction of vectors w r in (8) and tensors V r in (7) is expense with a cost of O(I I 2 I 3 min( 2, 3 )) The computations are significantly time-consuming when the tensor sizes are large, because of large memory operations associated with tensor permutations [2] Taking into account that the tensor product F r = Y 2 B T r 3 C T r is the common part involving in construction of V r and w w After updating a r, the tensors V r are computed quickly as V r = F r A r Therefore we suggest to update patterns H r after each update of loading components a r, b r and c r The update order is as follows: update [a r ], update [H r ], update [b r ], update [H r ], update [c r ], update [H r ],andsoon Finally, a direct evaluation of the cost function as a stopping criterion can be the most expensive step Since we complete each iteration by updating H r using (7), the cost value is simply computed without construction of the approximate tensor to Y as Table Clustering accuracy (%) using the low rank tensor deconvolution with rank R = 2 The values inside parentheses indicate the percentage of improvement of tensor deconvolution compared to the approach based on rank-2 CPD Digits (4) 945 (2) (35) 9 (3) 98 (9) 975 (35) (39) (45) (6) (335) 945 (23) (27) 895 (45) 725 (25) 87 (45) 3 98 (9) 67 (35) 97 (55) 94 (45) 8 (285) (25) (365) (85) (2) 96 (4) (23) 69 (75) 8 78 (3) D = 2 Y 2 F vec(v r ) T vec(h r ) (8) 3 APPLICATIONS TO FEATURE EXTRACTION This section introduces an application of tensor deconvolution to feature extraction Assuming that slides Y k for k =,,I 3 represent samples of certain entity, eg, two-dimensional images Then the third loading components c,, c R explain relation between I 3 samples In a particular case when 3 =, ie, patterns are matrices of size 2, vectors c r, for r =,,R, represent R feature vectors associated with R basis components H r (a r b T r ) = τ (a r ) H r τ 2 (b r ) T of rank min(, 2 ), respectively It is worth noting that CPD with rank-r also yields R feature vectors However, its R basis components are only of rank-, and do not explain complex structure as those in the tensor deconvolution We illustrate the tensor deconvolution in clustering applications on the MNIST handwritten digits We took the first images of size for each digit, and applied the tensor deconvolution to the data consisting of 2 images for each pair of digits, eg, and, 2 and 4 So far, there does not exist a method to determine the number of patterns, ie R, which is related to rank determination in the block tensor decomposition However, one can select a suitable R by balancing the approximation error versus the number of parameters [3], or through a cross-validation technique In this paper, the deconvolution estimated two common patterns H and H 2 of size, where varied in the range of [, ] Y H r (a r b r c r ) = (H r (a r b r )) c r (9) 27

4 (a) Digits and (b) Digits 2 and (c) Digits 4 and 7 Fig 2 Illustration of tensor deconvolution with two patterns of size for clustering of two digits and 3, 2 and 6, 4 and 7 Relative approximation errors and clustering accuracies are obtained with pattern sizes =, 2,, (a) CPD with =, R = 2 (b) Tensor deconvolution with = 8andR = 2 Fig 3 Approximate images for two digits and 3 using only two patterns when = and = 8 Parameters were initialised using the TEDIA algorithm (including the Tucker compression [22,23]) for two-sided blockdiagonalization of the compressed tensor of size R R 2 The two unconstrained factor matrices A unc r and B unc r obtained by TEDIA were then factorised to yield the Toeplitz matrices τ (a r )andτ (b r ) [2] The initial patterns H r were estimated such that they minimized the Frobenius norm Y R H r τ (a r ) 2 τ (b r ) 2 F, with fixed a r and b r Finally, we completed the initialisation by performing best rank- approximations to mode-3 matricization of H r, ie, approximations H r H r c r for r =,,R In Fig 2, the relative approximation errors and clustering accuracies using K-means are illustrated as functions of size for pairs and 3, 2 and 6, 4 and 7 The results for CPD with rank-2 are shown with = For these selected pairs (a) Digits and 3 (b) Digits 2 and 6 (c) Digits 4 and 7 Fig 4 Illustration of two basis images extracted for digits and 3, 2 and 6, 4 and 7 using rank-2 CPD and tensor deconvolution with R = 2and = 2 =, 3 = of digits, we did not achieve good clustering accuracies using two features extracted by CPD However, as seen on Fig 2, when = 8, 9 and for digits and 3, digits 2 and 6, and = 5 for digits 4 and 7, we obtained much better clustering accuracies ( 925%) using only two features estimated by the tensor deconvolution The high performance of the tensor deconvolution can be explained by complex structure of basis images F r = H r (a r b T r ) illustrated in Fig 4 Basis images of rank- by CPD, ie, a r b T r, do not express sufficient structure of digits With the more complex basis images, tensor deconvolution achieved lower approximation errors, which are confirmed in both Fig 2 and Fig 3 It is clear that in Fig 3, reconstructed digits and 3 could not be distinguished by rank- 2 CPD as compared with those by tensor deconvolution The improvement was also observed in clustering of other digits as summarised in Table 4 CONCLUSIONS The tensor deconvolution with rank- structures considered in this paper has shown advantage over the CP decomposition in providing complex structure basis patterns The tensor deconvolution explains the data better than CPD, while keeping the same number of features R With efficient implementation of update rules, initialisation and update strategy, our model and the proposed algorithm can be applied to feature extraction for other kinds of data Matlab implementation is provided in the TENSORBOX package, and available online at: http: //wwwbspbrainrikenp/ phan/tensorboxphp 272

5 5 REFERENCES [] L-H Lim and P Comon, Blind multilinear identification, CoRR,volabs/226663, 22, preprint [2] A Cichocki, R Zdunek, A-H Phan, and S Amari, Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation, Wiley, Chichester, 29 [3] A Cichocki, D P Mandic, A-H Phan, C Caiafa, G Zhou, Q Zhao, and L De Lathauwer, Tensor decompositions for signal processing applications from two-way to multiway component analysis, IEEE Signal Processing Magazine, accepted, 24 [4] P Comon, Tensors: a brief introduction, IEEE Signal Processing Magazine, vol 3, no, pp 44 53, 24 [5] RA Harshman, S Hong, and ME Lundy, Shifted factor analysis - Part I: Models and properties, ournal of Chemometrics, vol 7, no 7, pp , 23 [6] M Mørup, L K Hansen, S M Arnfred, L-H Lim, and K H Madsen, Shift-invariant multilinear decomposition of neuroimaging data, NeuroImage, vol 42, no 4, pp , 28 [7] D FitzGerald, M Cranitch, and E Coyle, Extended nonnegative tensor factorisation models for musical sound source separation, Computational Intelligence and Neuroscience, 28 [8] AT Cemgil, U Simsekli, and YC Subakan, Probabilistic latent tensor factorization framework for audio modeling, in Applications of Signal Processing to Audio and Acoustics (WASPAA), 2 IEEE Workshop on, Oct 2, pp 37 4 [9] P Smaragdis, Non-negative matrix factor deconvolution; Extraction of multiple sound sources from monophonic inputs, Lecture Notes in Computer Science,vol 395, pp , 24 [] P Smaragdis, Convolutive speech bases and their application to speech separation, IEEE Transactions of Speech and Audio Processing, vol 5, pp 2, an 27 [] R Zdunek, Improved convolutive and underdetermined blind audio source separation with MRF smoothing, Cognitive Computation, vol 5, no 4, pp , 23 [2] M Mørup and MN Schmidt, Sparse non-negative tensor factor double deconvolution (SNTF2D)for multi channel time-frequency analysis, Tech Rep, Technical University of Denmark, DTU, 26 [3] A Ozerov and C Févotte, Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation, Audio, Speech, and Language Processing, IEEE Transactions on, vol 8, no 3, pp , March 2 [4] A-H Phan, P Tichavský, A Cichocki, and Z Koldovský, Low-rank blind nonnegative matrix deconvolution, in Acoustics, Speech and Signal Processing (ICASSP), 22 IEEE International Conference on 22, pp , IEEE [5] L De Lathauwer and D Nion, Decompositions of a higher-order tensor in block terms Part III: Alternating least squares algorithms, SIAM ournal of Matrix Analysis and Applications, vol 3, no 3, pp 67 83, 28, Special Issue Tensor Decompositions and Applications [6] H Akaike, Block Toeplitz matrix inversion, SIAM ournal on Applied Mathematics, vol 24, no 2, pp , 973 [7] X-G Lv and T-Z Huang, The inverses of block Toeplitz matrices, ournal of Mathematics, vol 23, no ID 2776, pp 8 pages, 93 [8] P Alonso, M Badía-Contelles, and A M Vidal, Solving the block-toeplitz least-squares problem in parallel, Concurrency - Practice and Experience, vol 7, no, pp 49 67, 25 [9] P Tichavský, A-H Phan, and A Cichocki, Nonorthogonal tensor diagonalization, a tool for block tensor decompositions, arxiv, vol 42673, 24 [2] E Moulines, P Duhamel, Cardoso, and S Mayrargue, Subspace methods for the blind identification of multichannel fir filters, Signal Processing, IEEE Transactions on, vol 43, no 2, pp , Feb 995 [2] A-H Phan, P Tichavský, and A Cichocki, Fast alternating LS algorithms for high order CANDE- COMP/PARAFAC tensor factorizations, Signal Processing, IEEE Transactions on, vol 6, no 9, pp , 23 [22] L De Lathauwer, B De Moor, and Vandewalle, On thebestrank-andrank-(r,r2,,rn)approximation of higher-order tensors, SIAM ournal of Matrix Analysis and Applications, vol 2, no 4, pp , 2 [23] A-H Phan, A Cichocki, and P Tichavský, On fast algorithms for orthogonal Tucker decomposition, in Acoustics, Speech and Signal Processing (ICASSP), 24 IEEE International Conference on, May 24, pp