Nonnegative Tensor Factorization with Smoothness Constraints

Transcription

1 Nonnegative Tensor Factorization with Smoothness Constraints Rafal Zdunek 1 and Tomasz M. Rutkowski 2 1 Institute of Telecommunications, Teleinformatics and Acoustics, Wroclaw University of Technology, Wybrzeze Wyspianskiego 27, Wroclaw, Poland rafal.zdunek@pwr.wroc.pl 2 RIKEN Brain Science Institute, Wako-shi, Japan Abstract. Nonnegative Tensor Factorization (NTF) is an emerging technique in multidimensional signal analysis and it can be used to find partsbased representations of high-dimensional data. In many applications such as multichannel spectrogram processing or multiarray spectra analysis, the unknown features have locally smooth temporal or spatial structure. In this paper, we incorporate to an objective function in NTF additional smoothness constrains that considerably improve the unknown features. In our approach, we propose to use the Markov Random Field (MRF) model that is commonly-used in tomographic image reconstruction to model local smoothness properties of 2D reconstructed images. We extend this model to multidimensional case whereby smoothness can be enforced in all dimensions of a multi-dimensional array. We analyze different clique energy functions that are a subject to MRF. Some numerical results performed on a multidimensional image dataset are presented. Keywords: Nonnegative Tensor Factorization (NTF), multiarray spectra analysis, Markov Random Field (MRF). 1 Introduction Nonnegative Tensor Factorization (NMF) [1,2] is an extended version of Nonnegative Matrix Factorization (NMF) [3] to nonnegative multi-dimensional arrays. This method has already found a variety of applications, e.g. in multidimensional signal and image processing. Similarly as NMF, NTF also provides lower-rank sparse representations of nonnegative data. Moreover, spatial and temporal correlations between variables can be modeled more accurately with NTF than 2D matrix factorizations. Assuming the variables of interest have all smooth profiles (along each dimension) we can improve NTF with additional smoothness constraints. This approach is justified by observation that usually data representations have a locally smooth spatial and temporal structure, where the locality can be restricted to a few samples. The smoothness can be modeled in many ways, e.g. by the entropy measure or l 2 norm of the estimated components. In our approach, we used the Markov Random Field (MRF) model that is widely applied in image reconstruction and restoration. Such models, which are often expressed by the Gibbs prior, determine local D.-S. Huang et al. (Eds.): ICIC 2008, LNCS 5226, pp , c Springer-Verlag Berlin Heidelberg 2008

2 Nonnegative Tensor Factorization with Smoothness Constraints 301 roughness (smoothness) in the analyzed image with consideration of pair-wise interactions among adjacent pixels in a given neighborhood of a singe pixel. Thus, a total smoothness in an image can be expressed by a joint Gibbs distribution with a nonlinear energy function. In our approach, we use the Green s function for measuring strength of the pair-wise pixel interactions. Using a Bayesian framework, we get the Gibbs regularized cost function that is minimized with a gradient descent alternating minimization technique subject to nonnegativity constrains that can be imposed in many ways. One of them is achieved with standard multiplicative updates that were used, e.g. by Lee and Seung [3]. Let A be a tensor of the order N and dimension I 1 I 2... I n 1 I n I n+1... I N.Then-mode product of the tensor A and the matrix B R Jn In is the tensor A n B of the dimension I 1 I 2... I n 1 J n I n+1... I N [4]. For N =3,thek-th frontal slice of the tensor A is the matrix A k = A(:, :,k) R I1 I2. Applying the forward cycle matricizing by row-wise unfolding, the tensor A can be matricized as follows: Ā =[A 1, A 2,...,A k,...,a I3 ] R I1 I2I3. The paper has the following organization. The next section introduces the basic factorizations of nonnegative multi-dimensional arrays. Section 3 presents the main contribution, i.e. incorporation of smoothness constraints to the selected models. The simulation experiments are given in Section 5. Finally, some conclusions are drawn in the last section. 2 NTF Models I T K We assume the observed data are represented by the 3D array Y R+, where R + denotes the nonnegative orthant (subspace of nonnegative real numbers) of R. There are many models for factorizing the array Y, which follows from the underlying physical phenomena. Typical models [5,6] are as follows: PARAFAC with nonnegativity constraints Y = D 1 A 2 X T, (1) where A =[a ij ] R I J +, X =[x jt] R J T +,andd =[d jjk] R J J K + is a nonnegative tensor with diagonal frontal slices, i.e. k : D k =diag{d jj } R J J. NTF1 Y = X 1 A, (2) J T K where X =[x jtk ] R+ is a nonnegative tensor of lateral sources, and A =[a ij ] R I J + is a nonnegative mixing matrix. In practice, NTF 1 is an inexact factorization due to noisy perturbations, i.e. Y = X 1 A + N, (3) where N R I T K is a tensor of noisy perturbations. NTF2 Y = A 2 X T + N, (4)

3 302 R. Zdunek and T.M. Rutkowski where A =[a ijk ] R I J K + is a nonnegative tensor of weighting coefficients, X =[x jt ] R J T + is a nonnegative matrix of sources, N R I T K is a tensor of noisy perturbations. The objective is to estimate the multi-dimensional arrays D, X, A,andthematrices A and X, subject to nonnegativity constraints of all the entries, given the data tensor Y and possibly the prior knowledge on the nature of the true components to be estimated or on a statistical distribution of noisy disturbances in N. In the paper, we assume that the smoothness constraints apply only to the unknown sources, i.e. either to the matrix X or tensor X. The sources may be timeand space-varying with both smooth profiles. The prior knowledge on the sources is particularly strong when the sources have all the profiles locally smooth, i.e. the tensor X tends to be smooth in all 3 dimensions. This takes place very often when X models a series of smooth images with a temporal smooth profile. Thus, we restrict our considerations on the smoothness constraints only to the NTF 1 model, since the application of these constraints to the matrix X as in the PARAFAC and NTF 2 is straightforward, and it can be easily achieved. Furthermore, the MRFbased smoothing has been successfully applied to NMF with respect to blind separation of nonnegative signal in [7] and image processing in [8]. 3 Smoothness in NTF To estimate the factors A and X, we use the similar approach as to NMF [3]. The specific cost function D(Y X 1 A) that measures the distance between Y and X 1 A is minimized with the alternating minimization technique. Lee and Seung [3] were the first who proposed two types of NMF algorithms. One minimizes the Euclidean distance, which is optimal for a Gaussian distributed additive noise, and the other for minimization of the Kullback-Leibler (KL) divergence, which is suitable for a Poisson distributed noise. The NMF algorithms that are optimal for many other distribution of additive noise can be found, e.g. in [9,10,11]. Assuming a Poisson noise, which often occurs in image processing, D(Y X 1 A) is given by the KL divergence. Thus D(Y X 1 A)= itk y itk log y itk z itk + z itk y itk, (5) where Y =[y itk ]andz =[z itk ]=X 1 A. Using the Gibbs prior [12] to model total smoothness in X andwiththe Bayesian framework as in [7,8], we get the penalized KL divergence D(Y X 1 A)= itk y itk log y itk z itk + z itk y itk + βu(x ), (6) where β is a penalty parameter and U(X ) is a total energy function that measures total roughness (smoothness) in X.

4 Nonnegative Tensor Factorization with Smoothness Constraints Markov Random Field Model MRF models have been widely applied in many image reconstruction applications, especially in tomographic imaging. In our application, MRF models motivate the definition of the total energy function in (6). Thus U(X )= jtk p S (j) q S (t) w pqr (jtk) ψ (x jtk x pqr,δ), (7) r S (k) where S (j), S (t),ands (k) are sets of indices of the entries in the neighborhood of x jtk, w pqr (jtk) is a weighting factor between the entries x jtk and x pqr, δ is a scaling factor, and ψ (ξ,δ) is some potential function of ξ and δ, which can take various forms. Exemplary potential functions are listed in Table 1 Table 1. Potential functions Author(s) (Name) Reference Functions: ψ(ξ,δ) 2 ξ (Gaussian) [12,13] δ Besag (Laplacian) [13] fi fi fi ξ fi fi fi fi δ fi Hebert and Leahy [14] δ log»1+( ξδ )2 Geman and McClure [15] Geman and Reynolds [16] 16 3 (ξ/δ) 2 3 (1 + (ξ/δ) 2 ) ξ/δ 1+ ξ/δ Stevenson and Delp (Hubert) [17] min{ ξ δ 2, 2 ξ δ 1} Green [18] δ log[cosh(ξ/δ)] The sets S (j), S (t) and S (k), and the associated weighting factors w pqr (jtk) are usually defined by the MRF model. Taking into account the nearest neighborhood, we have S (j) = {j 1,j,j +1}, S (t) = {t 1,t,t +1}, ands (k) = {k 1,k,k+1}. In consequence, w pqr (jtk) = 1 for pixels adjacent along a horizontal or vertical line, and w pqr (jtk) = 1 2 for pixels adjacent along a diagonal line. For p = j, q = t, r = k, and otherwise, w (jtk) pqr =0.

5 304 R. Zdunek and T.M. Rutkowski 3.2 Algorithm Assuming the local stationarity of (6) with respect to A and X, and using the alternating minimization algorithm, NTF 1 can be obtained with the following algorithm: Set Randomly initialize with nonnegative real numbers: A (0), X (0), Ȳ =[Y 1, Y 2,...,Y K ] R J TK, % row-wise unfolding For s =1, 2,..., until convergence do θ jtk = p S (j) q S (t) r S x (s+1) jtk = x (s) jtk I i=1 a(s) ij + βθ jtk X (s+1) =[X (s+1) 1, X (s+1) a (s+1) ij = a (s) ij TK z=1 x(s+1) jz a (s+1) ij a(s+1) ij I i=1 a(s+1) ij (k) w(jtk) pqr x jtk ψ I i=1 2,...,X (s+1) K KT z=1 a (s) ij y itk J, c=1 a(s) ic x(s) ctk ȳ iz x (s+1) jz J c=1 a(s) ic x(s+1) cz, % normalization ( x jtk x (s) ] R J TK, % unfolding, pqr,δ), xjtk x (s) End Since the Green s function [18] in Table 1 satisfies all the properties mentioned in [19], i.e. at a constant δ>0, it is nonnegative, even, 0 at ξ = 0, strictly increasing for ξ>0, unbounded, convex, and has bounded first-derivative, we decided to select this function to our tests. Thus ( ) xjtk x pqr ψ (x jtk x pqr,δ)=tanh. x jtk δ jtk 4 Numerical Tests The proposed algorithm has been extensively tested for various benchmarks of nonnegative smooth and sparse signals and images. The exemplary four original 3D arrays of nonnegative smooth and focussed objects are illustrated in Fig. 1. The arrays are stored in 4D tensor X R The mixtures Y R are obtained by multiplying the tensor X across 4-mode with the uniformly distributed random matrix A R across the 2-nd dimension.

6 Nonnegative Tensor Factorization with Smoothness Constraints 305 Fig. 1. Volumetric slice plots of 4 original 3D arrays of nonnegative smooth and focussed objects Fig. 2. Volumetric slice plots of 2 selected 3D arrays from 4D tensor of linear mixtures To separate the original 3D arrays from the mixtures, we used the algorithm presented in Section 3.2. For β = 0, the algorithm becomes a standard rowunfolded algorithm for NTF 1 [5]. The quality of the separation is evaluated with the standard mean-sir measure, comparing the estimated 3D arrays with

7 306 R. Zdunek and T.M. Rutkowski 10 Parameters of the statistics: Mean = [db], Std = [db] Parameters of the statistics: Mean = [db], Std = [db] Mean SIRs [db] Mean SIRs [db] Fig. 3. Histograms of 100 mean-sir samples generated with: (left) standard algorithm for NTF 1 (β = 0), (right) our algorithm with the Green s function, β =0.2, and δ =0.001 the respective original ones. The algorithm is initialized with the random initial approximations for A (0) and X (0). Hence, the Monte Carlo (MC) analysis is applied to test the consistency (repeatability) of the results. Fig. 3 shows the histograms from 100 mean-sir samples for both β =0andβ =0.2. The results demonstrate that the smoothness constraints considerably improve the repeatability of the results (the STD of the histogram with β =0.2 ismuch smaller than with β = 0). This is a promising result towards a uniqueness of NTF. 5 Conclusions In the paper, we derived the new algorithm for NTF 1, which may be useful for estimation of locally smooth signals and images in BSS applications. The algorithm exploits the information on pair-wise interactions between adjacent pixels, which is motivated by MRF models in tomographic image reconstruction. The proposed approach can be further extended with additional constraints or different updating rules. Also, another extension may concern the application of data-driven hyperparameter estimation techniques, especially for the regularization parameter. References 1. Shashua, A., Hazan, T.: Non-negative tensor factorization with applications to statistics and computer vision. In: Proc. of the 22th International Conference on Machine Learning, Bonn, Germany (2005) 2. Hazan, T., Polak, S., Shashua, A.: Sparse image coding using a 3D non-negative tensor factorization. In: International Conference of Computer Vision (ICCV), pp (2005) 3. Lee, D.D., Seung, H.S.: Learning the parts of objects by nonnegative matrix factorization. Nature 401, (1999)

8 Nonnegative Tensor Factorization with Smoothness Constraints Bader, B.W., Kolda, T.G.: Algorithm 862: Matlab tensor classes for fast algorithm prototyping. ACM Trans. Math. Softw. 32, (2006) 5. Cichocki, A., Zdunek, R., Choi, S., Plemmons, R., Amari, S.I.: Novel multi-layer nonnegative tensor factorization with sparsity constraints. In: Beliczynski, B., Dzielinski, A., Iwanowski, M., Ribeiro, B. (eds.) ICANNGA LNCS, vol. 4432, pp Springer, Heidelberg (2007) 6. Cichocki, A., Zdunek, R., Choi, S., Plemmons, R., Amari, S.: Nonnegative tensor factorization using Alpha and Beta divergencies. In: Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2007), Honolulu, Hawaii, USA, vol. III, pp (2007) 7. Zdunek, R., Cichocki, A.: Gibbs regularized nonnegative matrix factorization for blind separation of locally smooth signals. In: 15th IEEE International Workshop on Nonlinear Dynamics of Electronic Systems (NDES 2007), Tokushima, Japan, pp (2007) 8. Zdunek, R., Cichocki, A.: Blind image separation using nonnegative matrix factorization with Gibbs smoothing. In: Ishikawa, M., Doya, K., Miyamoto, H., Yamakawa, T. (eds.) ICONIP 2007, Part II. LNCS, vol. 4985, pp Springer, Heidelberg (2008) 9. Dhillon, I., Sra, S.: Generalized nonnegative matrix approximations with Bregman divergences. In: Neural Information Proc. Systems, Vancouver, Canada, pp (2005) 10. Cichocki, A., Zdunek, R., Amari, S.: Csiszar s divergences for non-negative matrix factorization: Family of new algorithms. In: Rosca, J.P., Erdogmus, D., Príncipe, J.C., Haykin, S. (eds.) ICA LNCS, vol. 3889, pp Springer, Heidelberg (2006) 11. Kompass, R.: A generalized divergence measure for nonnegative matrix factorization. Neural Computation 19, (2006) 12. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6, (1984) 13. Besag, J.: Toward Bayesian image analysis. J. Appl. Stat. 16, (1989) 14. Hebert, T., Leahy, R.: A generalized EM algorithm for 3-D Bayesian reconstruction from Poisson data using Gibbs priors. IEEE Transactions on Medical Imaging 8, (1989) 15. Geman, S., McClure, D.: Statistical methods for tomographic image reconstruction. Bull. Int. Stat. Inst. LII-4, 5 21 (1987) 16. Geman, S., Reynolds, G.: Constrained parameters and the recovery of discontinuities. IEEE Trans. Pattern Anal. Machine Intell. 14, (1992) 17. Stevenson, R., Delp, E.: Fitting curves with discontinuities. In: Proc. 1st Int. Workshop on Robust Computer Vision, Seattle, Washington, USA (1990) 18. Green, P.J.: Bayesian reconstruction from emission tomography data using a modified EM algorithm. IEEE Trans. Medical Imaging 9, (1990) 19. Lange, K., Carson, R.: EM reconstruction algorithms for emission and transmission tomography. J. Comp. Assisted Tomo. 8, (1984)