Theoretical Performance Analysis of Tucker Higher Order SVD in Extracting Structure from Multiple Signal-plus-Noise Matrices

Transcription

1 Theoretical Performance Analysis of Tucker Higher Order SVD in Extracting Structure from Multiple Signal-plus-Noise Matrices Himanshu Nayar Dept. of EECS University of Michigan Ann Arbor Michigan Raj Rao Nadakuditi Dept. of EECS University of Michigan Ann Arbor Michigan Abstract The Tucker Higher Order SVD is a popular algorithm for uncovering structure from tensor datacubes. This algorithm has been successfully used in many signal processing machine learning data mining applications. In this work we use recent results from rom matrix theory to analyze the performance of the HOSVD algorithm. In particular we focus on the performance HOSVD on 3-D tensors for extraction of structure from signal-plus-noise tensors. We analyze the missing data setting where the entries of the signal-plus-noise tensor are romly deleted. Our analysis bring into focus a phase transition phenomenon which separates a regime where the HOSVD can accurately estimate the latent signal matrix from a regime where it cannot. The threshold depends on the signal-to-noise ratio the fraction of missing entries observed in a manner we make explicit. Finally we illustrate the predicted performance curves using numerical simulations illustrate implication of our predictions on an widely used facial recognition dataset. I. INTRODUCTION The Tucker HOSVD is a popular algorithm for extracting structure from datacubes [] [3]. In the signal processing literature the HOSVD has been shown to successfully tease out weak signals from multiple datasets that are organized as a tensor or a datacube; see for e.g. [4] [8]. The authors of these methods have pointed out the need to rigorously quantify the performance of the HOSVD so that it may be compared to the performance of other non-hosvd based methods for extracting structure from datacubes [9] []. In this work we consider a setting where we are given n m m 2 signal-plus-noise matrices X (: : )... X (: :n). Motivated by problems arising in EEG analysis image processing other such data fusion applications [] [5] we model these matrices as X (: :i) = k v(i j)a (j) + Z(: :i) for i =...n j= analyze the performance of the HOSVD algorithm via its ability to recover the latent A (j) matrices. The main contribution of this paper is the first mathematically rigorous characterization of the performance of the HOSVD in extracting underlying latent signal matrices. The results build on recent results from rom matrix theory [9] [2]. We expect that this work can form the basis for improve the understing of the fundamental performance limits of the HOSVD in the context of the many signal processing applications listed earlier. Our results bring into sharp focus a phase transition which separates a regime where the HOSVD recovers correlated estimates of the latent signal matrices from a regime where the HOSVD recovers (asymptotically) uncorrelated estimates of the latent signal matrices. We precisely characterize the phase transition boundary how it is affected in the missing data setting. The paper is organized as follows. In Section II we discuss the signal-plus-noise models for the datacube; we summarize the HOSVD algorithm the quantity of interest in Section III. We present the main theoretical results in Section IV provide a sketch of the proof in Section V corroborate the theoretical results with numerical simulations in Section VI. We present some concluding remarks in Section VII. II. SIGNAL-PLUS-NOISE TENSOR MODELS We are given a 3-D tensor X whose i th layer X (: :i) is modeled as X (: :i) = k v(i j)a (j) +Z(: :i) j= }{{} L(::i) for i =...n() where v(i j) are the entries of an arbitrarily chosen matrix v with orthonormal columns A(j) R m m2 : j {...k} for j {...k} are arbitrary 2-D matrices such that A (i) A (j) = Tr(A T (i) A (j)) =θ i θ j δ ij k is fixed with respect to n. The tensor L should be considered as the signal tensor. Each entry of noise-only tensor Z is assumed to be i.i.d. Gaussian with mean zero variance /n. Thus () describes X as a signal-plus-noise tensor. We also consider the missing data setting. Here we are given a 3-D tensor X whose i th layer for i =...n is modeled as X (: :i)=[ k v(i j)a (j) + Z(: :i)] M p (: :i) (2) j= where v {A (j) : j {...k}}k are defined as in () denotes the Hadamard or element-wise product. For x =

2 ...m y =...m 2 i =...n the tensor M in (2) is defined as { with probability p M p (x y i) = with probability ( p) where p ( ] is the probability of observing a given entry in the signal-plus-noise tensor. Note that when p = (2) is equivalent to (). III. THE HIGHER ORDER SVD If X R m m2 n is a 3-D tensor then its Higher Order SVD is given by [7 page 95] X = S U 2 U 2 3 U 3 where S R m m2 n is the so-called core tensor which satisfies the super orthogonality property i.e. S(i : :) S(j : :) = S(:i:) S(:j:) = S(: :i) S(: :j) = U R m m U 2 R m2 m2 U 3 R n n are orthogonal matrices which are obtained as U d = LSVD(unfold d (X )). (3) In (3) LSVD(.) returns the left singular vectors of the argument. If X R m m2 n then the unfold operator is defined as [7 page 93-94] unfold (X ):=[X (: :)... X (:m 2 :)] R n mm2 unfold 2 (X ):=[X (: : )... X (: :n)] R m2 mn unfold 3 (X ):= [ X ( : :) T... X (m : :) T ] R m nm2. The d-mode vectors X are the column vectors of unfold d (X ). For a vector v R mm2 the operation fold is defined as v()... v(m 2 ) fold(v [m m 2 ]) :=.... v((m )m 2 +)... v(m m 2 ) The unfold operation should be viewed as the inverse operation of fold operation. It can be shown that if the signal tensor L has HOSVD given by L = S U 2 U 2 3 U 3 then the matrix A (j) in () can be expressed as A (j) = S(: :j) U 2 U 2. These A (j) images are the primary HOSVD related objects that appear in detection classification algorithms that utilize the HOSVD of tensor datacubes; see for e.g. [8 eq. no. (6) page 996] for its appearance in the context of hwritten digit recognition. Thus it is of interest to quantify the accuracy of the estimates of A (j) formed from the signalplus-noise tensor X. To that end let the HOSVD of the tensor X be given by X = S Ũ 2 Ũ 2 3 Ũ 3. Then an estimate of A (j) which we denote by Ã(j) can be computed from X as à (j) = S(: :j) Ũ 2 Ũ 2. (4) In this paper we are interested in characterizing how well A (j) is approximated by its estimate Ã(j). We shall quantify this using their cosine similarity which is given by ( ) Ã(j) A (j) à T (j) A (j) =trace. (5) In what follows we will theoretically characterize when we can expect the images to be correlated when they will be uncorrelated. IV. MAIN RESULTS We now state our main results next. In what follows denotes almost sure convergence. Theorem. For the setup in () let θ >θ 2 > >θ k. Then as n m m 2 with m m 2 /n c we have that 2 Ã(j) A (j) c( + θ2 j ) θj 2(c + θ2 j ) if θ j >c 4 otherwise ( + θ 2 j )(c + θj 2) θj 2 if θ j >c 4 + c otherwise. This result implies that the estimate Ã(j) is asymptotically localized around the latent A (j) that this estimate is generically biased hence inconsistent. An additional insight from Theorem is that the cosine similarity undergoes a phase transition i.e. below a certain critical SNR the estimate à (j) is orthogonal to A (j). We now provide an analogous characterization for the missing data setting. Theorem 2. In (2) let θ >θ 2 > >θ k assume that A (j) v satisfy the low coherence condition i.e. there exist non-negative constants η A η v C A C v independent of n such that for each j =...k we have that max A (j) η A log CA (m m 2 ) j m m 2 max v(:j) η v log Cv n. j n Then for p ( ) asn m m 2 with m m 2 /n c we have that Ã(j) A (j) 2 ( + pθ 2 j )(c + pθj 2) p ( + c) c( + pθ2 j ) pθj 2(c + pθ2 j ) if θ j > c 4 p otherwise θ 2 j if θ j > c 4 p otherwise.

3 Theorem 2 shows that there is a phase transition induced by romly missing data. The key difference between Theorem Theorem 2 is that the SNR parameter θ i is scaled by p in Theorem 2. In other words the critical SNR for the phase transition has increased by a factor of / p. An inspection of Theorem 2 reveals that for fixed θ j there is a certain critical value p crit = c/θ 2 j such that if p<p crit the estimate Ã(j) is asymptotically orthogonal to A (j). Above this critical threshold the estimates are correlated. The low coherence assumption echoes the results in [6] the matrix completion literature. V. SKETCH OF THE PROOF The results follow from an extension of the recent results from rom matrix theory [9] [2]. To that end we first note that the underlying matrix A (j) its estimate Ã(j) can be written in terms of the matrix unfolding of the signal corrupted tensor along the third dimension. Let the SVD of the unfolded signal tensor L be given by unfold 3 (L) =U 3 Θ 3 V H 3 (6) where U 3 V 3 are respectively the left right singular vector matrices Θ 3 is the diagonal matrix of the singular values of the unfolded signal tensor. Analogously let the SVD of the unfolded signal-plus-noise tensor X be given by ( ) unfold 3 X = Ũ3 Θ 3 Ṽ3 H (7) where Ũ3 Ṽ3 are respectively the left right singular vector matrices Θ 3 is the diagonal matrix of singular values of the unfolded signal-plus-noise tensor. The key observation that is a straightforward consequence of tensor algebra is that A (j) can be expressed as A (j) =fold(v 3 (:j)[m m 2 ]) with =Θ 3 (j j) while Ã(j) can be expressed as à (j) =fold(ṽ3(:j) [m m 2 ]) with = Θ 3 (j j). Consequently the cosine similarity between the A (j) its estimate à (j) isgivenby Ã(j) Ã(j) A (j) = V 3 (:j) A (j) Ṽ3(:j). From (6) (7) we have that Ũ 3 Θ3 Ṽ H 3 = unfold 3 (L) + unfold 3 (Z) }{{} =U 3Θ 3V3 H where unfold 3 (Z) is a noise-only matrix with i.i.d. N ( /n) entries the matrix U 3 Θ 3 V3 H is a rank- k matrix. Thus a direct application of the result in [9 Section 3.] gives us 2 Ã(j) A (j) c( + θ2 j ) θj 2(c + θ2 j ) if θ j >c 4 otherwise Squared inner product p = p = / θ Fig.. The cosine similarity as a function of θ for different values of p for the model in (2). Here k = m = m 2 =5 n = 25. The empirical averages were computed over trials. The solid line is the theoretical prediction from Theorem 2. Squared inner product θ =.9375 θ = p Fig. 2. The cosine similarity as a function of p for different values of θ for the model in (2). Here k = m = m 2 =5 n = 25. The empirical averages were computed over trials. The solid line is the theoretical prediction from Theorem 2. as stated in Theorem. An application of the same result gives us the stated result for in Theorem. For the missing data setting in Theorem 2 the only difference is the presence of the masking matrix unfold 3 (M p ) on the right-h side of (2). Consequently we have that Ũ 3 Θ3 Ṽ H 3 = [ U 3 Θ 3 V H 3 + unfold 3 (Z) ] unfold 3 (M p ). Via the same argument as the preceding Theorem we can apply the results of [2 Theorem 2.4]. This gives us the results stated in Theorem 2. VI. SIMULATION RESULTS We now validate our theoretical predictions using numerical simulations. For the model in (2) we set m =25m 2 = 4 n = generated a rom (incoherent) 2-D matrix A () constructed a 3-D tensor by taking an outer product of this 2-D matrix with a rom isotropic vector v with v = θ. This signal tensor was then corrupted with additive Gaussian noise masked with a binary tensor as in (2). Figure shows the agreement between the prediction of the cosine similarity given by Theorem 2 the empirical average for the same computed over Monte-Carlo trials.

4 Figure 2 shows the agreement between the theoretical prediction in Theorem 2 empirical simulation for the model in (2). Figure 3 shows a heatmap of the empirically averaged cosine similarity on a logscale as a function of θ p. The heatmap clearly illustrates the phase transition confirms the accuracy of the threshold (shown with the solid white line) predicted in Theorem 2. θ =26 θ 2 =3.78 θ 3 =2.4 p θ 4 =.5 θ 5 =.26 θ 6 =.6 Observed:.998 Predicted:.998 (a) The ground truth images A (j) for j =...6. Observed:.949 Predicted:.982 Observed:.686 Predicted: θ Fig. 3. Empirical heatmap of the logarithm of the cosine similarity given by (5) averaged over trials for the model in (2) with k = m = 25m 2 =4 n =. The deep blue region is the regime where the estimates are nearly orthogonal to the latent signal matrix. The solid white line denotes the phase transition threshold θ crit = c /4 / p as predicted by Theorem 2. Observed:.3664 Predicted:.493 Observed:.279 Predicted:.676 Observed:.26 Predicted: Finally we present a simulation that shows how the results can provide a principled justification for the empirical observation that the HOSVD is robust to missing data []. To illustrate its robustness we compare the ground-truth A (j) s (see Figure 4(a)) with the Ã(j) s (see Figure 4(b)) for images from the Yale Faces dataset that were corrupted samples as in (2) with p =.75 an associated θ = [ ]; here k =in (). The shows that there is a strong correlation between the similarity of an estimated representative image à (j) with the original representative image A (j) the visual contrast of the estimated representative images. The higher the cosine similarity the more features are preserved in Ã(j). When the cosine similarity is low (for j = 6) then the images are uncorrelated as predicted by Theorem 2 utilizing them in an inferential scheme will result in a degradation in performance. VII. CONCLUSION We theoretically analyzed the performance of the HOSVD showed that when the SNR is strong enough then the HOSVD produces estimates of the signal matrix that are correlated with the latent signal matrix. When the SNR is below a critical threshold then the estimates are asymptotically uncorrelated. We analyzed the performance with missing data showed the presence of a similar critical threshold for the probability of observing an entry below which the estimates (b) Estimated images Ã(j) for j =...6. Fig. 4. The HOSVD is robust to missing data noise. The ground truth images in (a) were corrupted as in (2) with p =.75. The images estimated using (4) are shown in (b). The cosine similarity between the ground truth the estimated images was computed using (5) compared with the predictions from Section IV. were uncorrelated. We corroborated the theoretical predictions with empirical simulations. A natural next step is to analyze the performance of other tensor decomposition schemes such as the PARAFAC/CANDECOMP []. This can facilitate a comparison of their performance relative to the fundamental limits of the HOSVD uncovered in this paper provide a better understing of which technique(s) best uncover weak latent structure in the presence of missing data from noisy datacubes. ACKNOWLEDGMENT This work was supported by an ONR Young Investigator Award N466 an AFOSR Young Investigator Award

5 FA a NSF award CCF-65 a ARO MURI grant W9NF REFERENCES [2] R. Nadakuditi Optshrink: An algorithm for improved low-rank signal matrix denoising by optimal data-driven singular value shrinkage IEEE Transactions on Information Theory vol. 6 no. 6 pp. 7 May 24. [] T. G. Kolda B. W. Bader Tensor decompositions applications SIAM Review vol. 5 no. 3 pp [2] A. Cichocki Y. Washizawa T. M. Rutkowski H. Bakardjian A. H. Phan S. Choi H. Lee Q. Zhao L. Zhang Y. Li Noninvasive bcis: Multiway signal-processing array decompositions. IEEE Computer vol. 4 no. pp [3] G. Bergqvist E. G. Larsson The higher-order singular value decomposition: Theory an application [lecture notes] IEEE Signal Processing Magazine vol. 27 no. 3 pp [4] N. D. Sidiropoulos R. Bro G. B. Giannakis Parallel factor analysis in sensor array processing IEEE Trans. on Signal Processing vol. 48 no. 8 pp [5] M. A. O. Vasilescu D. Terzopoulos Multilinear analysis of image ensembles: Tensorfaces in Computer Vision - ECCV 22 pp [6] C. Tian G. Fan X. Gao Q. Tian Multiview face recognition: from tensorface to v-tensorface k-tensorface IEEE Trans. on Systems Man Cybernetics Part B: Cybernetics vol. 42 no. 2 pp [7] D. Terzopoulos Y. Lee M. A. O. Vasilescu Model-based image-based methods for facial image synthesis analysis recognition pp. 3 8 Proc. Sixth IEEE Int. Conf. on Automatic Face Gesture Recognition 24. [8] T. G. Kolda J. Sun Scalable tensor decompositions for multiaspect data mining Eighth IEEE International Conference on Data Mining pp [9] A. Karami M. Yazdi G. Mercier Compression of hyperspectral images using discerete wavelet transform tucker decomposition IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing vol. 5 no. 2 pp [] M. Signoretto R. Van de Plas B. De Moor J. A. Suykens Tensor versus matrix completion: a comparison with application to spectral data IEEE Signal Processing Letters vol. 8 no. 7 pp [] M. Weis F. Romer M. Haardt D. Jannek P. Husar Multidimensional space-time-frequency component analysis of event related eeg data using closed-form parafac Proc. of IEEE Int. Conf. on Acoustics Speech Signal Processing pp [2] Y. Washizawa H. Higashi T. Rutkowski T. Tanaka A. Cichocki Tensor based simultaneous feature extraction sample weighting for eeg classification in Neural Information Processing. Models Applications pp [3] M. Weis D. Jannek T. Guenther P. Husar F. Roemer M. Haardt Temporally resolved multi-way component analysis of dynamic sources in event-related eeg data using parafac2 in Proceedings of the 8th European Signal Processsing Conference (EUSIPCO-2) Aalborg Denmark 2. [4] S. L. Freire T. J. Ulrych Application of singular value decomposition to vertical seismic profiling Geophysics vol. 53 no. 6 pp [5] J.-F. Cardoso Eigen-structure of the fourth-order cumulant tensor with application to the blind source separation problem in Proc. of IEEE Conf. on Acoustics Speech Signal Processing pp [6] E. J. Cès X. Li Y. Ma J. Wright Robust principal component analysis? Journal of the ACM vol. 58 no. 3 p. 2. [7] L. Eldén Matrix methods in data mining pattern recognition vol. 4 SIAM 27. [8] B. Savas L. Eldén Hwritten digit classification using higher order singular value decomposition Pattern recognition vol. 4 no. 3 pp [9] F. Benaych-Georges R. R. Nadakuditi The singular values vectors of low rank perturbations of large rectangular rom matrices Journal of Multivariate Analysis vol. pp