Weighted Singular Value Decomposition for Folded Matrices

Transcription

1 Weighted Singular Value Decomposition for Folded Matrices SÜHA TUNA İstanbul Technical University Informatics Institute Maslak, 34469, İstanbul TÜRKİYE (TURKEY) N.A. BAYKARA Marmara University Mathematics Department Göztepe Campus, İstanbul TÜRKİYE (TURKEY) METİN DEMİRALP İstanbul Technical University Informatics Institute Maslak, 34469, İstanbul TÜRKİYE (TURKEY) Abstract: Multilinear arrays have attracted an increasing tendency in especially the last two decades. Treating the data in many science and engineering areas like signal processing, computer vision, neuroscience started to use these type of data even though the data were transformed to vectors or matrices in order to efficiently use linear algebraic tools. One of the main goals has been to decompose the given multilinear array data to sums of some products of less variate arrays. This has been accomplished although the uniqueness could have not been achieved because of certain features. Nevertheless the obtained form which was basically developed by Lathauwer has been based on unfolding and folding of multilinear arrays and then using standard linear algebra. Some reductive array decomposition methods have also been developed quite recently by our group members without using a folding or an unfolding of arrays. This work tries to extend this recently proposed idea, without expecting any reduction in the resulting structure via consecutive application of the method, and, does not exclude the possibility of using folding and unfolding. Key Words: Multilinear Arrays, Singular Value Decomposition, Reductive Array Decomposition, Folded Vectors, Folded Matrices, Folded Arrays. 1 Introduction An algebraic entity indexed by more than two integers is mostly called multilinear array. Hence they can be considered as extensions to one index arrays, vectors, or to two index arrays, matrices, which are the most basic and therefore standard components in linear algebra. One of the most important issues is the decomposition problem. Either a vector or a matrix is tried to be decomposed to same type but rather simple components. The expression of a vector in terms of the elements of a basis set is the simplest issue to this end. We consider a Cartesian space in which our vector lies and try to find a basis set for uniquely representing any vector in this space as a linear combination of the basis set elements. What we know is that the linear combination coefficients are peculiar to the vector under consideration and is unique as long as the so-called basis set is complete (sufficing to represent any vector in the space). The basis set elements need to be vectors of the same type as the one under consideration. If we denote the given n-element vector we want to decompose by a and the basis set elements by u i s which are to be n-element vectors also; then we can write n a = a i u i (1) i=1 where n may be any positive integer (it can grow even unboundedly) and the scalar a i coefficients can be determined from algebraic equations to be constructed through elementwise comparison of both sides. The most natural basis set for (1) is composed of the standard unit vectors ith of which has its only nonzero element with value 1 at the ith position. If this is used in (1) then a i becomes the ith element of a. The most important facilitating feature of this set is the mutual orthonormality. We could use some other basis set whose elements are mutually orthonormal for (1). Then the general coefficient term a i would be the inner product of u i with a. Therefore we can draw attention to the point of the basis set construction. To compute the linear combination coefficients of the representation in an easier way brings more efficiency in utilization. Although the above representation contains a finite number of terms, its truncation may gain great importance for approximating the orginal vector especially when n number of the elements grows unboundedly. If we use an orthonormal basis set then the absolute values of coefficients, a i s, become the norm of the corresponding summand. If the ordering of the basis set element is such that the sum in (1) becomes descending in absolute values of a i s then the truncation at the first few terms gain more importance since we take the terms most dominantly contributing to the norm of a. ISBN:

2 The situation is not different for the case of the two index arrays, that is, matrices. Similar to what we have done for vectors above, this time a certain matrix will be decomposed by using the basis set whose elements are in the same structure, meaning matrices, to contribute to the norm of the original array dominantly for truncating the representation at a certain number of terms. If we denote the given matrix by A which is assumed to be an m n array and the basis matrices of same type by U ij s, then we can write m n A = a ij U ij (2) i=1 j=1 where we have preferred to use double sum even though we could use just a single sum between 1 and mn inclusive. This form, in fact, somehow reflects two index array feature of the given matrix. If we are to use the abovementioned single sum, then we would need to utilize one index arrays which may be considered unfolded counterparts of the relevant two index arrays. The a ij coefficients in (2) can be determined from the algebraic equations which can be constructed from the elementwise comparison of both sides and depends on how the U ij basis elements are chosen. First, they must be linearly independent and should span the space where the given matrix A resides. The most natural construction for U is to choose U ij s as the matrices whose only nonzero element with value 1 reside in the intersection of ith row and jth column. If we denote the ith standard unit vectors of the Cartesian spaces of m and n element vectors by e (m) i respectively then we can write e (n) j U ij = e (m) i e (n) T j, and i = 1,2,...,m j = 1,2,...,n (3) where, as we mentioned above, e (m) i and e (n) j are the m and n element vectors whose only nonzero elements are located at the ith and jth positions respectively and have the value 1. This choice of the basis matrices in (3) implies that the coefficient a ij becomes the element of A at the intersection of the ith row and jth column. The matrices in (3) are outer products and therefore their ranks are just 1. The m n type matrices form a linear vector space where it is possible to define some geometrical entities like distance, norm, and angle. We can define the following inner product for any couple of m n type matrices, A 1 and A 2 as follows (A 1,A 2 ) Tr ( A T 1 A ) 2 (4) which takes us to the following Frobenius norm definition for any matrix A A Tr (A T A) (5) We can easily show that the U ij matrices form an orthonormal basis set under this norm definition. As we immediately notice, the mutual orthonormality appears again in the foreground. We do not need to use the outer products composed of the Cartesian unit vectors. As long as mutual orthonormality exists we can use any basis set with the same efficiency since the a ij coefficients are determined as follows a ij = (U ij,a), 1 i m, 1 j n (6) We can call the representation given through (2) Orthonormal Decomposition if the U ij s form an orthonormal basis set because of the above discussions. One of the most widely used orthonormal decompositions is the Spectral Decomposition of the symmetric real matrices (or equivalently Hermitian complex matrices). It can be written for an n n type symmetric real matrix A as follows n A = α i a i a T i (7) i=1 where αs stand for the eigenvalues while as symbolize the corresponding eigenvectors. In the case of Hermitian complex matrices a T i should be exchanged by the Hermitian conjugate. Even though the decomposition in (7) is restricted to the symmetric or Hermitian matrices it is possible to construct its extended form for the nonsymmetric matrices by using left and right eigenvectors as long as the target matrix is square type and its eigenvalues have all the same algebraic and geometric multiplicities. In these cases a T i should be exchanged by the left eigenvector of the relevant eigenvalue while the corresponding right eigenvector should be written in place of a i. If the algebraic and geometric multiplicities of an eigenvalue are different for a given matrix then the above analysis fails to remain valid. The formulae should be revised by adding certain terms stairwise structuring between nearest neighbours in eigenpairs. The additive structure for the expression of Jordan canonical form appears from the analysis. We do not intend to get into more detail since they are far beyond the goal of this work. The second section of the paper is devoted to singular value decomposition of the matrices with some addressings to the reductive array decomposition. The ISBN:

3 third section covers the definitions and some important aspects of the folvecs, folmats and folarrs. The fourth section includes the singular value decomposition of the folmats and the last section finalizes the paper by concluding remarks. 2 Singular Value and Reductive Array Decompositions We can not directly construct a spectral decomposition for the nonsquare matrices since a direct eigenvalue problem can not be defined. This happens since a rectangular matrix of m n type (m n) can not transform a vector from a space to the same space. If we denote the m n type matrix under consideration by A then A maps from the Cartesian space, K n, the space of n-element vectors to the Cartesian space, K m, the space of m-element vectors while the transpose of the same matrix, A, from K m to K n. This brings the idea of two way mapping between K m and K n. We can try to find the possible vector and scalars satisfying the following equations Au = σv, A T v = σu, u K n, v K m (8) which can be converted to the following ones A T Au = σ 2 u, u K n AA T v = σ 2 v, v K m. (9) These are algebraic eigenvalue problems in n and m dimensional Cartesian spaces. Since the relevant matrices are symmetric, the eigenvalues are all real, and beyond that, nonnegative. This guarantees the real values of the σ scalar. Because of the square nature there are two possibilities for each possible σ value, positive and negative for the same absolute value. The general tendency is to choose the positive ones. The corresponding u eigenvectors are mutually orthogonal and can be normalized to get orthonormal basis set. Same thing remains valid also for the v vectors. Due to certain scientific historical reasons σ values are called Singular Values while the vectors us and vs are named Right Singular Vectors and Left Singular Vectors respectively. The equations in (9) urges us to write the following decomposition for the matrix A A = max(m,n) i=1 σ i v i u T i (10) This can be confirmed as follows: 1-The postmultiplication of equation (10) by anyone of the right singular vectors, say u j, produces Au j = σ j v j ; confirming the satisfaction of the first part of the equations in (9); 2-The premultiplication of the equation (10) by the transpose of any left singular vector, say vj T, produces v T A = σ j vj T. This confirms the satisfaction of the second part of the equations (after transposing) in (9). The equation (10) is called Singular Value Decomposition (SVD). It is a widely used concept in data processing. For this purpose, first the data are taken into a form of rectangular matrix. Then that matrix is decomposed via SVD. The terms in SVD are ordered in descending singular values which are the positive square roots of the multiplicatively symmetrized original matrix. Due to the unit norm features of the outer products appearing in SVD the singular values measure the relevant term s contribution to the norm of the sum, therefore to the norm of the original matrix. By looking at the values of the singular values it is possible to truncate the SVD at certain number of the first terms in the sum. This issue is known as Principal Component Analysis and used to get efficient approximations to the related data matrices. The compression algorithms in computer science or dominant network analysis in neuroscientific applications can be given as good examples to this issue. The multilinear counterpart of the SVD is based on the unfolding of a multilinear array. The basic idea is again to decompose the target array into multilinear outer products each of which has the product type elements with factors depending on single but different indices. The algebra behind this logic is not so simple to grasp or envisualize, however it works well in applications although the decomposition is not unique. The sophisticated and rigorous structure of multilinear SVD has urged scientists to develop certain rather easier decomposition methods. To this end, High Dimensional Model Representation [5], Enhanced Multivariance Product Representation [6], Reductive Multilinear Array Decomposition(RMAD) [9] can be mentioned in a trice. This method has been developed by members of our research group. To explain how this approach works we can consider a multilinear array A i1...i N and write the following decomposition formula A i1...i N = m j=1 σ (j) i b (j) i 1 c (j) i 2...i N (11) which is a linear combination of again binary outer products in contrast to N-factor outer products of the vectors. The outer products here are based on vectors again in one factor but the other factor is a multilinear array of (N 1) indices which is one less than the original array s index number. The index number reduction in the multilinear factor of the outer product gives the adjective Reductive to the name of the ISBN:

4 decompostion method. The terms of this decomposition are determined by using weighted or unweighted Euclidean distance minimization between the original array and the approximant truncated from (11). The details of this recently proposed approach will be discussed elsewhere. 3 Folvecs, Folmats and Folarrs The folding and unfolding of an array enable us to use the basic tools of linear algebra. To explain how to use these, first consider a vector a and denote its ith element by a i. If i is an even number then we can write the upper half of the vector as the first column of a two column matrix while the second column of this matrix is formed by the lower half of the vector. This procedure is in fact a reordering and produces a matrix which is a two index array from the vector which is a one index array. We can call this action Folding because of its pictorial appearence. We could reverse this action starting from the two column matrix to a vector. This reverse action can be called Unfolding because of its pictorial appearence similar to the folding procedure. If we denote the two column matrix produced from the vector a by B and the relevant matrix elements by b ij then we can write { ai j = 1 1 i n b ij = 2 j = 2 1 i n 2 a i+ n 2 (12) which explains the basic points of the folding procedures. The above example is not the only possibility of constructing a matrix from a vector. If n is divisible by a positive integer m then the resulting matrix after the folding will be an n m m type matrix. The divisibility by an integer for the number of the elements in the vector to be folded can be relaxed either discarding the residual terms or populating the vector elements by zeros to obtain divisibility. The term discarding means data loss while zero addition may cause some unusual values. The folding examplified above can be realized not only from a vector to matrix but from a vector to a multilinear array by appropriately locating the vector elements in place of the entries of a multilinear array. Regarding this consideration a multilinear array can be considered as a folded vector. To proceed to our aim we call a multilinear array folded vector or briefly folvec by retaining first three letters of each word in the name. The number of the indices in a folvec can be called folding order. This urges us to use the name n-fold vector together with the name folvec equivalently. The folding order is not only item for unique definition of a folvec. The domain of each index must also be specified. The equality of two folvecs is meaningful only when the folding orders and the domain of the indices are the same for both folvecs. As it is valid for same type vectors, all requirements of forming a linear vector space are satisfied by the same type folvecs. In other words, same type folvecs form a linear vector space whose zero element is the folvec composed of only zero elements. To proceed in more detail it is better to precisely define the type of a folvec. For vectors, the type is a single entity which is the number of the elements in the vectors. For matrices there are two numbers, the number of the rows and the number of the columns in the matrix. For an N-fold folvec there are n number of type elements, which are the number of the values to be taken by the indices individually. Hence, we can write down the following definition to precisely define the type for a folvec whose indices, i 1,..., i N, vary between 1 and n 1,..., n N t (n 1,...,n N ) (13) which is in fact an N-tuple. We are going to denote the linear vector space of t type folvecs by F t. The inner product of two folvecs a and b is defined as (a,b) = n 1 i 1 =1... n N i N =1 which yields the following norm a (a,a) 1 n 1 2 =... i 1 =1 a i1...i N b i1...i N (14) 1 n N 2 a 2 i 1...i N i N =1 (15) As we do amongst Cartesian spaces it is possible to map from one folvec space to another by using arrays. We call these entities Folded Matrices or briefly Folmats by taking the first three letters of the words. The folmats mapping from one folvec space to itself will be called Square Folmats. These must have an even number of indices, second half of which are used to act on the folvec in the domain while the first half are used to determine the resulting folvec. This urges us to distinguish the indices into two groups: right and left indices. The left indices correspond to the row number of a matrix while the right indices are the folmat counterparts of the column numbers. We are going to use semicolon to separate the left and right indices. If we consider a folmat, from a folvec space to itself, denoted by the capital letter A then its elements can be given through the general indexed structure A i1...i N ;j 1...j N. ISBN:

5 Now the mapping from F t to F t can be expressed in both closed and explicit forms as follows b = Aa, b i1...i N = n 1 j 1 =1 a,b F t... n N j N =1 A i1...i N ;j 1...j N a j1...j N, 1 i k n k, 1 k N. (16) As we can notice immediately these are quite straightforward extensions of the vector and matrix concepts of the ordinary linear algebra. What we have used to distinguish folmats and folvecs has been the semicolon. However, we can use semicolon to separate the indices to subgroups not only as left and right but also to more than two directions. We call the multiindex arrays having more than one semicolon amongst the indices Folded Arrays or simply Folarr s even though we do not intend to focus on them any further. 4 Singular Value Decomposition for Folmats We can easily define the transpose of a folmat with an impression from the matrices although there are certain difficulties in the definition of a meaningful transpose for a multilinear array. If we consider a folmat A mapping from a folvec space F t of type t to another folvec space F t of type t then the transpose of folmat A is denoted by A T and satisfies the following elementwise equalities ( A T ) i 1...,i N ;j 1...j N A j1...j N ;i 1...i N, 1 i k n k, 1 k N, 1 j l n k, 1 l N (17) A T reversely maps between F t and F t, that is, from F t to F t. This and the contents of the previous section enable us to define the eigenvalue problem of a square folmat. The algebra to this end can be converted to ordinary routines of the linear algebra by unfolding the relevant matrix to an ordinary matrix while the vectors are unfolded to ordinary vectors. The rest is the well known algebraic eigenvalue problem and everything remains exactly the same for the case of folded items. However we can call the folded forms of the eigenvectors Eigenfolvecs to distinguish them from their unfolded forms. All these definitions permit us to construct spectral decompositions and singular value decompositions as we do in the ordinary eigenvalue problems of linear algebra. For the singular value decomposition of the folmat A we can write Au = σv, A T v = σu, u F t, v F t (18) which can be converted to the following equations A T Au = σ 2 u, u F t, AA T v = σ 2 v, v F t (19) These express eigenvalue problems of symmetric square and nonnegative definite folmats. These can be processed by unfolding the folded items first and then solving the resulting ordinary eigenvalue problems. Then the positive roots of the eigenvalues become the singular values while the normalized eigenvectors correspond to the singular vectors. The folding of these singular vectors to recover the original form of the vectors u and v produces the singular folvecs. This is the conceptual structure of the SVD for folvecs. 5 Concluding Remarks This work introduces certain multilinear items to facilitate the utilization of the ordinary linear algebraic tools in construction of a singular value decomposition for certain type multilinear arrays (we call folmats) mapping from one space of certain type of multilinear arrays to another space of some other type of multilinear arrays. We do not attempt to decompose the target array to products of unit norm arrays containing more than two factors. As we know from the reductive multilinear array decomposition the terms composed of binary products of arrays allow us to use all tools of ordinary linear algebra after an unfolding procedure. Having the structures in ordinary algebraic nature we can go back to the case of folded items by folding to the original folded structures. A multilinear array can always be considered as a folmat by grouping its indices to the left and right indices. This procedure is not unique and each different grouping defines a different type folmat between different type folvec spaces. This brings an uncertainty which leaves us to make a choice to find the better decomposition. In other words the decompositions are not unique in this sense and some of them may become more preferable for our needs. References: [1] L. E. Leurgans, T. Ross and R. B. Abel, A Decomposition for Three-way Arrays, SIAM J. Matrix Anal. Appl., 14, 1993, pp [2] J. B. Kruskal, Rank, Decomposition and Uniqueness for 3-way and N-way Arrays, in Multiway Data Analysis, R. Coppi and S. Bolasco,eds., North Holland, Amsterdam, 1989, pp.718 ISBN:

6 [3] M. Marcus, Finite Dimensional Multilinear Algebra, Dekker, New York, 1975 [4] L. D. Lathauwer, B. D. Moor and J. Vandewalle, Multilinear Singular Value Decomposition, SIAM J. Matrix Anal. Appl., 21(4), 2000, pp [5] M. Demiralp, High Dimensional Model Representation and Its Application Varieties, The 4th Int. Conf. on Tools for Mathematical Modelling, St. Petersburg, Russia, 2003 [6] A. Okan, N.A. Baykara, M. Demiralp, Weight Optimization in Enhanced Multivariance Product Representation (EMPR) Method, ICNA- AM 10, Rhodes, Greece, 2010 [7] M. Demiralp, E. Demiralp, An Orthonormal Decomposition Method For Multidimensional Matrices, AIP Proceedings on ICNAAM 09, Creete, Greece, September, 2009 [8] E. Demiralp, Application of Reductive Decomposition Method for Multilinear Arrays (RDMM- A) to Animations, (WSEAS) MMACTEE 09, 2009 [9] M. Demiralp and E. Demiralp, Reductive Multilinear Array Decomposition Based Support Functions in Enhanced Multivariance Product Representation(EMPR), International Conf. on Appl. Comp. Sci., Malta, [10] F. B. Hildebrand, Introduction to Numerical Analysis, Dover Publications, Inc., ISBN-13: , (1987) [11] B. Hunt, R. Limpson, J. Rosenberg with K. Coombes, J. Osborn and G. Stuck, A Guide to MATLAB for Beginners and Experienced Users, Cambridge University Press, (2000). ISBN: