Towards A New Multiway Array Decomposition Algorithm: Elementwise Multiway Array High Dimensional Model Representation (EMAHDMR)

Transcription

1 Towards A New Multiway Array Decomposition Algorithm: Elementwise Multiway Array High Dimensional Model Representation (EMAHDMR) MUZAFFER AYVAZ İTU Informatics Institute, Computational Sci. and Eng. Program, İTU Ayazağa Campus, 34469, İstanbul, TÜRKİYE (Turey), muzaffer.ayvaz@be.itu.edu.tr METİN DEMİRALP İTU Informatics Institute, Computational Sci. and Eng. Program, İTU Ayazağa Campus, 34469, İstanbul, TÜRKİYE (Turey), metin.demiralp@be.itu.edu.tr Abstract: In this study, very early steps of a new algorithm for multiway array decomposition via High Dimensional Model Representation (HDMR) is proposed. HDMR is originally developed for the multivariate functions and represents them as the sum of lower variate functions inluding the constant term, and thus HDMR is an inherent candidate for decomposing multiway arrays. The proposed algorithm represents given multiway array valued multivariate function as the sum of same type multiway array valued functions with lower multivariances starting from the constancy in ascending multivariance. This algorithm generalizes and unifies the recently proposed algorithms Vector HDMR and Matrix HDMR by Demiralp and his group and thus enlights the big picture of a new family of multiway array decomposition algorithms based on High Dimensional Model Representation. Key Words: Multilinear algebra, multidimensional arrays, tensor decomposition, singular value decomposition, High Dimensional Model Representation. 1 Introduction In the last decades, multilinear algebra has been an important scientific field with the primary emphasis on decomposing a multiway array (also called multidimensional array, tensor, N way array, etc.) Since processing large amount of multidimensional data generally requires its compression, decomposing a multiway array and approximating this N dimensional array by the lower ran N way arrays have great importance in the areas including physicometrics, chemometrics, neuroscience, signal and image processing, networ analysis, and human motion recognition amongst the others [1, 2]. The Tucer and CANDECOMP/PARAFAC decompositions are the most famous decompositions. The Tucer decomposition represents a given multiway array as the reduced dimensional core multiway array which is outer producted by the orthonormal matrices. HOSVD and N way SVD (SVD is abbreviation for singular value decomposition) are originally proposed algorithms to compute the Tucer decomposition of a multiway array [1]. Since these algorithms are natural generalizations of matrix SVD, they preserve some properties of the matrix SVD such as orthogonality, the equality of the Frobenious norm of given multiway array and core multiway array, etc. But some very important properties of the matrix SVD could not be preserved in these algorithms. While matrix SVD transforms given matrix to a diagonal form, HOSVD transforms given multiway array to a dimensionally reduced multiway array. While matrix SVD guarantees the best ran 1, ran 2,..., ran n approximations to a matrix by the Ecart Young theorem, HOSVD does not guarantee the best ran n approximation [1, 2] 1 CANDECOMOP (canonical decomposition) and PARAFAC (parallel factors) approximate to the given multiway array by a finite sum of ran one tensors. The main problem for the algorithms that produces CANDECOMP/PARAFAC decomposition is defining the ran of given higher order arrays [2]. Only the upper bound for multiway arrays ran is nown by Krusal theorem [2, 4, 5]. Defining the tensor or multiway array ran is an NP hard problem as recently proven. However there are lots of difficulties to define the ran of multiway arrays, and, there are lots of algorithms presented in the literature. Most of the in- 1 Also, a counter example to the Ecart Young Theorem for multidimensional case is given by Tamara Kolda. Interested reader should see reference [3]. ISBN:

2 troduced algorithms are iterative procedures that run until the desired accuracy is achieved. The higher order generalizations of the power method and Rayleigh Quotient Iteration algorithms are presented to calculate lower ran approximation to a multiway array. These algorithms do not guarantee the convergence to global optimum points. For different initial values, they may converge to different local optimum points [6]. Simultaneous diagonalization of a set of matrices is also an effective way to compute low ran approximation of a multiway array or to compute CAN- DECOMP/PARAFAC decomposition of a tensor under consideration. Several methods, including simultaneous generalized Schur decomposition, simultaneous EVD and Jacobi type algorithms, are developed. Jacobi type (also called Jacobi lie) algorithms are the generalization of the Jacobi algorithm for matrix diagonalization [7]. For specifically structured multiway arrays such as symmetric multiway arrays and multiway arrays having Hermitian slices Jacobi type algorithms are presented in the literature [8]. More general form of Jacobi lie algorithm presented for third order multiway arrays to maximize the trace and square sum of the elements on the multiway array diagonal. This algorithm produces quite good results in sense of diagonal dominancy by means of Frobenious norm [7]. 2 In the last 3 4 years, a new multiway array representation algorithm named TT (tensor train) and QTT (quantics tensor train) are also proposed [9 11]. TT and QTT represent a given multiway array as the products of lover dimensional multiway arrays and can be computed by a recursive procedure. These representations are used for some high complexity algorithms lie multiparticle Schrödinger equation, DMRG (Density matrix renormalization group) algorithm, and, produced very promising results to brea the bounds of curse of dimensionality 3. In this study, the introductory steps of the theory of a new framewor, Elementwise Multiway Array HDMR is discussed in some detail. The proposed algorithm represents a given multidimensional array valued function as the sum of same type multiway array valued functions with lower multivariances starting from constancy. A single constant multiway array with same type as the target array is followed by the same type multiway array valued univariate functions. Then bivariate arrays come and so on. In the 2 This paper does not cover all of the literature review of the presented wors. Comprehensive review of the multiway array decomposition and low ran approximation algorithms can be found in reference [2]. 3 The term curse of dimensionality is originally used by Bellman [12] representation all components are of same type array. The changing property is the multivariance as in Vector HDMR [13] and Matrix HDMR [14]. Since this algorithm is based on HDMR, it can be used for both function valued multiway arrays and applications where only discrete data is available. The theory presented in this paper can be extended for the EMPR (Enhanced Multivariate Product Representation) straightforwardly, which is out of the scope of this paper. The remaining part of the paper is organized as follows. The second section is devoted to the explanation of High Dimensional Model Representation. The detailed description of the proposed algorithm is discussed in the third section. The folded weight matrix construction, which is the most important part of the algorithm, is explained in the fourth section. The paper is finalized by the future directionings and concluding remars in the fifth section. 2 Basics of High Dimensional Model Representation High Dimensional Model Representation, originally introduced by I. M. Sobol, inspired from the theoretical wor of Kolmogorov [15], represents a multivariate function as the finite sum of lower variate and orthogonal functions and a constant term as follows [16]. f (x 1,...,x N ) = f 0 + N f i1 (x i1 ) i 1 =1 N + f (x i1i2 i1,x ) i2 i 1,i 2 =1 i 1 <i f 12...N (x 1,...,x N ) (1) The right hand side of the above equation consists 2 N terms. When all 2 N components are used, the multivariate function under consideration is expressed exactly. It is observed that the truncation at most at bivariate terms produces acceptable quality approximations to the function under consideration. All the right hand side components of HDMR can be determined uniquely under the vanishing under integration condition [16 18] w i (x i )dx i f i1 i 2...i (x i1,x i2,...,x i ) = 0, [a i,b i ] i = i 1,i 2,... i (2) Here, w i (x i ) denotes a weight function which has only one independent variable. The integration of this ISBN:

3 weight function over the independent variable domain should be equal to one for simplicity. w i (x i )dx i = 1 (3) [a i,b i ] The vanishing under integration condition also assures the orthogonality of the right hand side of the HDMR expansion. Using abovementioned properties every component of the expansion can be determined as follows. N f 0 = D[x] w (x )f(x 1,...,x N ) (4) Ω N f i (x i ) = D[x {x i }] Ω N 1 N w (x )f(x 1,...,x N ) f 0 (5) i f ij (x i,x j ) = D[x {x i,x j }] Ω N 2 N w (x )f(x 1,...,x N ) i,j f 0 f i (x i ) f j (x j ) (6) Here, Ω N denotes an N dimensional hyperprism while D[x] represents integration over all independent variables. D[x {x i }] stands for the integration over all independent variables except x i and so on. It is quite important here to note that the total weight functions used in the integrations has multiplicative nature. W(x 1,...,x N ) = N w i (x i ) (7) i=1 There may be lots of measures to determine the quality of the approximation. The most preferred one in the literature is the additivity measurers [18 29]. σ 0 = σ 1 =. σ N = 1 f 2 f f 2 N f i 2 + σ 0 i=1 1 f 2 f 12...N 2 + σ N 1 (8) Here, f 2 denotes the norm square, that is the inner product of the function under consideration by itself. It is quite easy to notice that all σ values are well ordered. 0 σ 1 σ 2... σ N = 1 (9) The presented HDMR method is generally called Plain HDMR. There are many other HDMR methods lie Generalized HDMR [24], Logarithmic HDMR, Factorized HDMR [26], Hybrid HDMR [23, 27], RS-HDMR [28], Transformational HDMR, EMPR amongst the others. All these methods can produce different results according to the function under consideration. In this study, only the Plain HDMR method is used for function valued multiway arrays and their decomposition. However, it is possible to extend the presented algorithm to the other methods and discrete data sets. 3 Elementwise Multiway Array High Dimensional Model Representation (EMAHDMR) In this section, we will demonstrate how to evaluate the HDMR components, in particular constant term, of a multiway array valued multivariate function F i1 i 2...i N (x 1,x 2,...,x M ). We do not intend to expand this entity to HDMR with respect to its indices. Indices will be ept unchanging in the expansion which will be therefore in independent variables x 1,..., x M. Hence what we are going to obtain will be called Elementwise Multiway Array High Dimensional Model Representation (EMAHDMR). F i1 i 2...i N (x 1,x 2,...,x M ) represents an N index array whose each entry is an M variate function. It is important to remar here that if the function values are discretely available, F i1 i 2...i N (x 1,x 2,...,x M ) represents (N + M) index multiway array. The simplest case is an M variate function F (x 1,x 2,...,x M ) and can be represented by Plain HDMR. The second and third cases are F i1 (x 1,x 2,...,x M ) and F i1 i 2 (x 1,x 2,...,x M ), and, can be respectively represented by Vector HDMR [13] and Matrix HDMR [14] which are the algorithms recently developed in related wors. The most general case is F i1 i 2...i N (x 1,x 2,...,x M ) and its HDMR expansion is as follows when we eep indices unchanged in the components F i1 i 2...i N (x 1,...,x M ) = F (i 1i 2...i N ) + M j 1 =1 0 F (i 1i 2...i N ) j 1 (x j1 ) + +F (i 1i 2...i N ) 12...M (x 1,...,x M ) (10) ISBN:

4 where we have used the superscript (i 1 i 2...i N ) to explicility show the index dependence (especially its unchanging nature). Here F (i 1i 2...i N ) 0 denotes an N index array whose elements are constants. (x i1 ) stands for an N index array whose elements are univariate functions depending on same independent variable, i 1, and so on. The evaluation F (i 1i 2...i N ) i 1 of F (i 1i 2...i N ) 0 can be realized explicitly using (4) and result is given as follows. F (i 1i 2...i N ) 0 = [ M ] D[x] W (x ) Ω N F i1 i 2...i N (x 1,...,x M ) (11) In this form the product type weight stands independent of i indices. This means no difference between this representation and the plain HDMR since a single HDMR decomposes all elements of the multiway array valued function in the same fashion. However our purpose is to bring more flexibility to this type HDMR. This can be done by extending the weight function structure. We are going to focus on this issue in the next section. 4 Weight Considerations in Elementwise Multiway Array High Dimensional Model Representation The product type weight in (11) is too restricted as we stated above. Weight does not depend on the i indices. So as an extension we may bring the index dependence to each factor of the weight. Then the overall weight can be rewritten as follows W (x 1,...,x M ) M W ({i}) (x ) (12) where we have used the shorthand notation {i} to represent all i indices. This extension separates the HDMR on each individual element of EMAHDMR above. However it does not permit the interaction to exist between different elements of the multiway array. Whereas this interaction is quite important to control the decomposition since it can bring more flexibilities as the vector and matrix HDMRs do. Therefore it is better to create more extension in the weight definition of (12) and to write the following formula for new extension. W (x 1,...,x M ) M W ({i},{j}) (x ) (13) where the new indices denoted by the shorthand notation {j} are individually symbolized as j 1,...,j M and j corresponds to i by taing same positive integer values. We have deliberately used comma to distinguish the roles of the i and j indices. As a matter fact i indices play the role of the row indices in a matrix while the j indices somehow correspond to the column indices of the same matrix. Now we propose to employ another widely used shorthand notation. The statement of the rule is as follows: If any index is repeated in a formula then it means that there is an implicit sum over the formula for all possible values of that index. The more number of repeated indices, the more number of sums. By having this rule we can rewrite (11) as follows [ M ] F ({i}) 0 = D[x] W ({i},{j}) (x ) Ω N F {j} (x 1,...,x M ) (14) where the multiway arrays given below [ ] W (x ) W ({i},{j}) (x ), = 1,2,...,M (15) act as if matrices. Each array maps from a multiway array of N indices to another multiway array of N indices. It behaves lie a square matrix. We call this entity as a folded matrix or briefly folmat [14]. This is because each row of a matrix can be divided into segments and those segments are reordered to get a multiway array so can be the columns of the same matrix, of course, by obeying certain compatibility conditions. Folded matrices or folmats in our shorthand terminology can be added, multiplied by scalars, and multiplied as in the case of matrices, standard items in linear algebra. In addition procedure the sum of the corresponding elements of two folmats gives the corresponding element of the resulting folmat. Similarly, multiplying by a scalar produces a folmat whose elements are the product of the corresponding elements of the target folmat by that scalar. In the multiplication of two folmats, first a set of row index values and a set of column index values are chosen, then the first folmat elements having these row index values are multiplied by the second folmat elements having these column index values if the column index of the first factor matches the row index of the second factor. The sum over the all possible matching index values is the resulting folmat s corresponding element which has the same row indices with the first folmat element and the same column indices with the second folmat element. All elements of zero folmat, which is actionless element in addition, are zero while the unit or identity ISBN:

5 folmat, which does not change its operand in multiplication, is defined as follows I N δ i j (16) where the row indices is and column indices js tae values from the index domain of the multiway array which is under consideration for HDMR and δ stands for the Kronecer s delta. The weight folmats have univariate function elements above. To facilitate the HDMR construction following features are expected to be possessed by them: The integral of each weight folmat factor over its independent variable and corresponding interval should be equal to unit or identity folmat. W (x )dx = I (17) [a,b ] All folded weight matrices must commute with each other. W i W j W j W i = 0, i,j (1,2,...,M) (18) All folded weight matrices should be positive definite and symmetric. Since the unfolding of folmats to ordinary matrices (two index arrays) converts all folmat sums and folmat products to ordinary matrix sums and products, EMAHDMR can be in fact considered as folded version of the vector HDMR [13]. Even matrix HDMR [14] can be related to this case via appropriate foldings. Of course unfolding is more suitable to computational aspects in computers as long as they use linear arrays for data storages. However, in the sense of lumping, EMAHDMR can be preferred. According to the application areas and/or structure of the multiway arrays under consideration, EMAHDMR may mean sometimes difficulty and sometimes mean flexibility. Setting a strategy to find the optimum unfolding of the multiway arrays for special types is left as the future wor. Before closing the section we need to mention about how the weight folmat is chosen. Of course the easiest form is proportional to the unit folmat. The next easiest case involves the super diagonal folmats where each element is proportional to the product of the Kronecer deltas appearing in the definition of the unit folmat with element depending proportionality functions. We do not intend to go beyond this level discussion here. 5 Conclusion In this study, fundamental concepts of the theory of a new family of multiway array valued multivariate function decomposition methods based on Plain HDMR philosophy is discussed in some detail. The proposed method is the generalization of the methods Vector HDMR [13] and Matrix HDMR [14]. Even though the method presented here is given for multiway array valued functions, it is quite straightforward to use it for the discretely available data as it is done for data partitioning through HDMR. An interpolation stage may tae us from discreteness to the realm of the functions. This ind of research is at the focus in our group and will be reported somewhere else in future. The presented method is also a powerful candidate for the decomposition of bloc structured multiway arrays. With the same philosophy mentioned in this study, it is possible to develop more efficient methods based on Enhanced Multivariate Product Representation (EMPR), which is left as a future wor. Acnowledgements: The first author thans Turish Scientific and Technological Research Council of Turey (TUBITAK) for its support and the second author is grateful to Turish Academy of Sciences, where he is a principal member, for its support and motivation. References: [1] De Lathauwer, L.; De Moor B; Vandewalle J.; A Multilinear Singular Value Decomposition, SIAM J. Matrix Anal. Appl, Vol. 21, No.4, pp [2] Kolda, T.G.; Bader B.W.; Tensor Decompositions and Applications, SIAM Review, Vol. 51, No.3, pp [3] Kolda, T.G.; A counterexample to the possibility of an extension of the Ecart-Young low-ran approximation theorem for the orthogonal ran tensor decomposition, SIAM J. Matrix. Anal. Appl, Vol. 24, No.3, pp [4] Krusal, J.B.; Three-way arrays: Ran and uniqueness of trilinear decompositions with applications to arithmetic complexity and statistics, Linear Algebra and its Applications, Vol. 18, pp ,1977 [5] Krusal, J.B.; Ran, decomposition, and u- niqueness for 3-way and N-way arrays, Multiway Data Analysis, R.Coppi and S. Bolasco (Eds.),pp. 7-18,Noth-Holland,1989 ISBN:

6 [6] Zhang T.; Golub G.H.; Ran-one approximation to high order tensors, SIAM J. Matrix. Anal. Appl, Vol.23, No.2, pp , 2001 [7] MArtini C.D.M.; Van Loan C.F.; A Jacobi-type Method for computing Orthogonal Tensor Decompositions, SIAM J. Matrix. Anal. Appl, Vol. 30, No.3, pp [8] Badeau, R.; Boyer R.; Fast Multilinear Singular Value Decomposition For Structured Tensors, SIAM J. Matrix Anal. Appl, Vol. 30, No.3, pp [9] Oseledets, I.V.; Tyrtyshniov, E.E.; Breaing the Curse of Dimensionality, or How to Use SVD in Many Dimensions, SIAM J. Sci. Compt, Vol. 33, No.5, pp [10] Oseledets, I.V.; Approximation of 2(d) x 2(d) Matrices Using Tensor Decompositions, SIAM J. Matrix Anal. Appl. Sci. Compt, Vol. 31, No.4, pp [11] Oseledets, I.V.; Tyrtyshniov, E.E.; Recursive decomposition of multidimensional tensors, Dolady Mathematics, Vol. 80, No.1, pp [12] Bellman, R.E.; Dynamic Programming, Princeton University Press, 1957 [13] Tunga, B.; Demiralp, M.; Basic Issues in Vector High Dimensional Model Representation, 9th International Conference of Numerical Analysis and Applied Mathematics, IC- NAAM 2011 (to be published). [14] Tuna S.; Demiralp M.; Matrix HDMR with the Weight Matrices Generated by Subspace Construction, 9th International Conference of Numerical Analysis and Applied Mathematics, IC- NAAM 2011 (to be published). [15] Kolmogorov, A. N.: On the Representation of Continuous Functions of Many Variables by Superposition of One Variable and Addition, English Translation: American Math. Soc., 2, 28, 1963, pp [16] Sobol, I. M.: Sensitivity Estimates for Nonlinear Mathematical Models, English Translation: MMCE, Vol.1, No.4,1993, pp [17] Alış, Ö. F.; Rabitz, H.: General Foundations of High Dimensional Model Representation, Journal of Mathematical Chemistry, 25, 1999, pp [18] Alış, Ö. F.; Shorter, J.; Shim, K.; Rabitz, H.: Efficient Input-Output Model Representation, Computer Physics Communications, 117,1999, pp [19] Alış, Ö. F.; Rabitz, H.: Efficient Implementation of HDMR, Journal of Mathematical Chemistry, 29-2,2001, pp [20] Li, G.; Rosenthal, C.; Rabitz, H.: High Dimensional Model Representation, Journal of Physical Chemistry A, ,2001, pp [21] Demiralp, M.: High Dimensional Model Representation and Its Application Varieties, The Fourth International Conference on Tools for Mathematical Modeling, St. Petersburg, Russia, June 23-28, 2003 [22] Kurşunlu, A.; Demiralp, M.: Additive and Factorized HDMR Applications to the Multivariate Diffusion Equation Under Vanishing Derivative Boundary Conditions, The Fourth International Conference on Tools for Mathematical Modeling, St. Petersburg, Russia, June 23-28,2003 [23] Tunga, B.; Demiralp, M.: Hybrid HDMR Approximants and Their Utilization in Applications, The Fourth International Conference on Tools for Mathematical Modeling, St. Petersburg, Russia, June 23-28, 2003 [24] Tunga, M. A.; Demiralp, M.: Data Partitioning Via Generalized HDMR and Multivariate Interpolative Applications, The Fourth International Conference on Tools for Mathematical Modeling, St. Petersburg, Russia, June 23-28,2003 [25] Yaman, İ.; Demiralp, M.: HDMR Approximation of an Evolution Operator with a First Order Partial Differential Operator Argument, App. Num. Anal. and Comp. Math., Wiley CHV, I, 2003, pp [26] Tunga, M.A.; Demiralp, M.: A Factorized High Dimensional Model Representation on the Nodes of a Finite Hyperprismatic Regular Grid, Applied Mathematics and Computation, Volume 164, Issue 3,2005,, [27] Tunga, B.; Demiralp, M.: A Novel Hybrid High Dimensional Model Representation (HHDMR) Based on Combination of Plain and Logarithmic High Dimensional Model Representations, WSEAS-2007 Proceedings, WSEAS 12-th International Conference on Applies Mathematics for Science and Engineering, Vol.1, 2007, pp [28] Li, G.; Wang, S.; Rabitz, H.: Fractical Approaches to Construct RS-HDMR Component Functions, Journal of Physical Chemistry A, 106, 2002, [29] Li, G.; Schoendorf, J.; Ho, T.; Rabitz, H.; Multicut-HDMR with an Application to an Ionospheric Model, Journal of Computational Chemistry, 25-9, 2004, ISBN: