Dimitri Nion & Lieven De Lathauwer

Size: px

Start display at page:

Download "Dimitri Nion & Lieven De Lathauwer"

Kenneth McDonald
5 years ago
Views:

1 he decomposition of a third-order tensor in block-terms of rank-l,l, Model, lgorithms, Uniqueness, Estimation of and L Dimitri Nion & Lieven De Lathauwer.U. Leuven, ortrijk campus, Belgium s: Dimitri.Nion@kuleuven-kortrijk.be Lieven.DeLathauwer@kuleuven-kortrijk.be CP 2009, Nurià, Spain, une 4th-9th, 2009

2 ntroduction ensor Decompositions Powerful multi-linear algebra tools that generalize matrix decompositions. Motivation: increasing number of applications involving manipulation of multi-way data, rather than 2-way data. Some key research axes: Development of new models/decompositions Development of algorithms to compute decompositions Uniqueness of tensor decompositions Use these tools in new applications, or existing applications where the multi-way nature of data was ignored until now ensor decompositions under constraints e.g. imposing nonnegativity or algebraic structures, specific to applications under consideration 2

3 From matrix SVD to tensor HOSVD U D H V Matrix SVD d u v H + + d u v H ensor HOSVD third-order case U L N H M W V L M N y uv w h ijk il jm kn lmn l m n H U V W 2 3 One unitary matrix U, V, W per mode H is the representation of in the reduced spaces. We may have L M N H is not diagonal difference with matrix SVD. 3

4 From matrix SVD to PFC U D H V Matrix SVD d u v H + + d u v H C H B PFC decomposition H is diagonal if ijk, h ijk, else, h ijk 0 c b + + c b Sum of rank- tensors: + + a C B a set of matrices of the form: :,:,k diagck,: B

5 From PFC/HOSVD to Block Components Decompositions BCD [De Lathauwer and Nion] BCD in rank L r,l r, terms L c L B + + L c L B BCD in rank L r, M r,. terms B L H M H + + L M B BCD in rank L r, M r, N r terms N L H M B C + + L N H M C B 5

6 Content of this talk BCD - L r,l r, L c L B + + L c L B Model ambiguities lgorithms Uniqueness Estimation of the parameters L r r,, and n application in telecommunications 6

7 BCD - L r,l r, : Model ambiguities c c L F F B + + L F F B Unknown matrices: L... L L B L B... B C... BCD-L r,l r, is said essentially unique if the only ambiguities are: c c rbitrary permutation of the blocks in and B and of the columns of C + Each block of and B post-multiplied by arbitrary non-singular matrix, each column of C arbitrarily scaled. and B estimated up to multiplication by a block-wise permuted blockdiagonal matrix and C up to a permuted diagonal matrix.

8 BCD - L r,l r, : lgorithms Usual approach: estimate, B and C by minimization of Φ r r B r c r 2 F outer product he model is fitted for a given choice of the parameters {L r, } Exploit algebraic structure of matrix unfoldings k i j... 8

9 9 Z, Z 2 and Z 3 are built from 2 matrices only and have a block-wise hatri- ao product structure.,,, C B Z C Z B B Z C F F F,,, C B Z C Z B B Z C Φ Φ Φ BCD - L r,l r, : LS lgorithm [ ] [ ] [ ] 3 ˆ, ˆ ˆ 2 ˆ, ˆ ˆ ˆ, ˆ ˆ, ˆ, ˆ > Φ Φ k k while k k k k k k k k k k k k B C Z C Z B B Z C B e.g. nitialisation: ε ε

10 LS algorithm: problem of swamps Observation: Long swamp LS is fast in many problems, but sometimes, a long swamp is encountered before convergence iterations! Long Swamps typically occur when: he loading matrices of the decomposition i.e. the objective matrices are ill-conditioned he updated matrices become ill-conditioned impact of initialization One of the tensor-components in + + has a much higher norm than the - others e.g. «near-far» effect in telecommunications 0

11 mprovement of LS: Line Search Purpose: reduce the length of swamps Principle: for each iteration, interpolate, B and C from their estimates of 2 previous iterations and use the interpolated matrices in input of LS.Line Search: new k 2 C C + ρ C B new new B k 2 k 2 + ρ B + ρ 2.hen LS update Cˆ Bˆ ˆ k k k k k + k k k C B k 2 k 2 k 2 [ ˆ ] new ˆ new Z B, [ ˆ, ˆ ] new k Z2 C [ ˆ, ˆ ] k k Z C B 3 Search directions Choice of ρ ρ crucial annihilates LS step i.e. we get standard LS 2 3

12 mprovement of LS: Line Search [Harshman, 970] «LSH» Choose ρ. 25 [Bro, 997] «LSB» / 3 Choose ρ k and validate LS step if decrease in Fit [ajih, Comon, 2005] «Enhanced Line Search ELS» For EL tensors Φ new, S new, H new Optimal ρ is the root that minimizes Φ Φ ρ 6 new, S new th, H order new polynomial. [Nion, De Lathauwer, 2006] «Enhanced Line Search with Complex Step ELSCS» For complex tensors,look for optimal ρ m. e We have Φ new, S new, H new Φ m, θ lternate update of m andθ : Φ m, θ Update m : for θ fixed, 5 m Φ m, θ Update θ : for m fixed, 6 θ th th iθ order polynomialin m order polynomialin t θ tan 2 2

13 mprovement of LS: Line Search «easy» problem «difficult» problem 2000 iterations iterations ELS Large reduction of the number of iterations at a very low additional complexity w.r.t. standard LS 3

14 mprovement 2 of LS: Dimensionality reduction L N H M C B C + + B SEP 3: SEP : HOSVD of Come back to original space + a few refinement iterations in original space SEP 2: BCD of the small core tensor H compressed space Compression Large reduction of the cost per iteration since the model is 4 fitted in compressed space.

15 mprovement 3 of LS: Good initialization Comparison LS and LS+ELS, with three random initializations nstead of using random initializations, could we use the observed tensor itself? ES For the BCD-L,L,, if and B are full column rank so and have to be long enough, there is an easy way to find a good intialization, in same spirit as Direct rilinear Decomposition DLD used to initialize PFC not detailed in this talk. 5

16 Other algorithms Existing algorithms for PFC can be adapted to Block-Component- Decompositions. Examples: Levenberg-Marquardt algorithm Gauss-Newton type method, Simultaneous Diagonalization SD algorithms let s say a few words on this technique. SD for PFC De Lathauwer, 2006 nitial condition to reformulate PFC in terms of SD: min, PFC decomposition can be computed by solving a SD problem: M WD W, n,...,, D is diagonal n n n dvantage: Low complexity only matrices of size x to diagonalize + direct use of existing fast algorithms designed for SD or for PFC SD reformulation yields a uniqueness bound generically more relaxed than ruskal bound et

17 BCD - L,L, : computation via Simultaneous Diag. Nion & De Lathauwer, 2007 esults established for BCD-L,L,, i.e., same L for the terms nitial condition to reformulate BCD-L,L, in terms of SD: min, hen the decomposition can be computed by solving a SD problem: M WD W, n,...,, D is diagonal n n n dvantage: Low complexity only matrices of size x to diagonalize + direct use of existing fast algorithms designed for SD SD reformulation yields a new, more relaxed uniqueness bound next slide n case of exact decomposition no noise, the solution is found directly. 7

18 BCD - L,L, : Uniqueness Nion & De Lathauwer, 2007 Sufficient bound [De Lathauwer 2006] L and min,+ min,+ min, 2+ L L Sufficient bound 2 [Nion & De Lathauwer, 2007] : min, and C L +. C L+ k C n C L+ + L n! k! n k! 2 New Bound much more relaxed 8

19 Concluding remarks on algorithms Standard LS sometimes slow swamps LS+ELS drastically reduces swamp length at low additional complexity Levenberg-Marquardt convergence very fast, less sensitive to ill-conditioned data, but higher complexity and memory dimensions of acobian matrix Simultaneous diagonalization: a very attractive algorithm low complexity and good accuracy. mportant practical considerations: - Dimensionality reduction pre-processing step e.g. via ucker/hosvd - Find a good initialization if possible. lgorithms have to be adapted to include constraints specific to applications: - preservation of specific matrix-structures oeplitz, Van der Monde, etc - Constant Modulus, Finite lphabet, e.g. in elecoms pplications - non-negativity constraints e.g. Chemometrics applications 9

20 BCD - L r,l r, : estimation of and L r Problem: Given a tensor, how to estimate the number of terms and the rank L r of the matrices r and B r that yield a reasonable L r, L r, model? c L L B + + c L L B Criterion : Simple approach: examinate singular values of matrix unfoldings. x generically rank x generically rank x generically rank N N r L r if if if min, min, min, f noise level not too high and if conditions on dimensions satisfied, the number of significant singular values yields an estimate for and/or N. N N 20

21 COCOND Core Consistency Diagnostic Core idea: PFC can be seen as a particular case of ucker model, where the core tensor is diagonal. C H B H is diagonal if ijk, h ijk, else, h ijk 0 Method [Bro et al.] Choose a set of plausible values for. For a given test i.e., for a given, fit a PFC model and compute the Least Squares estimate of the core tensor H, and measure the diagonality of the core tensor: Examinate the core consistency measurements to select C 00 H Hˆ 2 2 F

22 Block-L r,l r, COCOND Core idea: BCD-L r,l r, can be seen as a particular case of ucker model, where the core tensor is «block-diagonal». L c L B + + L c L B L L C L L... N B... B N H N r L r 22

23 Block-L r,l r, COCOND Criterion 2: So we can proceed in a way similar to COCOND for PFC Choose a set of plausible values for and L r, r,,. For a given test i.e., for given and L r s, fit a BCD-L r,l r, model and compute the Least Squares estimate of the core tensor H, and measure the block - diagonality of the core tensor: C CO 00 H Hˆ Examinate the multiple core consistency measurements to select the most plausible parameters Criterion 3: Similarly to PFC, better to couple Block-COCOND to other criteria, e.g., examination of the relative Fit to the L r, L r, model: L 2 F C Fit ˆ 00 2 F 2 F 23

24 Block-L r,l r, COCOND Example : 2, 2, 50, L2, 3 LL L 2 L 3 Complex data random, and SN0 db est: try {,2,3,4,5,6} and L try {,2,3,4} Note: For each,l pair, the decomposition is computed via LS+ELS algorithm and 5 different starting points. L try C Fit C CO try L try < 0 < < 0 < < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 try L2 and 3 corresponds to the intersection of the acceptable values of Fit and the ones for Core Consistency. 24

25 Block-L r,l r, COCOND Example 2: 2, 2, 50, L3, 3 LL L 2 L 3 Complex data random, and SN0 db est: try {,2,3,4,5,6} and L try {,2,3,4,5} L try L try C Fit try C CO < 0 < < 0 < < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0 < 0,L3,2 and,l3,3 could be chosen. Find with other criteria to help in the final decision 25

26 Block-L r,l r, COCOND dea: fit a rank-n PFC model N is the number of rank- terms and compute correlation of estimated c vectors Criterion 4: use the BCD-L,L, structure c L L B + + c L L B c c c c b + + b L + + b + + b L a a L a a L Can be seen as PLND Parallel profiles with Linear Dependencies [Bro, Harshman, Sidiropoulos] epetition of the vectors c r in each term.

27 Block-L r,l r, COCOND From example 2, ambiguous choice:,l3,2 or,l3,3? Fit a rank-6 and a rank-9 PFC model and check if the pairing of the estimated c vectors clearly appears Clustering in 3 groups of 2 vectors «not good» Clustering in 3 groups of 3 vectors «good» 27

28 pplications n application of the BCD-L r,l r,: Blind Source Separation in telecommunications CDM «Code Division Multiple ccess» signals Used in 3rd generation wireless standard UMS llows users to communicate simultaneously in the same bandwidth User wants to transmit s [ - -]. CDM code allocated to user : c [ - -]. User transmits [+ c - c - c ] User 2 transmits his symbols spread by his own CDM code c 2, orthogonal to c, etc Signals received by an antenna array. Signal received by each antenna mixture of signals transmitted by users, affected by wireless channel effects. Purpose: Separate these signals, from exploitation of the received signals 28 only.

29 n application of the BCD-L r,l r,: Blind Source Separation in telecommunications receive antennas Chip rate sampling times faster than symbol rate Observation during symbol periods Code Diversity Spatial Diversity emporal Diversity Build the 3rd order observed tensor Decompose to blindly estimate the transmitted symbols. Which decomposition to use? the one that best reflects the algebraic structure of the data 29

30 n application of the BCD-L r,l r,: Blind Source Separation in telecommunications Case : single path propagation no inter-symbol-interference Use PFC [Sidiropoulos et al.] a s + + a s Spatial Diversity emporal Diversity c c User User length of the CDM codes number of symbols number of antennas at the receiver Code Diversity «Blind» receiver: uniqueness of PFC does not require prior knowledge of the CDM codes, neither of pilot sequences to blindly estimate the symbols of all users. 30

31 n application of the BCD-L r,l r,: Case 2: Multi-path propagation with inter-symbol-interference but far-field reflections only. Use PLND [Sidiropoulos & Dimic] or BCD-L,L, [De Lathauwer & de Baynast] Blind Source Separation in telecommunications L r interfering symbols r L r H r L r a r S r oeplitz structure convolution H r Channel matrix channel impulse response convolved with CDM code S r Symbol matrix, holds the symbols of interest for user r a r esponse of the antennas to the angle of arrival steering vector 3

32 n application of the BCD-L r,l r,: Blind Source Separation in telecommunications 2, 00, L2 for all users 4 antennas and 5 users 6 antennas and 3 users 32

33 Conclusion Block Component Decomposition in rank-l r,l r, terms is a generalization of PFC. Other BCD, even more general, have also been proposed [De Lathauwer] lgorithms: LS coupled with Enhanced Line Search good compromise between complexity / convergence speed. lgorithms based on Simultaneous Diagonalization SD also merits consideration lower complexity than LS and better accuracy on-going research Uniqueness: SD-based reformulation also yields relaxed uniqueness bound on-going research Selection of the number of terms and the rank L r is important in practice e.g. in telecoms number of users, L r user-dependent channel length 33

Coupled Matrix/Tensor Decompositions:

Coupled Matrix/Tensor Decompositions: An Introduction Laurent Sorber Mikael Sørensen Marc Van Barel Lieven De Lathauwer KU Leuven Belgium Lieven.DeLathauwer@kuleuven-kulak.be 1 Canonical Polyadic Decomposition