A new truncation strategy for the higher-order singular value decomposition

Transcription

1 A new truncation strategy for the higher-order singular value decomposition Nick Vannieuwenhoven K.U.Leuven, Belgium Workshop on Matrix Equations and Tensor Techniques RWTH Aachen, Germany November 21, 2011 Joint work with Raf Vandebril and Karl Meerbergen

2 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

3 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

4 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

5 Context I Everything is a tensor d = 0 Scalar a = d = 1 Vector a = d = 2 Matrix A = d 3 Tensor A =

6 Context II Everything is a tensor Multidimensional data appears in many applications: image and signal processing, pattern recognition, data mining and machine learning, chemometrics, biomedicine, psychometrics,... Two major problems associated with this data: 1 Storage cost is very high, and 2 analysis and interpretation of patterns and phenomena in data.

7 Context III Tensor decompositions for compression Stable tensor decompositions suitable for compression in the Low-dimensional case: Tucker (Tucker 1966), Higher-Order SVD (De Lathauwer et al 2000), Cross-approximation (Oseledets et al. 2008), Sequentially Truncated HOSVD (Vannieuwenhoven et al. 2011). High-dimensional case: Hierarchical Tucker (Hackbusch and Kühn 2009, Grasedyck 2010), Tensor-Train (Oseledets 2011).

8 Context IV Compression Low-rank representation of matrices by SVD: A USV T := (U, V ) S A U S V T Low-rank representation of tensors by Tucker: A (Û 1, Û 2, Û 3 ) Ŝ Û 3 A Û 1 Ŝ Û 2

9 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

10 Notation I Mode k vector space A tensor A of order d is an object in the tensor product of d vector spaces: A R n 1 R n 2 R n d R n 1 n 2 n d A 3 rd order tensor has 3 associated vector spaces: Mode 1 vectors (R n 1 ) Mode 2 vectors (R n 2 ) Mode 3 vectors (R n 3 )

11 Notation II Unfolding R n 1 n 2 n 3 A = Mode 2 unfolding A (2) = R n 2 n 1 n 3

12 Notation III Frobenius norm: A 2 := i 1,i 2,i 3 A 2 i 1,i 2,i 3. Multilinear multiplication: [(I, M 2, I ) A] (2) := M 2 A (2). (M 1, M 2, M 3 ) A := (M 1, I, I ) (I, M 2, I ) (I, I, M 3 ) A. Projection of mode 2 vectors on span of U 2 (orthogonal columns): π 2 A := (I, U 2 U2 T, I ) A Projection of mode 2 vectors on complement of U 2 : π2 A := A π 2 A

13 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

14 Orthogonal Tucker approximation problem Best rank-(r 1, r 2, r 3 ) approximation problem: min A B F = rank(b) (r 1,r 2,r 3 ) min A (U 1 U1 T, U 2 U2 T,..., U d Ud T ) A F. U i O(n i,r i ) with O(n i, r i ) the group of n i r i matrices with orthonormal columns. Optimum is found by orthogonal projection onto a new, optimal tensor basis, but no closed solution known.

15 Orthogonal Tucker approximation I Definition A (Û 1, Û 2, Û 3 ) Ŝ Û 3 A Û 1 Ŝ Û 2 Rank (r 1, r 2, r 3 ) orthogonal Tucker approximation to A Columns of Û 1 R n 1 r 1 Û 2 R n 2 r 2 Û 3 R n 3 r 3 can be extended to a basis of R n 1 R n 2 R n 3

16 Orthogonal Tucker approximation II Error [VVM2011] If A Â := π 1 π 2 π 3 A = (U 1 U1 T, U 2 U2 T, U 3 U3 T ) A. Then an error expression is = A π 1 π 2 π 3 A 2 = + + π2 A 2 + π1 π 2A 2 + π3 π 1π 2 A 2 with upper bound + + A π 1 π 2 π 3 A 2 π2 A 2 + π1 A 2 + π3 A 2

17 Orthogonal Tucker approximation II Error [VVM2011] If A Â := π 1 π 2 π 3 A = (U 1 U1 T, U 2 U2 T, U 3 U3 T ) A. Then an error expression is = A π 1 π 2 π 3 A 2 = + + π2 A 2 + π1 π 2A 2 + π3 π 1π 2 A 2 with upper bound + + A π 1 π 2 π 3 A 2 π2 A 2 + π1 A 2 + π3 A 2 Dependence on the processing order of the modes.

18 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

19 Truncated Higher-Order SVD (T-HOSVD) [DDV2000] Recall upper bound? + + A π 1 π 2 π 3 A 2 π2 A 2 + π1 A 2 + π3 A 2 Minimize it! min A π 1 π 2 π 3 A 2 π 1,π 2,π 3 = min π 1,π 2,π 3 k=1 3 k=1 3 πk A 2, min π k π k A 2. Minimum given by r k first singular vectors in every mode k!

20 Algorithm Rank (r 1, r 2, r 3 ) T-HOSVD: for every mode k do Compute rank r k truncated SVD: A (k) = [ Ū k Ū k end for Project: S = (Ū T 1, ŪT 2, ŪT 3 ) A ] [ S k S k ] [ V k V k ] T

21 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

22 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

23 Sequentially truncated HOSVD (ST-HOSVD) [VVM2011] Recall error expression? = A π 1 π 2 π 3 A 2 = + + π2 A 2 + π1 π 2A 2 + π3 π 1π 2 A 2 (Try to) minimize it! min A π 1 π 2 π 3 A 2 π 1,π 2,π 3 = min π 1,π 2,π 3 [ = min π 1 π1 A 2 + π2 π 1 A 2 + π3 π 1 π 2 A 2] [ π2 π 1 A 2 + min π π 3 π 1 π 2 A 2 3 [ π 1 A 2 + min π 2 ]]

24 Sequentially truncated HOSVD (ST-HOSVD) [VVM2011] The sequentially truncated HOSVD computes solution to π 1 = arg min π 1 π 1 A 2 π 2 = arg min π 2 π 2 π 1A 2 π 3 = arg min π 3 π 3 π 1π 2A 2 π k given by r k first singular vectors of [π 1 π k 1 A] (k)!

25 Algorithm Rank (r 1, r 2, r 3 ) ST-HOSVD: Ŝ = A for every mode k do Compute rank r k trunctated SVD: Ŝ (k) = [ Û k Project: Ŝ (k) = Ûk T end for Û k Ŝ(k) ] [ Ŝ k Ŝ k ] [ ˆV k ˆV k ] T Ŝ = A Ŝ (1) = Û T 1 Ŝ(1) Ŝ (2) = Û T 2 Ŝ(2) Ŝ (3) = Û T 3 Ŝ(3)

26 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

27 Operation count Theorem Let A R n n n be truncated to rank (r, r,..., r) by the ST-HOSVD and T-HOSVD. Assume an O(m 2 n) algorithm to compute the SVD of an m n matrix, m n. Then, the ST-HOSVD requires ( d ) d O r k 1 n d k+2 + r k n d k k=1 k=1 operations, and T-HOSVD requires ( ) d O dn d+1 + r k n d k+1 k=1 operations to compute the approximation.

28 Operation count Speedup r d = 3 d = 4 d = 5 Speedup of ST-HOSVD over T-HOSVD for an order-d tensor of size which is truncated to rank (r, r,..., r). Speedups greater than d possible with non-cubic tensors.

29 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

30 Approximation error I Hypothesis Hypothesis Let A be an ST-HOSVD approximation, and A the T-HOSVD approximation of corresponding rank. Then, A A F? A A F. Not valid in general.

31 Approximation error II Counterexample Rank-(1, 1, 1, 1) approximation of [ ] [ ] A :,:,1,1 =, A :,:,2,1 =, [ ] [ ] A :,:,1,2 =, A :,:,2,2 = T-HOSVD: A = ST-HOSVD: A = , , The approximation errors:, , , , [ ] [ ] A A 2 F = and A A 2 F = ,..

32 Approximation error III Sufficient condition Theorem Let A R n 1 n 2 n 3. Let A be the rank-(1, r, r) ST-HOSVD of A and let A be the T-HOSVD of A of the same rank. Then, A A F A A F. ST-HOSVD yields better rank-1 approximation for order-3 tensors.

33 Approximation error IV Bounds ST-HOSVD error bound: = A π 1 π 2 π 3 A 2 = + + π2 A 2 + π1 π 2A 2 + π3 π 1π 2 A 2 T-HOSVD error bound: + + A π 1 π 2 π 3 A 2 π2 A 2 + π1 A 2 + π3 A 2

34 Approximation error IV Bounds ST-HOSVD error bound: = A π 1 π 2 π 3 A 2 = + + π2 A 2 + π1 π 2A 2 + π3 π 1π 2 A 2 T-HOSVD error bound: + + A π 1 π 2 π 3 A 2 π2 A 2 + π1 A 2 + π3 A 2 Both bounded by: A A F d A A opt F.

35 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

36 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

37 Compression of images I Compression of the Olivetti Research Laboratory faces database [SH1994, VT2003]. Tensor of size (Texel Subject Expression). Unstructured, 4.1 million nonzeros. HOOI, T-HOSVD and ST-HOSVD approximations of 6560 different ranks.

38 Compression of images II Relative difference between approximation error of T-HOSVD and HOOI: (err HOSVD err HOOI )/err HOOI. Subject mode rank Expression mode rank = Texel mode rank 7% 6% 5% 4% 3% 2% 1% 0% Subject mode rank Expression mode rank = Texel mode rank 7% 6% 5% 4% 3% 2% 1% 0% Average relative error: 2.115% Maximum relative error: 6.340%

39 Compression of images II Relative difference between approximation error of ST-HOSVD and HOOI: (err HOSVD err HOOI )/err HOOI. Subject mode rank Expression mode rank = Texel mode rank 7% 6% 5% 4% 3% 2% 1% 0% Subject mode rank Expression mode rank = Texel mode rank 7% 6% 5% 4% 3% 2% 1% 0% Average relative error: 0.099% Maximum relative error: 1.012%

40 Dinner tonight La Finestra at 19h30!

41 Compression of images III Total compression time (min) of HOOI, ST-HOSVD and T-HOSVD:

42 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

43 Handwritten digit classification I Classification of handwritten digits by T-HOSVD [SE2007]. Tensor of size (Texel Example Digit). Unstructured, 42.6 million non-zeros. T-HOSVD and ST-HOSVD truncated to relative error of 10%. T-HOSVD ST-HOSVD Rel. model error 9.90% 9.68% Model rank (94, 511, 10) (94, 511, 10)

44 Handwritten digit classification II T-HOSVD ST-HOSVD Classification error 4.94% 4.94% Factorization time 49m 26.0s 1m 8.7s 43x speedup!

45 Handwritten digit classification III Why? Recall truncation rank? (94, 511, 10). T-HOSVD requires: SVD of matrix, SVD of matrix, and SVD of matrix. ST-HOSVD (only) requires: 1 SVD of matrix, 2 SVD of matrix, and 3 SVD of matrix.

46 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

47 Compression of simulation results I Compression of numerical solution of the heat equation on a square domain computed by explicit Euler. Inspired by [LVV2010]. Tensor of size (x y t). Partially symmetric, million non-zeros. T-HOSVD and ST-HOSVD truncated to absolute error of 10 4 (discretization accurary).

48 Compression of simulation results II T-HOSVD ST-HOSVD Abs. error Rank (22, 22, 20) (22, 21, 19) T-HOSVD ST-HOSVD Storage (nb. of values) Factorization time 2h 46m 1m 14.7s 133x speedup!

49 Overview 1 Introduction Context Notation 2 Orthogonal Tucker approximation 3 Truncated HOSVD 4 Sequentially truncated HOSVD Definition Operation count Approximation error 5 Numerical examples Compression of images Handwritten digit classification Compression of simulation results 6 Conclusions

50 Conclusions Early projection, as in ST-HOSVD, can greatly improve the performance of T-HOSVD.

51 Thank you for your attention.

52 References DDV2000 L. De Lathauwer, B. De Moor, and J. Vandewalle, A multilinear singular value decomposition, SIMAX, 21 (2000) LVV2010 L.S. Lorente, J.M. Vega, and A. Velazquez, Compression of aerodynamic databases using higher-order singular value decomposition, Aerosp. Sc. and Tech., 14 (2010) SÉ2007 B. Savas and L. Eldén, Handwritten digit classification using higher-order singular value decomposition, Patt. Recog., 40 (2007) VVM2011 N. Vannieuwenhoven, R. Vandebril, and K. Meerbergen, A new truncation strategy for the higher-order singular value decomposition, 2011, Submitted. Also: On the truncated multilinear singular value decomposition, Tech. Rep. TW589, K.U.Leuven, March 2011

53 References SH1994 F.S. Samaria and A.C. Harter, Parameterization of a stochastic model of human face identification, Proc. Second IEEE W. Appl. Comp. Vision, VT2003 M.A.O. Vasilescu and D. Terzopoulos, Multilinear subspace analysis of image ensembles, Comp. Vision Patt. Recog. IEEE Conf. 2003