Dealing with curse and blessing of dimensionality through tensor decompositions

Transcription

1 Dealing with curse and blessing of dimensionality through tensor decompositions Lieven De Lathauwer Joint work with Nico Vervliet, Martijn Boussé and Otto Debals June 26, 2017

2 2 Overview Curse of dimensionality Algorithms Variants and applications

3 3 Curse of dimensionality The curse Tensor decompositions as a remedy Immunization by low rank

4 The curse 4 Tensors 1 I 1 I 2 I N

5 The curse 5 Tensors Multidimensional array of numerical values General Nth order tensor T C I 1 I 2 I N Number of elements: O ( I N)

6 The curse 5 Tensors Multidimensional array of numerical values General Nth order tensor T C I 1 I 2 I N Number of elements: O ( I N) Curse of Dimensionality The problems arising from the exponential increase in memory and computational requirements Example: # entries in a Nth order tensor of size exceeds # atoms in observable universe for N > 41

7 The curse 6 Alleviating or breaking the curse of dimensionality Use decompositions Canonical Polyadic Decomposition Low Multilinear Rank Approximation (Tucker, MLSVD) Tensors Trains Hierarchical Tucker Tensor Networks

8 The curse 6 Alleviating or breaking the curse of dimensionality Use decompositions Canonical Polyadic Decomposition Low Multilinear Rank Approximation (Tucker, MLSVD) Tensors Trains Hierarchical Tucker Tensor Networks Scientific computing vs signal processing/data analysis

9 The curse 6 Alleviating or breaking the curse of dimensionality Use decompositions Canonical Polyadic Decomposition Low Multilinear Rank Approximation (Tucker, MLSVD) Tensors Trains Hierarchical Tucker Tensor Networks Scientific computing vs signal processing/data analysis Use incomplete tensors Because we do not have the full tensor Because we do not want the full tensor

10 7 Curse of dimensionality The curse Tensor decompositions as a remedy Immunization by low rank

11 Decompositions as a remedy Low Multilinear Rank Approximation Multilinear transform of a core tensor A (3) T = G A (2) A (1) Mathematically, for a general Nth order tensor T C I I T = G 1 A (1) 2 A (2) N A (N) G; A (1), A (2),..., A (N) Number of variables: O ( NIR + R N) Curse not broken, but can be computed via QR/SVD Truncation error bound: 2 T ˆT MLSVD, trunc N min T ˆT F rank ( ˆT ) (R 1,R 2,...,R N ) 2 F 8

12 Decompositions as a remedy 9 Canonical polyadic decomposition Sum of rank-1 terms c 1 c R T = b b R a 1 a R Mathematically, for a general Nth order tensor T C I I T = R r=1 a (1) r a (2) r a (N) r = A (1), A (2),..., A (N) Number of variables: O (NIR) Curse broken, but possibly ill-conditioned/ill-posed problem

13 Decompositions as a remedy 10 Tensor Trains or Matrix Product States Write tensor as a train of lower-order tensors [Oseledets 2011] A 1 A 2 A 3 A 4 Mathematically, for a general Nth order tensor T C I I t i1 i 2 i N = a (1) i 1 r 1 a (2) r 1 i 2 r 2 a (N) r N 1 i N r 1,r 2,...,r N 1 Number of variables: O ( 2IR + (N 2)IR 2) Curse broken and can be computed via QR/SVD Truncation error bound T ˆT TT, trunc 2 F (N 1) min T ˆT 2 F rank TT ( ˆT ) (R 1,R 2,...,R N 1 )

14 11 Curse of dimensionality The curse Tensor decompositions as a remedy Immunization by low rank

15 Immunization by low rank 12 Key assumption: low rank Matrix: decaying singular value spectrum Power law Exponential polynomial structure (see further) Rank-1 terms: T = R r=1 u (1) r u (2) r u (N) r T [1,2,...;n+1,n+2,...] = (U (1) U (2) ) (U (n+1) U (n+2) ) T all matrix representations have rank R

16 13 Overview Curse of dimensionality Algorithms Variants and applications

17 14 Algorithms for large-scale tensors Missing entries/partially sampled tensor and CPD Missing entries / partially sampled tensors and CPD Randomized block sampling for CPD

18 15 How to handle large tensors? Use incomplete tensors Missing entries/partially sampled tensor and CPD CPWOPT [Acar et al. 2011] CPDI NLS [Vervliet et al. 2014; Vervliet et al. 2016a] Exploit sparsity GigaTensor [Kang et al. 2012] ParCube [Papalexakis et al. 2012] Compress the tensor PARACOMP algorithm [Sidiropoulos et al. 2014] Tensor Trains [Oseledets and Tyrtyshnikov 2010] Decompose subtensors and combine results ParCube [Papalexakis et al. 2012] Grid PARAFAC [Phan and Cichocki 2011] Parallel ADMoM [Liavas and Sidiropoulos 2015] Most of the above

19 16 Optimization for CPD Optimization problem: 1 min W A (1),A (2),...,A (N) 2 Algorithms Missing entries/partially sampled tensor and CPD ( T A (1), A (2),..., A (N) ) 2 F CPWOPT [Acar et al. 2011] Nonlinear Conjugate Gradients INDAFAC [Tomasi and Bro 2005] Gauss Newton CPD/SDF [Sorber et al. 2015] Quasi-Newton and (approximate) inexact Gauss Newton Tensorlab: cpd_nls, sdf_nls CPD(L)I [Vervliet et al. 2016a; Vervliet et al. 2016d] Inexact Gauss Newton with possible linear constraints Tensorlab: cpd_nls with UseCPDI option, cpdli_nls Samples investigated: N samples

20 Randomized block sampling for CPD 17 Algorithms for large-scale tensors Missing entries / partially sampled tensors and CPD Randomized block sampling for CPD

21 Randomized block sampling for CPD 18 Randomized block sampling CPD: idea + + [Vervliet and De Lathauwer 2016]

22 Randomized block sampling for CPD 18 Randomized block sampling CPD: idea + + [Vervliet and De Lathauwer 2016]

23 Randomized block sampling for CPD 18 Randomized block sampling CPD: idea + + Take sample [Vervliet and De Lathauwer 2016]

24 Randomized block sampling for CPD 18 Randomized block sampling CPD: idea + + Take sample Initialization Compute step + + [Vervliet and De Lathauwer 2016]

25 Randomized block sampling for CPD 18 Randomized block sampling CPD: idea + + Take sample Initialization Update Compute step + + [Vervliet and De Lathauwer 2016]

26 Randomized block sampling for CPD 19 Detection of hazardous gasses using e-noses Classify 900 experiments containing 72 time series with samples each. [Vervliet and De Lathauwer 2016]

27 Randomized block sampling for CPD 20 Classify hazardous gasses Does the sample contain CO, acetaldehyde or ammonia? Sensor Experiment Time Strategy: classify using coefficients of spatiotemporal patterns R = 5 Unknown

28 Randomized block sampling for CPD 21 Results Resulting factor matrices time sensor experiment

29 Randomized block sampling for CPD 21 Results Resulting factor matrices time sensor experiment Performance after clustering Iterations Time (s) Error (%) No restriction Restriction

30 22 Overview Curse of dimensionality Algorithms Variants and applications

31 23 Variants and applications Compression as preprocessing in the computation of unconstrained and constrained decompositions Thermodynamic data and curse of dimensionality Quantization and blessing of dimensionality

32 Compression as preprocessing Exploiting low multilinear rank for tensor decompositions Strategy without constraints: 1 Compress tensor, e.g, using (randomized) MLSVD [Vervliet et al. 2016c] 2 Compute CPD of core tensor 3 Expand CPD using factor matrices of compression 4 Refine result if necessary Orthogonal factor matrices preserve length and distance in compression 24

33 Compression as preprocessing Exploiting low multilinear rank for tensor decompositions Strategy with constraints 1 Compute LMLRA 2 Decompose while exploiting structure Core operations like norms, inner products and mtkrprod exploit structure of the tensor - [Vervliet et al. 2016c]: 2-2 Many combinations of structures and decompositions possible 25

34 Compression as preprocessing 26 Exploiting efficient representations in tensor decompositions Tensorlab can compute tensors given in an efficient format CPD, LMLRA, Tensor Train, Hankel, Löwner,... using possibly coupled and/or symmetric decompositions CPD, LL1, LMLRA, BTD with possible constraints nonnegativity, Hankel, Vandermonde, polynomial, orthogonal,... Example: compute a nonnegative rank-5 CPD of a tensor after randomized MLSVD compression using mlsvd_rsi

35 Compression as preprocessing Nonnegative CPD using MLSVD compression Compute nonnegative rank-10 CPD of I I I tensor with SNR 20 db: Time (s) Projected GN Parametric GN (SDF) Full 7.6 Compressed I I [Vervliet et al. 2016b] 27

36 Compression as preprocessing 28 Nonnegative CPD using TT compression Compute nonnegative rank-5 CPD of Nth-order tensor with SNR 20 db: Time (s) Full 10.8 Compressed Order N [Vervliet et al. 2016b]

37 29 Variants and applications Compression as preprocessing in the computation of unconstrained and constrained decompositions Thermodynamic data and curse of dimensionality Quantization and blessing of dimensionality

38 Thermodynamic data 30 Modeling multiway thermodynamic data Modes: fraction atom/molecule n in a multi-component material Value: Gibbs free energy, chemical potential, melting temperature (e.g., computed using thermodynamic software)

39 Thermodynamic data Modeling multiway thermodynamic data Modes: fraction atom/molecule n in a multi-component material Value: Gibbs free energy, chemical potential, melting temperature (e.g., computed using thermodynamic software) Second-order example Alloy with c 1 % iron, c 2 % carbon and 100 c 1 c 2 % nickel Discretize c 1 and c 2 in 100 steps Grid of size [Vervliet et al. 2014] 30

40 Thermodynamic data 31 Multiway dataset # constituent materials: 10 (thus N = 9) Size: elements # Samples: of which are validation samples Model: T = R r=1 a (1) r a (2) r a (N) r

41 Thermodynamic data 31 Multiway dataset # constituent materials: 10 (thus N = 9) Size: elements # Samples: of which are validation samples Model: Algorithm T = R r=1 a (1) r a (2) r a (N) r Tensorlab 3.0 with cpd_nls and UseCPDI option [Vervliet et al. 2014; Vervliet et al. 2016d] Initialization: optimally scaled best-out-of-five strategy

42 Thermodynamic data 32 Visualization 1,600 1,400 Tmelt 1,200 1, c c 1 5

43 Thermodynamic data 33 Fitting the model Error E Time (s) R Figure: Errors on training E tr ( ) and validation E val ( ) set and the 99% quantile error E quant ( ) for different CPDs. The computation time for each model is indicated by ( ) on the right y-axis.

44 Thermodynamic data 33 Fitting the model Error E Time (s) R Figure: Errors on training E tr ( ) and validation E val ( ) set and the 99% quantile error E quant ( ) for different CPDs. The computation time for each model is indicated by ( ) on the right y-axis.

45 Thermodynamic data 34 From a discrete model a9r

46 Thermodynamic data From a discrete model a9r to a continuous (e.g., polynomial) model t i1 i 9 f (c 1,..., c N ) = R r=1 n=1 a (n) r (c n ), Advantage: allows interpolation, derivation and integration, parameter reduction,... 34

47 Thermodynamic data 35 Recap From a ninth order tensor with elements...

48 35 Thermodynamic data Recap From a ninth order tensor with elements... we took samples...

49 35 Thermodynamic data Recap From a ninth order tensor with elements... we took samples... to get a rank-1 model with parameters...

50 35 Thermodynamic data Recap From a ninth order tensor with elements... we took samples... to get a rank-1 model with parameters... to get a continuous model with O (100) parameters...

51 35 Thermodynamic data Recap From a ninth order tensor with elements... we took samples... to get a rank-1 model with parameters... to get a continuous model with O (100) parameters... in 3 min

52 36 Variants and applications Compression as preprocessing in the computation of unconstrained and constrained decompositions Thermodynamic data and curse of dimensionality Quantization and blessing of dimensionality

53 Quantization and blessing of dimensionality 37 Low-rank matrices can be used as compact models for large-scale vectors matricize M vectorize I J P M = IJ P(I + J)

54 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials

55 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials 1 z f = z 2 z 3 z 4 z 5

56 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials 1 z f = z 2 z 3 z 4 z 5 1 z 3 R = z z 4 z 2 z 5

57 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials 1 z f = z 2 z 3 z 4 z 5 1 z 3 1 R = z z 4 = z ( 1 z 3) z 2 z 5 z 2

58 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials 1 z f = z 2 z 3 z 4 z 5 1 z 3 1 R = z z 4 = z ( 1 z 3) z 2 z 5 z 2 R can be interpreted as a compact form of the Hankel matrix H 1 z z 2 z 3 H = z z 2 z 3 z 4 z 2 z 3 z 4 z 5

59 Quantization and blessing of dimensionality 38 The approach holds exactly for (exponential) polynomials 1 z f = z 2 z 3 z 4 z 5 1 z 3 1 R = z z 4 = z ( 1 z 3) z 2 z 5 z 2 R can be interpreted as a compact form of the Hankel matrix H 1 z z 2 z 3 1 H = z z 2 z 3 z 4 = z ( 1 z z 2 z 3) z 2 z 3 z 4 z 5 z 2

60 Quantization and blessing of dimensionality 39 f (t) r(h) f (t) r(h) az t 1 a sin(bt) a cos(bt) 2 az t sin(bt) 2 p(t) = Q a q t q Q + 1 q=0 p(t)z t Q + 1 R a r zr t r=1 R a r sin(b r t) r=1 R a r zr t sin(b r t) r=1 R p r (t) r=1 R p r (t)zr t r=1 R 2R 2R R Q r + R r=1 R Q r + R r=1 [Boussé et al. 2017]

61 Quantization and blessing of dimensionality Periodic signals can be reshaped into low-rank matrices one period f = 40

62 Quantization and blessing of dimensionality 40 Periodic signals can be reshaped into low-rank matrices f = one period [ ] R = r(r) = 1 Regardless of the type of signal, e.g., discontinuities are allowed.

63 Quantization and blessing of dimensionality Periodic signals can be reshaped into low-rank matrices one period f = r(r) = 1 R = Regardless of the type of signal, e.g., discontinuities are allowed. 40

64 Quantization and blessing of dimensionality 40 Periodic signals can be reshaped into low-rank matrices f = one period [ ] R = r(r) = 2 half a period Regardless of the type of signal, e.g., discontinuities are allowed.

65 Quantization and blessing of dimensionality 41 The approach also works well for more general compressible functions ɛ = f vec( R) 2 F Underlying function f (t) Low-rank approximation of R = reshape(f)

66 Quantization and blessing of dimensionality 41 The approach also works well for more general compressible functions ɛ = f vec( R) 2 F Underlying function f (t) Low-rank approximation of R = reshape(f) Functions with rapidly converging Taylor series admit an approximate low-rank model f (t) p(t) ɛ Taylor Taylor polynomial

67 Quantization and blessing of dimensionality 41 The approach also works well for more general compressible functions ɛ = f vec( R) 2 F Underlying function f (t) Low-rank approximation of R = reshape(f) Functions with rapidly converging Taylor series admit an approximate low-rank model f (t) p(t) ɛ Taylor Taylor polynomial [Grasedyck et al. 2013; Boussé et al. 2017]

68 Quantization and blessing of dimensionality 42 The singular values of R often decay fast, hence, f often admits a good representation for low rank values. Gaussian 1 original function rank-1 model Sigmoid Rational original function rank-2 model

69 43

70 44 Tensorlab 3.0 A MATLAB toolbox for tensor decompositions Variety of tensor decompositions CPD, LMLRA, MLSVD, BTD, LL1,... Support large scale and incomplete tensors Randomized block sampling, MLSVD RSI, Structured tensors Constrained, symmetric and coupled decompositions Structured data fusion framework Tensorization of data Segmentation, Hankelization, cumulants,... Cumulants, tensor visualization, estimating a tensor s rank or multilinear rank,...

71 45

72 46

73 Conclusion Tensor problems are often large-scale Alleviate/break curse of dimensionality using decompositions for analysis, compression,... by computations using randomization, incompleteness, efficient representations,... Dimensionality is also a blessing segmentation/quantization 47

74 Dealing with curse and blessing of dimensionality through tensor decompositions Lieven De Lathauwer Joint work with Nico Vervliet, Martijn Boussé and Otto Debals June 26, 2017

75 2 Survey papers Cichocki, A. et al. (2015). Tensor Decompositions for Signal Processing Applications: From two-way to multiway component analysis. In: IEEE Signal Processing Magazine 32.2, pp Sidiropoulos, N.D., L. De Lathauwer, X. Fu, K. Huang, E.E. Papalexakis, and C. Faloutsos (2017). Tensor Decomposition for Signal Processing and Machine Learning. In: IEEE Transactions on Signal Processing 65.13, pp

76 3 References I Acar, E., D.M. Dunlavy, T.G. Kolda, and M. Mørup (2011). Scalable tensor factorizations for incomplete data. In: Chemometrics and Intelligent Laboratory Systems 106.1, pp Boussé, M., O. Debals, and L. De Lathauwer (2017). A Tensor-Based Method for Large-Scale Blind Source Separation using Segmentation. In: IEEE Transactions on Signal Processing 65.2, pp Grasedyck, L., D. Kressner, and Tobler C. (2013). A literature survey of low-rank tensor approximation techniques. In: GAMM-Mitteilungen 36.1, pp

77 4 References II Kang, U., E. Papalexakis, A. Harpale, and C. Faloutsos (2012). GigaTensor: scaling tensor analysis up by 100 times-algorithms and discoveries. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp Liavas, A. and N. Sidiropoulos (2015). Parallel Algorithms for Constrained Tensor Factorization via the Alternating Direction Method of Multipliers. In: IEEE Transactions on Signal Processing PP.99, pp Oseledets, I.V. (2011). Tensor-Train Decomposition. In: SIAM J. Sci. Comput. 33.5, pp Oseledets, I.V. and E.E. Tyrtyshnikov (2010). TT-cross approximation for multidimensional arrays. In: Linear Algebra and its Applications 432.1, pp

78 5 References III Papalexakis, E., C. Faloutsos, and N. Sidiropoulos (2012). ParCube: Sparse Parallelizable Tensor Decompositions. English. In: Machine Learning and Knowledge Discovery in Databases. Ed. by PeterA. Flach, Tijl De Bie, and Nello Cristianini. Vol Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp Phan, A.-H. and A. Cichocki (2011). PARAFAC algorithms for large-scale problems. In: Neurocomputing 74.11, pp Sidiropoulos, N., E. Papalexakis, and C. Faloutsos (2014). Parallel randomly compressed cubes: A scalable distributed architecture for big tensor decomposition. In: IEEE Signal Processing Magazine 31.5, pp

79 6 References IV Sorber, L., M. Van Barel, and L. De Lathauwer (2015). Structured Data Fusion. In: IEEE Journal of Selected Topics in Signal Processing 9.4, pp Tomasi, G. and R. Bro (2005). PARAFAC and missing values. In: Chemometrics and Intelligent Laboratory Systems 75.2, pp Vervliet, N. and L. De Lathauwer (2016). A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors. In: IEEE Journal of Selected Topics in Signal Processing 10.2, pp Vervliet, N., O. Debals, and L. De Lathauwer (2016a). Canonical polyadic decomposition of incomplete tensors with linearly constrained factors. Technical Report , ESAT-STADIUS, KU Leuven, Belgium.

80 7 References V Vervliet, N., O. Debals, and L. De Lathauwer (2016b). Exploiting efficient data representations in tensor decompositions. Technical Report , ESAT-STADIUS, KU Leuven, Belgium. (2016c). Tensorlab 3.0 Numerical optimization strategies for large-scale constrained and coupled matrix/tensor factorization. Technical Report , ESAT-STADIUS, KU Leuven, Belgium. Vervliet, N., O. Debals, L. Sorber, and L. De Lathauwer (2014). Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis. In: IEEE Signal Processing Magazine 31.5, pp

81 8 References VI Vervliet, N., O. Debals, L. Sorber, M. Van Barel, and L. De Lathauwer (2016d). Tensorlab 3.0. Available online at