A randomized block sampling approach to the canonical polyadic decomposition of large-scale tensors

Transcription

1 A randomized block sampling approach to the canonical polyadic decomposition of large-scale tensors Nico Vervliet Joint work with Lieven De Lathauwer SIAM AN17, July 13, 2017

2 2 Classification of hazardous gasses using e-noses Classify 900 experiments containing 72 time series with samples each. Sensor Experiment Time

3 3 Overview Decomposing large-scale tensors Randomized block sampling Experimental results Chemo-sensing application

4 4 Canonical polyadic decomposition Sum of R rank-1 terms c 1 c R T = a 1 b a R b R

5 4 Canonical polyadic decomposition Sum of R rank-1 terms c 1 c R T = a 1 b a R b R Mathematically, for a general Nth order tensor T T = R r=1 a (1) r a (2) r a (N) r = A (1), A (2),..., A (N)

6 5 Computing a CPD Optimization problem: 1 min A (1),A (2),...,A (N) 2 T A (1), A (2),..., A (N) 2 F

7 5 Computing a CPD Optimization problem: 1 min A (1),A (2),...,A (N) 2 T A (1), A (2),..., A (N) 2 Algorithms Alternating least squares CPOPT [Acar et al. 2011a] (Damped) Gauss Newton [Phan et al. 2013] (Inexact) nonlinear least squares [Sorber et al. 2013] F

8 6 Curse of dimensionality Suppose Nth order T C I I I, then number of entries: I N memory and time complexity: O ( I N)

9 6 Curse of dimensionality Suppose Nth order T C I I I, then number of entries: I N memory and time complexity: O ( I N) number of variables: NIR

10 6 Curse of dimensionality Suppose Nth order T C I I I, then number of entries: I N memory and time complexity: O ( I N) number of variables: NIR Example [Vervliet et al. 2014] Ninth-order tensor with I = 100 and rank R = 5: number of entries: number of variables: 4500

11 7 How to handle large tensors? Use incomplete tensors Acar et al. 2011b; Vervliet et al. 2014; Vervliet et al. 2016a Exploit sparsity Kang et al. 2012; Papalexakis et al. 2012; Bader and Kolda 2007 Compress the tensor Sidiropoulos et al. 2014; Oseledets and Tyrtyshnikov 2010; Vervliet et al. 2016b Decompose subtensors and combine results Papalexakis et al. 2012; Phan and Cichocki 2011 Parallel Liavas and Sidiropoulos many of the above

12 8 Overview Decomposing large-scale tensors Randomized block sampling Experimental results Chemo-sensing application

13 9 Randomized block sampling CPD: idea + +

14 9 Randomized block sampling CPD: idea + +

15 9 Randomized block sampling CPD: idea + + Take sample

16 9 Randomized block sampling CPD: idea + + Take sample Initialization Compute step + +

17 9 Randomized block sampling CPD: idea + + Take sample Initialization Update Compute step + +

18 10 Randomized block sampling CPD: algorithm input : Data T and initial guess A (n), n = 1,..., N output: A (n), n = 1,..., N such that T A (1),..., A (N) while k < K and not converged do Create sample T s and corresponding A (n) s, n = 1,..., N Let Ā (n) s be the result of 1 iteration in a restricted CPD algorithm on T s with initial guess A (n) s, n = 1,..., N and restriction Update the affected variables A (n) using Ā (n) s, n = 1,..., N k k + 1

19 10 Randomized block sampling CPD: algorithm input : Data T and initial guess A (n), n = 1,..., N output: A (n), n = 1,..., N such that T A (1),..., A (N) while k < K and not converged do Create sample T s and corresponding A (n) s, n = 1,..., N Let Ā (n) s be the result of 1 iteration in a restricted CPD algorithm on T s with initial guess A (n) s, n = 1,..., N and restriction Update the affected variables A (n) using Ā (n) s, n = 1,..., N k k + 1

20 11 Ingredient 1: randomized block sampling For a 6 6 tensor and block size 3 2: I 1 = {3, 1, 2, 6, 5, 4} I 2 = {1, 2, 4, 6, 3, 5}

21 11 Ingredient 1: randomized block sampling For a 6 6 tensor and block size 3 2: I 1 = {3, 1, 2, 6, 5, 4} I 2 = {1, 2, 4, 6, 3, 5} I 1 = {3, 1, 2, 6, 5, 4} I 2 = {1, 2, 4, 6, 3, 5}

22 11 Ingredient 1: randomized block sampling For a 6 6 tensor and block size 3 2: I 1 = {3, 1, 2, 6, 5, 4} I 2 = {1, 2, 4, 6, 3, 5} I 1 = {3, 1, 2, 6, 5, 4} I 2 = {1, 2, 4, 6, 3, 5} I 1 = {6, 1, 4, 2, 5, 3} I 2 = {1, 2, 4, 6, 3, 5}

23 12 Ingredient 2: restricted CPD algorithm ALS variant A (n) k+1 = (1 α)a(n) k + αt (n) V (n) ( W (n) ) 1 Enforce restriction by α = k.

24 12 Ingredient 2: restricted CPD algorithm ALS variant A (n) k+1 = (1 α)a(n) k + αt (n) V (n) ( W (n) ) 1 Enforce restriction by α = k. NLS variant in which 1 min p k 2 vec (F(x k)) J k p k 2 s.t. p k k F = T A (1),..., A (N)

25 13 Ingredient 3: restriction Use restriction of form { k = 0 ˆ 0 α (k Ksearch)/Q if if k < K search k K search Iteration k

26 13 Ingredient 3: restriction Use restriction of form { k = 0 ˆ 0 α (k Ksearch)/Q if if k < K search k K search Iteration k Example (Selecting Q) For a tensor and block size , Q = 4

27 14 Ingredient 4: A stopping criterion Function evaluation f val = 0.5 T A (1),..., A (N) f val CPD Error Iteration k

28 14 Ingredient 4: A stopping criterion Function evaluation f val = 0.5 T A (1),..., A (N) f val CPD Error Iteration k Step size

29 15 Intermezzo: Cramér Rao bound Uncertainty of an estimate 68% 3σ 2σ σ 0 σ 2σ 3σ

30 15 Intermezzo: Cramér Rao bound Uncertainty of an estimate 68% CRB σ 2 3σ 2σ σ 0 σ 2σ 3σ

31 15 Intermezzo: Cramér Rao bound Uncertainty of an estimate 68% CRB σ 2 3σ 2σ σ 0 σ 2σ 3σ C = τ 2 (J H J) 1

32 16 Ingredient 4: Cramér Rao bound based stopping criterion Experimental bound (n) Use estimates A k Use f val to estimate noise τ

33 16 Ingredient 4: Cramér Rao bound based stopping criterion Experimental bound (n) Use estimates A k Use f val to estimate noise τ Stopping criterion: 1 D CRB = R n I n N I n R n=1 i=1 r=1 A (n) k (i, r) A(n) k K CRB (i, r) C (n) (i, r)

34 16 Ingredient 4: Cramér Rao bound based stopping criterion Experimental bound (n) Use estimates A k Use f val to estimate noise τ Stopping criterion: 1 D CRB = R n I n γ N I n R n=1 i=1 r=1 A (n) k (i, r) A(n) k K CRB (i, r) C (n) (i, r)

35 17 Unrestricted phase vs restricted phase CPD Error Iteration k Unrestricted phase (1 + 2): converge to a neighborhood of an optimum Restricted phase (3): pull iterates towards optimum

36 17 Unrestricted phase vs restricted phase CPD Error Iteration k Unrestricted phase (1 + 2): converge to a neighborhood of an optimum Restricted phase (3): pull iterates towards optimum

37 17 Unrestricted phase vs restricted phase CPD Error Iteration k Unrestricted phase (1 + 2): converge to a neighborhood of an optimum Restricted phase (3): pull iterates towards optimum Assumptions CPD of rank R exists SNR is high enough Most block dimensions > R

38 18 Overview Decomposing large-scale tensors Randomized block sampling Experimental results Chemo-sensing application

39 19 Experiment overview Experiments Comparison ALS vs NLS (see paper) Influence of block size Influence of step size (see paper)

40 19 Experiment overview Experiments Comparison ALS vs NLS (see paper) Influence of block size Influence of step size (see paper) Performance 50 Monte Carlo experiments CPD error max A (n) n 0 A (n) res / A (n) 0

41 19 Experiment overview Experiments Comparison ALS vs NLS (see paper) Influence of block size Influence of step size (see paper) Performance 50 Monte Carlo experiments CPD error max A (n) n 0 A (n) res / A (n) 0 cpd rbs in Tensorlab 3.0 [Vervliet et al. 2016c]

42 20 Influence of block size: setup (4 4 2) ν U(0, 1) = N R = 20 No noise

43 21 Influence of block size on computation time 150 Time (s) full ν (4 4 2) ν R = 20, U(0, 1) No noise

44 22 Influence of block size on data accesses Data accesses (%) 1000 Full tensor full ν (4 4 2) ν R = 20, U(0, 1) No noise

45 23 Influence of block size on accuracy 10 0 Unrestricted ECPD full ν (4 4 2) ν R = 20, U(0, 1) 20 db

46 23 Influence of block size on accuracy ECPD Unrestricted Restricted full ν (4 4 2) ν R = 20, U(0, 1) 20 db

47 24 Overview Decomposing large-scale tensors Randomized block sampling Experimental results Chemo-sensing application

48 25 Classify hazardous gasses Does the sample contain CO, acetaldehyde or ammonia? Sensor Experiment Time Strategy: classify using coefficients of spatiotemporal patterns R = 5 Unknown

49 26 Classify hazardous gasses: results Resulting factor matrices time sensor experiment

50 26 Classify hazardous gasses: results Resulting factor matrices time sensor experiment Performance after clustering Iterations Time (s) Error (%) No restriction Restriction

51 27 Conclusion The randomized block sampling CPD algorithm enables the decomposition of larger tensors, using fewer data points and less memory Block size controls accuracy, data accesses and time Step size restriction improves accuracy Cramér Rao bound based stopping criterion combines noise and step information

52 28 More details: N. Vervliet and L. De Lathauwer [2016]. A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors. In: IEEE Journal of Selected Topics in Signal Processing 10.2, pp

53 A randomized block sampling approach to the canonical polyadic decomposition of large-scale tensors Nico Vervliet Joint work with Lieven De Lathauwer SIAM AN17, July 13, 2017

54 2 References I Acar, E., D.M. Dunlavy, and T.G. Kolda (2011a). A scalable optimization approach for fitting canonical tensor decompositions. In: Journal of Chemometrics 25.2, pp Acar, E. et al. (2011b). Scalable tensor factorizations for incomplete data. In: Chemometrics and Intelligent Laboratory Systems 106.1, pp Bader, B.W. and T.G. Kolda (2007). Efficient MATLAB computations with sparse and factored tensors. In: SIAM J. Sci. Comput. 30.1, pp Kang, U. et al. (2012). GigaTensor: scaling tensor analysis up by 100 times-algorithms and discoveries. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp

55 3 References II Liavas, A. and N. Sidiropoulos (2015). Parallel Algorithms for Constrained Tensor Factorization via the Alternating Direction Method of Multipliers. In: IEEE Trans. Signal Process. PP.99, pp Oseledets, I.V. and E.E. Tyrtyshnikov (2010). TT-cross approximation for multidimensional arrays. In: Linear Algebra and its Applications 432.1, pp Papalexakis, E., C. Faloutsos, and N. Sidiropoulos (2012). ParCube: Sparse Parallelizable Tensor Decompositions. English. In: Machine Learning and Knowledge Discovery in Databases. Ed. by PeterA. Flach, Tijl De Bie, and Nello Cristianini. Vol Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp

56 4 References III Phan, A.-H. and A. Cichocki (2011). PARAFAC algorithms for large-scale problems. In: Neurocomputing 74.11, pp Phan, A.-H., P. Tichavský, and A. Cichocki (2013). Low Complexity Damped Gauss Newton Algorithms for CANDECOMP/PARAFAC. In: SIAM J. Appl. Math. 34.1, pp Sidiropoulos, N., E. Papalexakis, and C. Faloutsos (2014). Parallel randomly compressed cubes: A scalable distributed architecture for big tensor decomposition. In: IEEE Signal Process. Mag. 31.5, pp

57 5 References IV Sorber, L., M. Van Barel, and L. De Lathauwer (2013). Optimization-Based Algorithms for Tensor Decompositions: Canonical Polyadic Decomposition, Decomposition in Rank-(L r, L r, 1) Terms, and a New Generalization. In: 23.2, pp Vervliet, N. and L. De Lathauwer (2016). A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors. In: IEEE Journal of Selected Topics in Signal Processing 10.2, pp Vervliet, N., O. Debals, and L. De Lathauwer (2016a). Canonical polyadic decomposition of incomplete tensors with linearly constrained factors. Technical Report , ESAT-STADIUS, KU Leuven, Belgium.

58 6 References V (2016b). Tensorlab 3.0 Numerical optimization strategies for large-scale constrained and coupled matrix/tensor factorization. In: th Asilomar Conference on Signals, Systems and Computers. Vervliet, N. et al. (2014). Breaking the Curse of Dimensionality Using Decompositions of Incomplete Tensors: Tensor-based scientific computing in big data analysis. In: IEEE Signal Process. Mag. 31.5, pp Vervliet, N. et al. (2016c). Tensorlab 3.0. Available online at