Dynamical low-rank approximation

Transcription

1 Dynamical low-rank approximation Christian Lubich Univ. Tübingen Genève, Swiss Numerical Analysis Day, 17 April 2015

2 Coauthors Othmar Koch 2007, 2010 Achim Nonnenmacher 2008 Dajana Conte 2010 Thorsten Rohwedder & Reinhold Schneider 2013 Bart Vandereycken 2013, 2015 Ivan Oseledets 2014, 2015 Jutho Haegeman & Frank Verstraete 2014 pre Emil Kieri & Hanna Walach 2015 pre

3 Motivation: Data reduction / model reduction Encyclopedia: term-document matrix latent semantic indexing: low-rank approximation by a truncated SVD: A r j=1 σ ju j vj T time-dependent encyclopedia? (à la Wikipedia) Multi-particle quantum dynamics time-dependent Schrödinger equation i ψ t = Hψ for the wavefunction ψ = ψ(x 1,..., x N, t) MCTDH: reduced model via low-rank tensor approximation

4 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

5 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

6 Best low-rank approximation given a matrix A R m n (huge m, n) best approximation to A of rank r: matrix X in the manifold M r of rank-r matrices with X M r such that X A = min! Frobenius norm: problem solved by truncated SVD X = UΣ r V T = r i=1 σ i u i v T i

7 Best low-rank approximation given matrices A(t) R m n for 0 t T (huge m, n) best approximation to A(t) of rank r: matrix X (t) in the manifold M r of rank-r matrices with X (t) M r such that X (t) A(t) = min! Frobenius norm: problem solved by truncated SVD, but expensive! Need an updating technique

8 Dynamical low-rank approximation X (t) M r such that X (t) A(t) = min! replaced by initial value problem of differential equations on M r : Ẏ (t) T Y (t) M r such that Ẏ (t) Ȧ(t) = min!

9 Motivation Ẏ (t) T Y (t) M r such that Ẏ (t) Ȧ(t) = min! use sparse increments Ȧ(t) instead of the complete data matrix A(t): time-continuous updating linear projection onto the tangent space instead of a nonlinear, nonconvex minimization problem extends to matrix differential equations Ȧ = F (A): minimum defect approximation... Ẏ (t) T Y (t) M r such that Ẏ (t) F (Y (t)) = min!

10 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

11 Non-unique factorisation of rank-r matrices Y = USV T with invertible S R r r U R m r and V R n r with orthonormal columns SVD has diagonal S, is not assumed here!

12 Unique decomposition in the tangent space non-unique decomposition Y = USV T M r tangent matrix Ẏ T Y M r : Ẏ = USV T + UṠV T + US V T with Ṡ Rr r and skew-symmetric U T U, V T V Ṡ, U, V are uniquely determined by Ẏ and S, U, V under the orthogonality constraints U T U = 0, V T V = 0 gauge conditions enforce uniqueness

13 Equivalent formulations of dynamical low-rank approximation Ẏ T Y M r such that Ẏ Ȧ = min! Ẏ Ȧ, Z = 0 for all Z T Y M r Ẏ = P(Y )Ȧ with P(Y ) = orth. projection onto T Y M r

14 ODEs for dynamical low-rank approximation Y = USV T with U = (I m UU T )ȦV S 1 V = (I n VV T )Ȧ T US T Ṡ = U T ȦV Koch & L cf. ODEs for SVD (Wright 1992 and Dieci & Eirola 1999) but here, no singularities arise for coalescing singular values

15 Low-rank approximation of matrix ODEs Ȧ = F (A) solution A approximated by Y = USV T with U = (I m UU T )F (Y )VS 1 V = (I n VV T )F (Y ) T US T Ṡ = U T F (Y )V minimum defect: Ẏ T Y M r with Ẏ F (Y ) = min!

16 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

17 Equivalent formulations of dynamical low-rank approximation Ẏ T Y M r such that Ẏ Ȧ = min! Ẏ Ȧ, Z = 0 for all Z T Y M r Ẏ = P(Y )Ȧ with P(Y ) = orth. projection onto T Y M r : P(Y )Ȧ = ȦP R(Y T ) P R(Y ) ȦP R(Y T ) + P R(Y ) Ȧ Idea: split the projection the integrator (wait and see!) L. & Oseledets 2014

18 Splitting integrator, abstract form 1. Solve the differential equation Ẏ I = ȦP R(Y T I ) with initial value Y I (t 0 ) = Y 0 for t 0 t t Solve Ẏ II = P R(YII )ȦP R(Y T II ) with initial value Y II (t 0 ) = Y I (t 1 ) for t 0 t t Solve Ẏ III = P R(YIII )Ȧ with initial value Y III (t 0 ) = Y II (t 1 ) for t 0 t t 1. Finally, take Y 1 = Y III (t 1 ) as an approximation to Y (t 1 ).

19 Solving the split differential equations The solution of 1. is given by Y I = U I S I V T I with (U I S I ) = ȦV I, VI = 0 : U I (t)s I (t) = (A(t) A(t 0 ))V I (t 0 ), V I (t) = V I (t 0 ). and similarly for 2. and 3.

20 Splitting integrator, practical form Start from Y 0 = U 0 S 0 V T 0 M r. 1. With the increment A = A(t 1 ) A(t 0 ), set and orthogonalize: K 1 = U 0 S 0 + A V 0 K 1 = U 1 S1, where U 1 R m r has orthonormal columns, and S 1 R r r. 2. Set S 0 = S 1 U1 T A V Set L 1 = V T 0 S 0 + A T U 1 and orthogonalize: L 1 = V 1 S1 T, where V 1 R n r has orthonormal columns, and S 1 R r r. The algorithm computes a factorization of the rank-r matrix Y 1 = U 1 S 1 V1 T Y (t 1 ).

21 Splitting integrator, cont. use symmetrized variant (Strang splitting) for a matrix differential equation in substep 1. solve Ȧ = F (A): K = F (KV T 0 )V 0, K(t 0 ) = U 0 S 0 by a step of a numerical method (Runge-Kutta etc.), and similarly in substeps 2. and 3.

22 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

23 ODEs for dynamical low-rank approximation Y = USV T with U = (I m UU T )ȦV S 1 V = (I n VV T )Ȧ T US T Ṡ = U T ȦV What if S is ill-conditioned? (effective rank smaller than r)

24 Numerical experiment ε-perturbed rank-10 matrices for t = 0, 0.2,..., 1, ε=1e

25 Numerical experiment: errors at t = 1 Table: Approximation rank r = 10. ε X A Y A S 1 1e e e e 01 1e e e e+00 1e e e e+00 1e e e e+00 Table: Approximation rank r = 20. ε X A Y A S 1 1e e e e+00 1e e e e+01 1e e e e+02 1e e e e+03

26 Robustness under over-approximation Theorem A(t) = X q (t) + E(t) with E(t) ε 0, Ė(t) ε 1 for X q (t) M q with q r σ q (A(t)) ρ > 0 Ȧ(t) µ Y (0) = X r (0) M r Then, Y (t) X q (t) ε 0 + 6tε 1 for t ρ 16µ good rank-r approximation even in case of effective rank q < r

27 Numerical experiment: integrator errors (h = 10 3 ) Table: ε = 10 3, r = 20 Table: ε = 10 6, r = 20 p Appr. err. Midpoint KLS KLS(symm) KSL KSL(symm) p Appr. err. Midpoint KLS KLS(symm) KSL KSL(symm) Table: ε = 10 3, r = 10 Table: ε = 10 6, r = 10 p Appr. err. Midpoint KLS KLS(symm) KSL KSL(symm) p Appr. err. Midpoint - failed KLS KLS(symm) KSL e-05 KSL(symm) e-05

28 An exactness result for the splitting method Theorem If A(t) has rank at most r, then the splitting integrator is exact: Y 1 = A(t 1 ) Ordering of the splitting is essential! (KSL, not KLS)

29 Approximation is robust to small singular values Ȧ = F (t, A), F is locally Lipschitz-continuous A(t 0 ) = Y 0 M r (I P(Y ))F (t, Y ) ε for all Y M r. Y n M r result of the projector-splitting integrator after n steps Theorem Y n A(t n ) c 1 ε + c 2 h for t n T, where c 1, c 2 depend only on the local Lipschitz constant and bound of F, and on T. Kieri, L. & Walach 2015 pre

30 Remarks on the proof The method splits P(Y ) = P I (Y ) P II (Y ) + P III (Y ) in Ẏ = P(Y )F (t, Y ). Difficulty: cannot use the Lipschitz continuity of the tangent space projection P( ) and its subprojections, because the Lipschitz constants become large for small singular values Rescue: use the previous exactness result use the conservation of the subprojections in the substeps

31 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

32 Tensors in Tucker format Approximation of time-dependent tensors A(t) R n 1 n d with entries A(k 1,..., k d ; t) by tensors Y (t) R n 1 n d of the form, with r i n i, Y (k 1,..., k d ; t) = r 1 j 1 =1 r d j d =1 c j1,...,j d (t) u (1) j 1 Tucker format of multilinear rank r = (r 1,..., r d ): (k 1 ; t)... u (d) j d (k d ; t), Each 1-mode matrix unfolding of the core tensor has full rank. The vectors u (i) 1 (t),..., u(i) r i (t) R n i are orthonormal. Dynamical tensor approximation: Koch & L Projector-splitting integrator: L pre

33 Tensor trains / matrix product states Non-commutative separation of variables: Y (k 1,..., k d ) = G 1 (k 1 )... G d (k d ) where the G(k i ) are r i 1 r i matrices, with r 0 = r d = 1. Attractive because of low data requirement: dkr 2 Matrix product states: e.g., Verstraete, Murg & Cirac 2008 Tensor trains: Oseledets 2011 Manifold structure: Holtz, Rohwedder & Schneider 2012 Uschmajew & Vandereycken 2013 Dynamical approximation in the tensor train format: L., Rohwedder, Schneider, Vandereycken 2013 Projector-splitting integrator: L., Oseledets & Vandereycken 2015 in physics context: Haegeman, L., O., V. & Verstraete 2014 pre Error analysis: Kieri, L. & Walach 2015 pre

34 Outline Dynamical low-rank approximation Differential equations Splitting integrator Robustness to small singular values Extensions to tensors

35 Review C.L., Low-rank dynamics in Extraction of Quantifiable Information from Complex Systems (S. Dahlke et al., eds.), Springer Lecture Notes Comput. Sci. Eng. 102, na.uni-tuebingen.de