Tensor networks, TT (Matrix Product States) and Hierarchical Tucker decomposition

Size: px

Start display at page:

Download "Tensor networks, TT (Matrix Product States) and Hierarchical Tucker decomposition"

Buck McKinney
6 years ago
Views:

1 Tensor networks, TT (Matrix Product States) and Hierarchical Tucker decomposition R. Schneider (TUB Matheon) John von Neumann Lecture TU Munich, 2012

2 Setting - Tensors V ν := R n, H d = H := d ν=1 V ν d-fold tensor product Hilbert-s., H {(x 1,..., x d ) U(x 1,..., x d ) R : x i = 1,..., n i }. The function U H will be called an order d-tensor. For notational simplicity, we often consider n i = n. Here x 1,..., x d {1,..., n} will be called variables or indices. k U(k 1,..., k d ) = (U k1,...,k d ), k i = 1..., n i. Or in index (vectorial) notation U = ( U k1,...,k d ) ni k i =0,1 i d dim H = n d curse of dimensions!!! E.g. wave function Ψ(r 1, s 1,..., r N, s N )

3 Contracted tensor representations Espig & Hackbusch & Handschuh & S. Let V ν := R rν K K = K := K ν=1 V ν d-fold tensor product space, K {(k 1,..., k K ) V (k 1,..., k K ) : k i = 1,..., r i }. Here k 1,..., k N will be called contraction variables, x 1,..., x d will be called exterior variables. We define an order d + K tensor product space H := H K {(x 1,... x d, k 1,..., k K ) Ũ(x 1,..., x d, k 1,..., k K ) : x i = 1,..., n; 1 i d, k j = 1,..., r j, 1 j K }.

4 Contracted tensor representations Definition Subset V r H: Let d < d d + K, r 1 r K d U V r U =... u ν (x ν, k ν(1), k ν(2), k ν(3) ) k 1 ν=1 where k ν(i) {k 1,..., k K }, i = 1, 2, 3, and u ν (x ν, k ν(1), k ν(2) ) if ν = 1,..., d u ν (k ν(1), k ν(2), k ν(3) ) if ν = d + 1,..., d, k K I.e. the d tensors u ν (.,.,.) depends at most on 3 variables! We will write shortly u ν (x ν, k ν ) R, keeping in mind that u ν depends only on at most three variables x ν, k ν(1), k ν(2),k ν(3). Example Canonical format: K = 1, d = d, u ν (x ν, k), k = 1,..., r.

5 Contracted tensor representations Definition Subset V r H: Let d < d d + K, r 1 r K d U V r U =... u ν (x ν, k ν(1), k ν(2), (k ν(3)) ) k 1 ν=1 k K the d tensors u ν (.,.,.) depends at most on 3 variables! (Not necessary!) Complexity: Let r := max{r i : i = 1,..., K } then storage requirement is DOF s max{d n r 2, d r 3 }.

6 Contracted tensor representations Due to the multi-linear ansatz, the present extremely general format allow differentiation and local optimization methods! (See e.g. previous talk!) Redundancy has not been removed, in general redundancy is enlarged. Closedness would be violated in general. Tensor networks Definition A minimal tensor network is a representation in the above format, where each contraction variable k i, i = 1,..., K, appear exactly twice. In this case, they can be labeled by i (ν, µ), 1 ν d, 1 µ < ν. It is often assumed, that each exterior variable x i appear at most once!

7 Tensor networks Diagram - (Graph) of a tensor network: vector matrix tensor x x x 1 2 x 1 u [x] u [x 1, x 2 ] = x 1 x 2 k u 1 [x 1, k] x 1 k 2 = u [x 1,..., x 8 ] u 2 u 4 [x 4, k 3, k 4 ] = k i TT format HT format Diagramm: x 1 x 8 Each node corresponds to a factor u ν, ν = 1,..., d, each line to a variable x ν or a contraction variable k µ. Since k ν is an edge connecting 2 tensors u µ1 and u µ2, one has to sum over connecting lines (edges).

8 Tensor networks has been introduced in quantum information theory. Example 1. TT (tensor trains, Oseledets & Tyrtyshnikov ) and HT (Hackbusch & Kühn) formats are tensor networks. 2. the canonical format is not (directly) a tensor network. tensor networks x 1 tensor networks x 1 k 1 ki u 2 [x k i k j ] 2 closed tensor network

9 Tensor formats Hierarchical Tucker format (HT; Hackbusch/Kühn, Grasedyck, Kressner : Tree-tensor networks)

10 Tensor formats Hierarchical Tucker format (HT; Hackbusch/Kühn, Grasedyck, Kressner : Tree-tensor networks) Tucker format (Q: MCTDH(F)) U(x 1,.., x d ) = r 1... r d d B(k 1,.., k d ) U i (k i, x i ) k 1 =1 k d =1 i=1 {1,2,3,4,5}

11 Tensor formats Hierarchical Tucker format (HT; Hackbusch/Kühn, Grasedyck, Kressner : Tree-tensor networks) Tucker format (Q: MCTDH(F)) Tensor Train (TT-)format (Oseledets/Tyrtyshnikov, MPS-format of quantum physics) U(x) = r 1 k 1 =1... r d 1 k d 1 =1 i=1 d B i (k i 1, x i, k i ) = B 1 (x 1 ) B d (x d ) {1,2,3,4,5} {1} {2,3,4,5} {2} {3,4,5} U 1 U 2 U 3 U 4 U 5 r 1 r 2 r 3 r 4 n 1 n 2 n 3 n 4 n 5 {3} {4,5} {4} {5}

12 Noteable special case of HT: TT format (Oseledets & Tyrtyshnikov, 2009) (matrix product states (MPS), Vidal 2003, Schöllwock et al.) TT tensor U can be written as matrix product form = r 1 k 1 =1.. r d 1 k d 1 =1 U(x) = U 1 (x 1 ) U i (x i ) U d (x d ) U 1 (x 1, k 1 )U 2 (k 1, x 1, k 2 )...U d 1 (k d 2 x d 1, k d 1 )U d (k d 1, x d, k d ) with matrices U i (x i ) = ( u k i k i 1 (x i ) ) R r i 1 r i, r 0 = r d := 1 U 1 U 2 U 3 U 4 U 5 r 1 r 2 r 3 r 4 n 1 n 2 n 3 n 4 n 5 Redundancy: U(x) = U 1 (x 1 )GG 1 U 2 (x 2 ) U i (x i ) U d (x d ).

13 TT approximations of Friedman data sets f 2 (x 1, x 2, x 3, x 4 ) = (x (x 2x 3 1 x 2 x 4 ) 2, f 3 (x 1, x 2, x 3, x 4 ) = tan 1 ( x 2 x 3 (x 2 x 4 ) 1 x 1 ) on 4 D grid, n points per dim. n 4 tensor, n {3,..., 50}. full to tt (Oseledets, successive SVDs) and MALS (with A = I) (Holtz & Rohwedder & S.)

14 Solution of U = b Dimension d = 4,..., 128 varying Gridsize n = 10 Right-hand-side b of rank 1 Solution U has rank 13

15 Hierarchical Tucker as subspace approximation 1) D = {1,..., d}, tensor space V = j D V j 2) T D dimension partition tree, vertices α T D are subsets α T D, root: α = D 3) V α = j α V j for α T D 4) U α V α subspaces of dimension r α with the characteristic nesting U α U α1 U α2 (α 1, α 2 sons of α) 5) v U D (w.l.o.g. U D = span (v)).

16 Subspace approximation: formulation with bases U α = span {b (α) i : 1 i r α } b (α) l = r α1 r α2 c (α,l) ij b (α 1) i b (α 2) j (α 1, α 2 sons of α T D ). i=1 j=1 Coefficients c (α,l) ij form the matrices C (α,l). Final representation of v is v = r D c (D) i b (D) i, (usually with r D = 1).

17 Orthonormal basis and recursive description We can choose {b (α) i : 1 i r α } orthonormal basis for α T D. This is equivalent to 1) At the leaves (α = {j}, j D): {b (j) i : 1 i r j } is chosen orthonormal 2) The matrices { C (α,l) } : 1 l r α are orthonormal (w.r.t. the Frobenius scalar product).

18 Hierarchical Tucker tensors Canonical decomposition Subspace approach (Hackbusch/Kühn, 2009) (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

19 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

20 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

21 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) B {1,2,3,4,5} B {1,2,3} B {4,5} B {1,2} U 3 U 4 U 5 U 1 U 2 (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

22 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) B {1,2,3,4,5} B {1,2,3} B {4,5} B {1,2} U 3 U 4 U 5 U 1 U 2 U {1,2} (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

23 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) B {1,2,3,4,5} B {1,2,3} B {4,5} B {1,2} U 3 U 4 U 5 U 1 U 2 U {1,2} U {1,2,3} (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

24 Hierarchical Tucker tensors Canonical decomposition not closed, no embedded manifold! Subspace approach (Hackbusch/Kühn, 2009) B {1,2,3,4,5} B {1,2,3} B {4,5} B {1,2} U 3 U 4 U 5 U 1 U 2 U {1,2} U {1,2,3} (Example: d = 5, U i R n k i, B t R k t k t1 k t2 )

25 HOSVD bases (Hackbusch) 1) Assume r D = 1. Then C (D,1) = Σ α := diag{σ (D) 1,...}, where σ (D) i are the singular values of M α1 (v) 2) For non-leaf vertices α T D, α D, we have rα l=1 (σ (α) l ) 2 C (α,l) C (α,l)h = Σ 2 α 1, rα l=1 (σ (α) l ) 2 C (α,l)t C (α,l) = Σ 2 α 2, where α 1, α 2 are the first and second son of α T D and Σ αi diagonal of the singular values of M αi (v). the

26 Historical remarks hierarchical Tucker approx. Physics: (I do not have suff. complete knowledge) The ideas are well established in many body quantum physics, quantum optics and quantum information theory 1. DMRG: S. White (91) 2. MPS: Roemmer & Ostlund (94), Vidal (03), Verstraete et al. 3. (tree) tensor networks : Verstrate, Cirac, Schollwöck, Wolf, Eisert... (?) 4. MERA, PEPS, Chemistry: Multilevel MCTDH HT : Meyer et al. (2000) Mathematics: Completely new in tensor product approximation 1. Khoromskij et al. (2006), remark about multi level Tucker 2. Lubich (book) (2008): review about Meyers paper 3. HT - Hackbusch (2009) et al.: Introduction of the subspace approximation!!!! 4. Grasedyck (2009): HT -sequential SVD 5. Oseledets & Tyrtishnikov (2009): TT and sequential SVD

27 Successive SVD for e.g. TT tensors - Vidal (2003), Oseledets (2009), Grasedyck (2009) 1. Matricisation - unfolding F(x 1,..., x d ) F x 2,...,x d x 1. low rank approximation up to an error ɛ 1, e.g. by SVD. F x 2,...,x d x 1 = r 1 k 1 =0 u k 1 x 1 v x 2,...,x d k 1, U 1 (x 1, k 1 ) := u k 1 x 1, k 1 = 1,..., r Decompose V (k 1, x 2,..., x d ) via matricisation up to an accuracy ɛ 2, V x 3,...,x d k 1,x 2 = r 2 k 2 u k2 k 1,x 2 v x 3,...,x d k 2 U 2 (k 1, x 2, k 2 ) := u k 2 k 1,x repeat with V (k 2, x 3,... x d ) until one ends with [ vkd 1,x d ] Ud (k d 1, x d ).

28 Example The function U(x 1,..., x d ) = x x d H in canonical representation U(x 1,..., x d ) = x x x d In Tucker format, let U i,1 = 1, U i,2 = x i be the basis of U i (non-orthogonal), r i = 2. U intt or MPS representation U(x) = x x d = ( x 1 1 ) ( ) x 2 1 (non-orthogonal), r 1 =... = r d 1 = 2 ( ) ( ) x d 1 1 x d

29 Successive SVD for e.g. TT tensors The above algorithm can be extended to HT tensors! Error analysis Quasi best approximation F(x) U(x 1 )... U d (x d ) 2 ( d 1 i=1 ɛ 2 ) 1/2 i d 1 inf U T r F(x) U(x) 2. Exact recovery: if U T r, then is will be recovered exactly! (up ot rounding errors)

30 Ranks of TT and HT tensors The minimal numbers r i for a TT or HT respresentation of a tensor U is well defined. 1. The optimal ranks r i, r = (r 1,..., r d 1 ), of the TT decomposition are equal to the ranks of the following matrices, i = 1,..., d 1 r i = rank of A i = U x i+1,...,x d x 1,...,x i. 2. If ν D then µ := D\µ the the mode ν-rank r ν is given by r ν = rank of A i = U x µ x ν. (If ν = {1, 2, 7, 8} and µ = {3, 4, 5, 6} then x ν = (x 1, x 2, x 7, x 8 ). )

31 Closedness A tree T D is characterized by the property, if one remove one egde yields two seperate trees. Observation: Let A i with ranks ranka i r. If lim i A i A 2 = 0 then ranka r: closedness of Tucker and HT tensor in T r (Falco & Hackbsuch). T r = s r Ts H is closed! due to Hackbusch & Falco Landsberg & Ye: If a tensor network has not a tree structure, the set of all tensor of this form need not to be closed!

32 Landsbergs result - tensor networks Let µ are the vertices and s µ S µ are incident edges of a vertex µ, and V µ the vector space attached to it. (it can be empty!) A vertex µ is called super critical if dim V µ Π s(µ) S(µ) r s(µ), and critical if dim V µ = Π s(µ) S(µ) r s(µ) Lemma Any supercritical vertex can be reduced to a critical one by Tucker decomposition. Theorem (Landsberg-Ye) If a (proper) loop a tensor network contains only (super)-critical vertices, then the network is not (Zariski)-closed. I.e. if the ranks are too small! For d i=1 C2, i.e. n = 2, Landsbergs result has no consequences.

33 Summary For Tucker and HT redundancy can be removed (see next talk) Table: Comparison canonical Tucker HT complexity O(ndr) O(r d + ndr) O(ndr + dr 3 ) TT- O(ndr 2 ) ++ + rank no defined defined r c r T r HT, r T r HT closedness no yes yes essential redundancy yes no no recovery?? yes yes quasi best approx. no yes yes best approx. no exist exist but NP hard but NP hard

34 Thank you for your attention. References: UNDER CONSTRUCTION, G. Beylkin, M.J. Mohlenkamp Numerical operator calculus in higher dimensions, Proc. nat. Acad. Sci. USA, 99 (2002), G. Beylkin, M.J. Mohlenkamp Algorithms for numerical analysis high dimensions, SIAM J. Sci. Comput. 26, (2005), W. Hackbusch, Tensor spaces and numerical tensor calculus,sscm, vol. 42, to appear, Springer, W. Hackbusch, S. Kühn, A new scheme for the tensor representation, Preprint 2, MPI MIS, A. Falco, W. Hackbusch, On minimal subspaces in tensor representations Preprint 70, MPI MIS, M. Espig, W. Hackbusch, S. Handschuh and R. Schneider Optimization Problems in Contracted Tensor Networks, MPI, MIS Preprint 66/2011, submitted to Numer. Math. L. Grasedyck, Hierarchical singular value decomposition of tensors, SIAM. J. Matrix Anal. & Appl. 31, p. 2029, C.J. Hillar and L.-H. Lim, Most tensor problems are NP hard preprint, (2012) arxiv: v3 S. Holtz, Th. Rohwedder, R. Schneider, On manifolds of tensors with fixed TT rank, Num. Math. DOI: /s , O. Koch, C. Lubich, Dynamical tensor approximation, SIAM J. Matrix Anal. Appl. 31, p. 2360, I. Oseledets, E. E. Tyrtyshnikov, Breaking the curse of dimensionality, or how to use SVD in many dimensions, SIAM J. Sci. Comput., Vol.31 (5), 2009, J. M. Landsberg, Yang Qi, Ke Ye, On the geometry of tensor network states arxiv: v2 [math.ag] to appear Quantum Information Computation D. Kressner and C. Tobler htucker A Matlab toolbox for tensor in hierarchical Tucker format Tech Report, SAM ETH Zürich (2011) G. Vidal Phys. Rev. Lett 101, (2003) I do not have suff. complete knowledge about physics literatur

Tensor Product Approximation

Tensor Product Approximation R. Schneider (TUB Matheon) Mariapfarr, 2014 Acknowledgment DFG Priority program SPP 1324 Extraction of essential information from complex data Co-workers: T. Rohwedder (HUB),