Max-Planck-Institut für Mathematik in den Naturwissenschaften Leipzig

Transcription

1 Max-Panck-Insttut für Mathematk n den Naturwssenschaften Lepzg Truncaton of Tensors n the Herarchca Format by Wofgang Hackbusch Preprnt no.:

2

3 Truncaton of Tensors n the Herarchca Format Wofgang Hackbusch Max-Panck-Insttut Mathematk n den Naturwssenschaften Insestr. 22, D Lepzg Abstract Tensors are n genera arge-scae data whch requre a speca representaton. These representatons are aso caed a format. After mentonng the r-term and tensor subspace formats, we descrbe the herarchca tensor format whch s the most fexbe one. Snce operatons wth tensors often produce tensors of arger memory cost, truncaton to reduced ranks s of utmost mportance. The so-caed hgherorder snguar-vaue decomposton HOSVD) provdes a save truncaton wth expct error contro. The paper expans n deta how the HOSVD procedure s performed wthn the herarchca tensor format. Fnay, we state speca favourabe propertes of the HOSVD truncaton. AMS Subject Cassfcatons: 15A69, 15A18, 15A99, 65F99, 65T99 Key words: herarchca tensor representaton, HOSVD truncaton 1 Introducton In the standard case, tensors or order d are quanttes v ndexed by d ndces,.e., the entres of v are v[ 1,..., d ], where e.g. a ndces run from 1 to n. Hence the data sze s n d. Ths shows that even number n and d of moderate sze yed a huge vaue so that t s mpossbe to store a entres. Instead one needs some data-sparse tensor representaton. In ths paper we menton such representatons. The optma one s the herarchca representaton expaned n Chapter 4. A sght generasaton s the tree-based format descrbed n Facó et a. [3]. Chapter 2 contans an ntroducton nto tensor spaces and the used notaton. We many restrct ourseves to the fnte-dmensona case, n whch we do not have to dstngush between the agebrac and topoogca tensor spaces. The atter tensor spaces are dscussed n [3]. Snce true tensor spaces have ess peasant propertes than matrces whch are tensors of order 2), one tres to nterpret tensors as matrces. Ths eads to the technque of matrcsaton expaned n 2.2. The range of the obtaned matrces defnes the mnma subspaces ntroduced n 2.3. The dmenson of the mnma subspaces yeds the assocated ranks. The snguar-vaue decomposton apped to the matrcsatons eads to the so-caed hgher-order snguar-vaue decomposton HOSVD) whch w be mportant ater. Fnay, n 2.5, we dscuss bass transformatons. In Chapter 3 we brefy dscuss two cassca representatons of tensors: the r-term format aso caed CP format) and the tensor subspace format aso caed Tucker format). For the atter format the HOSVD s expaned n 3.3: nstead of appyng SVD to the fu tensor, we can appy t to the smaer core tensor. As a resut of HOSVD we can ntroduce speca HOSVD bases. These bases aow a smpe truncaton to smaer ranks.e., the data sparsty s mproved; cf. 3.4). Chapter 4 s devoted to the herarchca tensor format. In prncpe t s a recursve appcaton of the tensor subspace format. It s connected wth a bnary tree. The generasaton to a genera tree yeds the tree-based format n [3]. However, for practca reasons one shoud use a bnary tree. a key pont s the ndrect codng of the bases dscussed n 4.2. As a resut ony transfer matrces are stored nstead of arge-szed bass vectors. Bass transformatons can be performed by smpe matrx operatons cf. 4.3). Smary, the orthonormasaton of the bases are performed by an orthonormasaton of the transfer matrces cf. 4.4). The man chaenge s the computaton of the HOSVD bases. As demonstrated n 4.5 one can obtan these bases by snguar-vaue decompostons ony nvovng the transfer matrces. The correspondng truncaton can be performed as n the prevous chapter cf. 4.6). The SVD truncaton to ower ranks can be regarded as a projecton onto smaer subspaces. However, dfferent from genera projectons, the SVD projecton has partcuar propertes whch are dscussed n the 1

4 fna Chapter 5. In 5.1 we consder the case of the tensor subspace representaton of 3.2. It turns out that certan propertes of the gven tensor e.g., sde condtons or smoothness propertes) are nherted by the projected truncated) approxmaton. As proved n 5.2, the same statement hods for the best approxmaton n the format of ower ranks. Fnay, ths statement s generased to the herarchca tensor representaton. 2 Tensor Spaces 2.1 Defntons, Notaton Let V j 1 j d) be arbtrary vector spaces over the fed K, where ether K = R or K = C. Then the d agebrac tensor space V := a V j conssts of a fnte) near combnatons of eementary tensors d vj) v j) V j ). The agebrac defnton of V and of the tensor product : V 1... V d V reads as foows cf. Greub [4, Chap. I, 2]): Let U be any vector space over K. Then, for any mutnear mappng ϕ : V 1... V d U, there exsts a unque near mappng Φ : V U such that ϕv 1), v 2),..., v d) ) = Φ d vj) ) for a v j) V j. In the case of nfnte-dmensona tensor spaces, one can equp the tensor space wth a norm. The d competon wth respect to the norm yeds the topoogca tensor space V j cf. Hackbusch [5, 4]). In ths artce, we restrct ourseves to the fnte-dmensona case. Then the agebrac tensor space ntroduced above s aready compete wth respect to any norm and therefore t concdes wth the topoogca d tensor space. Ths fact aows us to avod the affx a n V = a V j. Instead, V = d V j s suffcent. The smpest exampe of a tensor space s based on the vector spaces V j = K nj, where the vectors v K nj are ndexed by I j := {1,..., n j }. Instead of K nj we aso wrte K Ij. Then the eementary product v := d vj) s ndexed 1 by d-tupes I := I 1... I d : v[] = v[ 1,..., d ] = d v j) [ j ] for = 1,..., d ) I. 2.1) Therefore the tensor space V s somorphc to K I. The second exampe s based on the matrx spaces U j = K mj nj. Then the tensor space U := d U j can be nterpreted as foows. Set V = d V j wth V j = K nj as above, whe W = d W j s generated by W j = K mj. Matrces M j U j defne a near map beongng to the vector space LV j, W j ). Now the eementary tensor M := d M j U can be regarded as a near map of LV, W) defned by 2 d M v j) = d M j v j)) for a v j) V j. 2.2) The tensor product d M j s aso caed the Kronecker product. In the fnte-dmensona case, U concdes wth LV, W). The defnton of V = d V j by a near combnatons of eementary tensors ensures that any v V has a representaton v = r d =1 v j) v j) V j ). 2.3) The tensor rank of v s the smaest possbe nteger r n 2.3). It s denoted by rankv). If d = 2 and V j = K nj, the tensor space V = V 1 V 2 s somorphc to the matrx space K n1 n2. The eementary tensor v w corresponds to the rank-1 matrx vw T. In ths case the tensor rank concdes wth the usua matrx rank. d = 1 s the trva case where V = d V j concdes wth V 1. For d = 0, the empty product V = d V j s defned by the underyng fed K. 1 The ndces are wrtten n square brackets snce we want to avod secondary ndces. 2 Here we use that a near map s unquey defned by ts acton on eementary tensors. 2

5 Remark 2.1 The dmenson of V = d V j s dmv) = d dmv j). Ths fact mpes that, e.g., U V W, U V ) W, U V W ), and U W ) V are somorphc as vector spaces. Here U V ) W s the tensor space of order 2 based on the vector spaces X := U V and W. However, these spaces are not somorphc as tensor spaces. For nstance they have dfferent eementary tensors. V = d V j and V = d W j are somorphc as tensor spaces f, for a 1 j d, the vector spaces V j and W j are somorphc. 2.2 Matrcsaton Wthn the theory of tensor spaces, the matrx case correspondng to d = 2 s an exceptona case. Ths means that most of the propertes of matrces do not generase to tensors of order d 3. An exampe s the tensor rank for d 3. In genera ts determnaton s NP hard cf. Håstad [9]). Tensors n d Rnj can aso be regarded as eements n d Cnj, but the correspondng tensor ranks may be dfferent. Matrx decompostons ke the Jordan norma form or the snguar-vaue decomposton do not have an equvaent for d 3. To overcome these dffcutes one may try to nterpret tensors are matrces. Accordng to Remark 2.1, for a 1 j d, the tensor space V = d k=1 V k s somorphc to V j V [j], where V [j] := k j V k. Here k j means k {1,..., d}\{j}. The vector space somorphsm M j : d k=1 V k V j V [j] s defned by d k=1 vk) v j) v [j] wth v [j] = k j vk) and caed the j-th matrcsaton. In the case of V k = K I k, the mage M j v) of v V s a matrx M K Ij I [j] wth I [j] = k j I k and the entres M[ j, [j] ] = v[ 1,..., d ] for a j I j and [j] = 1,..., j 1, j+1,..., d ) I [j]. An obvous generasaton reads as foows. Set D := {1,..., d} and choose a subset α D wth = α D. The compement s α c := D\α. Defne I α := j α I j and V α = j α V j. The matrcsaton wth respect to α uses the somorphsm M α : V V α V α c, M α v)[ α, α c] = v[] for = α, α c) I D. 2.4) M α v) can be regarded as a matrx n K Iα I α c. Fo = {j} we obtan the j-th matrcsaton M j from above. Fo = D, the set α c s empty. The forma defnton j V j = K expans that M D : V V K. Regardng M D v) as a matrx means that there s ony one coumn contanng the vectorsed tensor v. Anaogousy, M v) = M D v) T contans v as row vector. The α-rank of a tensor v s aready defned by Htchcock [10, p. 170] va the matrx rank of M α v): rank α v) := rank M α v)). For dfferent α the α-ranks are dfferent. The ony reatons are rank α v) = rank α cv), rank α v) rank α v) rank α v) fo = α α cf. Hackbusch [5, Lemma 6.19], s the dsjont unon). The connecton wth the tensor rank ntroduced n 2.1 s rank α v) rankv) cf. Hackbusch [5, Remark 6.21]). Fo = {j} we wrte rank j v). The tupe rank 1 v),..., rank d v)) s aso caed the mutnear rank of v. Let M := d M j) be an eementary Kronecker product of matrces M j). The tensor Mv s defned by 2.2) and satsfes M α Mv) = M α M α c) M α v), 2.5a) where M α = j α M j) and M α c = j α M j) are parta Kronecker products. Interpretng M c α v) and M α Mv) as matrces, the equvaent statement s M α Mv) = M α M α v) M T α c. 2.5b) 3

6 2.3 Mnma Subspaces Gven v V = d V j, there may be smaer subspaces U j V j such that v U = d U j. The subspaces of mnma dmenson are caed the mnma subspaces and denoted by Uj mn v). They satsfy whe v d U j mpes U mn j v) U j. v d U mn j v) 2.6) The generasaton to subsets α D = {1,..., d} uses the somorphsm V V α V α c. U mn α s defned as the subspace of mnma dmenson such that v U mn α be charactersed by v) V α v) V α c. The mnma subspaces can U mn α v) := range M α v)). 2.7) Ths ncudes the case α = {j}, for whch U mn α v) s wrtten as Uj mn v). In the nfnte-dmensona case one cannot nterpret M α v) as a matrx. Then the defnton 4) must be repaced by } U mn α v) := {φ α cv) : φ α c a V j α c j, 2.8) where V j s the dua space3 of V j. The appcaton of φ α c = j α c ϕj) to v s defned by d φ α c v j) := ϕ j) v j) ) v j) V α. j α c j α In the genera case the α-rank s defned by rank α v) = dmu mn α v)). An mportant property s that under natura condtons weak convergence v n v mpes cf. Hackbusch [5, Theorem 6.24], Facó Hackbusch [2]). rank α v) m nf n rank αv n ) 2.9) 2.4 Hgher-Order Snguar-Vaue Decomposton HOSVD) In the foowng we assume that a V j are pre-hbert spaces equpped wth the Eucdean scaar product,. Eucdean scaar product n V α = j α V j satsfes j α v j), j α w j) = j α v j), w j). Interpretng M α v) as a matrx, one may determne ts snguar-vaue decomposton SVD). A possbe tensor representaton s M α v) = =1 σ α) b α) b αc ), 2.10) where = rank α v) s the rank and σ α) 1 σ α) 2... σ r α) α > 0 are the snguar vaues, whe {b α) : 1 } V α and {b αc ) : 1 } V α c are orthonorma systems. De Lathauwer De Moor Vandewae [1] ntroduced the name HOSVD for the smutaneous SVD of the matrcsatons M j v), 1 j d. Note that n genera the SVD spectra σ j) ) 1 rj as we as r j = rank j v) do not concde. Compare aso Hackbusch Uschmajew [8]. 3 In the case of nfnte-dmensona normed spaces one has to dstngush the agebrac dua space V j contnuous near functonas. Defnton 2.8) hods wth V j as we as wth V j. from the space V j of 4

7 It w turn out that the mportant quanttes n 2.10) are the snguar vaues σ α) and the eft snguar vectors b α). These quanttes are aso charactersed by the dagonasaton of the matrx M α v)m α v) H : M α v)m α v) H = =1 σ α) ) 2 α) b In the case of α = {j}, M j v) s a matrx wth n j = dmv j ) rows and n [j] = k j dmv k) coumns. Note that n [j] may be a huge quantty. However, M j v)m j v) H s ony of the sze n j n j. 2.5 Bass Representatons, Transformatons The notaton 2.1) refers to the unt vectors e j) 1 n j ) of V j = K nj,.e., the tensor s v = n1 n d 1=1 d =1 v[ 1,..., d ] d ej) j. We may choose another bass b j) 1 n j ) of V j = K nj and obtan n 1 n d d v = c[ 1,..., d ] b j) j 2.11) 1=1 d =1 wth another coeffcent tensor c V := d Knj. The bass b j) 1 n j ) yeds the reguar matrx [ ] B j = b j) 1,..., bj) n j. Formng the Kronecker product B := d B j LV, V), Eq. 2.11) becomes v = Bc. If B j and B j are two bases of V j, there are transformatons T j) and S j) = T j) ) 1 wth B j = B jt j) and B j = B j S j),.e., b j) = Form T := d T j) and S := d Sj) = T 1. Then b α) ) H. r j k=1 T j) k b j) k. v = Bc = B c hods wth B = B T, B = B S, c = Sc, c = Tc. 2.12) Remark 2.2 Accordng to 2.5b), the matrcsaton satsfes M α v) = B α M α c) B T α c wth B α := j α B j. 3 Tensor Representatons 3.1 r-term Format Often, the dmenson d n j of d s much arger than the avaabe computer memory. Therefore Knj a nave representaton of a tensor va ts entres 2.1) s mpossbe. A cassca tensor representaton s the r-term format aso caed the canonca or CP format) reated to 2.3). Let V = d V j. We fx an nteger r N 0 = N {0} and defne the set r d R r := v V : v = v j), v j) V j, =1.e., v s represented by r eementary tensors wth the factors v j). Assumng dmv j ) = n j n, the memory cost of v R r s rnd unt: numbers n K). One checks that R r = {v : rankv) r}. As ong as rankv) r hods wth r of moderate sze, ths format yeds a sutabe representaton. If rankv) s too arge, one may try to fnd an approxmatng v of smaer rank. Another queston s the mpementaton of tensor operatons wthn ths format. Addng u R r and v R s, one obtans the representaton of the sum w := u + v n R r+s. Other operatons et the representaton rank ncrease even more. Therefore one needs a truncaton procedure whch approxmates a tensor n R t t too arge) by an approxmaton n R r for a sutabe r < t. Unfortunatey, ths task s rather dffcut cf. Hackbusch [5, 7, 9]). 5

8 3.2 Tensor-Subspace Format A remedy s the Tucker format or tensor-subspace format, whch s reated to 2.6) and 2.11). Let n j = dmv j ). Assume that we know that v d U j hods for subspaces U j V j of hopefuy much) smaer dmenson than n j. Choose any bass or even ony a generatng system) b j) 1 r j ) of U j,.e., U j = span{b j) : 1 r j }. 3.1a) Then there s a tensor c d Krj the so-caed core tensor such that v = r 1 1=1 r d d =1 c[ 1,..., d ] d b j) j. 3.1b) Note the dfference to 2.11). The sums n 2.11) have n j terms, whereas 3.1b) ony uses r j < n j as upper bound. Defnton 3.1 We denote the set of a tensors n V wth a representaton 3.1b) by T r, where r = r 1,..., r d ) s a mut-ndex. Remark 3.2 The optma choce of U j s gven by U j = Uj mn v) cf. 2.3), snce then r j = rank j v) s mnma. The memory cost for the core tensor s d r j. Therefore ths representaton s unfavourabe for arger d. [ ] Bud the rectanguar) matrces B j = b j) 1,..., bj) r j K nj rj and the Kronecker product B := d B j L d, V) as n 2.5. Then 3.1b) s equvaent to Krj v = Bc. 3.1c) 3.3 HOSVD The frst step are transformatons nto orthonorma bases B j wth B j = B j T j e.g., a QR decomposton of B j yeds B j = Q and T j = R). Accordng to 2.12), we have v = B c wth c := Tc. Denotng B and c agan by B and c, we obtan the representaton 3.1b) wth orthonorma bases b j) ) 1 rj. The second step s the HOSVD apped to the core tensor c d. Assume that the matrcsaton Krj C α := M α c) has the snguar-vaue decomposton C α = X α Σ α Yα T wth dagona Σ α and untary matrces X α, Y α. By Remark 2.2 the matrcsaton of v s M α v) = B α M α c) B T α = B c α X α Σ α Yα T B T αc. Ths s the snguar-vaue decomposton of M α v) wth the untary matrces B α X α and B α cy α. Takng α = {j}, we obtan a new bass transform by B j := B j X j 1 j d). The new bass b j) ) 1 rj s caed the j-th HOSVD bass. The core tensor has to be transformed nto c as above. Agan denotng B and c by B and c, we obtan the representaton 3.1b) wth respect to the HOSVD bases. Snce we do not need the rght snguar vectors n Y α, the practca computaton frst forms the product P j := M j c)m j c) H K rj rj. Ths s the most expensve step wth an arthmetc cost of O d r j) d r j). The second step s the snguar-vaue decomposton of P j cost: O d r3 j )). The representaton of v T r by the HOSVD bases aows two types of truncatons. The number r j = dmu j ) may be arger than necessary,.e., arger than rank j v) = dmuj mn v)). Ths s detected by vanshng snguar vaues. Assume that σ s j) j > 0, whereas σ j) = 0 for s j < r j. Then the sums n 3.1b) can be shortened repace r j by s j ). After ths step, v d U j T s hods wth U j = Uj mn v) and s j = rank j v). Note that the descrbed procedure yeds a shorter representaton whe the tensor s unchanged. A truncaton changng the tensor s descrbed next. 6

9 3.4 HOSVD Truncaton Assume agan that the representaton 3.1b) of v T r uses the HOSVD bases. We are ookng for an approxmaton u T s wth smaer dmensons s j < r j of the correspondng subspaces U j. Ths probem has two answers. Frst there s a not necessary unque) best approxmaton u best T s wth v u best = nf{ v u : u T s } s the Eucdean norm). The computaton must be done teratvey. It s hard to ensure that the correspondng mnmsaton method converge to the goba mnmum. A much easer approach s the HOSVD truncaton: Gven v T r wth HOSVD bases n 3.1b), omt a terms nvovng ndces j > s j. The other terms are unchaned. Obvousy, the resutng tensor u HOSVD beongs to T s and ts computaton requres no arthmetca operaton. In the case of matrces one knows that u HOSVD = u best. However, for d 3, u HOSVD s not the best, but the quas-optma approxmaton: v u HOSVD d r j =s j+1 σ j) ) 2 d v ubest 3.2) cf. [5, Theorem 10.3]). Snce the snguar vaues σ j) are known, the frst nequaty n 3.2) yeds a precse error estmate. Gven a toerance ε, one can choose r j such that the error s beow ε. The second nequaty proves quas-optmaty. 4 The Herarchca Tensor Format 4.1 Defnton, Notaton The tree-based tensor formats use a so-caed dmenson partton tree T D. The root of the tree s D = {1,..., d}, whe the eaves are {1},..., {d}. The tree descrbes how D s dvded recursvey. The vertces of the tree are subsets of D. Ether a vertex α s a sngeton and therefore a eaf) or t has sons α wth the property that α s the dsjont unon of the α. Exampes for d = 4 are gven beow: a) b) c) d) The frst nterpretaton s that the tree a) corresponds to V 1 V 2 V 3 V 4, b) to the somorphc space V 1 V 2 ) V 3 V 4 ), c) to V 1 V 2 ) V 3 ) V 4, and d) to V 1 V 2 V 3 ) V 4. The second nterpretaton nvoves the assocated subspaces. The tree a) corresponds to the Tucker format n 3.2: A subspaces U 1,..., U d are joned nto U 1 U 2 U 3 U 4. In the case of tree b) one frst forms the subspaces U 1 U 2 and U 3 U 4 and determnes subspaces U {1,2} U 1 U 2 and U {3,4} U 3 U 4. Fnay U {1,2} U {3,4} s defned. The trees c) and d) ead to anaogous constructons. The fna subspace U D must be such that v U D hods for the tensor v whch we want to represent. Obvousy, the onedmensona subspace U D = span{v} s suffcent. Restrctng ourseves to bnary trees T D, we obtan the herarchca tensor format cases a) c) n 4.1); cf. Hackbusch Kühn [7]). The practca advantage of a bnary tree s the fact that the quanttes appearng n the ater computatons are matrces. The further restrcton to near trees as n case c) of 4.1) eads to the so-caed TT format or matrx product format cf. Verstraete Crac [14], Oseedets Tyrtyshnkov [11, 12]). Consder a vertex α D of the bnary tree T D together wth ts sons α 1 and α 2 : 4 4.1) α U α U α1 U α2 4.2) U α1 α 1 α 2 U α2 7

10 The sons are assocated wth subspaces U α V α = j α V j = 1, 2), whe U α U α1 U α2 V α s the characterstc property of U α. If α = D, v U D s requred. Therefore we can choose U D = span{v}. 4.3) The mnma subspaces U mn α v) ntroduced n 2.7) and 2.8) satsfy 4 the ncuson Ths proves the foowng remark. U mn α v) U mn v) U mn v). α 1 Remark 4.1 The exstence of subspaces U α α T D ) wth the requred propertes s ensured by the optma choce U α = U mn α v). Vce versa, U α U mn α v) hods for a subspaces U α satsfyng v U D und U α U α1 U α Impementaton of the Subspaces In prncpe, a subspaces U α α T D ) are descrbed by bass 5 vectors: α 2 U α = span{b α) : 1 }. However, b α) are aready tensors of order #α whch shoud not be stored expcty. Therefore we dstngush two cases. Case A. α = {j} s a eaf. Then the bass vectors b j) of U α = U j are stored expcty. Case B. α s a non-eaf vertex wth sons α 1 and α 2. Note that {b α1) b α2) j : 1 1, 1 j 2 } s a bass of U α1 U α2. The ncuson U α U α1 U α2 mpes that the bass vector b α) U α must have a representaton 6 b α) =,j wth coeffcents c α,) j formng an 1 2 matrx c α,) j b α1) b α2) j 4.4) C α,) = c α,) j ) 1 ). 4.5) The tupe C α,)) 1 of matrces can be regarded as a tensor C α of order 3 wth entres C α [, j, ] = C α,) j. In the case of α = D, r D = 1 hods. The desred representaton of the tensor v s v = c D) 1 b D) ) Remark 4.2 The representaton of a tensor v by the herarchca format uses the data b j) 1 j d, 1 r j ), C α,) 1, α non-eaf vertex of T D ), and c D) 1. The memory cost of the herarchca format s bounded by dnr + d 1) r 3 + 1, where n := max j dmv j ) and r := max α TD. Athough the representaton of v by the quanttes b j), C α, c D) 1 ) s rather ndrect, a tensor operatons can be performed by a recurson n the tree T D ether from the eaves to the root or n the opposte drecton). Beow we descrbe transformatons, the orthonormasaton of the bases, and the HOSVD computaton. Concernng other operatons we refer to [5, 13]. 4.3 Transformatons We reca that the bases {b α) : 1 } are we-defned by 4.4), but they are not drecty accessbe except for eaves α T D. Transformatons of the bases [ are descrbed] by the correspondng modfcatons of the matrces C α,). As n 3.2 we form matrces B α = b α) 1... b α) reated to a near map n LK rα, V α ). For smpcty we w ca B α the bass of the spanned subspace). 4 Ths statement aso hods for more than two sons. Hence, the constructon 4.2) and Remark 4.1 can be generased to any tree-based format. 5 More generay, the bass vectors may be repaced by any set of spannng vectors. 6 In the case of a genera tree-based format wth δ α sons α 1 δ α) of α, Eq. 4.4) becomes b α) = 1,..., δα c α,) [ 1,..., δα ] δ α b α j ) j cf. 3.1b)). 8

11 α, Β α, Β α C α α, Β 1 α 1 2 α2 The eft fgure ustrates the connecton of the bass B α wth B α1 and B α2 at the son vertces va the data C α. Whenever one of these bases changes, aso C α must be updated. Eq. 4.7) descrbes the update caused by a transformaton of B α, whe 4.8) consders the transformatons of B α1 and B α2. two bases reated by B α = B α S α),.e., b α) k C α,) and C α,) satsfy Bass transformaton n α. Assume that α s not a eaf and that B α and B α are = 1 k ). Then the coeffcent matrces C α,k) = Sα) jk bα) j S α) jk Cα,j) 1 k r α). 4.7) Usng the tensor C α, ths transformaton becomes C α = I I S α) ) T) C α. Bass transformaton n the son vertces α. Let α 1, α 2 be the sons of α. Let B α and B α be two bases reated by B α T α) = B α = 1, 2). The correspondng coeffcent matrces C α,) and C α,) are reated by C α,) = T α1) C α,) T α2) ) T for ) Ths s equvaent to C α = T α1) T α2) I ) C α. 4.4 Orthonormasaton Orthonormaty of the non-accessbe) bases {b α) } can be checked by correspondng propertes of the coeffcent matrces C α,). The foowng suffcent condton s easy to prove. Remark 4.3 Let α be a non-eaf vertex. The bass {b α) } s orthonorma, f a) the bases {b α1) } and } of the sons α 1, α 2 are orthonorma and b) the matrces C α,) n 4.5) are orthonorma wth respect {b α2) j to the Frobenus scaar product: C α,), C α,m) = F j cα,) j c α,m) j = δ m. The bases can be orthonormased as foows. Orthonormase the expcty gven bases at the eaves e.g., by QR). As soon as {b α1) } and {b α2) j } are orthonorma, orthonormase the matrces C α,). The new matrces C new α,) defne a new orthonorma bass {b α),new }. The above mentoned cacuatons requre bass transformatons. Here the foowng has to be taken nto account cf. 4.3 and [5, ]). Case A1. bass {b α1),new remans unchanged. Case A2. If b α2) nto C α,) T T. Let α 1 be the frst son of α. Assume that the bass {b α1) } s transformed nto a new } so that bα1) = k T k b α1) k,new. Changng Cα,) nto C new α,) := T C α,), the bass {b α) } = k T kb α2) k,new s a transformaton of the second son of α, Cα,) must be changed Case B. Consder a non-eaf vertex α. If the bass {b α) } shoud be transformed nto b α) T b α), one has to change the coeffcent matrces C α,) by C new α,) ths transformaton causes changes at the father vertex accordng to Case A1 or Case A2). As n 3.3, the bases are to be orthonormased before the HOSVD bases are computed. 4.5 HOSVD Bases,new := := T C α,). In addton, The chaenge s the computaton of the HOSVD, more precsey of the snguar vaues σ α) and the eft snguar vectors tensors) b α) of M α v). We reca that these data requre the dagonasaton of the square matrx 7 M α v)m α v) H. In the case of the tensor subspace representaton of 3.3 t was possbe to reduce M α v)m α v) H to M α c)m α c) H nvovng the smaer) core tensor. Now we reduce the computaton of M α v)m α v) H to matrx operatons nvovng the data C α. 7 Here we nterpret M αv) as a matrx, not as a tensor. 9

12 The bass B α spans the subspace U α V α = j α V j. The requrement v U D mpes that U mn α v) U α cf. Remark 4.1). Together wth U mn α v) = rangem α v)) cf. 2.7)) we concude that M α v)m α v) H must be of the form wth some coeffcents e α) j X α := M α v)m α v) H =, e α) j whch form an matrx E α = e α) ) rα j,. bα) b α) ) H = B α E α B H α 4.9) To smpfy matters we assume that the bases are aready orthonorma 8 cf. 4.4). We start wth the root α = D of the tree T D. Snce r D = 1, E D = e D) 11 s a scaar. The defnton of M D v) n 2.2 shows that X α = vv H. On the other hand, the equaty v = c D) 1 b D) 1 n 4.6) mpes E D = e D) 11 = c D) 1 2 and σ D) 1 = c D) 1, 4.10) where σ D) 1 s the ony snguar vaue of M D v). Its eft snguar vector s v. The foowng recurson starts wth α = D. We assume that for some non-eaf vertex α T D the snguar vaues σ α) and the matrx E α are known. Now we want to determne E α1 and E α2 for the sons α 1 and α 2 of α. Concernng X α and X α1, we reca the defnton of M α v) by 2.4). The entres of X α are X α [ α, j α ] = M α v)[ α, k α c] M α v)[j α, k α c] = v[ α, k α c)] v[j α, k α c)]. 4.11) k α c I α c k α c I α c On the eft-hand sde, e.g., α I α = j α I j and k α c form the par of matrx ndces, whe α, k α c) I D s the ndex of v. Anaogousy we have X α1 [ α1, j α1 ] = ) α1, k α c 1 ] v[ jα1, k α c 1) ]. k α c 1 I α c 1 v[ The compement of α 1 s α1 c = α c α 2 so that I α c 1 = I α c I α2. Hence the summaton over k α c 1 I α c 1 becomes a doube sum over k α c I α c and k α2 I α2. The sum over k α c I α c aready appears n 4.11) so that X α1 [ α1, j α1 ] = v[ α1, k α2, k α c)] v[j α1, k α2, k α c)] = X α [ α1, k α2 ), j α1, k α2 )]. 4.11) k α2 I α2 k α2 I α2 k α c I α c Returnng to the matrces M α v) and M α1 v), the atter sum can be regarded as a matrx mutpcaton when we nterpret b α) n 4.9) as an 1 2 matrx: X α1 = =, k α2 I α2 X α [, k α2 ),, k α2 )] = e α) j bα) [, k α2 ] b α) [, k α2 ] = 4.4) k α2 I α2, e α) j Snce the bass s orthonorma, we obtan b α1) ν b α2) µ, b α2) κ b α1) ν b α1) λ ) H = δ µκ b α1) ν X α1 =, e α) j b α2) µ νµ ) T bα) b α) ) H c α,) νµ b α1) ν b α1) λ b α2) κ [, k α2 ),, k α2 )] b α2) µ ) T ) b α1) λ ) H δ µκ : Kronecker deta). Hence, e α) j ν,µ,λ c α,) νµ c α,j) λµ b α1) ν λκ c α,j) λκ bα1) λ ) ) T H = b α1) ν b α2) b α1) λ ) H. µ ) T b κ α2) b α2) κ ) T ) H. b α1) λ ) H = 8 The genera case s treated n [5, Theorem 5.14]. There the proof s based on the tensor nterpretaton of the quanttes, whereas here we use the somorphc matrx nterpretaton. 10

13 Ths proves that 4.9) hods fo 1 nstead of α wth coeffcents e α1) νλ =, eα) j λ cα,) νµ c α,) λµ formng the matrx E α1 =, eα) j Cα,) C α,j) ) H cf. 4.5)). A smar treatment of X α2 proves the foowng theorem. Theorem 4.4 The matrces E α, E α1, E α2 E α1 =, are connected by e α) j Cα,) C α,j) ) H, E α2 =, e α) j Cα,) ) T C α,j). As n 3.3 the HOSVD bases {b α),hosvd } s defned by the dagonasaton of X α := M α v)m α v) H = σα) ) 2 b α),hosvd bα),hosvd )H. A comparson wth 4.9) shows that X α s dagonased f and ony f E α = dag{σ α) 1 ) 2, σ α) 2 ) 2,...}. 4.12) Snce r D = 1 at the root α = D, we have σ D) 1 = c D) 1 cf. 4.10)) and b D) 1 = b D) 1,HOSVD. Assume that the HOSVD bass {b α),hosvd } s aready chosen for the representaton we reca that the defnton of bα),hosvd s mpcty gven by the coeffcent matrces C α,) ). Combnng 4.12) wth Theorem 4.4 we obtan E α1 = =1 σ α) ) 2 C α,) C α,) ) H, E α2 = Dagonasaton of the expcty gven matrces E α1 and E α2 yeds =1 E α1 = UΣ 2 α 1 U H, E α2 = V Σ 2 α 2 V H σ α) ) 2 C α,) ) T C α,). wth orthogona matrces U, V and dagona matrces Σ α = dag{σ α) 1,...}. Snce aso B α s orthogona.e., B H α B α = I), the dagonasaton s gven by X α1 = B α1 U)Σ 2 α 1 B α1 U) H, X α2 = B α2 V )Σ 2 α 2 B α2 V ) H. Hence B HOSVD α 1 = B α1 U and B HOSVD α 2 = B α2 V are the desred HOSVD bases at the vertces α 1 and α 2. If α s a eaf, ths transformaton s performed expcty. Otherwse the coeffcent matrces are modfed accordng to 4.3. The procedure s repeated for the sons of α 1, α 2 unt we reach the eaves. Then at a vertces HOSVD bases are ntroduced together wth snguar vaues σ ν α). If there are vanshng snguar vaues σ α), the correspondng contrbutons can be omtted. Ths reduces the assocated subspace U α cf. 4.2)) to the mnma subspace U mn α v). Correspondngy the vaue of becomes rank α v). 4.6 HOSVD Truncaton We assume that foowng the procedure descrbed above the herarchca representaton uses the HOSVD bases. The format H r wth r = ) α TD conssts of a tensors v V wth rank α v). Gven v H r we ask for an approxmaton u H s for a tupe s wth s r. The HOSVD truncaton s smar to the procedure n 3.4. In terms of the mpcty defned) bases the approxmaton u HOSVD s obtaned by omttng a contrbutons nvovng the HOSVD bass vectors b α) for s α <. In practce ths means that the coeffcent matrx C α,) ) sα< are omtted, whe the remanng 1 2 matrces C α,) are reduced to sze s α1 s α2 by deetng the ast 1 s α1 rows and 2 s α2 coumns. If α = {j} s a eaf, the expcty gven bass {b j),hosvd : 1 r j} s repaced by {b j),hosvd : 1 s j}. The approxmaton error v u HOSVD satsfes cf. [5, Theorem 11.58]): v v HOSVD α ν s α+1 σ α) ν ) 2 2d 3 v v best. 4.13) The frst nequaty aows us to expcty contro the error wth respect to the Eucdean norm by the choce of the omtted snguar vaues. The second nequaty proves quas-optmaty of ths truncaton. u best H s s the best approxmaton. The parameter d s the order of the tensor. 11

14 The number 2d 3 on the rght-hand sde becomes smaer f s α = hods for some vertces α. For nstance, the TT format as descrbed n [11] uses the maxma vaue s j = r j = dmv j ) for the eaves. Then 4.13) hods wth d 1 nstead of 2d 3. 5 Propertes of the SVD Projecton 5.1 Case of the Tensor-Subspace Format The HOSVD truncaton of the tensor-subspace format n 3.4 s the Kronecker product Π := d P j, where P j : V j span{b j) : 1 s j } wth s j < r j s the orthogona projecton. Π s agan an orthogona projecton onto d span{bj) : 1 s j }. The tensor product of the snge projectons P j can aso be wrtten as a usua product d P j of P j := I [j] P j wth I [j] := k j I j, 5.1) where I j s the dentty map on V j. Snce the projectons P j commute, the order of the factors n d P j does not matter. We reca the snguar-vaue decomposton of the matrcsaton M j v) cf. 2.4): M j v) = r j =1 σ j) b j) b [j] ) T, where the superscrpt [j] = {j} c denotes the compement of the eaf α = {j}. Usng 2.5b), we get M j P j v) = P j M j v) = r j =1 σ j) P j b j) ) b [j] ) T = s j =1 σ j) b j) b [j] ) T. However, we may aso defne ˆP j := P [j] I j, 5.2) where P [j] s the orthogona projecton of V onto span{b [j] M j ˆP j v) = M j v)p T [j] = r j =1 σ j) : 1 s j }. Snce b j) P [j] b [j] ) T = s j =1 σ j) b j) b [j] ) T, we obtan the dentca vaue P j v = ˆP j v athough the projectons are dfferent. Ths property has nterestng consequences. We ntroduce ) Π j := ˆP j k j P k and observe that Π j v = k j P k)ˆp j v = k j P k)p j v =Πv hods for the speca tensor v athough Π j Π. Note that a maps P k and ˆP j are eementary tensors contanng the dentty I j : V j V j wth respect to the j-th drecton. Ths proves the next emma for whch we ntroduce the foowng notaton. Let ϕ j : V j W j be a near map. It gves rse to the eementary Kronecker product φ j := I 1... I j 1 ϕ j I j+1... I d. Lemma 5.1 Let ϕ j : V j W j and φ j as above. Then φ j Π j = Π j φ j hods the atter Π j contans the dentty I j : W j W j nstead of I j : V j V j ). Ths aows the foowng estmate wth respect to the Eucdean norm. Concuson 5.2 Gven v V, et u HOSVD T s be the HOSVD approxmaton defned n 3.4. Wth φ j from above we have φ j u HOSVD φ j v. 5.3) 12

15 Proof. u HOSVD = Π j v shows that φ j u HOSVD = φ j Π j v = Π j φ j v. Snce Π j s a product of orthogona projecton, Π j φ j v φ j v foows. In the case of nfnte-dmensona Hbert spaces V we may consder unbounded near maps φ j. The subspace of eements v for whch φ j v s defned, s caed the doman of φ j. Concuson 5.3 If v V beongs to the doman of φ j, then aso u HOSVD beongs to the doman and satsfes 5.3). An mportant exampe s the topoogca tensor space V = L 2 Ω) = d k=1 L2 Ω j ), where Ω s the Cartesan product of the Ω j. Set φ j = k / x k j. If the functon v V possesses a k-th dervatve wth respect to x j, then by Concuson 5.3 aso u HOSVD s k-tmes dfferentabe n the L 2 sense and satsfes k u HOSVD / x k j L 2 k v/ x k j L 2. Assumng suffcent smoothness of v and usng the Gagardo Nrenberg nequaty, we proved n [6] estmates of v u HOSVD wth respect to the maxmum norm by means of the L 2 norm of v u HOSVD. Ths s mportant when we evauate the functon pontwse. Another trva concuson from 5.3) s that φ j v = 0 mpes φ j u HOSVD = 0. For nstance, et ϕ j V j be a functona on V j.e., W j = K). Exampes of ϕ j are the mean vaue ϕ j u) = 1 T u or a zero at a certan ndex : ϕ j u) = u = 0. We say that v satsfes the sde condton ϕ j f φ j v = 0. We concude that u HOSVD satsfes the same sde condton. In the case of ϕ j u) = 1 T u, aso u HOSVD has a vanshng mean. If ϕ j u) = u, u HOSVD [] = 0 hods for wth j =. In the case of matrx spaces V j, structura propertes ke symmetry or sparsty can be descrbed by functonas. One concudes that the HOSVD approxmatons ead to the same matrx structures. 5.2 Best Approxmaton u best We reca that the HOSVD approxmaton u HOSVD T r of v V s not the necessary) the best approxmaton defned by u best T r wth v u best = nf{ v u : u T r } 5.4) cf. Defnton 3.1). Nevertheess, u best has smar propertes as u HOSVD. Defne U k := Uk mn u best ) for 1 k d. Let P k : V k U k be the orthogona projecton onto U k. Based on these projectons we defne P k and Π as n 5.1. Now we fx one ndex j and defne Π [j] := k j P k. Set v j := Π [j] v U 1... U j 1 V j U j+1... U d and note that P j v j = u best. Based on the SVD of M j v j ) we can determned ts HOSVD approxmaton u HOSVD T r. Snce t s the mnmser of mn u Tr v j u, we have v j u HOSVD v j u best. For an ndrect proof assume that v j u HOSVD < v j u best. Both u HOSVD and u best are n the range of Π [j],.e., I Π[j] ) uhosvd = I Π [j] ) ubest = 0. Pythagoras equaty yeds v u HOSVD 2 = Π[j] v u HOSVD ) 2 + I Π [j] ) v uhosvd ) 2 = v j u HOSVD 2 + I Π [j] ) v 2 < vj u best 2 + I Π [j] ) v 2 = Π [j] v u best ) 2 + I Π [j] ) v ubest ) 2 = v u best 2 n contradcton to the optmaty of u best. Hence, v u HOSVD = v u best must hod. Dependng on the mutpcty of certan snguar vaues, the SVD approxmaton may be unque. In ths case u HOSVD = u best hods. If the SVD approxmaton s not unque, we may choose u best as u HOSVD = P j v j. Knowng that P j s a SVD projecton, we may repace P j by ˆP j as defned n 5.2). The projecton Π j := Π [j] ˆP j has the same propertes as Π j n 5.1. Ths proves the foowng cf. Uschmajew [13]). Theorem 5.4 The statements of Lemma 5.1 and the Concusons 5.2 and 5.3 aso hod for the best approxmaton u best n 5.4) and the reated mappng Π j. 13

16 5.3 Case of the Herarchca Format The HOSVD truncaton wthn the herarchca format cf. 4.6) can be expressed by orthogona projectons P α for a vertces α of the tree T D. However, dfferent from 5.1, projectons P α and P β commute f and ony α β =. The truncaton s descrbed by the product Π := α T D P α, where the factors are ordered n such a way that P α s apped before P α1 and P α2 α 1, α 2 sons of α) foow. Because of these restrctons, the anayss s more nvoved. We refer the reader to Hackbusch [6, 4]. As a resut the statements n 5.1 aso hod for the herarchca format. Acknowedgements. The author kes to thank Antono Facó for frutfu dscussons durng a vst at the Unversdad Cardena Herrera-CEU. References [1] De Lathauwer, L., De Moor, B.L.R., Vandewae, J.: A mutnear snguar vaue decomposton. SIAM J. Matrx Ana. App. 21, ) [2] Facó, A., Hackbusch, W.: On mnma subspaces n tensor representatons. Found. Comput. Math. 12, ) [3] Facó, A., Hackbusch, W., Nouy, A.: Tree-based tensor formats. Same ssue of SEMA 2018) [4] Greub, W.H.: Mutnear Agebra, 2nd ed. Sprnger, New York 1978) [5] Hackbusch, W.: Tensor Spaces and Numerca Tensor Cacuus, SSCM, vo. 42. Sprnger, Bern 2012) [6] Hackbusch, W.: L estmaton of tensor truncatons. Numer. Math. 125, ) [7] Hackbusch, W., Kühn, S.: A new scheme for the tensor representaton. J. Fourer Ana. App. 15, ) [8] Hackbusch, W., Uschmajew, A.: On the nterconnecton between the hgher-order snguar vaues of rea tensors. Numer. Math. 135, ) [9] Håstad, J.: Tensor rank s NP-compete. J. Agorthms 11, ) [10] Htchcock, F.L.: The expresson of a tensor or a poyadc as a sum of products. Journa of Mathematcs and Physcs 6, ) [11] Oseedets, I.V.: Tensor-tran decomposton. SIAM J. Sc. Comput. 33, ) [12] Oseedets, I.V., Tyrtyshnkov, E.E.: TT-cross approxmaton for mutdmensona arrays. Lnear Agebra App. 432, ) [13] Uschmajew, A.: Reguarty of tensor product approxmatons to square ntegrabe functons. Constr. Approx. 34, ) [14] Verstraete, F., Crac, J.I.: Matrx product states represent ground states fathfuy. Phys. Rev. B 73, 094, ) 14