Computational Linear Algebra

Transcription

1 Computational Linear Algebra PD Dr. rer. nat. habil. Ralf-Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2018/19

2 Part 6: Some Other Stuff PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 2

3 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 3

4 Motivation something about tensors tensors are multidimensional arrays of numerical values and therefore generalise matrices to higher dimensions first emerged in psychometrics community and since then spread to numerous other disciplines (e.g. statistics, data science, machine learning) not to be confused with tensor (fields) in physics and engineering typical applications compression of image / video data dimensional reduction classification provide new insight into data (e.g. complex latent relationships) deep learning Richard W.HAMMING source: scihi.org The purpose of computation is insight, not numbers. PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 4

5 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 5

6 Notations tensor order number of dimensions (also referred to as ways or modes) hence, designators for tensor of order one (cf. vector) x tensor of order two (cf. matrix) X tensor of order three or higher X tensor entries ith entry of a vector x x i element (i, j) of a matrix X x ij element (i, j, k) of a third-order tensor X x ijk general: element (i 1, i 2,, i N ) of an N-mode tensor X x, where indices typically range from 1 to their capital version, e.g. i k = 1,, I k PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 6

7 Notations tensor norm norm of a tensor X to be computed as which is analogous to matrix FROBENIUS norm inner product, of two same-sized tensors X, Y to be computed as from which follows immediately sequences nth element in a sequence denoted by a superscript in parentheses hence, nth tensor in a sequence X (n) PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 7

8 Notations (some more ) tensor fibres higher-order analogue of matrix rows and columns defined by fixing every index but one example third-order tensor X with column, row, and tube fibres denoted as x :jk, x i:k, and x ij:, resp. mode-1 fibres x :jk mode-2 fibres x i:k mode-3 fibres x ij: PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 8

9 Notations (some more ) tensor slices two-dimensional sections of a tensor defined by fixing all but two indices example third-order tensor X with horizontal, lateral, and frontal slices denoted as X i::, X :j:, and X ::k, resp. (or more compactly as X i, X j, or X k representing ith horizontal, jth lateral, or kth frontal slice, resp.) horizontal slices X i:: lateral slices X :j: frontal slices X ::k PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 9

10 Notations (some more ) rank-one tensor an N-way tensor X is of rank one if it can be written as X = x (1) x (2) x (N), i.e. as outer product ( ) of N vectors x (i) x (3) = x (2) X x (1) example: rank-one third-order tensor X = x (1) x (2) x (3) hence, each element of rank-one tensor X to be computed as for all 1 i n I N PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 10

11 Notations (and even more ) matricisation (a.k.a. unfolding or flattening) process of reordering the elements of an N-way tensor into a matrix special case: mode-n matricisation of tensor X denoted as X (n) rearrangement of mode-n fibres as columns of the resulting matrix formal notation tensor element (i 1, i 2,, i N ) maps to matrix element (i n, j), where with PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 11

12 Notations (and even more ) matricisation (cont d) example: let X be given with frontal slices X 1 = X 2 = mode-1 unfolding of X X (1) = PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 12

13 Notations (and even more ) matricisation (cont d) example: let X be given with frontal slices X 1 = X 2 = mode-2 unfolding of X X (2) = PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 13

14 Notations (and even more ) matricisation (cont d) example: let X be given with frontal slices X 1 = X 2 = mode-3 unfolding of X X (3) = PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 14

15 Notations (nearly done ) matrix products KRONECKER product of matrices A and B is denoted A B resulting matrix of size (IK) (JL) is defined by a 11 B a 12 B a 1J B A B = a 12 B a 22 B a 2J B a I1 B a I2 B a IJ B = [ a 1 b 1 a 1 b 2 a 1 b 3... a J b L 1 a J b L ] where a i and b j denote the ith and jth, resp., column of A and B PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 15

16 Notations (nearly done ) matrix products (cont d) KATHRI-RAO product (the matching columnwise KRONECKER product) of matrices A and B is denoted as A B resulting matrix of size (IJ) K is defined by A B = [ a 1 b 1 a 2 b 2... a K b K ] HADAMARD product of matrices A and B, both of size I J, is denoted A B resulting matrix also of size I J is defined by a 11 b 11 a 12 b 12 a 1Jb 1J A B = a 12 b 12 a 22 b 22 a 2J b 2J a I1 b I1 a I2 b I2 a IJ b IJ PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 16

17 Notations (nearly done ) matrix products (cont d) these products have useful properties necessary for further discussion (A B)(C D) = AC BD (A B) + = A + B + A B C = (A B) C = A (B C) (A B) T (A B) = A T A B T B (A B) + = (A T A B T B) + (A B) T MOORE-PENROSE inverse For matrix A, a pseudoinverse A + must satisfy following conditions: 1) AA + A = A 2) A + AA + = A + 3) (AA + ) = AA + 4) (A + A) = A + A PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 17

18 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 18

19 CANDECOMP / PARAFAC Decomposition prelude idea: polyadic form of a tensor, i.e. expressing a tensor as finite sum of rank-one tensors c 2 c c R b 1 b 2 b R X a 1 a 2 a R example: decomposition of a third-order tensor X concept proposed in 1970 as CANDECOMP (canonical decomposition) and PARAFAC (parallel factors); further referred to as CP PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 19

20 CANDECOMP / PARAFAC Decomposition prelude (cont d) let third-order tensor X be given then, CP should decompose X such that (1) where R denotes a positive integer and a r, b r, and c r factor matrices refer to the combination of vectors from the rank-one components, i.e. A = [a 1 a 2... a R ], B = [b 1 b 2... b R ], and C = [c 1 c 2... c R ] hence, (1) can be rewritten in matricised from X (1) A(C B) T X (2) B(C A) T X (3) C(B A) T PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 20

21 CANDECOMP / PARAFAC Decomposition prelude (cont d) summarising, CP decomposition to be expressed as often useful to normalise columns of A, B, and C to length one weights to be absorbed into vector λ so that for general N-mode tensor X the CP decomposition is where λ and A (n) PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 21

22 CANDECOMP / PARAFAC Decomposition tensor rank definition: smallest number of rank-one tensors that generate X as their sum, denoted as rank(x) in other words, rank(x) defines the smallest number of components (of factor matrices A (n) ) in an exact CP decomposition with R = rank(x), i.e. an exact CP decomposition is also called rank decomposition problem: no straightforward algorithm to determine rank(x) (NP-hard!) for general three-way tensors X some weak upper bound on its maximum rank exists rank(x) min{ij, IK, JK} PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 22

23 CANDECOMP / PARAFAC Decomposition (something about) low-rank approximations & border rank let A be a matrix, R its rank, and an SVD be given by with σ 1 σ 2 σ r > 0 a rank-k approximation then minimises with unfortunately, this result does not hold true for higher-order tensors (e.g. best rank-k approximation of a third-order tensor of rank R by summing k of its factors) problem of degeneracy: a tensor is called degenerate if it may be approximated arbitrary well by a factorisation of lower rank concept of border rank: minimum number of rank-one tensors that sufficiently approximate given tensor with arbitrarily small nonzero error PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 23

24 CANDECOMP / PARAFAC Decomposition CP decomposition again, there is no (finite) algorithm for determining rank(x) most procedures fit multiple CP decompositions with different numbers of components until one is sufficient hence, if data are noise-free (which is the rare case) and some procedure for fixed number of components is given compute for R = 1, 2, 3, components until one gives a perfect (100%) fit for further consideration, assume number of components is fixed standard algorithm for CP: alternating least squares (ALS) method let X, then compute CP decomposition with R components that approximates X best, i.e. find with PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 24

25 CANDECOMP / PARAFAC Decomposition CP decomposition (cont d) idea: repeat fix B and C solve for A fix A and C solve for B fix A and B solve for C until convergence with all but one matrix fixed reduces to linear least-squares problems for instance (with B and C fixed) with hence, optimal solution given by which can be rewritten / simplified to (i.e. computing pseudoinverse on R R matrix rather than JK R matrix) finally, normalise columns of  to obtain A, i.e. let and PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 25

26 CANDECOMP / PARAFAC Decomposition CP decomposition (cont d) full ALS procedure for an N-way tensor initialise all A (n) repeat for n = 1,, N do V A (1)T A (1) A (n 1)T A (n 1) A (n+1)t A (n+1) A (N)T A (N) A (n) X (n) (A (N) A (n+1) A (n 1) A (1) )V + normalise columns of A (n) (storing norms as λ) od until fit ceases to improve maximum iteration exhausted factor matrices to be initialised in any way, e.g. randomly or by setting all A (n) = R leading left singular vectors of X (n) cons: number of components must a priori be specified, convergence can take many iterations, convergence to global minimum not guaranteed PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 26

27 CANDECOMP / PARAFAC Decomposition (simple) example assume there are two diseases A and B which might (not necessarily) evoke three different indications: fever, nausea, and headache evocation of any symptom is independent of the other two doctors have medicine M A and M B for both diseases A and B, resp. but medicine M A and M B must never applied together medicine M A is ineffective for disease B and vice versa at first sight, doctors can prescribe by chance medicine M A or M B and, thus, help the patient with a 50:50 opportunity as some of the doctors know tensors, they collect data of 400 patients and put the results together in a tensor X PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 27

28 CANDECOMP / PARAFAC Decomposition (simple) example (cont d) collected results fever no fever headache no head. headache no head. nausea nausea no nausea no nausea to be written as tensor (using frontal slices) X 1 = X 2 = does this tensor help doctors to prescribe the right medicine if a) specific symptoms of a patient are known or b) are not known? PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 28

29 CANDECOMP / PARAFAC Decomposition (simple) example (cont d) decomposition leads to two rank-one tensors X = a 1 b 1 c 1 a 2 b 2 c 2 decomposition supposes 300 patients suffer from disease A and only 100 patients from disease B hence, without knowing specific symptoms doctors should decide for first disease (with a 3:1 opportunity) if patients suffers, for instance, from nausea w/o fever and w/o headache (according data collection 31 persons), only = 3 suffer from first disease hence, doctors should prescribe medicine for second disease observation: R = 2 (two diseases) with three random variables (symptoms) PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 29

30 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 30

31 Tucker Decomposition prelude first introduced by TUCKER in 1963 decomposition is form of high-order principle component analysis (PCA), i.e. decomposes tensor into core tensor and one matrix along each mode hence, for a third-order tensor X the decomposition yields with (usually orthogonal) factor matrices A, B, and C that can be thought of as principal components in each mode and tensor G called the core tensor elementwise, the TUCKER decomposition is for i = 1,, I, j = 1,, J, k = 1,, K PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 31

32 Tucker Decomposition prelude (cont d) in case P, Q, R are smaller than I, J, K, the core tensor G can be thought of as a compressed version of X C X A G B example: TUCKER decomposition of three-way tensor matricised forms of TUCKER decomposition are X (1) AG (1) (C B) T X (2) BG (2) (C A) T X (3) CG (3) (B A) T PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 32

33 Tucker Decomposition computation first approach known as higher-order SVD (HOSVD) idea: find those components that best capture the variation in mode n for n = 1,, N do A (n) R n leading left singular vectors of X (n) do G X 1 A (1)T 2 A (2)T N A (N)T where R n = rank n (X) for n = 1,, N is the column rank of X (n), i.e. the dimension of the vector space spanned by the mode-n fibres not to be confused with rank, i.e. the minimum number of rank-one components based on HOSVD (used for initialisation only) we can now construct a more efficient technique called higher-order orthogonal iteration (HOOI) PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 33

34 Tucker Decomposition computation (cont d) let X, then find with G, all A (n) and columnwise orthogonal HOOI method (ALS method for TUCKER decomp. of Nth-order tensor) initialise all A (n) using HOSVD repeat for n = 1,, N do Y X 1 A (1)T n 1 A (n 1)T n+1 A (n+1)t N A (N)T A (n) R n leading left singular vectors of Y (n) od until fit ceases to improve maximum iteration exhausted G X 1 A (1)T 2 A (2)T N A (N)T PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 34

35 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 35

36 Other Decomposition INDSCAL individual difference in scaling special case of CP for three-way tensors that are symmetric in two modes CANDELINC canonical decomposition with linear constraints including domain/user knowledge for CP with linear constraints on one or more factor matrices DEDICOM decomposition into directional objects for I objects and asymmetric relations according to X, the algorithm groups the objects into R latent components and describes their interaction via X ARA T nonnegative tensor factorisations for analysis of nonnegative data (such as environmental models, greyscale images) in order to retain nonnegative characteristics of original data PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 36

37 overview motivation (some) notations CANDECOMP / PARAFAC decomposition tucker decomposition other decompositions PD Dr. Ralf-Peter Mundani Computational Linear Algebra Winter Term 2018/19 37