The Fréchet derivative of a generalized matrix function. Network Science meets Matrix Functions Univerity of Oxford, 1-2/9/16

Transcription

1 The Fréchet derivative of a generalized matrix function Vanni Noferini (University of Essex) Network Science meets Matrix Functions Univerity of Oxford, 1-2/9/16 September 1st, 2016 Vanni Noferini The Fréchet derivative of a generalized matrix function 1 / 33

2 Classical matrix functions Classical definition in linear algebra: given f : C C and a square matrix A C n n we wish to define f (A) in a way that mimics the property of f. Vanni Noferini The Fréchet derivative of a generalized matrix function 2 / 33

3 Classical matrix functions Classical definition in linear algebra: given f : C C and a square matrix A C n n we wish to define f (A) in a way that mimics the property of f. For example: A 1/2 should solve X 2 = A e ta should integrate y(t) = Ay(t), y(0) = v.... Vanni Noferini The Fréchet derivative of a generalized matrix function 2 / 33

4 Defining matrix functions from the eigendecomposition From Linear Algebra 1: A Jordan block is a bidiagonal Toeplitz matrix with unit superdiagonal: λ λ λ λ A matrix is in Jordan canonical form if it is the direct sum of Jordan blocks: λ λ λ µ Vanni Noferini The Fréchet derivative of a generalized matrix function 3 / 33

5 Defining matrix functions from the eigendecomposition Rules: The function of a Jordan block is an upper triangular Toeplitz matrix: λ f (λ) f (λ) f (λ)/2! f (λ)/3! f 0 λ λ 1 = 0 f (λ) f (λ) f (λ)/2! 0 0 f (λ) f (λ) λ f (λ) f (A 1 A 2 ) = f (A 1 ) f (A 2 ) f (ZAZ 1 ) = Zf (A)Z 1 Vanni Noferini The Fréchet derivative of a generalized matrix function 4 / 33

6 Defining matrix functions from the eigendecomposition Rules: The function of a Jordan block is an upper triangular Toeplitz matrix: λ f (λ) f (λ) f (λ)/2! f (λ)/3! f 0 λ λ 1 = 0 f (λ) f (λ) f (λ)/2! 0 0 f (λ) f (λ) λ f (λ) f (A 1 A 2 ) = f (A 1 ) f (A 2 ) f (ZAZ 1 ) = Zf (A)Z 1 Hence f (A) is defined as long as f is defined and sufficiently many times differentiable on the spectrum of A. Vanni Noferini The Fréchet derivative of a generalized matrix function 4 / 33

7 Computing matrix functions On a computer, it is important to know the sensitivity of the attempetd computation. Vanni Noferini The Fréchet derivative of a generalized matrix function 5 / 33

8 Computing matrix functions On a computer, it is important to know the sensitivity of the attempetd computation. A backward stable algorithm computes the exact solution of a nearby problem: f (A) = f (A + E). Vanni Noferini The Fréchet derivative of a generalized matrix function 5 / 33

9 Computing matrix functions On a computer, it is important to know the sensitivity of the attempetd computation. A backward stable algorithm computes the exact solution of a nearby problem: f (A) = f (A + E). Whether this is close to f (A) or not depends on the sensitivity of the problem and, intuitively, on the first derivative of f at A. For a scalar, analytic f : f (A + E) = f (A) + f (A)E + O(E 2 ). Vanni Noferini The Fréchet derivative of a generalized matrix function 5 / 33

10 Derivatives in Banach spaces Let X, Y be Banach spaces over R, x X and f : X Y. Vanni Noferini The Fréchet derivative of a generalized matrix function 6 / 33

11 Derivatives in Banach spaces Let X, Y be Banach spaces over R, x X and If the limit f : X Y. f (x + te) f (x) lim t 0 t exists for all e X then f is Gâteaux differentiable at x. Vanni Noferini The Fréchet derivative of a generalized matrix function 6 / 33

12 Derivatives in Banach spaces Let X, Y be Banach spaces over R, x X and If the limit f : X Y. f (x + te) f (x) lim t 0 t exists for all e X then f is Gâteaux differentiable at x. If there exists a bounded R-linear map L f (x, ) s.t. f (x + h) f (x) L f (x, h) Y lim h X 0 h X exists for all h X then f is Fréchet differentiable at x. Vanni Noferini The Fréchet derivative of a generalized matrix function 6 / 33

13 Derivatives in Banach spaces Vanni Noferini The Fréchet derivative of a generalized matrix function 7 / 33

14 Derivatives in Banach spaces Fréchet differentiable Gâteaux differentiable, but Vanni Noferini The Fréchet derivative of a generalized matrix function 7 / 33

15 Derivatives in Banach spaces Fréchet differentiable Gâteaux differentiable, but If f is Fréchet differentiable then cond abs (f, A) = L f (A, ) Vanni Noferini The Fréchet derivative of a generalized matrix function 7 / 33

16 The Fréchet derivative of a classical matrix function When A is diagonalizable, an explicit formula is known. Vanni Noferini The Fréchet derivative of a generalized matrix function 8 / 33

17 The Fréchet derivative of a classical matrix function When A is diagonalizable, an explicit formula is known. Theorem (Daleckǐi and Kreǐn 1965) Let A = ZDZ 1 and f differentiable on the spectrum of A. Then L f (A, E) = Z(F (Z 1 EZ))Z 1 with F ij = f (D ii ) f (D jj ) D ii Djj if D ii D jj or F ij = f (D ii ) otherwise. Vanni Noferini The Fréchet derivative of a generalized matrix function 8 / 33

18 Example Take Then A = 0 2 0, f (A) = e A, E = e 2 e 2 sinh(2) e 2 2e 2 sinh(2) F = e 2 e 2 sinh(2) L f (A, E) = sinh(2) sinh(2) e e 2 Vanni Noferini The Fréchet derivative of a generalized matrix function 9 / 33

19 Consequences The eigenvector matrix Z appears in the Daleckǐi-Kreǐn theorem. This leads to issues caused by non-normality of the argument matrix. Vanni Noferini The Fréchet derivative of a generalized matrix function 10 / 33

20 Consequences The eigenvector matrix Z appears in the Daleckǐi-Kreǐn theorem. This leads to issues caused by non-normality of the argument matrix. Theorem (N. Higham 2008) cond abs (f, A) κ 2 (Z) 2 F. Vanni Noferini The Fréchet derivative of a generalized matrix function 10 / 33

21 Generalized matrix functions Given A C m n with SVD A = USV and a function f : [0, ) R we wish to define f (A) in some sensible way. Vanni Noferini The Fréchet derivative of a generalized matrix function 11 / 33

22 Gmf: definition From Linear Algebra 2: In an SVD A = USV, U, V are unitary and S is diagonal with nonincreasing nonnegative elements. This leads to the CSVD A = U r S r Vr where U r, V r are submatrices of U, V and S r is square invertible (empty if A = 0). Vanni Noferini The Fréchet derivative of a generalized matrix function 12 / 33

23 Gmf: definition Definition (Hawkins and Ben-Israel 1973): Equivalent definition: A = U r S r V r f (A) = U r f (S r )V r. A = USV f (A) = Uf (S)V, where f (S) is diagonal with diagonal entries { f f (σ i ) if σ i 0 (σ i ) = 0 if σ i = 0 Vanni Noferini The Fréchet derivative of a generalized matrix function 13 / 33

24 Why do we care? Among the applications: complex networks (Arrigo, Benzi, Estrada, Fenu, D. Higham, Klymko... ); computing classical matrix functions of structured matrices (Del Buono, Lopez, Politi... ); computer vision (Bylow, Kahl, Larsson, Olsson... ); the unitary factor of the polar decompisition of a full rank matrix is a gmf; the Moore-Penrose pseudoinverse of A is a gmf of A. Vanni Noferini The Fréchet derivative of a generalized matrix function 14 / 33

25 Why do we care? Among the applications: complex networks (Arrigo, Benzi, Estrada, Fenu, D. Higham, Klymko... ); computing classical matrix functions of structured matrices (Del Buono, Lopez, Politi... ); computer vision (Bylow, Kahl, Larsson, Olsson... ); the unitary factor of the polar decompisition of a full rank matrix is a gmf; the Moore-Penrose pseudoinverse of A is a gmf of A. (So they are familiar to at least some people in the room...) Vanni Noferini The Fréchet derivative of a generalized matrix function 14 / 33

26 Example Toy matrix: A = Vanni Noferini The Fréchet derivative of a generalized matrix function 15 / 33

27 Example Toy matrix: for f (σ) = σ 3, A = f (A) = Vanni Noferini The Fréchet derivative of a generalized matrix function 15 / 33

28 Example Toy matrix: for f (σ) = σ 3, for f (σ) = e σ, Note that e A = f (A) = e f (A) = 0 e Vanni Noferini The Fréchet derivative of a generalized matrix function 15 / 33

29 Basic observations In spite of their name, gmf do not reduce to classical mf when m = n; Vanni Noferini The Fréchet derivative of a generalized matrix function 16 / 33

30 Basic observations In spite of their name, gmf do not reduce to classical mf when m = n; Even when m = n, f (A) for a polynomial function f is not a polynomial in A in the classical sense; Vanni Noferini The Fréchet derivative of a generalized matrix function 16 / 33

31 Basic observations In spite of their name, gmf do not reduce to classical mf when m = n; Even when m = n, f (A) for a polynomial function f is not a polynomial in A in the classical sense; If f (0) 0 and A is rank deficient then f (A) is not continuous let alone differentiable at A. Vanni Noferini The Fréchet derivative of a generalized matrix function 16 / 33

32 Basic observations In spite of their name, gmf do not reduce to classical mf when m = n; Even when m = n, f (A) for a polynomial function f is not a polynomial in A in the classical sense; If f (0) 0 and A is rank deficient then f (A) is not continuous let alone differentiable at A. If A = QH is a polar decomposition then f (A) = Qf (H). Vanni Noferini The Fréchet derivative of a generalized matrix function 16 / 33

33 Basic observations In spite of their name, gmf do not reduce to classical mf when m = n; Even when m = n, f (A) for a polynomial function f is not a polynomial in A in the classical sense; If f (0) 0 and A is rank deficient then f (A) is not continuous let alone differentiable at A. If A = QH is a polar decomposition then f (A) = Qf (H). If H is Hpd then f (H) = f (H); if H is Hpsd this is not generally true unless f (0) = 0. Vanni Noferini The Fréchet derivative of a generalized matrix function 16 / 33

34 Intuitions on computational advantage Although you do not compute classical mf via the (unstable?) eigendecomposition, their definition may lead to numerical ill conditioning. Vanni Noferini The Fréchet derivative of a generalized matrix function 17 / 33

35 Intuitions on computational advantage Although you do not compute classical mf via the (unstable?) eigendecomposition, their definition may lead to numerical ill conditioning. Gmf are defined via the (stable) SVD. Does this imply that they are always well conditioned? Vanni Noferini The Fréchet derivative of a generalized matrix function 17 / 33

36 Gmf: conditions on differentiability Theorem Let A C m n and f : [0, ) R differentiable on an open subset containing the positive singular value of A. Moreover, if ranka < min(m, n), suppose that f (0) = 0 and f is right differentiable at 0. Then f (X ) is Fréchet differentiable at X = A. Vanni Noferini The Fréchet derivative of a generalized matrix function 18 / 33

37 Gmf: conditions on differentiability Theorem Let A C m n and f : [0, ) R differentiable on an open subset containing the positive singular value of A. Moreover, if ranka < min(m, n), suppose that f (0) = 0 and f is right differentiable at 0. Then f (X ) is Fréchet differentiable at X = A. Important: in this theorem, the Fréchet derivative is the real Fréchet derivative. Generally, generalized matrix functions are not complex differentiable. Vanni Noferini The Fréchet derivative of a generalized matrix function 18 / 33

38 Some notation... Given X C m n, Υ : C m n C m n is the following real-linear operator: [ X 1 X 2 ] Υ (X ) = X if m = n; if m > n and X = [ X 1 X 2 ] ; ] [X 1 X 2 if m < n and X = [X 1 X 2 ]. Vanni Noferini The Fréchet derivative of a generalized matrix function 19 / 33

39 Notation again... Given the singular values of A C m n we introduce two operators F, G R m n : Vanni Noferini The Fréchet derivative of a generalized matrix function 20 / 33

40 Notation again... Given the singular values of A C m n we introduce two operators F, G R m n : σ i f (σ i ) σ j f (σ j ) σi 2 σ2 j if i j, max(i, j) ν, and σ i σ j ; σ i f (σ i )+f (σ i ) 2σ i if i j, max(i, j) ν, and σ i = σ j 0; F ij = f (σ j ) σ j if i > n, and σ j 0; f (σ i ) σ i if j > m, and σ i 0; f (σ i ) otherwise; Vanni Noferini The Fréchet derivative of a generalized matrix function 20 / 33

41 Notation again... Given the singular values of A C m n we introduce two operators F, G R m n : σ i f (σ i ) σ j f (σ j ) σi 2 σ2 j if i j, max(i, j) ν, and σ i σ j ; σ i f (σ i )+f (σ i ) 2σ i if i j, max(i, j) ν, and σ i = σ j 0; F ij = f (σ j ) σ j if i > n, and σ j 0; f (σ i ) σ i if j > m, and σ i 0; f (σ i ) otherwise; σ j f (σ i ) σ i f (σ j ) if i j, i, j ν, and σ σi 2 i σ j ; σ2 j G ij = σ i f (σ i ) f (σ i ) 2σ i if i j, i, j ν, and σ i = σ j 0; 0 otherwise. Vanni Noferini The Fréchet derivative of a generalized matrix function 20 / 33

42 I am lost! Help! Forgetting about the degenerate cases... Vanni Noferini The Fréchet derivative of a generalized matrix function 21 / 33

43 I am lost! Help! Forgetting about the degenerate cases... F is f (σ i ) on the main diagonal, and it is off the main diagonal. σ i f (σ i ) σ j f (σ j ) σ 2 i σ 2 j Vanni Noferini The Fréchet derivative of a generalized matrix function 21 / 33

44 I am lost! Help! Forgetting about the degenerate cases... F is f (σ i ) on the main diagonal, and it is off the main diagonal. G is 0 on the main diagonal, and it is off the main diagonal. σ i f (σ i ) σ j f (σ j ) σ 2 i σ 2 j σ j f (σ i ) σ i f (σ j ) σ 2 i σ 2 j Vanni Noferini The Fréchet derivative of a generalized matrix function 21 / 33

45 Daleckǐi-Kreǐn formula for gmf Theorem Given A = USV, under the assumptions of the previous theorem, ) L f (A, E) = U (F Ê + ih Im(Ê) + G Υ (Ê) V, where Ê = U EV, F, G are as in the previous slides, and H is diagonal with H ii = f (σ i )/σ i F ii if σ i 0 or H ii = f (0) F ii otherwise. Vanni Noferini The Fréchet derivative of a generalized matrix function 22 / 33

46 Daleckǐi-Kreǐn formula for real gmf U, V, E are real, so that the formula simplifies a little. Theorem Given A = USV T, under the assumptions of the previous theorem, ) L f (A, E) = U (F Ê + G Υ (Ê) V T, where Ê = UT EV, F, G are as a few slides ago. Vanni Noferini The Fréchet derivative of a generalized matrix function 23 / 33

47 Special cases For special f, formuale may simplify. For example, f = 1 unitary factor of a polar decomposition. (f (0) 0, hence differentiability A has full rank). The theorem then leads to efficient algorithms for the computation of the Fréchet derivative, see Arslan, Noferini and Tisseur (in preparation). Vanni Noferini The Fréchet derivative of a generalized matrix function 24 / 33

48 Example Toy matrices: A = 0 2 0, E = Vanni Noferini The Fréchet derivative of a generalized matrix function 25 / 33

49 Example Toy matrices: for f (σ) = σ 2, A = 0 2 0, E = / /3 F = 3 4 7/3, G = 1 0 2/3, 7/3 7/3 2 2/3 2/ /3 L f (A, E) = 1 0 2/3. 7/3 7/3 2 Vanni Noferini The Fréchet derivative of a generalized matrix function 25 / 33

50 Example Toy matrices: A = 0 2 0, E = Vanni Noferini The Fréchet derivative of a generalized matrix function 26 / 33

51 Example Toy matrices: for f (σ) = 1 (polar decomposition), A = 0 2 0, E = /4 1/3 0 1/4 1/3 F = 1/4 0 1/3 = G, L f (A, E) = 1/4 0 1/3. 1/3 1/3 0 1/3 1/3 0 Vanni Noferini The Fréchet derivative of a generalized matrix function 26 / 33

52 Application to conditioning Theorem Assuming m n and A R m n, cond abs (f, A) max{max i F ii, max j<i F ij, max i<j F ij + G ij, max F ij G ij }. i<j Vanni Noferini The Fréchet derivative of a generalized matrix function 27 / 33

53 Application to conditioning Theorem Assuming m n and A R m n, cond abs (f, A) max{max i F ii, max j<i F ij, max i<j F ij + G ij, max F ij G ij }. i<j Unlike for classical matrix function, no condition number of some potentially ill conditioned matrix appears! Vanni Noferini The Fréchet derivative of a generalized matrix function 27 / 33

54 Some more bounds In practice, only a few singular values may be known: Theorem If A R m n has full rank and smallest singular value σ r, suppose that M is the sup norm of f on [σ r, A 2 ] and that f is Lipschitz continuos on the same interval with constant K. Then cond abs (f, A) max{k, Mσ 1 r }. Vanni Noferini The Fréchet derivative of a generalized matrix function 28 / 33

55 Some more bounds In practice, only a few singular values may be known: Theorem If A R m n has full rank and smallest singular value σ r, suppose that M is the sup norm of f on [σ r, A 2 ] and that f is Lipschitz continuos on the same interval with constant K. Then Theorem cond abs (f, A) max{k, Mσ 1 r }. If A R m n and f (0) = 0, suppose that f is Lipschitz continuos on on [0, A 2 ] with constant K. Then cond abs (f, A) K. Vanni Noferini The Fréchet derivative of a generalized matrix function 28 / 33

56 Some more bounds Again, for special choices of f more can be said. Vanni Noferini The Fréchet derivative of a generalized matrix function 29 / 33

57 Some more bounds Again, for special choices of f more can be said. The following is known but can be recovered as a special case: Theorem (Kenney and Laub, 1991) If A R m n has full rank r, then for f (x) = 1 cond abs (f, A) = 2 σ r + σ r 1. Vanni Noferini The Fréchet derivative of a generalized matrix function 29 / 33

58 Some more bounds Again, for special choices of f more can be said. The following is known but can be recovered as a special case: Theorem (Kenney and Laub, 1991) If A R m n has full rank r, then for f (x) = 1 Theorem cond abs (f, A) = 2 σ r + σ r 1. If A R m n has full rank r, then for f (x) = e x, defining g(x) = e x /x: cond abs (f, A) = max{g(σ r ), g( A 2 ), e A 2 }. Vanni Noferini The Fréchet derivative of a generalized matrix function 29 / 33

59 Relative condition number..it is the relative condition number that is of interest, but it is more convenient to state results for the absolute condition number. Vanni Noferini The Fréchet derivative of a generalized matrix function 30 / 33

60 Relative condition number..it is the relative condition number that is of interest, but it is more convenient to state results for the absolute condition number. Vanni Noferini The Fréchet derivative of a generalized matrix function 30 / 33

61 Some more bounds Define µ = f ( A 2 ) 2 + f (σ r ) 2. Theorem If A R m n has full rank and smallest singular value σ r, suppose that M is the sup norm of f on [σ r, A 2 ] and that f is Lipschitz continuos on the same interval with constant K. Then cond rel (f, A) max(m, n) A 2 max{k/µ, σ 1 r }. Vanni Noferini The Fréchet derivative of a generalized matrix function 31 / 33

62 Some more bounds Define µ = f ( A 2 ) 2 + f (σ r ) 2. Theorem If A R m n has full rank and smallest singular value σ r, suppose that M is the sup norm of f on [σ r, A 2 ] and that f is Lipschitz continuos on the same interval with constant K. Then Theorem cond rel (f, A) max(m, n) A 2 max{k/µ, σ 1 r }. If A R m n and f (0) = 0, suppose that f is Lipschitz continuos on on [0, A 2 ] with constant K. Then cond rel (f, A) max(m, n)k A 2 µ 1. Vanni Noferini The Fréchet derivative of a generalized matrix function 31 / 33

63 The big picture If f (0) = 0, then f has essentially the same condition number as the scalar function f. Unlike classical mf, gmf are never numerically dodgier than the scalar case. Vanni Noferini The Fréchet derivative of a generalized matrix function 32 / 33

64 The big picture If f (0) = 0, then f has essentially the same condition number as the scalar function f. Unlike classical mf, gmf are never numerically dodgier than the scalar case. If f (0) 0, trouble may happen only if max i f (σ i )/σ i max f (σ i ). i Vanni Noferini The Fréchet derivative of a generalized matrix function 32 / 33

65 Can trouble happen? If f (0) 0, f is discontinuos at a rank deficient A. Intuitively, we expect numerical issues for an ill conditioned A. Let Then, Yet, A = [ ] ɛ 0 0, f (x) = 1 + (x ɛ) cond rel (f, ɛ) = 0, cond rel (f, 1) = 1 + O(ɛ 2 ). cond rel (f, A) = 1 ɛ 5 + O(1). Vanni Noferini The Fréchet derivative of a generalized matrix function 33 / 33

66 Can trouble happen? If f (0) 0, f is discontinuos at a rank deficient A. Intuitively, we expect numerical issues for an ill conditioned A. Let Then, Yet, A = [ ] ɛ 0 0, f (x) = 1 + (x ɛ) cond rel (f, ɛ) = 0, cond rel (f, 1) = 1 + O(ɛ 2 ). cond rel (f, A) = 1 ɛ 5 + O(1) ρ Vanni Noferini ǫ The Fréchet derivative of a generalized matrix function 33 / 33