Introduction to Numerical Linear Algebra II

Transcription

1 Introduction to Numerical Linear Algebra II Petros Drineas These slides were prepared by Ilse Ipsen for the 2015 Gene Golub SIAM Summer School on RandNLA 1 / 49

2 Overview We will cover this material in relatively more detail, but will still skip a lot... 1 Norms {Measuring the length/mass of mathematical quantities} General norms Vector p norms Matrix norms induced by vector p norms Frobenius norm 2 Singular Value Decomposition (SVD) The most important tool in Numerical Linear Algebra 3 Least Squares problems Linear systems that do not have a solution 2 / 49

3 Norms

4 General Norms How to measure the mass of a matrix or length of a vector Norm is function R m n R with 1 Non-negativity A 0, A = 0 A = 0 2 Triangle inequality A + B A + B 3 Scalar multiplication α A = α A for all α R. Properties Minus signs A = A Reverse triangle inequality A B A B New norms For norm on R m n, and nonsingular M R m m def A M = M A is also a norm 4 / 49

5 Vector p Norms For x R n and integer p 1 x p def = ( n ) 1/p x i p i=1 One norm x 1 = n j=1 x j Euclidean (two) norm x 2 = n j=1 x j 2 = x T x Infinity (max) norm x = max 1 j n x j 5 / 49

6 Exercises 1 Determine 11 p for 11 R n and p = 1, 2, 2 For x R n with x j = j, 1 j n, determine closed-form expressions for x p for p = 1, 2, 6 / 49

7 Inner Products and Norm Relations For x, y R n Cauchy-Schwartz inequality Hölder inequality x T y x 2 y 2 Relations x T y x 1 y x T y x y 1 x x 1 n x x 2 x 1 n x 2 x x 2 n x 7 / 49

8 Exercises 1 Prove the norm relations on the previous slide 2 For x R n show x 2 x x 1 8 / 49

9 Vector Two Norm Theorem of Pythagoras For x, y R n x T y = 0 x ± y 2 2 = x y 2 2 Two norm does not care about orthonormal matrices For x R n and V R m n with V T V = I n V x 2 = x 2 9 / 49

10 Vector Two Norm Theorem of Pythagoras For x, y R n x T y = 0 x ± y 2 2 = x y 2 2 Two norm does not care about orthonormal matrices For x R n and V R m n with V T V = I n V x 2 = x 2 V x 2 2 = (V x)t (V x) = x T V T V x = x T x = x / 49

11 Exercises For x, y R n show 1 Parallelogram equality x + y x y 2 2 = 2 ( x y 2 ) 2 2 Polarization identity x T y = 1 4 ( x + y 2 2 x y 2 2) 10 / 49

12 Matrix Norms (Induced by Vector p Norms) For A R m n and integer p 1 def Ax p A p = max x 0 x p = max y p=1 A y p One Norm: Maximum absolute column sum A 1 = max 1 j n m i=1 a ij = max 1 j n A e j 1 Infinity Norm: Maximum absolute row sum A = max 1 i m n j=1 a ij = max 1 i m AT e i 1 11 / 49

13 Matrix Norm Properties Every norm realized by some vector y 0 A p = Ay p y p = Az p where z y y p z p = 1 {Vector y is different for every A and every p} Submultiplicativity Matrix vector product: For A R m n and y R n Ay p A p y p Matrix product: For A R m n and B R n l AB p A p B p 12 / 49

14 More Matrix Norm Properties For A R m n and permutation matrices P R m m, Q R n n Permutation matrices do not matter P A Q p = A p Submatrices have smaller norms than parent matrix ( ) B A12 If P A Q = then B A 21 A p A p / 49

15 Exercises 1 Different norms realized by different vectors Find y and z so that A 1 = Ay 1 and A = Az when ( ) 1 4 A = If Q R n n is permutation then Q p = 1 3 Prove the two types of submultiplicativity 4 If D = diag ( d 11 d nn ) then D p = max 1 j n d jj 5 If A R n n nonsingular then A p A 1 p 1 14 / 49

16 Norm Relations and Transposes For A R m n Relations between different norms 1 n A A 2 m A 1 m A 1 A 2 n A 1 Transposes A T 1 = A A T = A 1 A T 2 = A 2 15 / 49

17 Proof: Two Norm of Transpose {True for A = 0, so assume A 0} Let A 2 = Az 2 for some z R n with z 2 = 1 Reduce to vector norm ( ) A 2 2 = Az 2 2 = (Az) T (Az) = z T A T Az = z T A T Az }{{} vector Cauchy-Schwartz inequality and submultiplicativity imply ( ) z T A T Az z 2 A T Az 2 A T 2 A 2 Thus A 2 2 AT 2 A 2 and A 2 A T 2 Reversing roles of A and A T gives A T 2 A 2 16 / 49

18 Matrix Two Norm A R m n Orthonormal matrices do not change anything If U R k m with U T U = I m, V R l n with V T V = I n U A V T 2 = A 2 Gram matrices For A R m n A T A 2 = A 2 2 = A A T 2 Outer products For x R m, y R n xy T 2 = x 2 y 2 17 / 49

19 Proof: Norm of Gram Matrix Submultiplicativity and transpose A T A 2 A 2 A T 2 = A 2 A 2 = A 2 2 Thus A T A 2 A 2 2 Let A 2 = Az 2 for some z R n with z 2 = 1 Cauchy-Schwartz inequality and submultiplicativity imply ( ) A 2 2 = Az 2 2 = (Az) T (Az) = z T A T A z }{{} vector z 2 A T A z 2 A T A 2 Thus A 2 2 AT A 2 18 / 49

20 Exercises 1 Infinity norm of outer products For x R m and y R n, show x y T = x y 1 2 If A R n n with A 0 is idempotent then (i) A p 1 (ii) A 2 = 1 if A also symmetric 3 Given A R n n Among all symmetric matrices, 1 2 (A + AT ) is a (the?) matrix that is closest to A in the two norm 19 / 49

21 Frobenius Norm The true mass of a matrix For A = ( a 1 a n ) R m n A F def = n n m a j 2 2 = a ij 2 = j=1 j=1 i=1 trace(a T A) 20 / 49

22 Frobenius Norm The true mass of a matrix For A = ( a 1 a n ) R m n A F def = n n m a j 2 2 = a ij 2 = j=1 j=1 i=1 trace(a T A) Vector If x R n then x F = x 2 Transpose Identity A T F = A F I n F = n 20 / 49

23 More Frobenius Norm Properties A R m n Orthonormal invariance If U R k m with U T U = I m, V R l n with V T V = I n Relation to two norm U A V T F = A F A 2 A F rank(a) A 2 min{m, n} A 2 Submultiplicativity A B F A 2 B F A F B F 21 / 49

24 Exercises 1 Show the orthonormal invariance of the Frobenius norm 2 Show the submultiplicativity of the Frobenius norm 3 Frobenius norm of outer products For x R m and y R n show x y T F = x 2 y 2 22 / 49

25 Singular Value Decomposition (SVD)

26 Full SVD Given: A R m n Tall and skinny: m n ( ) 1 Σ A = U V T where Σ σ... 0 Short and fat: m n A = U ( Σ 0 ) V T where Σ σ 1... σ n 0 σ m 0 U R m m and V R n n are orthogonal matrices 24 / 49

27 Names and Conventions A R m n with SVD ( ) Σ A = U V T 0 or A = U ( Σ 0 ) V T Singular values (svalues): Diagonal elements σ j of Σ Left singular vector matrix: U Right singular vector matrix: V Svalue ordering σ 1 σ 2... σ min{m, n} 0 Svalues of matrices B, C: σ j (B), σ j (C) 25 / 49

28 SVD Properties A R m n Number of svalues equal to small dimension A R m n has min{m, n} singular values σ j 0 Orthogonal invariance If P R m m and Q R n n are orthogonal matrices then P A Q has same svalues as A Gram Product Nonzero svalues (= eigenvalues) of A T A are squares of svalues of A Inverse A R n n nonsingular σ j > 0 for 1 j n If A = U Σ V T then A 1 = V Σ 1 U T SVD of inverse 26 / 49

29 Exercises 1 Transpose A T has same singular values as A 2 Orthogonal matrices All singular values of A R n n are equal to 1 A is orthogonal matrix 3 If A R n n is symmetric and idempotent then all singular values are 0 and/or 1 4 For A R m n and α > 0 express svalues of (A T A + α I) 1 A T in terms of α and svalues of A ( ) 5 If A R m n In with m n then singular values of are A equal to 1 + σj 2 for 1 j n 27 / 49

30 Singular Values A R m n with svalues σ 1 σ p p min{m, n} Two norm A 2 = σ 1 Frobenius norm A F = σ σ2 p Well conditioned in absolute sense Product σ j (A) σ j (B) A B 2 1 j p σ j (A B) σ 1 (A) σ j (B) 1 j p 28 / 49

31 Matrix Schatten Norms A R m n, with singular values σ 1 σ ρ > 0, and integer p 0, the family of the Schatten p-norms is defined as ( ρ ) 1/p def A p = σ p i. i=1 Different than the vector-induced matrix p-norms 1. Schatten zero norm 2 : equal to the matrix rank. Schatten one norm: the sum of the singular values of the matrix, also called the nuclear norm. Schatten two norm: the Frobenius norm. Schatten infinity norm: the spectral (or two) norm. Schatten p-norms are unitarily invariant, submultiplicative, satisfy Hölder s inequality, etc. 1 Notation is, unfortunately, confusing. 2 Not really a norm / 49

32 Exercises 1 Norm of inverse If A R n n nonsingular with svalues σ 1 σ n then A 1 2 = 1/σ n 2 Appending a column to a tall and skinny matrix If A R m n with m > n, z R m, B = ( A z ) then σ n+1 (B) σ n (A) σ 1 (B) σ 1 (A) 3 Appending a row to a tall and skinny matrix If A R m n with m n, z R n, B T = ( A T z ) then σ n (B) σ n (A) σ 1 (A) σ 1 (B) σ 1 (A) 2 + z / 49

33 Rank rank(a) = number of nonzero (positive) svalues of A Zero matrix rank(0) = 0 Rank bounded by small dimension If A R m n then rank(a) min{m, n} Transpose rank(a T ) = rank(a) Gram product rank(a T A) = rank(a) = rank(aa T ) General product rank(a B) min{rank(a), rank(b)} If A nonsingular then rank(a B) = rank(b) 31 / 49

34 SVDs of Full Rank Matrices All svalues of A R m n are nonzero Full column-rank rank(a) = n ( ) Σ A = U V T Σ R n n nonsingular 0 {Linearly independent columns} Full row-rank rank(a) = m {Linearly independent rows} A = U ( Σ 0 ) V T Σ R m m nonsingular Nonsingular rank(a) = n = m {Lin. indep. rows & columns} A = U Σ V T Σ R n n nonsingular 32 / 49

35 Exercises 1 Rank of outer product If x R m and y R n then rank(xy T ) 1 2 If A R n n nonsingular then ( A B ) has full row-rank for any B R n k 3 Orthonormal matrices If A R m n has orthonormal columns then rank(a) = n and all svalues of A are equal to 1 4 Gram products For A R m n (i) rank(a) = n A T A nonsingular (ii) rank(a) = m AA T nonsingular 33 / 49

36 Thin SVD A R m n with A = U ( ) Σ V T or A = U ( Σ 0 ) V T 0 Singular values σ 1 σ p 0, p min{m, n} Singular vectors U = ( u 1 u m ) V = ( v 1 v n ) If rank(a) = r then thin (reduced) SVD {only non zero svalues} 1 A = ( u 1 ) u r σ... σ r v T 1. v T r = r σ j u j vj T j=1 34 / 49

37 Optimality of SVD Given A R m n, rank(a) = r, thin SVD r j=1 σ j u j v T j Approximation from k dominant svalues A k k σ j u j vj T j=1 1 k < r 35 / 49

38 Optimality of SVD Given A R m n, rank(a) = r, thin SVD r j=1 σ j u j v T j Approximation from k dominant svalues A k k σ j u j vj T j=1 1 k < r Absolute distance of A to set of rank-k matrices min rank(b)=k A B 2 = A A k 2 = σ k+1 min A B F = A A k F = rank(b)=k r j=k+1 σ 2 j 35 / 49

39 Exercises 1 Find an example to illustrate that a closest matrix of rank k is not unique in the two norm 36 / 49

40 Moore Penrose Inverse Given A R m n with rank(a) = r 1 and SVD A = U ( ) Σr 0 V T = 0 0 r σ j u j vj T j=1 Moore Penrose inverse ( ) A def Σ 1 r 0 = V U T = 0 0 r j=1 1 σ j v j u T j Zero matrix 0 m n = 0 n m 37 / 49

41 Special Cases of Moore Penrose Inverse Nonsingular If A R n n with rank(a) = n then A = A 1 Inverse A 1 A = I n = A A 1 Full column rank If A R m n with rank(a) = n then A = (A T A) 1 A T Left inverse A A = I n Full row rank If A R m n with rank(a) = m then A = A T (AA T ) 1 Right inverse A A = I m 38 / 49

42 Necessary and Sufficient Conditions A is Moore Penrose inverse of A A satisfies 1 A A A = A 2 A A A = A 3 (A A ) T = A A {symmetric} 4 (A A) T = A A {symmetric} 39 / 49

43 Exercises 1 If x R m and y R n with y 0 then x y 2 = x 2 / y 2 2 For A R m n the following matrices are idempotent: A A A A I m A A I n A A 3 If A R m n and A 0 then A A 2 = A A 2 = 1 4 If A R m n then (I m A A ) A = 0 m n A (I n A A) = 0 m n 5 If A R m n with rank(a) = n then (A T A) 1 2 = A If A = B C where B R m n has rank(b) = n and C R n n is nonsingular then A = C 1 B 40 / 49

44 Matrix Spaces and Singular Vectors: A A R m n Column space range(a) = {b : b = A x for some x R n } R m Null space (kernel) null(a) = {x : A x = 0} R n If rank(a) = r 1 A = ( ) ( ) ( ) Σ U r U r 0 V T r m r 0 0 Vn r T range(a) = range(u r ) null(a) = range(v n r ) 41 / 49

45 Matrix Spaces and Singular Vectors: A T A R m n Row space range(a T ) = {d : d = A T y for some y R m } R n Left null space null(a T ) = {y : A T y = 0} R m If rank(a) = r 1 A = ( ) ( ) ( ) Σ U r U r 0 V T r m r 0 0 Vn r T range(a T ) = range(v r ) null(a T ) = range(u m r ) 42 / 49

46 Fundamental Theorem of Linear Algebra A R m n range(a) null(a T ) = R m implies m = rank(a) + dim null(a T ) range(a) null(a T ) range(a T ) null(a) = R n implies n = rank(a) + dim null(a) range(a T ) null(a) 43 / 49

47 Spaces of the Moore Penrose Inverse {Need this for least squares} Column space range(a ) = range(a T A) = range(a T ) null(a) Null space null(a ) = null(a A T ) = null(a T ) range(a) 44 / 49

48 Least Squares (LS) Problems

49 General LS Problems Given A R m n and b R m min x Ax b 2 General LS solution y = A b + q for any q null(a) All solutions have same LS residual r b A y {A y = A A b since q null(a)} LS residual orthogonal to column space A T r = 0 LS solution of minimal two norm y = A b Computation: SVD 46 / 49

50 Exercises 1 Use properties of Moore Penrose inverses to show that the LS residual is orthogonal to the column space of A 2 Determine the minimal norm solution for min x A x b 2 if A = 0 m n 3 If y is the minimal norm solution to min x A x b 2 and A T b = 0, then what can you say about y? 4 Determine the minimal norm solution for min x A x b 2 if A = c d T where c R m and d R n? 47 / 49

51 Full Column Rank LS Problems Given A R m n with rank(a) = n and b R m min x Ax b 2 Unique LS solution y = A b Computation: QR decomposition 1 Factor A = Q R where Q T Q = I n and R is 2 Multiply c = Q T b 3 Solve R y = c Do NOT solve A T A y = A T b 48 / 49

52 Exercises 1 If A R m n with rank(a) = n has thin QR factorization A = Q R where Q T Q = I n and R is then A = R 1 Q T 2 If A R m n has orthonormal columns then A = A T 3 If A R m n has rank(a) = n then I m A A 2 = min{1, m n} 49 / 49