2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian

Size: px

Start display at page:

Download "2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian"

Kory Sullivan
5 years ago
Views:

1 FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian 2-1

2 Vector notation n-vector x: x = x 1 x n x 2. also written as x = (x 1, x 2,..., x n ) set of n-vectors is denoted R n (Euclidean space) x i : ith element or component or entry of x x is also called a column vector y = [ ] y 1 y 2 y n is called a row vector unless stated otherwise, a vector typically means a column vector Linear algebra 2-2

3 zero vectors: x = (0, 0,..., 0) Special vectors all-ones vectors: x = (1, 1,, 1) (we will denote it by 1) standard unit vectors: e k has only 1 at the kth entry and zero otherwise (standard unit vectors in R 3 ) e 1 = 1 0, e 2 = 0 1, e 3 = unit vectors: any vector u whose norm (magnitude) is 1, i.e., u example: u = (1/ 2, 2/ 6, 1/ 2) u u u2 n = 1 Linear algebra 2-3

4 Inner products definition: the inner product of two n-vectors x, y is x 1 y 1 + x 2 y x n y n also known as the dot product of vectors x, y notation: x T y properties (αx) T y = α(x T y) for scalar α (x + y) T z = x T z + y T z x T y = y T x Linear algebra 2-4

5 properties x = Euclidean norm x x x2 n = x T x also written x 2 to distinguish from other norms αx = α x for scalar α x + y x + y (triangle inequality) x 0 and x = 0 only if x = 0 interpretation x measures the magnitude or length of x x y measures the distance between x and y Linear algebra 2-5

6 Matrix notation an m n matrix A is defined as A = a 11 a a 1n a 21 a a 2n a m1 a m2... a mn, or A = [a ij] m n a ij are the elements, or coefficients, or entries of A set of m n-matrices is denoted R m n A has m rows and n columns (m, n are the dimensions) the (i, j) entry of A is also commonly denoted by A ij A is called a square matrix if m = n Linear algebra 2-6

7 Special matrices zero matrix: A = 0 A = a ij = 0, for i = 1,..., m, j = 1,..., n identity matrix: A = I A = a square matrix with a ii = 1, a ij = 0 for i j Linear algebra 2-7

8 diagonal matrix: a square matrix with a ij = 0 for i j A = a a a n triangular matrix: a square matrix with zero entries in a triangular part upper triangular a 11 a 12 a 1n A = 0 a 22 a 2n..... A = 0 0 a nn a ij = 0 for i j lower triangular a a 21 a a n1 a n2 a nn a ij = 0 for i j Linear algebra 2-8

9 Block matrix notation example: 2 2-block matrix A A = [ ] B C D E for example, if B, C, D, E are defined as B = [ ] 2 1, C = 3 8 then A is the matrix [ ] 0 1 7, D = [ 0 1 ], E = [ ] A = note: dimensions of the blocks must be compatible Linear algebra 2-9

10 Column and Row partitions write an m n-matrix A in terms of its columns or its rows A = [ a 1 a 2 a n ] = b T 1 b T 2. b T m a j for j = 1, 2,..., n are the columns of A b T i for i = 1, 2,..., m are the rows of A example: A = [ 1 2 ] [ [ [ a 1 =, a 4] 2 =, a 9] 3 =, b 0] T 1 = [ ], b T 2 = [ ] Linear algebra 2-10

11 Matrix-vector product product of m n-matrix A with n-vector x Ax = a 11 x 1 + a 12 x a 1n x n a 21 x 1 + a 22 x a 2n x n. a m1 x 1 + a m2 x a mn x n dimensions must be compatible: # columns in A = # elements in x if A is partitioned as A = [ a 1 a 2 a n ], then Ax = a 1 x 1 + a 2 x a n x n Ax is a linear combination of the column vectors of A the coefficients are the entries of x Linear algebra 2-11

12 Product with standard unit vectors post-multiply with a column vector 0 a 11 a a 1n Ae k = a 21 a a 2n a m1 a m2... a mn 1. 0 = a 1k a 2k. a mk = the kth column of A pre-multiply with a row vector e T k A = [ ] a 11 a a 1n a 21 a a 2n a m1 a m2... a mn = [ a k1 a k2 a kn ] = the kth row of A Linear algebra 2-12

13 Trace Definition: trace of a square matrix A is the sum of the diagonal entries in A tr(a) = a 11 + a a nn example: trace of A is = 7 properties tr(a T ) = tr(a) A = tr(αa + B) = α tr(a) + tr(b) tr(ab) = tr(ba) Linear algebra 2-13

14 Inverse of matrices Definition: a square matrix A is called invertible or nonsingular if there exists B s.t. AB = BA = I B is called an inverse of A it is also true that B is invertible and A is an inverse of B if no such B can be found A is said to be singular assume A is invertible an inverse of A is unique the inverse of A is denoted by A 1 Linear algebra 2-14

15 assume A, B are invertible Facts (αa) 1 = α 1 A 1 for nonzero α A T is also invertible and (A T ) 1 = (A 1 ) T AB is invertible and (AB) 1 = B 1 A 1 (A + B) 1 A 1 + B 1 Linear algebra 2-15

16 Inverse of 2 2 matrices the matrix A = [ ] a b c d is invertible if and only if and its inverse is given by A 1 = ad bc 0 1 ad bc [ d ] b c a example: A = [ ] 2 1, A 1 = [ ] Linear algebra 2-16

17 Invertible matrices Theorem: for a square matrix A, the following statements are equivalent 1. A is invertible 2. Ax = 0 has only the trivial solution (x = 0) 3. the reduced echelon form of A is I 4. A is invertible if and only if det(a) 0 Linear algebra 2-17

18 Inverse of special matrices diagonal matrix A = a a a n a diagonal matrix is invertible iff the diagonal entries are all nonzero a ii 0, i = 1, 2,..., n the inverse of A is given by A 1 = 1/a /a /a n the diagonal entries in A 1 are the inverse of the diagonal entries in A Linear algebra 2-18

19 triangular matrix: upper triangular a 11 a 12 a 1n A = 0 a 22 a 2n..... A = 0 0 a nn a ij = 0 for i j lower triangular a a 21 a a n1 a n2 a nn a ij = 0 for i j a triangular matrix is invertible iff the diagonal entries are all nonzero a ii 0, i = 1, 2,..., n product of lower (upper) triangular matrices is lower (upper) triangular the inverse of a lower (upper) triangular matrix is lower (upper) triangular Linear algebra 2-19

20 symmetric matrix: A = A T for any square matrix A, AA T and A T A are always symmetric if A is symmetric and invertible, then A 1 is symmetric if A is invertible, then AA T and A T A are also invertible Linear algebra 2-20

21 Symmetric matrix A R n n is called symmetric if A = A T Facts: if A is symmetric all eigenvalues of A are real all eigenvectors of A are orthogonal A admits a decomposition A = UDU T where U T U = UU T = I (U is unitary) and D is diagonal (of course, the diagonals of D are eigenvalues of A) Linear algebra 2-21

22 a matrix U R n n is called unitary if Unitary matrix U T U = UU T = I [ ] 1 example: 2 1 1, 1 1 Facts: [ cos θ ] sin θ sin θ cos θ a real unitary matrix is also called orthogonal a unitary matrix is always invertible and U 1 = U T columns vectors of U are mutually orthogonal norm is preserved under a unitary transformation: y = Ux = y = x Linear algebra 2-22

23 Orthogonal projection matrix P is said to be an orthogonal projection if P = P T and P 2 = P examples: P = P is bounded, i.e., P x x [ ] 1 0, P = 0 0 [ 1/2 ] 1/2 1/2 1/2 y = (y P y)+p y, and P y (y P y) (by using P = P T and P 2 = P ) hence, y 2 = P y 2 and that the norm of P y must be less than y if P is an orthogonal projection onto a line spanned by a unit vector u, P = uu T (we see that rank(p ) = 1 as the dimension of a line is 1) another example: P = A(A T A) 1 A T for any matrix A Linear algebra 2-23

24 Positive definite matrix a symmetric matrix A is positive semidefinite, written as A 0 if x T Ax 0, x R n and positive definite, written as A 0 if x T Ax > 0, for all nonzero x R n Facts: A 0 if and only if all eigenvalues of A are non-negative all principle minors of A are non-negative Linear algebra 2-24

25 example: A = [ 1 ] or we can check from 0 because x T Ax = [ ] [ ] [ ] 1 1 x1 x 1 x x 2 = x x 2 2 2x 1 x 2 = (x 1 x 2 ) 2 + x eigenvalues of A are 0.38 and 2.61 (real and positive) the principle minors are 1 and = 1 (all positive) note: A 0 does not mean all entries of A are positive! Linear algebra 2-25

26 Schur complement a consider a symmetric matrix X partitioned as [ ] A B X = B T C Schur complement of A in X is defined as S 1 = C B T A 1 B, if det A 0 Schur complement of C in X is defined as S 2 = A BC 1 B T, if det C 0 we can show that det X = det A det S 1 = det C det S 2 Linear algebra 2-26

27 Schur complement of positive definite matrix Facts: X 0 if and only if A 0 and S 1 0 if A 0 then X 0 if and only if S 1 0 analogous results for S 2 X 0 if and only if C 0 and S 2 0 if C 0 then X 0 if and only if S 2 0 Linear algebra 2-27

28 Linear equations a general linear system of m equations with n variables is described by a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2. =. a m1 x 1 + a m2 x a mn x n = b m where a ij, b j are constants and x 1, x 2,..., x n are unknowns equations are linear in x 1, x 2,..., x n existence and uniqueness of a solution depend on a ij and b j Linear algebra 2-28

29 Linear equation in matrix form the linear system of m equations in n variables a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2. =. a m1 x 1 + a m2 x a mn x n = b m in matrix form: Ax = b where A = a 11 a a 1n a 21 a a 2n a m1 a m2... a mn, x = x 1 x 2. x n, b = b 1 b 2. b m Linear algebra 2-29

30 Three types of linear equations square if m = n [ ] [ ] a11 a 12 x1 = a 22 x 2 a 21 [ b1 b 2 ] (A is square) underdetermined if m < n [ ] a11 a 12 a 13 x 1 x a 21 a 22 a 2 = 23 x 3 [ b1 b 2 ] (A is fat) overdetermined if m > n a 11 a 12 [ ] a 21 a 22 x1 = x a 31 a 2 32 b 1 b 2 b 3 (A is skinny) Linear algebra 2-30

31 Existence and uniqueness of solutions existence: no solution a solution exists uniqueness: the solution is unique there are infinitely many solutions every system of linear equations has zero, one, or infinitely many solutions there are no other possibilities Linear algebra 2-31

32 Nullspace the nullspace of an m n matrix is defined as N (A) = {x R n Ax = 0} the set of all vectors that are mapped to zero by f(x) = Ax the set of all vectors that are orthogonal to the rows of A if Ax = b then A(x + z) = b for all z N (A) also known as kernel of A N (A) is a subspace of R n Linear algebra 2-32

33 Zero nullspace matrix A has a zero nullspace if N (A) = {0} if A has a zero nullspace and Ax = b is solvable, the solution is unique columns of A are independent equivalent conditions: A R n n A has a zero nullspace A is invertible or nonsingular columns of A are a basis for R n Linear algebra 2-33

34 Range space the range of an m n matrix A is defined as R(A) = {y R m y = Ax for some x R n } the set of all m-vectors that can be expressed as Ax the set of all linear combinations of the columns of A = [ ] a 1 a n R(A) = {y y = x 1 a 1 + x 2 a x n a n, x R n } the set of all vectors b for which Ax = b is solvable also known as the column space of A R(A) is a subspace of R m Linear algebra 2-34

35 Full range matrices A has a full range if R(A) = R m equivalent conditions: A has a full range columns of A span R m Ax = b is solvable for every b N (A T ) = {0} Linear algebra 2-35

36 Rank and Nullity rank of a matrix A R m n is defined as rank(a) = dim R(A) nullity of a matrix A R m n is nullity(a) = dim N (A) Facts rank(a) is maximum number of independent columns (or rows) of A rank(a) min(m, n) rank(a) = rank(a T ) Linear algebra 2-36

37 Full rank matrices for A R m n we always have rank(a) min(m, n) we say A is full rank if rank(a) = min(m, n) for square matrices, full rank means nonsingular (invertible) for skinny matrices (m n), full rank means columns are independent for fat matrices (m n), full rank means rows are independent Linear algebra 2-37

38 Theorems Rank-Nullity Theorem: for any A R m n, rank(a) + dim N (A) = n the system Ax = b has a solution if and only if b R(A) the system Ax = b has a unique solution if and only if b R(A), and N (A) = {0} Linear algebra 2-38

39 Derivative and Gradient Suppose f : R n R m and x int dom f the derivative (or Jacobian) of f at x is the matrix Df(x) R m n : Df(x) ij = f i(x) x j, i = 1,..., m, j = 1,..., n when f is scalar-valued (i.e., f : R n R), the derivative Df(x) is a row vector its transpose is called the gradient of the function: f(x) = Df(x) T, f(x) i = f(x) x i, i = 1,..., n which is a column vector in R n Linear algebra 2-39

40 Second Derivative suppose f is a scalar-valued function (i.e., f : R n R) the second derivative or Hessian matrix of f at x, denoted 2 f(x) is 2 f(x) ij = 2 f(x) x i x j, i = 1,..., n, j = 1,..., n example: the quadratic function f : R n R where P S n, q R n, and r R f(x) = P x + q 2 f(x) = P f(x) = (1/2)x T P x + q T x + r, Linear algebra 2-40

41 Chain rule assumptions: f : R n R m is differentiable at x int dom f g : R m R p is differentiable at f(x) int dom g define the composition h : R n R p by h(z) = g(f(z)) then h is differentiable at x, with derivative Dh(x) = Dg(f(x))Df(x) special case: f : R n R, g : R R, and h(x) = g(f(x)) h(x) = g (f(x)) f(x) Linear algebra 2-41

42 example: h(x) = f(ax + b) Dh(x) = Df(Ax + b)a h(x) = A T f(ax + b) example: h(x) = (1/2)(Ax b) T P (Ax b) h(x) = A T P (Ax b) Linear algebra 2-42

43 References Chapter 1 in H. Anton, Elementary Linear Algebra, 10th edition, Wiley, 2010 R.A. Horn and C.R. Johnson, Matrix analysis, Cambridge press, 2012 K.B. Petersen, M.S. Pedersen, et.al., The Matrix Cookbook, Technical University of Denmark, 2008 Linear algebra 2-43

Chapter 3 Transformations

Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases