Linear Algebra Formulas Ben Lee January 27, 2016 Definitions and Terms Diagonal: Diagonal of matrix A is a collection of entries A ij where i = j. Diagonal Matrix: A matrix (usually square), where entries outside the main diagonal are 0. Diagonal entries may or may not be 0. Identity: Square matrix with ones in the main diagonal and zero elsewhere. Matrix multiplication If then (AB) ij = m A ik B kj k=1 U = u 1... u n, V = v 1... v n UV T = n T u i v i If D = diag(d 1,..., d n ) = d 1... d n then UD = d 1 u 1... d n u n 1
Linearly Independence span{x 1,..., x k } = {x : x = k j=1 α jx j for some α j, j = 1,..., k} Set of vectors {x 1,...x k } is linearly independent if k α i x i = 0 = α i = 0, i = 1,..., k This can also be written as Av = 0 = v = 0 where the columns of A are x i Linearly Independence also means 1. Each x span {x 1,..., x k } is a unique linear combination of x 1,..., x k k α i x i = k β i x i = α i = β i, i = 1,..., k 2. No elements of {x 1,..., x k } can be written as a linear combination of the others Invertible Matrix If A, B are invertible, then (AB) 1 = B 1 A 1 If A is n by n matrix The following are equivalent: 1. A is invertible (AKA nonsingular, nondegenerate) 2. det A 0 3. A has full rank. ranka = n 4. Only solution to Ax = 0 is x = 0 5. null A = {0} 6. Ax = b for each b R n has exactly one solution. 7. Columns of A are linearly independent 8. Columns of A span R n 9. Columns of A form a basis of R n 10. There is a matrix B such that AB = I n = BA 11. The number 0 is not an eigenvalue of A 2
Subspaces, Bases, Dimensions Subspace of R n is U R n that is closed under linear combination of its elements. We typically specify a subspace as the span{x 1,...x k }. A basis for a subspace U is the set of linearly independent vectors that spans U. 1. span{x 1,...x k } is the smallest subspace of R n that contains the vectors x 1,...x k. 2. If U is a finite-dimensional vector space, then any two bases of U have the same finite number of elements. Since a vector space has the same number of elements, we call that dim V. 1. If dim V = n, (a) Any subset of V with more than n vectors is linearly dependent (b) No subset of V with fewer than n vectors can span V Orthogonal Complement Orthogonal subspace U is defined by U R n = {x R n : u U, x u} 1. (U ) = U 2. U U = {0 } 3. Any x R n can be written as x = v + w where v U, w U 4. U V implies V U Orthogonal Matrix If Q has orthonormal columns, Q O nxn 1. Q T Q = QQ T = I n 2. If U O nxk, then U T U = I k but UU T I n 3. Q T = Q 1 4. For each x, y R n, Qx, Qy = x, y and Qx, Qx = x, x = x 2 5. If Q, W O nxn, then QW O nxn and Q 1 O nxn That is, the set O nxn is closed under multiplication and inverse. 3
Rank Rank of A is dimension of vector space spanned by its columns. Rank is also the same as the dimension spanned by rows. Note that dimensions refer to vector spaces and ranks refer to matrices. Assume A R m n 1. rank(a) min(m, n) If rank(a) = min(m, n), A has full rank. Otherwise, A is rank deficient. 2. Only a zero matrix has rank zero. 3. Square matrix A R n n is invertible if and only if A has full rank. 4. If B R n k, rank(ab) min(ranka, rankb) 5. If B R n k and has rank n, rank(ab) = rank(a) 6. If C R l m and has rank m, rank(ca) = rank(a) 7. If A is real, rank(a T A) = rank(aa T ) = rank(a) = rank(a T ) Kernel, NullSpace, Image (Range) Kernel (AKA nullspace) of a linear map is L : V W between vector spaces V and W is the set of all elements v for which L(v) = 0 ker(l) = {v V L(v) = 0} or in terms of matrix multiplication N(A) =Null(A) = ker(a) = {x R n Ax = 0} The nullity of A is the dimension of the kernel of A. 1. Null(A) contains the zero vector 2. If x, y Null(A), then x + y Null(A) 3. If x Null(A) and c is scalar, then cx Null(A) 4. Rank-nullity Theorem: If A is m-by-n matrix, rank(a) + nullity(a) = n Older books call the range of a function as the set of the outputs of the function. R(A) = {y R m : y = Ax, x R n } 4
Modern books call this image: im(a) = R(A) Theorem: N(A) = R(A T ) Eigenvalues and Eigenvectors x 0 is an eigenvector of A R n n if Ax = λx for some scalar λ λ is the eigenvalue. Eigendecomposition of Matrix: Square n by n matrix A has n eigenvectors q i (i = 1,..., n). λ 1 A = QΛQ 1 = q 1... q n... λ n q 1... q n where the columns of Q are q i and Λ is diagonal with corresponding eigenvalues. Symmetric Matrix Matrix A is symmetric if A = A T 1. Given symmetric matrices A, B A + B is symmetric 2. AB is NOT necessarily symmetric AB is symmetric if and only if A and B commute, i.e., if AB = BA 3. Real n n matrix A is symmetric if and only if Ax, y = x, Ay x, y R n 4. If A is real, all eigenvalues are real 5. A has n orthonormal eigenvectors Real symmetric matrix A is Positive-Definite if z T Mz > 0 z R n, z 0 Real symmetric matrix A is Positive-Semidefinite if z T Mz 0 z R n Note: Positive-definite and positive-semidefinite is defined for symmetric matrices. Properties of PD symmetric matrices: 1 1. All eigenvalues are positive 2. It is invertible 5
Trace Trace of a n n square matrix is the sum of the elements in the main diagonal. tr(a) = a 11 + a 22 +... + a nn = n a ii 1. tr(a + B) = tr(a) + tr(b) 2. tr(ca) = ctr(a) for all scalar c and square matrices A, B 3. tr(a T ) = tr(a) 4. Trace is invariant under cyclic permutations tr(ab) = tr(ba) tr(abcd) = tr(bcda) = tr(cdab) = tr(dabc) 5. Arbitrary permutation is NOT true in general tr(abc) tr(acb) 6. tr(p 1 AP ) = tr(a) 7. If n n square matrix (real or complex) A has λ 1,...λ n eigenvalues, then tr(a) = i λ i Determinants Determinant of 2 by 2 matrix is a c b d = ad bc 1. det(i n ) = 1 2. det(a T ) = A 3. det(a 1 ) = det(a) 1 4. For square matrices A, B, det(ab) = det(a) det(b) 5. For square n by n matrix A det(ca) = c n det(a) 6. Solutions to det(a xi n ) = 0 are the eigenvalues of A 6
Norms Properties 1. x 0 with equality if and only if x = 0 2. αx = α x 3. x + y x + y Frobenius Norm For some matrix A, B of the same size, A, B = i,j A ij B ij = tr(a T B) Then Properties A F = A, A 1/2 1. For each x, y R n, xy T F = x y 2. Squared Frobenius norm is the sum of squares of the singular values: A 2 F = r σ 2 i References Formulas are based on wikipedia. 7