as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations Multiplication with a constant c cx { cx cx = if c < c > cx m contraction expantion Addition of x, y R n x + y = x x + y y = x + y x + y x m y m x m + y m Inner Product or Scalar Product x = x y = y x n y n x y = x y + x y + + x n y n L x = length of x = x x Let x, y R and θ i denotes the angle between the vector and the x axis Then cos θ = x L x sin θ = x L x cos θ = y L y sin θ = y L y θ = angle between x and y θ = θ θ cos θ = cosθ θ = cos θ cos θ + sin θ sin θ
In general for x, y R n cos θ = x L x y L y + x L x y L y = x y L x Ly cos θ = x y L x L y or x y = L x L y cos θ Remark: cos θ = x and y are perpendicular Length of a vector x R n : L x = x + x + + x n How multiplication with a constant c changes the length? L x = c x + c x + + c x n = c L x Remarks: Let c = L x, then is a vector with unit length and with direction of x L x x If ax + bx = then x = b a x and x have the same direction but different length Unit vectors in R : e = e = Unit vectors in R n : Let e = x = x e = e n = = x e + x e + + x m e n x n Definition: The space of all n-tuples with scalar multiplication and addition as defined above, is called a vector space Definition: by = a x + a x + + a k x k is a linear combination of the vectors x,, x k The zero vector is defined as =
Definition: The vectors x, x,, x k are said to be linearly dependent if there exist k numbers a,, a k not all zero, such that a x + + a k x k = Otherwise x, x,, x k is said to be linearly independent Examples: i e = and e = are linearly independent, because if a e + a e = then a a = a = a = ii Similarly you can prove in R n that e,, e n are linearly independent iii Let x = x = 5 x 3 = Then x, x and x 3 are linearly dependent since x x + 3x 3 = Definition: A set of m linearly independent vectors in R m called a basis for the vector space of m-tuples Theorem: basis Every vector in R m can be expressed as a unique linear combination of a fixed Example Let e,, e m be a basis in R m Then x = x e + x e + + x m e m Definition: The length of a vector x is L x = x + x + + x m Definition: The inner product or dot product of two vectors x, y R m is x y = y x = x y + x y + + x m y m Remark: ilength of a vector x: L x = x x 3
ii Let us denote θ the angle between two vectors x, y R m Then cosθ = x y x x y y Definition: When the angle between two vectors x and y is θ = 9 or 7 we say that x and y are perpendicular or orthogonal Since cos 9 = cos 7 = x and y are perpendicular if x y = Notation Example x y The basis vectors e,, e m are mutually perpendicular e ie j =, i j bf Remark: For any vector x, the unit vector with direction x is fracx x x Results: a z is perpendicular to every vector x R m z = b If z is perpendicular to each vector x,, x k then z is perpendicular to their linear span If z x i =, i =,, k then z a x + a x + + a k x k = c Mutually perpendicular vectors are linearly independent Definition: The projection of x on y P y x = x y y L y If L y = then P y x = x yy Lemma: Let z = x y L y y Then x z is orthogonal to y Proof: y x x y y = y x x y y y L y L y = 4
GRAM-SCHMIDT ORTHOGONALIZATION PROCESS: Given linearly independent vectors x,, x k, there exists mutually perpendicular vectors u,, u k with the same linear span This may be constructed sequentially by setting u = x u = x x u u u u u 3 = x 3 x 3 u u u u x 3 u u u u u k = x k x k u u u u x k u k u u k u k k We can use unit length vectors z,, z k instead of u,, u k ; z j = u j u j u j j =,, k Example Compute u, u, z, z for x = 4 x = 3 5
MATRICES A matrix is a rectangular array of real numbers An m k columns A, R, Σ boldface letters, denote matrices a a a k a A m k = a a k a m a m a mk matrix has m rows and k A = {a ij } i =,, m, j =,, k For example an m matrix has m rows and column, it is an m dimensional vector or column matrix m matrix is a column vector, m matrix is a row vector Definition: Let A = a ij B = b ij be two m k matrices We say that the two matrices are equal A = B a ij = b ij i, j I e two matrices are equal if i their dimensionality is the same, ii every corresponding element is the same MATRIX OPERATIONS Let A, B be two m k matrices Matrix Addition: C = A + B is an m k matrix with elements c ij = a ij + b ij i =,, m j =,, k Scalar Multiplication: c arbitrary scalar A = a ij ca = Ac = B = b ij where b ij = ca ij = a ij c i =,, m j =,, k Matrix Substraction A = a ij B = b ij i =,, m j =,, k A B = A + B = C where C = c ij c ij = a ij b ij i =,, m, j =,, k Definition: Transpose of a matrix A = a ij, i =,, m j =,, k, A is defined as a k m matrix with element a ji j =,, k i =,, m That is the transpose of a matrix A is obtained from A by interchanging the rows and columns 6
Example: A = 3 A = 3 Theorem: Let A, B, C be m k matrices, c, d are scalars a A + B + C = A + B + C b A + B = B + A c ca + B = ca + cb d c + da = ca + da e A + B = A + B f cda = cda g ca = ca Definition:If numbers of rows are equal to the numbers columns of a matrix A, then it is called square matrix Definition: Let A be a k k square matrix The matrix A is said to be symmetric if A = A That is A is symmetric a ij = a ji i, j =,, k Examples: i Let I denotes the m m matrix with -s in the main diagonal and zeros elswhere Then I is symmetric ii 4 A = 4 Definition: I the k k matrix is defined as the identity matrix if it has ones only in the main diagonal and zeros elsewhere Definition: Matrix Product Suppose that A = a ij is an m n matrix B = b ij is an n k matrix Then AB = C = c ij, where C is and m k matrix with elements c ij, i =,, m j =,, k, where c ij is the scalar product of ith row of A with jth column of B, n c ij = a il b lj i =,, m j =,, k l= Example: A = 3 = 3 B = 3 = AB = 3 3 3 7 3 3 = 4 4
Theorem: Properties of matrix multiplication A, B, C defined such that the indicated products are defined and a scalar c is given acab = cab babc = ABC cab + C = AB + AC db + CA = BA + CA eab = B A Important Remarks: AB BA AB = does not imply that A =, or B = Example: 3 4 = However, it is true, that if A = or B = then A B = Definition: The row rank of a matrix A is the maximum number of linearly independent rows, considered as vectors The column rank of A is the maximum number of independent columns considered as vectors Theorem: For any matrix A, the number of independent rows is equal to the number of independent columns Definition: Let A be a k k square matrix The matrix A is nonsingular if Ax = x = where x is a k-dimensional vector A is singular if there exists x such that Ax = Remark: Let a x + a x + + a k x k a Ax = x + a x + + a k x k a k x + a k x + + a kk x k = x a + x a + + x k a k where a,, a k are the column vectors of A Thus, for a nonsingular matrix A x a + + x k a k = x = x = = x k = i e nonsingularty is equivalent that the columns of A are linearly independent 8
Theorem: Let A be a k k nonsingular matrix Then there exists one and only one k k matrix B such that AB = BA = I where I is the identity Definition: The matrix B, such that AB = BA = I is called the inverse of A and it is denoted by A In fact if BA = I or AB = I then B = A 4 3 Example: A = A 3 = AA = 4 3 3 = + 3 6 6 + 3 = I Theorem: a Inverse of a matrix A = a a a a A = A is given by a a a a b In general A is given the following way a ij = A A ji i+j where A ij is the value of the determinant of A ji obtained from A deleting the j-th row and i-th column of A Definition: The determinant of a square matrix A is a scalar denoted by A defined recursively if k = A = a, k A = a ij A ij i+j j= where A ij is the determinant of the k k matrix, obtained as deleting the ith row and jth column of A We can use the first row: A = k a j A j +j j= Theorem: The following statements are equivalent for a k k square matrix A; a Ax = x = ie A nonsingular 9
b A c there exists a matrix A such that AA = A A = I Theorem: Let A and B be k k square matrices, and suppose that both have inverse matrix Then a A = A b AB = B A Proof: a I = A A = A A b ABAB = ABB A = I Theorem: Let A and B be k k matrices a A = A b If each element of a row column of A is zero then A = c If any two rows or columns of A are identical, then A = d If A is nonsingular, then A = ie A A = A e AB = A B f ca = c k A Definition: Let A = a ij be a k k square matrix The trace of the matrix A is tra = k a ii i= Theorem: Let A, B be k k matrices, c scalar Then a trca = ctra b tra ± B = tra ± trb ctrab = trba d trb AB = tra e traa = k k a ij i= j= Definition: A square matrix A is said to be orthogonal if its rows considered as vectors are mutually perpendicular and have unit length, which means that AA = I Theorem: A matrix A is orthogonal if and only if A = A For an orthogonal matrix AA = A A = I Example: A = A A = is orthogonal A = = = I
Definiton: Let A be a k k matrix and I be the k k identity Then the scalars λ, λ,, λ k satisfy the polynomial equation A λi = called eigenvalues of A A λi = called characteristic equation λ Example: A = A λi = 3 4 3 4 λ A λi = λ4 λ = λ = λ = 4 are the eigenvalues Definiton: Let A be a k k matrix Let λ be an eigenvalue of A If x such that Ax = λx then x is said to be an eigenvector of A associated with the eigenvalue λ Ax = λx = A λix = x col A λi + x col A λi + + x k col k A λi If x it means that the columns of A λi are linearly dependent Example: A = λ = λ = 4 3 4 From the first expression, 3 4 3 4 x x x x = = x x 4x 4x x = x 3x + 4x = x x = x = From the second expression x = 4x 3x + 4x = 4x x = x = Usual practise to determine an eigenvector with length one In the example: e i = x i x i x i i =, e = x e = x
Definition: Quadratic form Qx in k variables x,, x k, where x = x,, x k R k, is defined as Qx = x Ax where A is a fix k k matrix The quadratic form is a quadratic function of x, x,, x k Example: x, x x x = x, x x + x x x x + x x x = x + x x + x = x + x Theorem Let A be a k k symmetric matrix, i e A = A then A has k pairs of eigenvalues and eigenvectors λ, e, λ, e,, λ k, e k The eigenvectors can be chosen to satisfy = e e = = e ke k and be mutually perpendicular The eigenvectors are unique unless two or more eigenvalues are equal Definition: Positive Definite Matrices Let A be a symmetric matrix k k A is said to be positive definite if x Ax > for all x R k x A is positive semi-definite if x Ax for all x R k Theorem: Spectral decomposition of a symmetric k k matrix A is given: A = λ e e + λ e e + + λ k e k e k λ,, λ k are eigenvalues of A and e,, e k are the associated normalized eigenvectors Theorem A, k k symmetric matrix is positive definite if and only if every eigenvalue of A is positive A is positive semi-definite if and only if every eigenvalue of A is nonnegative Proof: Trivial from the spectral decomposition theorem A = λ e e + λ e e + + λ k e k e k x Ax = λ x e e x + λ x e e x + + λ k x ke k e kx = λ y + λ y + + λ k y k where y i = x e i Choosing x = e j j =,, k follows the theorem
Another form of the spectral decomposition: λ A = e e k e = PΛP λ k e k INVERZE AND SQUARE ROOT OF A POSITIVE DEFINIT MATRIX By the spectral decomposition, if A is positive definit: A = PΛP Inverse of A If A is positive definite, then the eigenvaluse of A are λ λ, λ λ k > i =,, k The inverse of A is A = PΛ P where Λ = Because; A A = PΛ P PΛP = PΛ ΛP = PP = I Square Root Matrix defined for positive semi-definite A Notice: A A = PΛ P PΛ P = A A = A k A = λ i e i e i = PΛ P i= λ k MATRIX INEQUALITIES AND MAXIMIZATION Cauchy-Schwarz Inequality: Let b, d R p, then b d b bd d, with equality if and only if b = cd Proof: The inequality is obvious when one of the vectors is the zero one For b xd we have that, < b xd b xd = b b xb d xd b + x d d = b b xb d + x d d Adding and substratcting b d /d d, we have that < b b b d d d + d d x b d d d 3
Choosing x = b d/d d follows the statement Extended Cauchy Schwartz Inequality: Let b, d R p and let B be a positive definite p p matrix Then b d b Bbd B d, with equality if and only if b = cb d for some constant c Maximization of Quadratic forms on the Unit Sphere: Let B be a p p positive definite matrix with eigenvectors e,, e p and associated eigenvalues λ,, λ p eigenvectors, where λ λ λ p Then Morover max x x Bx x x = λ x Bx min x x x x Bx max x e,,e k x x and the equality is attained when x = e k+ = λ k+, = λ p 4