Review of Vectors and Matrices

A P P E N D I X D Review of Vectors and Matrices D. VECTORS D.. Definition of a Vector Let p, p, Á, p n be any n real numbers and P an ordered set of these real numbers that is, P = p, p, Á, p n Then P is an n-vector (or simply a vector). The ith component of P is given by p i. For example, P =, is a two-dimensional vector. D.. Addition (Subtraction) of Vectors Consider the n-vectors P = p, p, Á, p n Q = q, q, Á, q n R = r, r, Á, r n For R = P ; Q, component i is computed as r i = p i ; q i. In general, given the vectors P, Q, and S, P + Q = Q + P Commutative law P ; Q ; S = P ; Q ; S Associative law P + - P = 0 zero or null vector CD-45

CD-46 Appendix D Review of Vectors and Matrices D..3 D..4 Multiplication of Vectors by Scalars Given a vector P and a scalar (constant) quantity u, the new vector Q = up = up, up, Á, up n is the scalar product of P and u. In general, given the vectors P and S and the scalars u and g, up + S = up + us Distributive law ugp = ugp Associative law Linearly Independent Vectors The vectors P, P, Á, P n are linearly independent if, and only if n a u j P j = 0 Q u j = 0, j =,, Á, n j = If n a u j P j = 0, for some u j Z 0 j = then the vectors are linearly dependent.for example, the vectors P =,, P =, 4 are linearly dependent because for u = and u = -, u P + u P = 0 D. MATRICES D.. Definition of a Matrix A matrix is a rectangular array of elements. The element a ij of the matrix A occupies the ith row and jth column of the array. A matrix with m rows and n columns is said to be of size (or order) m * n. For example, the following matrix is of size 4 * 3. a a a 3 D.. a A = a a 3 = a a 3 a 3 a ij 4 * 3 33 a 4 a 4 a 43 Types of Matrices. A square matrix has m = n.. An identity matrix is a square matrix in which all the main diagonal elements equal and all the off-diagonal elements equal zero. For example, a 3 * 3 identity matrix is given by 0 0 I 3 = 0 0 0 0

D. Matrices CD-4 3. A row vector is a matrix with one row and n columns. 4. A column vector is a matrix with m rows and one column. 5. The matrix A T is the transpose of A if the element a ij in A is equal to element in A T for all i and j.for example, 4 A = 5 Q A T = a 3 4 5 6 b 3 6 6. A matrix B = 0 is a zero matrix if every element of B is zero.. Two matrices A = a ij and B = b ij are equal if, and only if, they have the same size and a ij = b ij for all i and j. a ji D..3 Matrix Arithmetic Operations In matrices only addition (subtraction) and multiplication are defined. Division, though not defined, is replaced by inversion (see Section D..6). Addition (Subtraction) of Matrices. Two matrices A = a ij and B = b ij can be added together if they are of the same size m * n. The sum D = A + B is obtained by adding the corresponding elements. Thus, If the matrices A, B, and C have the same size, then Product of Matrices. The product D = AB of two matrices, A = a ij and B = b ij, is defined if, and only if, the number of columns of A equals the number of rows of B. If A is of size m * r and B is of size r * n, then D must be of size m * n, where m and n are arbitrary positive integer values. In this case, the elements of D are computed as For example, given we have A + B = B + A A ; B ; C = A ; B ; C Associative law A ; B T = A T ; B T d ij m * n = a ij + b ij m * n d ij = a r k = A = a 3 9 b, B = a5 4 6 8 0 b D = a 3 4 ba5 9 * 5 + 3 * 6 * + 3 * 8 * 9 + 3 * 0 b = a 6 8 0 * 5 + 4 * 6 * + 4 * 8 * 9 + 4 * 0 b 3 3 9 = a 34 46 8 b In general, AB Z BA even if BA is defined. a ik b kj, for all i and j Commutative law

CD-48 Appendix D Review of Vectors and Matrices Matrix multiplication follows these general properties: Multiplication of Partitioned Matrices. Let A be an m * r matrix and B an r * n- matrix. Assume that A and B are partitioned as follows: The partitioning assumes that the number of columns of rows of for all i and j.then B ij For example, I m A = AI n = A, I m and I n are identity matrices ABC = ABC CA ; B = CA ; CB A ; BC = AC ; BC aab = aab = AaB, a is a scalar A = a A A A 3 b, B = A A A 3 B B B B B 3 B 3 is equal to the number of A * B = a A B + A B + A 3 B 3 A B + A B + A 3 B 3 A B + A B + A 3 B 3 A B + A B + A 3 B 3 b A ij 3 4 0 5 = 5 6 8 4 + 3a 8 b a 5 b4 + a0 5 0 ba 8 b = 4 + + 4 a 4 8 b + a40 53 b 30 = 44 6 D..4 Determinant of a Square Matrix Consider the n-square matrix a a Á an a A = a Á an o o o o a n a n Á ann Next, define the product P j j Áj n = a j a j Á a njn such that each column and each row of A is represented exactly once among the subscripts of j, j, Á, and j n. Next, define H j j Áj n = e, j j Á j n even permutation 0, j j Á j n odd permutation

D. Matrices CD-49 Let r represents the summation over all n! permutations, then the determinant of A, det A or ƒ A ƒ, is computed as a r H j j Áj n P j j Áj n Then As an illustration, consider The properties of a determinant are: a a a 3 A = a a a 3 a 3 a 3 a 33 ƒ A ƒ = a a a 33 - a 3 a 3 - a a a 33 - a 3 a 3 + a 3 a a 3 - a a 3. The value of the determinant is zero if every element of a row or a column is zero.. ƒ A ƒ = ƒ A T ƒ. 3. If B is obtained from A by interchanging any two rows or any two columns, then ƒ B ƒ = -ƒ A ƒ. 4. If two rows (or two columns) of A are multiples of one another, then ƒ A ƒ = 0. 5. The value of ƒ A ƒ remains the same if scalar a times a column (row) vector is added to another column (row) vector. 6. If every element of a column or a row of a determinant is multiplied by a scalar a, the value of the determinant is multiplied by a.. If A and B are two n-square matrices, then Definition of the Minor of a Determinant. The minor of the element in the determinant ƒ A ƒ is obtained from the matrix A by striking out the ith row and jth column of B. For example, for a a a 3 A = a a a 3 a 3 a 3 a 33 a M = ` a 3 a `, M a = ` a 3 `, Á 33 a 33 a 3 ƒ AB ƒ = ƒ A ƒƒb ƒ Definition of the Adjoint Matrix. Let A ij = - i + j M ij be defined as the cofactor of the element a ij of the square matrix B. Then, the adjoint matrix of A is the transpose of A ij, and is defined as: a 3 M ij A A Á A n adj A = A ij T A = A Á A n o o o o A n A n Á A nn a ij

CD-50 Appendix D Review of Vectors and Matrices For example, if 3 A = 3 3 3 4 then, A = - 3 * 4 - * 3 = 6, A = - 3 * 4-3 * = -, Á, 6-5 adj A = - -5 4-3 3 - or D..5 Nonsingular Matrix A matrix is of a rank r if the largest square array in the matrix having a non-zero determinant is of size r.a square matrix with a non-zero determinant is called a full-rank or nonsingular matrix. For example, consider 3 A = 3 4 3 5 A is a singular matrix because ƒ A ƒ = * - 0 - * 4 - + 3 * 0-9 = 0 But A has a rank r = because a 3 b = - Z 0 D..6 Inverse of a Nonsingular Matrix If B and C are two n-square matrices such that BC = CB = I, then B is called the inverse of C and C the inverse of B.The common notation for the inverse is B - and C -. Theorem. If BC = I and B is nonsingular, then C = B -, which means that the inverse is unique. Proof. then or By assumption, BC = I B - BC = B - I IC = B -

D. Matrices CD-5 or C = B - Two important results can be proved for nonsingular matrices.. If A and B are nonsingular n-squre matrices, then AB - = B - A -. If A is nonsingular, then AB = AC implies that B = C. Matrix inversion is used to solve n linearly independent equations. Consider a a Á a n x b a a Á a n x b = o o o o o o a n a n Á a nn x n b n where x i represents the unknowns and a ij and b i are constants. These n equations can be written in matrix form as AX = b Because the equations are independent, A must be nonsingular. Thus A - AX = A - b or X = A - b D.. Methods of Computing the Inverse of a Matrix Adjoint Matrix Method. Given A, a nonsingular matrix of size n, A - = adj A = ƒ A ƒ ƒ A ƒ A A Á A n A A Á A n o o o o A n A n Á A nn For example, for 3 A = 3 3 3 4 6-5 adj A = - -5 4, ƒ A ƒ = - -3 3 - TORA s inverse module is based on LU decomposition method. See Press and Associates (986)

CD-5 Appendix D Review of Vectors and Matrices Hence A - = 6-5 - 6 - - 5 - -5 4 = -3 3-3 - 3 5-4 Row Operations (Gauss-Jordan) Method. Consider the partitioned matrix where A is nonsingular. Premultiplying by A -, we obtain A I, A - A A - I = I A - Thus, applying a specific sequence of row transformations, A is changed to I and I is changed to A -. To illustrate the procedure, consider the system of equations: 3 x 3 3 x = 4 3 3 4 x 3 5 The solution of X and the inverse of the basis matrix can be obtained directly by considering A - A I b = I A - A - b The following iterations detail the transformation operation: Iteration 0 Iteration Iteration Iteration 3 3 0 0 3 3 3 0 03 4 3 3 4 0 0 5 3 0 - -4 3 0-3 -5 0 0-03 -3 0 0-5 -3 0-0 43-03 0 0 3-3 0 0-6 - 0 03 5 0 0 3-3 5-4 3 6 3 3 - -4

D. Matrices CD-53 This gives x = 3 and x 3 =, x = 6,. The inverse of A is given by the right-hand-side matrix, which is the same as obtained by the method of adjoint matrix. Product Form of the Inverse. Suppose that two nonsingular matrices, B and B next, differ exactly in one column. Further, assume that B - is given. Then the inverse B next can - be computed using the formula The matrix E is computed in the following manner. If the column vector P j in B is replaced with the column vector P r to produce B next, then E is constructed as an m-identity matrix with its rth column replaced by If B - P j r = 0, then B next does not exist. - The validity of the formula B next is proved as follows. Define F as an m-identity matrix whose rth column is replaced by B - P j that is, Because differs from B only in that its rth column is replaced with P j, then Thus, B next j = B - P j r The formula follows by setting E = F -. The product form can be used to invert any nonsingular matrix, B, in the following manner. Start with B 0 = I = B - 0. Next, construct B as an identity matrix, except that the first column is replaced with the first column in B.Then B i In general, if we construct as an identity matrix with its first i columns replaced with the first i columns of B, then B - ī = E i B i - - B next - = E i E i - B i - This means that for the original matrix B, - - B next -B - P j -B - P j o + o -B - P j m = EB - ; rth place, B - P j r Z 0 F = e, e r -, B - P j, e r +, Á, e m B next = BF = BF - = F - B - B - = E B 0 - = E I = E B - = E n E n - Á E = Á = E i E i - Á E

CD-54 Appendix D Review of Vectors and Matrices The following example illustrates the application of the product form of the inverse. Consider Iteration 0 Iteration Iteration 0 B = 0 0 = B 4 0 B - 0 0 P = 0 0 = - 0 0 0 B = 0 0 4 0 0 0 B 0 = B - 0 = 0 0 0 0 0 0 B = 0 0 4 0 B - 0 P = P = 0 4 + 0 0 E = - 0 0-4 0 0 0 B - = 0 0-0 - ;r = ; r = - 0-4 0 E = 0 + 0 = 0 0 0 - - 0 - B - = B - = E B - 4 0 0 0-4 0 = 0 0 0 0 = 0 0 0-0 -

D. Matrices CD-55 Partitioned Matrix Method. Suppose that the two n-nonsingular matrices A and B be partitioned as shown as follows: A A p * p p * q A =, A A A nonsingular q * p q * q B B p * p p * q B = B B q * p q * q If B is the inverse of A, then from AB = I n, we have Also, from BA = I n, we get Because is nonsingular, exists. Solving for B, B, B, and B, we get where such that A - A A B + A B = I p A B + B A = 0 B A + B A = 0 B A + B A = I q B = A - + A - A D - A A - B = -A - A D - B = -D - A A - B = D - D = A - A A - A To illustrate the use of these formulas, partition the matrix 3 A = 3 3 3 4 A =, A =, 3, A = a 3 b, A = a 3 3 4 b

CD-56 Appendix D Review of Vectors and Matrices In this case, Thus, A - = and which directly give B = A - D = a 3 3 4 b - a - -4 b, 3 = a 3-3 -5 b D - = - a -5 4 5 3 - b = a B = A - 6 B, B = A - B = a 3 b, B = a - 4-3 5 B 5-4 - 3 b b D..8 Matrix Manipulations Using Excel Excel provides facilities for automatically performing the following matrix manipulations:. Transpose.. Multiplication. 3. Inverse of a nonsingular matrix. 4. Determinant value of a nonsingular matrix. Figure D. provides illustrative examples (file excelmatmanip.xls). In Example (Transpose), A is a * 3 matrix whose elements are entered in the range A4:C5. Transpose(A), or A T, appears in the user-specified range E4:F6. The steps for obtaining the output in the selected range are:. Enter the formula = TRANSPOSEA4:C5 in cell E4.. Select (highlight) the output cells E4:F6. 3. Press F. 4. Press CTRL + SHIFT + ENTER. In Example, the elements of the input matrices A and B are entered in the respective ranges A0:C3 and A6:A8. The output matrix is in the (user-selected) range E0:E3. Next enter the formula = MMULT(A0:C3,A6:A8) in cell E0 and follow steps through 4 exactly as in Example (replacing E4:F6 with E0:E3). Notice that MMULT(A6:A8,A0:C3) is undefined. In Example 3, the inverse of the matrix in the range A:C4 is assigned to the range E:G4 by entering the formula = MINVERSEA : C4 in cell E, then following steps, 3, and 4 as in Example. Finally, in Example 4, the determinant of the matrix in the range A8:C30 is obtained by entering the formula = MDETERMA8:C30 in the user-selected cell E8.

D.3 Quadratic Forms CD-5 FIGURE D. Matrix manipulations using Excel (file excelmatmanip.xls) D.3 QUADRATIC FORMS Given and X = x, x, Á, x n T a a Á a n the function a A = a Á a n o o o o a n a n Á a nn QX = X T AX = a n n a i = j = a ij x i x j

4 CD-58 Appendix D Review of Vectors and Matrices is called a quadratic form. The matrix A can always be assumed symmetric because a ij + a ji each element of every pair of coefficients a ij and a ij i Z j can be replaced by without changing the value of Q(X). As an illustration, the quadratic form is the same as 0 x QX = x, x, x 3 6 x 3 0 x 3 x QX = x, x, x 3 3 x 3 x 3 Note that A is symmetric in the second case. We will assume henceforth that A is always symmetric. The quadratic form is said to be. Positive-definite if QX 0 for all X Z 0.. Positive-semidefinite if QX Ú 0 for all X, and there exists X Z 0 such that QX = 0. 3. Negative-definite if -QX is positive-definite. 4. Negative-semidefinite if -QX is positive-semidefinite. 5. Indefinite in all other cases. It can be proved that the necessary and sufficient conditions for the realization of the preceding cases are. Q(X) is positive-definite (-semidefinite) if the values of the principal minor determinants of A are positive (nonnegative). In this case, A is said to be positive definite (semidefinite).. Q(X) is negative-definite if the value of the kth principal minor determinant of A has the sign of - k, k =,, Á, n. In this case, A is called negative-definite. 3. Q(X) is negative-semidefinite if the kth principal minor determinant of A either is zero or has the sign of - k, k =,, Á, n. The kth principal minor determinant of is defined by A n * n a a Á a k a a Á a k 4, k =,, Á, n o o o o a k a k Á a kk

Problems CD-59 D.4 CONVEX AND CONCAVE FUNCTIONS A function f(x) is said to be strictly convex if, for any two distinct points flx + - lx 6 lfx + - lfx and X, where 0 6 l 6. Conversely, a function f(x) is strictly concave if -fx is strictly convex. A special case of the convex (concave) function is the quadratic form (see Section D.3) fx = CX + X T AX where C is a constant vector and A is a symmetric matrix. It can be proved that f(x) is strictly convex if A is positive-definite and f(x) is strictly concave if A is negative definite. X PROBLEMS. Show that the following vectors are linearly dependent. (a) (b). Given find (a) (b) (c) - - 4-3 - - 4-3 -6 4 8 5 0 A + B A - 3B A + B T 3. In Problem, show that AB Z BA 4. Consider the partitioned matrices 4 9 - A = 5-8, B = 9 4 8 3 3 6 0 5 3-4 5-6 9 A =, B = 6 3 3 0 9 4 9 Find AB using partitioned matrix manipulation. 5. In Problem, find A - and B - using the following: (a) Adjoint matrix method (b) Row operations method (c) Product form of the inverse (d) Partitioned matrix method

CD-60 Appendix D Review of Vectors and Matrices 6. Consider B = 0, B - = 4 0 5 5 4-5 8-3 8 4-4 - Suppose that the third vector P 3 is replaced with the V 3 = P + P. This means that the resulting matrix is singular. Show how the product form of the inverse discovers the singularity of the matrix.. Use the product form of the inverse to verify whether each of the following equations has a unique solution, no solution, or an infinity of solutions. (a) x + x = 3 x + 4x = (b) x + x = 5 -x - x = -5 (c) x + x + x 3 = 5 4x + x + 3x 3 = 8 x + 3x - x 3 = 3 8. Verify the formulas given in Section B.. for obtaining the inverse of a partitioned matrix. 9. Find the inverse of A = a H G b, B nonsingular B 0. Show that the following quadratic form is negative definite. Qx, x = 6x + 3x - 4x x - x - 3x - 4. Show that the following quadratic form is positive definite. Qx, x, x 3 = x + x + 3x 3 + x x + x x 3. Show that the function fx = e x is strictly convex over all real values of x. 3. Show that the quadratic function fx, x, x 3 = 5x + 5x + 4x 3 + 4x x + x x 3 is strictly convex. 4. In Problem 3, show that -fx, x, x 3 is strictly concave. SELECTED REFERENCES Hadley, G., Matrix Algebra, Addison-Wesley, Reading, MA, 96. Hohn, F., Elementary Matrix Algebra, nd ed., Macmillan, New York, 964. Press, W., B. Flannery, B. Teukolsky, and W. Vetterling, Numerical Recipes: The Art of Scientific Computing, Cambridge University Press, Cambridge, England, 986.