LINEAR ALGEBRA AND MATRICES. n ij. is called the main diagonal or principal diagonal of A. A column vector is a matrix that has only one column.

Transcription

1 PART 1 LINEAR ALGEBRA AND MATRICES Generl Nottions Mtri (denoted by cpitl boldfce letter) A is n m n mtri n 1... n A ij m1 m... mn ij denotes the component t row i nd column j of A. If m n, A is squre mtri nd its digonl contining 11,,..., nn is clled the min digonl or principl digonl of A A submtri of A is obtined by omitting some rows or columns (or both) froma. Vector (denoted by lowercse boldfce letter) A row vector is mtri tht hs only one row n A column vector is mtri tht hs only one column. b1 b b : bn 1-1

2 Trnsposition T The trnspose A of n m n mtri A ij is the mtri n m tht hs the first row of A s its first column nd the second row of A s its second column, nd so on. Thus the trnspose of A is A T m m ji n n... mn A symmetric mtri is squre mtri tht A T A For emple, A skew-symmetric mtri is squre mtri tht A T A For emple,

3 Mtri Addition, Sclr Multipliction Definition: Equlity of mtrices Mtrices A nd B re equl, written s A=B, if nd only if they hve the sme size nd ll of the corresponding entries re equl. Definition: Mtri ddition Addition is defined only for mtrices A nd B of the sme size, nd their sum, written s A+B, is then obtined by dding the corresponding entries. Emple Given A 0 1 nd B Then AB 3 Definition: Sclr multipliction The product of ny m n mtri A nd ny sclr c, written s ca, is the m n mtri ca obtined by multiplying ech entry in A by c. 1-3

4 Emple Given A A 0 0.9, A 0 1, A According to bove definitions, these properties follow A BBA UVW UVW A 0 A A A 0 c A B cacb c k AcAkA A ck c k 1A A A T T T A B A B T T ca ca 1-4

5 Mtri Multipliction The product C AB of n m n mtri A nd n r p mtri B is defined if nd only if r n, otherwise C is undefined. c ik C is n m p mtri with entries n c b b b... b ik ij jk i1 1k i k in nk j 1 Mtri multipliction is not commuttive AB BA For instnces, wheres AB 0 does not necessrily imply A 0 or B 0 or BA Other properties re kab kab AkB ABC ABC A BCACBC CA BCACB 1-5

6 Specil mtrices A squre mtri whose entries bove the min digonl re ll zero is clled lower tringulr mtri Similrly, n upper tringulr mtri is squre mtri whose entries below the min digonl re ll zero Digonl mtri is squre mtri whose entries bove nd below the min digonl re ll zero nd A digonl mtri whose entries on the min digonl re ll equl is clled sclr mtri. c c... 0 S c AS SA ca 1-6

7 A unit mtri or identity mtri I is sclr mtri whose entries on the min digonl re ll 1. I Trnspose of Product T T T AB B A Inner Product of Vectors b 1 n T 1... n : b i ib nb n i1 bn b b The inner product of vectors is lwys sclr. Product in Terms of Column Vectors Let B b1 b... b n where b1, b,..., b n re column vectors AB Ab1 Ab... Ab n 1-7

8 LINEAR SYSTEMS OF EQUATIONS A liner system of m equtions in n unknowns 1,,..., n is set of equtions of the form 111 1nnb1 b b 1 1 n n m1 1 mn n m jk re given nd clled the coefficients of the systems. b re lso given numbers. i If b i re ll zero, the system is clled homogeneous system. If t lest one b i is not zero, then the system is clled nonhomogeneous system. A solution of the system of equtions is set of number 1,,..., n tht stisfy ll the m equtions. If the system is homogenous, it hs t lest the trivil solution where 1... n 0 Consider the number of equtions nd number of unknowns If there re more equtions thn unknowns, the system is overdetermined. If m n, the system is determined. If there re more unknowns thn equtions, the system is underdetermined. An underdetermined system lwys hs solutions, wheres the other two cses, solutions my or my not eist. 1-8

9 Geometric Interprettion If m=n=, we hve two eqution of two unknowns. If we interpret the two unknowns s pir of coordintes in -D plne, then ech eqution is n eqution for stright line. Ech point on the line is pir of vlues tht stisfies the eqution. Emple Cse () y 1 y 0 No solution if the two lines re prllel. y Cse (b) y 1 y 0 y There is precisely one solution if the two lines intersect Cse (c) y 1 y y There re infinitely mny solutions if the two lines coincide. 1-9

10 Coefficient Mtri From the definition of mtri multipliction we see tht the m equtions my be written s single vector eqution is equivlent to or b n n 1 b 1 1 n n b m1 1 mn n m n 1 b n b : :... b m1 m mn n m A b where the coefficient mtri A jk is the m n mtri A n 1 n m1 m mn, nd 1 : n nd b b b b 1 : m re column vectors. Note tht hs n components (n unknowns) wheres b hs m components (m equtions). 1-10

11 The mtri A n b n b m1 m... mn bm is clled ugmented mtri becuse it is obtined by ugmenting the column vector b to the coefficient mtri A. This ugmented mtri cn completely describe the liner system of equtions. Elementry Row Opertions When we solve the originl set of liner equtions, we cn (1) Interchnge the order of two equtions () Multiply n eqution by nonzero constnt (3) Add constnt multiple of one eqution to nother eqution without ffecting the solution. Therefore, we cn (1) Interchnge two rows of the ugmented mtri () Multiply row by nonzero constnt (3) Add constnt multiple of one row to nother row without chnging the solution. These mnipultions, which produce mtri tht is row-equivlent to the originl one, i.e., yielding the sme set of solution, re clled elementry row opertions. 1-11

12 Guss Elimintion Guss Elimintion is method to obtin the solution of the liner system by performing the elementry row opertions on the ugmented mtri of the liner system. By subtrcting multiple of the first row from ll other rows, it mkes the first column of ll other rows become zero. This process is clled pivoting whereby the first row is the pivot row. Then, by subtrcting multiple of the second row from ll other rows below, it mkes the second column of ll other rows below become zero. This process uses the second row s the pivot row. Repeting similr process for ll the rows below will result in rowequivlent mtri tht is n upper-tringulr mtri, which is clled Echelon form. If the i th column of the i th row hppens to be equl zero, it cn not be used s pivot eqution, so we need to interchnge the i th row with nother row below it. This is clled prtil pivoting. 1-1

13 Emple Augmented mtri Pivot by the first row: Row = Row (-1)*Row 1 Row 3 = Row 3 Row 4 = Row 4 0*Row Move Row to the lst row nd move Row 3 nd 4 up

14 Pivot by the second row: Row 3 = Row 3 (30/10)*Row Now we obtin n upper-tringulr mtri tht is row-equivlent to the originl ugmented mtri. The corresponding system is clled reduced system From this system, we cn esily determine the solution by first clculting 3, then clculting from the known 3, nd then clculting 1 from the known 3 nd. We then obtin, 4,. This lst process is clled bck substitution. 3 1 In this emple, the system is overdetermined becuse there re more equtions thn unknowns. However, there eists one unique solution. 1-14

15 Emple Guss Elimintion for n underdetermined system Augmented mtri is Pivot by Row 1 Row = Row (0.6/3.0)*Row 1 Row3 = Row 3 (1./3.0)*Row 1 Pivot by Row Row 3 = Row 3 (-1)*Row

16 Bck substitution Since 3 nd 4 remin rbitrry, the system hs infinitely mny solutions. If we choose vlue of 3 nd vlue of 4, then the corresponding vlues of 1 nd re uniquely determined. 1-16

17 Emple Guss Elimintion if unique solution eists Pivot by Row Row = Row + 3*Row 1 Row 3 = Row 3 Row Pivot by Row : Row 3 = Row 3 Row Bck substitution, 1, The system hs one unique solution. 1-17

18 Emple Guss Elimintion if no solution eists The ugmented mtri is Row = Row (/3)*Row 1 Row 3 = Row 3 (6/3)*Row 1 Row 3 = Row 3 6*Row /3 1/ /3 1/ The lst row indictes contrdiction. Thus, there is no solution tht will mke the equtions hold. 1-18

19 To determine if the liner system hs unique solution, mny solutions, or no solution, consider the echelon form (upper tringulr mtri t the end of Guss elimintion). It hs generl form of b b : : : :... b 0 b 0 : 0 b n 1 n rr rn r If ny of b,..., r 1 bm is nonzero, no solution eists Otherwise, r 1 m If r If r n, the system hs one unique solution. n, the system hs infinitely mny solutions. 1-19

20 Liner Independence Given ny set of m vectors 1,,..., m, liner combintion of these vectors is of the form c11 c... cm m where c1, c,..., c m re ny constnt sclrs. The set of vectors 1,,..., m is linerly independent if c1 c... c m 0 is the only set of c1, c,..., c m tht mkes liner combintion of them equl to zero c11c... c m m 0 If there eist choice of c1, c,..., c m (not ll zero) tht mkes the liner combintion of 1,,..., m equl to zero, then set of vectors 1,,..., m re sid to be linerly dependent becuse then we cn epress one of them s liner combintion of the others, for emple, Emple 1 c... cm m 1 c re linerly dependent becuse Although this cn be checked esily, it is not esy to discover. 1-0

21 Linerly dependent vectors v v 3 v 1 v 1 + v 1.75v 3 = v 3 v v 1 Linerly independent vectors v 4 v v 1 1-1

22 VECTOR SPACE Spn of set of vectors 1,,..., m is the set of ll liner combintions of 1,,..., m. A vector spce V is set of vectors with the two lgebric opertions of ddition nd sclr multipliction defined such tht the following holds 1) The sum +b of ny vectors nd b in V is lso in V nd the product k of ny vector in V nd sclr k is lso in V. ) For ll vectors nd sclrs we hve the fmilir rules bb bcbc 0 0 k b kkb k l kl kl k l 1 1 -

23 Dimension of V is the mimum number of linerly independent vectors in V nd is denoted by dim V A linerly independent set consisting of mimum possible number of vectors in V is clled bsis. Thus, the number of vectors in bsis for V equls dim V. n Rel n -dimensionl vector spce R is the spce of ll vectors with n rel numbers s components nd rel numbers s sclrs. Rnk of mtri Rnk of mtri is the number of linerly independent row vectors of mtri nd is denoted by rnk A Note tht rnk A = 0 if nd only if A =

24 Theorem 1: The rnk of mtri A equls the mimum number of linerly independent column vectors of A. Hence A nd its trnspose A T hve the sme rnk. Proof: Let rnk A = r. By definition, A hs linerly independent set of r row vectors, clled v 1, v v r. Ech row of A, clled 1, m, cn be written s 1 c11v1c1v... c1rvr c1v1cv... crvr : c v c v... c v m m1 1 m mr r These re vector equtions. Ech of the bove eqution contins n component. If we consider the k th component of ech of the bove equtions, we get 1k c11v1 k c1vk... c1 rvrk c v c v... c v : c v c v... c v k 1 1k k r rk mk m1 1k m k mr rk where k = 1,, n. This cn be written s 1k c11 c1 c1r k c 1 c c r v 1k v k... v rk : : : : c c c mk m1 m mr 1-4

25 The left hnd side is the k th column of A. This shows tht ech columns of A cn be written s liner combintion of set of r vectors, which mens tht the mimum number of linerly independent column vectors of A is no more thn r (=mimum number of linerly independent row vector of A). We cn imgine tht this finding is lso pplicble to A T. The mimum number of linerly independent column vectors of A T (row of A) is no more thn mimum number of linerly independent row vectors of A T (column of A). Thus, they must be equl. Hence the mimum number of linerly independent column vectors of A equls to the rnk of mtri A nd the T rnk A rnk A The spn of the row vectors of mtri A is clled the row spce of A nd the spn of the column vectors the column spce of A. From Theorem 1, we thus hve Theorem : The row spce nd the column spce of mtri A hve the sme dimension equl to rnk A. Theorem 3: Row-equivlent mtrices hve the sme rnk. By definition, rnk of mtri is equl to the dimension of its row spce nd row spces of row-equivlent mtrices re the sme, so row-equivlent mtrices hve the sme rnk. By this theorem, we cn immeditely determine the rnk of mtri from its echelon form. 1-5

26 Emple Determine of the rnk of Row = Row + *Row 1 Row 3 = Row 3 7 *Row 1 Row 3 = Row 3 + ½*Row Therefore, rnk A = 3 0 A

27 Theorem 4: p row vectors 1,,..., p (with n components ech) re linerly independent if the mtri with row vectors 1,,..., p hs rnk p; they re linerly dependent if tht rnk is less thn p. A n 1... n : : : : : p p1 p... pn Theorem 5: p vectors with n components re lwys linerly dependent if n < p. Proof: From Theorem 1, the mimum number of linerly dependent row vector of A equls the mimum number of linerly dependent column vector of A. Therefore, there cn be t most n linerly independent row vectors. From definition of dimension, we thus hve Theorem 6: The vector spce R n consisting of ll vectors with n components hs dimension n. 1-7

28 Generl Properties of Solutions of Liner Systems Theorem 7: Fundmentl Theorem for liner systems ) A liner system of m equtions in n unknowns 1,, n hs solutions if nd only if the coefficient mtri A nd the ugmented mtri A hve the sme rnk. b n n 1 b 1 1 n n b m1 1 mn n m (1) b) If this rnk r equls n, the system (1) hs precisely one solution. c) If r < n, the system (1) hs infinitely mny solutions, ll of which re obtined by determining r suitble unknowns (whose submtri of coefficients must hve rnk r) in terms of the remining n - r unknowns, to which rbitrry vlues cn be ssigned. d) If solutions eist, they cn ll be obtined by the Guss elimintion. 1-8

29 Proof: ) Write A in term of column vector c 1,, c n. A c1... c n From A = b, if A hs solution 1,, n, then b cn then be written s b c c c n n b is liner combintion of column vectors of A. Adding linerly dependent column vector b to coefficient mtri A does not chnge the rnk of A, thus the ugmented mtri A hs the sme rnk s A. Conversely, if rnk A = rnk A, b must be liner combintion of column vectors of A, nd solution must eist. b) If this rnk A = rnk A = r = n, the n column vectors of A re linerly independent. If 1,,..., n re solution, the representtion b c c c n n which is liner combintion of linerly independent vectors must be unique. Becuse if there is y 1,, y n such tht then b y c y c y c n n ( y ) c ( y ) c... ( y ) c 0 n n n From the definition of liner independence, i y i must be 0, so y i must be the sme s i nd solution is unique. 1-9

30 c) If rnk A = r < n, the column vectors of A consists of r linerly independent vectors nd n r dependent vectors. Suppose we reorder the unknowns nd columns of A such tht the first n columns re linerly independent nd the lst n r columns re liner combintion of the first r. When solution eists, b cn be written s liner combintion of column vectors of A. b c c... c c... c 1 1 r r r1 r1 n n All of columns vectors b, c 1,, c n re in the sme vector spce of dimension r. Then, the vector b( c... c ) c c... c r1 r1 n n 1 1 r r cn lso be represented by liner combintion of only the first r column vectors. Even if the unknowns,..., r 1 n is chnged rbitrrily, the right hnd side cn be re-clculte to restore the equlity. 1-30

31 Homogeneous System n 1 1 n m1 1 mn n n n () Theorem 8: Homogeneous system ) A homogeneous system () lwys hs the trivil solution 1 = = = n = 0. b) Nontrivil solution eist if nd only if rnk A < n. c) If rnk A = r < n, these solutions, together with = 0, form vector spce of dimension n - r. In prticulr, if 1 nd re solution vectors of (), then = c c, where c 1 nd c ny sclrs, is solution vector of (). The vector spce of ll solution of () is clled the null spce of the coefficient mtri A, becuse if we multiply ny in this null spce by A we get 0. The dimension of the null spce is clled the nullity of A. In term of these concepts, Theorem 8 sttes tht rnk A + nullity A = n where n is the number of unknowns (number of columns of A) Theorem 9: System with fewer equtions thn unknowns A homogeneous system of liner equtions with fewer equtions thn unknowns lwys hs nontrivil solutions. 1-31

32 Nonhomogenous System Theorem 10: Nonhomogenous System If nonhomogenous liner system of equtions of the form (1) hs solutions, then ll these solutions re of the form 0 h where 0 is ny fied solution of (1) nd h runs through ll the solutions of the corresponding homogeneous system (). Proof: Let be ny given solution to the nonhomogeneous system (1) nd 0 be n rbitrrily chosen solution of (1). Then, Then, A = b nd A 0 = b. A( 0 ) = 0 ( 0 ) is solution to the homogeneous system () = h. Thus, = 0 + h 1-3

33 INVERSE OF A MATRIX In this section, we consider only squre mtrices. The inverse of n n n mtri A is denoted by A -1 nd is n n n mtri such tht where I is the n 1 1 AA A A I n unit mtri. If A hs n inverse, then A is clled nonsingulr mtri. If A hs no inverse, then A is clled singulr mtri If A hs n inverse, the inverse is unique. If both B nd C re inverses of A, then tht we obtin the uniqueness from AB I nd CA I so B IBCAB CAB CI C Theorem 1: Eistence of the inverse The inverse A -1 of n n n mtri A eists if nd only if rnk A = n. Hence A is nonsingulr if rnk A = n, nd is singulr if rnk A < n. 1-33

34 Determintion of the Inverse From the definition of inverse of mtri AA 1 I 1 If we write A s mtri consisting of column vector 1,,..., n nd write I s mtri consisting of column vector e1, e,..., e n where the i th component of e i equl to 1 nd other components re zero A e e 1 n 1 n We cn determine 1 A if we solve for ech of i from A e. i To do this, we ugment the coefficient mtri by ll column vectors e i nd perform row opertion until the left prt of ugmented mtri becomes unit mtri (Guss-Jordn 1 Elimintion) then we will obtin 1,,..., n (= A ) in the right prt of the reduced mtri. i Guss-Jordn Elimintion is similr to the Guss Elimintion but lso eliminting the entries bove the digonl nd scled ll of the digonl entries to be equl to one. 1-34

35 Emple Find the inverse of A A I = Row = Row + 3*Row1 Row 3 = Row 3 Row 1 Row 3 = Row 3 Row This is the end of the Guss Elimintion. Net, perform Guss-Jordn elimintion to obtin unit mtri in the left prt. Scle ech row by 1/digonl to get digonl equl to

36 Row 1 = Row 1 + *Row 3 Row = Row 3.5*Row 3 Row 1 = Row 1 + Row The lst three columns is 1 A. Let s check it

37 Useful Formuls For nonsingulr mtri, we obtin A A det A 1 11 where det A Emple A A Useful Formuls For nonsingulr digonl mtri, we hve A nn A 1 1/ / nn Emple A A Inverse of the Inverse is the given mtri 1 A 1 A 1-37

38 Inverse of Product AC C A To prove this, we strt from the definition 1 AC AC I Pre-multiplying by 1 A gives Agin, pre-multiplying by 1 1 A CAC 1 C gives AC C A We cn similrly generlize this formul to the products of more thn two mtrices. AC PQ Q P C A Vnishing of Products From the strnge fct tht AB 0 does not necessrily imply A 0 or B 0 or BA Ech of these two mtrices hs rnk less thn n =. This sitution chnges when n n mtrices hve rnk n. 1-38

39 Theorem : Cncelltion Lw Let A, B, C be n n mtrices. Then: () If rnk A = n nd AB = AC, then B = C (b) If rnk A = n, then AB = 0 implies B = 0. Hence if AB = 0, but A 0 s well s B 0, then rnk A < n nd rnk B < n. (c) If A is singulr, so re AB nd BA. Proof: () If rnk A = n, A is nonsingulr. We cn pre-multiply 1 A on both sides of AB = AC, nd then obtin B = C. (b) Similrly, we cn pre-multiply 1 A on both sides. (c) A is singulr rnk A < n A=0 hs nontrivil solution. Multipliction by B gives BA=0 rnk of BA < n BA is singulr. From the theorem tht the trnspose of ny mtri hs the sme rnk of the originl mtri, rnk A T < n A T is singulr B T A T is singulr rnk B T A T < n rnk (AB) T < n AB is singulr. 1-39

40 DETERMINANTS Determinnts were first defined for solving liner systems, lthough they re imprcticl in computtions. They hve importnt engineering pplictions in eigenvlue problem nd differentil equtions. An n th -order determinnts is n epression ssocited with n n n squre mtri beginning with n=. A determinnt of second order is denoted nd defined by D = det A = Emple

41 Crmer s rule For liner system of equtions b b 1 1 the solutions cn be obtined from 1 D1/ D, D/ D where b1 1 D1 b 1 1b b b D b b b Emple D 14, D , D /14 6 nd 56/

42 Third-Order Determinnts A determinnt of third order cn be defined by D Note tht the signs on the right re + +. Ech term on the right is the entry of the first column multiplying its minor, which is the determinnt of submtri obtined by deleting the row nd column of tht entry. D Crmer s rule For liner system of equtions b1 b b D 1, D D D3, 3 D D where D is the determinnt of the system nd b D b, 1 3 b b D b, 1 3 b b D b 3 1 b

43 Determinnt of Any Order n A determinnt of order n is sclr ssocited with n n n mtri n 1... n D det A : : : :... It is defined for n 1 by nd for n by n1 n nn D 11 D j1cj1j Cj... jncjn (j = 1,,, or n) or D 1 kc1k kc k... nkcnk (k = 1,,, or n) where 1 j C k M jk jk nd M jk is determinnt of order n 1 of the submtri of A obtined by deleting the row nd column of the entry jk (the j th row nd the k th column). C jk is clled the cofctor of jk in D nd M jk is clled the minor of jk in D. The epression for D my be written s D D n jk 1 jkm jk (j = 1,,, or n) k 1 n jk 1 jkm jk (k = 1,,, or n) j1 1-43

44 Emple: Minors nd cofctors of third order determinnt The minors re M M M M M M M M M nd the cofctors re C C C M C1 M1 C13 M M C M C3 M3 1 1 M C3 M3 C33 M

45 Emple D The epnsion by the first row is D The epnsion by the lst column is D Determinnt of tringulr mtri The determinnt of ny tringulr mtri equls the product of ll the entries of the min digonl. This cn be shown by epnding by rows if the mtri is lower tringulr nd epnding by columns if the mtri is upper tringulr. Emple

46 GENERAL PROPERTIES OF DETERMINANTS Theorem 1: Trnsposition The vlue of determinnt is not ltered if its rows re written s columns, in the sme order. Emple det A T det A Theorem : Multipliction by constnt If ll the entries in one row (or one column) of determinnt re multiplied by the sme fctor k, the vlue of the new determinnt is k times the vlue of the given determinnt. Note tht Emple det ka k n det A Theorem 3: If ll the entries in row (or column) of determinnt re zero, the vlue of the determinnt is zero. Theorem 4: If ech entry in row (or column) of determinnt is epressed s binomil, the determinnt cn be written s the sum of two determinnts. 1-46

47 Emple d b c b c d b c d b c b c d b c d b c b c d b c Theorem 5: Interchnges of rows or columns If ny two rows (or two columns) of determinnt re interchnged, the vlue of the determinnt is multiplied by 1. Emple Theorem 6: Proportionl rows or columns If corresponding entries in two rows (or two columns) of determinnt re proportionl, the vlue of the determinnt is zero. Emple

48 Theorem 7: Addition of row or column The vlue of determinnt is left unchnged if the entries in row (or column) re ltered by dding to them ny constnt multiple of the corresponding entries in ny other row (or column, respectively). Emple The determinnt of mtri cn be determined by using row opertions (Guss elimintion) to obtin n upper tringulr mtri. Keep in mind tht interchnging ny two rows will ffect the vlue of the determinnt by fctor of -1. D

49 In ech cycle of elimintion, the determinnt cn be epnded s the summtion of products of entries in the column nd their corresponding cofctors. Only one term will remin; thus, we cn write them in compct form s the following D Theorem 8: Determinnt of product of mtrices For ny n Emple n mtrices A nd B det AB det BA det A det B

50 If the entries of squre mtri re sclrs (constnts), its determinnt is lso constnt. If the entries of squre mtri re functions, its determinnt is lso function nd we cn determine its derivtive from the following useful formul. Theorem 9: Derivtive of determinnt The derivtive D ' of determinnt D of order n whose entries re differentible functions cn be written s D' D D... D n 1 where D j is obtined from D by differentiting the entries in the j th row. Emple: Derivtive of third-order determinnt d f g h f ' g' h' f g h f g h p q r p q r p ' q ' r ' p q r d u v w u v w u v w u ' v ' w ' Rnk in Terms of Determinnts Theorem 1: Rnk in terms of determinnts An m n mtri A hs rnk r 1 if nd only if A hs n r r submtri with nonzero determinnt, wheres the determinnt of every squre submtri with r 1 or more rows tht A hs is zero. In prticulr, if A is squre mtri, A is nonsingulr, so tht the inverse A -1 eists, if nd only if det A

51 Theorem : Crmer s Theorem () If the determinnt D det A of liner system of n equtions nnb1 11 nnb b n1 1 nn n n in the sme number of unknowns is not zero, the system hs precisely one solution. This solution is given by the formuls 1 D 1, D D Dn,, n D D where D k is the determinnt obtined from D by replcing in D the k th column by the column with entries b 1,, b n. (b) Hence if the system is homogeneous nd hs only the trivil solution 1 0, 0,, n 0. If D=0, the homogeneous system lso hs nontrivil solutions. Theorem 3: Inverse of mtri The inverse of nonsingulr n n mtri A jk is given by C11 C1... Cn T 1 C1 C... C n A C det A det A : : : : C1n Cn... Cnn where C jk is the cofctor of jk in det A. T C is clled djoint of A. 1-51

52 Emple A det A C , C1, C C , C 1 1 4, C C , C , C A

53 PROGRAMMING OF GAUSS ELIMINATION ) ALGORITHM GAUSS ( A jk A b Input: Augmented n n 1 A jn, 1 b j mtri jk, where Output: Solution j or messge tht the system hs no unique solution. For k = 1 to n-1 (k refers to the unknown being eliminted) If kk =0 then Find the smllest j k such tht jk 0 If no such j eists, then output No unique solution eists nd stop Else Echnge the contents of row j nd k of A For j = k +1 to n m jk jk kk (j refers to row being eliminted) End End For p = k+1 to n+1 (p refers to column on row j) End jp jp mjkkp If 0 nn then output No unique solution eists nd stop Else Strt Bck Substitution 1-53

54 nn, 1 n nn For i = n-1 downto 1 (i refers to unknown being determined) End n 1 i i, n1 ij j ii j i 1 Output j nd stop End GAUSS m jk is clled the multiplier becuse the pivot eqution is multiplied by this fctor before it is subtrcted from the j th eqution. Opertion count The qulity of numericl method is judged in terms of Amount of storge Amount of time (=number of opertions) Effect of round-off error For Guss elimintion, the number of opertion is s follows. In step k we eliminte k from n-k equtions. This needs n-k divisions in computing the m jk nd (n-k)(n-k+1) multiplictions nd subtrctions. We do such elimintion steps for k from 1 to n

55 The totl number of opertions for Guss elimintion is n1 n1 1 f n n k n k n k k1 k1 1 f n n1n n 1n n where n /3 is obtined by dropping the lower powers of n. 3 We see tht f(n) grows bout proportionl to n. We sy tht 3 f(n) is of order n nd write 3 f n O n where O suggest order. The generl definition of O is if f n/ f n O h n h n remins bounded (does not grow to infinity) For bck substitution, we mke n-i multiplictions nd subtrctions nd 1 division. The totl number of opertions in the bck substitution is n 1 bn n i n nn n n On i1 We see tht it grows more slowly thn the number of opertions in the forwrd elimintion of Guss lgorithm. Emple n Elimintion Bck substitution sec sec min 0.5 sec 1-55

56 Difficulty with smll pivot entries If pivot entry kk is zero, we must interchnge tht row with the other. In ddition to this, if kk is too smll, should still interchnge the equtions becuse m jk will be very lrge nd we will subtrct lrge multiple of one row from the others. This will crete more round-off error nd ffect ccurcy of the result. Emple We know tht the ect solution is 1 = 10 nd = 1. Let s try to solve this problem by Guss elimintion using 4-digit floting-point numbers. () m / Subtrct m*eqution 1 from Eqution, we get Then / nd from the first eqution * This lrge error occurs becuse 11 is smll compre to 1 so the smll round-off error in led to lrge error in

57 (b) Let s use the second eqution s the pivot eqution for the first unknown. m=0.0004/0.4003= Subtrcting m* Eqution from Eqution 1, we get nd from the second eqution = = 1 1 = 10 This success occurs becuse 1 is not very smll compred to, so smll round-off error in would not led to lrge error in 1. Therefore, we would prefer to use the pivot eqution such tht the pivot is lrgest coefficient of unknowns in bsolute vlue in its eqution. In prctice, one picks the pivot eqution such tht the rtio between the pivot term nd the lrgest coefficient in its row is the lrgest mong ll vilble rows. Emple: Choice of pivot eqution We hve 11 3 m 5 1k m 3 k 31 6 m 8 3k 1-57

58 Therefore, we select the second eqution s the pivot eqution nd eliminte We now hve m * * k 6 m * 3 * 3k 1 1 Here we select the lst eqution s the pivot eqution nd get the tringulr system Bck substitution gives 3 1, 3, nd

59 ILL-CONDITIONING AND NORMS Sometimes we observe tht some systems A=b give n ccurte solution while some other systems do not. We cll A=b well-conditioned if smll errors in the coefficients or in the solution process hve only smll effect on the solution nd ill-conditioned if the effect on the solution is lrge. For two equtions in two unknowns, ill-conditioning occurs if nd only if the equtions represent two nerly prllel lines, so tht the intersection moves very much if the line is shifted only slightly. Emple y = 1 y = 1 The ect solution to this system is = 0.5 nd y = -0.5 wheres the system y = 1 y = 1+e hs the solution = e nd y = e This shows tht if the eqution is modified slightly by e the solution will chnge significntly by 5000e. Note tht the lines of both equtions re nerly prllel. 1-59

60 RESIDUAL Residul of n pproimte solution of A=b is defined s nd becuse b=a, thus r b A r A r is smll when is very ccurte but smll r does not imply ccurcy of. Emple = =.0001 The ect solution is 1 = 1 nd = 1. Suppose tht we hve bd solution 1 = nd = r One might be fooled if he only inspects the residul nd thinks tht those solutions re ccurte becuse of its smll residul. To mesure the ill-conditioning of liner system, we need to define the followings. 1-60

61 VECTOR NORM Vector norm of denoted by is generlized length of the vector. It must hve the following properties: is nonnegtive rel number =0 if nd only if =0 k k for ll k y y (Tringulr Inequlity) Definition of p-norm 1/ p 1... p n where p is fied number nd p 1 Emple is clled l 1 -norm n 1 is clled Eucliden or... n m is clled l -norm j j p p p l -norm For n=3, l -norm is the usul length of vector in 3-D spce. Emple T If , then If nd re the ect nd pproimte solutions, is mesure of the distnce between them, or the error of. 1-61

62 MATRIX NORM If A is n n n mtri nd is ny vector with n components, then A is vector with n components. We cn tke the vector norm of A nd. One cn prove tht there is number c such tht A c Among ny nonzero, the smllest c is the mtri norm of A. Alterntively, A A m m 1 A A Note tht A depends on the corresponding vector norm. One cn show tht A equl the lrgest column bsolute sum for l 1 -norm A equl the lrgest row bsolute sum for l -norm The mtri norm hs the following properties A A (by definition) AB A B n A A n 1-6

63 Emple: An ill-conditioned system A nd A For l 1 -norm, A =.0001 nd 1 A = For l -norm, A = nd 1 A = Note tht 1 A >> A. 1-63

64 CONDITION NUMBER OF A MATRIX The condition number of squre nonsingulr mtri is defined s 1 A A A Theorem 1: Condition number A liner system of eqution A=b whose condition number is smll is well-conditioned. A lrge condition number indictes ill-conditioning. It cn be shown tht r A b where r ba nd is the pproimte solution. δ δa δ A A nd A δb b This mens tht the reltive error in the solution is bounded by the condition number multiplied with reltive residul, reltive error in A, or reltive error in b. Emple From previous emple, A =0001. The condition number of I is equl to 1, which is the lowest possible vlue of condition number. For the mtri tht is closed to singulr mtri ( det A is smll compred to the mimum entry of A), the condition number will be lrge. 1-64

65 LU-FACTORIZATION A given squre mtri A cn be written s A=LU where L is lower tringulr nd U is upper tringulr. For emple, A 8 5 = LU = It cn be proved tht for ny nonsingulr mtri, the rows cn reordered so tht the resulting mtri A hs n LUfctoriztion, where the mtri L contin multipliers m jk in the Guss elimintion wheres U is the upper tringulr reduced mtri. The importnt ide is tht L nd U cn be computed directly without using the Guss elimintion. These computtion needs bout n 3 /3 opertions, which is only hlf s mny s the Guss elimintion, which needs bout n 3 /3. Once we hve the L nd U, we cn determine the solution by bck substitution in two steps, ech of which involves only n opertions. Note from A = LU = b tht we cn solve for y from Ly = b nd then from U = y. The method to compute L nd U is clled Doolittle s method. This method gives lower tringulr tht hs ll entries on the digonl equl to 1. Another method is clled Crout s method. It gives n upper tringulr tht hs ll digonl entries equl to

66 Doolittle s method u k = 1,, n j1 j =,, n u 1k 1k m j1 11 j 1 k = j,, n; j u m u jk jk js sk s1 k 1 1 jk jk u kk s1 js sk m m u j = k+1,, n; k Crout s method m u j = 1,, n 1 k k =,, n m j1 j1 1k 11 k 1 m m u j = k,, n; jk jk js sk s1 j 1 k 1 ujk jk mjsusk k = j+1,, n; j m jj s1 Emple A u 11 = 11 = u 1 = 1 = 3 m 1 = 1 /u 11 = 8/ = 4 u = -m 1 u 1 = 5-4*3 =

67 Cholesky method Another specil method especilly for symmetric positive definite (s.p.d.) mtri (A=A T nd T A > 0 for ll ) is clled Cholesky s method. We cn choose U = L T, so A=LL T. Emple Cholesky method m m j1 j1 j =,, n m 11 j1 jj jj js s1 m m j =,, n k 1 1 jk jk js ks m kk s1 m m m j = k+1,, n; k If A is not positive definite, it will result in comple numbers. 1-67

68 METHOD OF LEAST SQUARE When the number of equtions is greter thn number of unknowns (m > n), the system is clled overdetermined. No solution cn stisfy the eqution ectly. However, we cn determine the best solution tht gives the lest residul r. To minimize r, we could choose ny norm, such s r, r, 1 r. The lst one which corresponds to the minimizing the sum of the squred residuls problem. Norml eqution T r is equl to T minimizes T minimum vlue of condition m ri is liner lest squre i1 A b A b ; therefore, tht A b A b lso minimizes r. At the A b A b, we hve the following d d T A b A b 0 T A A b 0 T AA T Ab This is clled the Norml eqution. T The mtri AA is squre n n mtri. The problem becomes n liner equtions in n unknowns, which cn be solved by Guss elimintion. 1-68

69 Emple: Liner regression Determine liner eqution tht best fit the eperimentl dt given below. p q The liner eqution hs the form q p 1 p nd the set of equtions re A b

70 T 8 36 AA T 36 Ab q p The condition number of T AA using -norm (A) =

71 Emple Determine the prmeters of cubic eqution tht best fits the given set of dt in the previous emple. The set of equtions re q( p) p p p A b

72 T AA T Ab q( p) p1.07 p p q p T The condition number of AA using -norm (A) = 5.03E06 T In prctice, the mtri AA is often ill-conditioned. There re other techniques tht void solving the norml equtions directly. 1-7

73 QR-Fctoriztion (decomposition) Let A be n m n mtri with m n. Suppose tht A hs full column rnk. Then there eist n m n orthogonl mtri Q nd n n n upper tringulr mtri R with positive digonl r ii > 0 such tht A=QR. Orthogonl (Unitry) mtri T 1 Q Q or T T QQQQ I The condition number of n orthogonl mtri equls to 1 The solution of the lest squre problem cn be determined by 1 T R Q b Emple Let A The QR decomposition is Q R One cn verify tht QR=A nd Q T Q=I. 1-73

74 Singulr Vlues Decomposition (SVD) Let A be n m n mtri with m n. Then we cn write T A UV, where U is m n nd U T U = I, V is n n nd V T V = I, nd is n n n digonl mtri=dig(,..., 1 n) with 1... n 0. The columns u 1,, u n of U re clled left singulr vectors. The columns v 1,, v n of V re clled right singulr vectors. The i re clled singulr vlues. The solution of the lest squre problem cn be determined by 1 T VΣ U b Emple Determine the SVD of A in the previous emple A U Σ V One cn verify tht T UV A, U T U = I, nd V T V = I. 1-74

75 PSEUDO-INVERSE Suppose tht A is n m n mtri with m n nd hs full T rnk, with A QR UV being its QR decomposition nd SVD, respectively. Then 1 T T 1 T 1 T A A A A R Q VΣ U is clled the (Moore-Penrose) pseudoinverse of A. If m < n, then A + =A T (AA T ) -1. The pseudoinverse lets us write the solution of the full-rnk, overdetermined lest squres problem s simply =A + b. If A is squre nd full rnk, this formul reduces to =A -1 b s epected. MATLAB COMMANDS det(a) return the determinnt of A inv(a) return the inverse of A cond(a) return the condition number of A [q r]=qr(a) return the Q nd R mtrices for QR [u s v]=svd(a) return the U,, nd V mtrices for SVD pinv(a) return the pseudoinverse of A =A\b help elmt help mtfun solve the liner system of eqution lists Elementry mth functions. lists more numericl liner lgebr. 1-75

76 EIGENVALUES AND EIGENVECTORS Eigenvlue problems re mong the most importnt problems in engineering in connection with mtrices. For given squre n n mtri A, consider the vector eqution A where is sclr number. It is cler tht 0 is trivil solution for ny vlue of. For the cse of nontrivil solutions, vlue of is clled n eigenvlue or chrcteristic vlue of the mtri A nd the corresponding vector is clled eigenvector or chrcteristic vector of A corresponding to tht eigenvlue. The spn of ll eigenvectors of A is clled the eigenspce of A. Emple Given: Eigenvlue problem: A 5 A

77 A I 0 This is homogeneous liner equtions. By Crmer s Theorem, it hs nontrivil solution 0 if nd only if its coefficient determinnt is zero. D detα I We cll D the chrcteristic determinnt nd D 0 the chrcteristic eqution of A. The solutions re 1 1 nd. These re the eigenvlues of A. 6 The eigenvector corresponding to 1 cn be obtined by solving the homogeneous eqution 1 A I Obviously, the coefficient mtri of this system does not hve full rnk becuse its determinnt is zero. Hence, there re infinitely mny solutions. If we choose 1 rbitrrily equl to k, we get = k. The eigenvectors corresponding to 1 1 re k 1 1 where k

78 We cn check tht A The eigenvector corresponding to cn be obtined by solving the homogeneous eqution A I If we choose 1 rbitrrily equl to k, we get = -k. The eigenvectors corresponding to 6 re Theorem 1: Eigenvlues k 1 where k 0 The eigenvlues of squre mtri A re the roots of the corresponding chrcteristic eqution. Hence n n n mtri hs t most n numericlly different eigenvlues. The set of ll eigenvlues of A is clled the spectrum of A. The lrgest of the bsolute vlues of the eigenvlues of A is clled the spectrl rdius of A. Theorem : Eigenvectors If is n eigenvector of A corresponding to n eigenvlue, so is k with ny k

79 Applictions of Eigenvlue Problems Stress Representtion The stte of stress in -D cn be written in mtri form. σ The principl stresses re the eigenvlues of σ. For the cse 3-D, this concept is lso pplicble, but the mtri σ will become 33 mtri. 1-79

80 Emple 1 1 σ = , 1 For 1 1, For 1, The eigenvector points towrd the direction of the principl stress (corresponding eigenvlue)

81 Roots of Chrcteristic Eqution The chrcteristic eqution D A I n... : :... : 1 n det 0... n1 n nn is n n th degree polynomil, which generlly hve n roots (some my be comple numbers). The polynomil mybe written s M1 M j M j Suppose tht there re j different roots. Some roots my be repeted, e.g., M. M is clled the lgebric multiplicity of. The number of linerly independent eigenvectors corresponding to the eigenvlue is clled the geometric multiplicity m. In generl, m M. 1-81

82 Emple The chrcteristic eqution is 3 A det AI Epnding the determinnt gives which cn be written s The solution is 1 5 nd 3 3. We cn see tht the lgebric multiplicity of 3 is equl to. The eigenvector corresponding to 1 cn be determined from

83 If we rbitrrily choose =, we get 3 = -1 nd 1 = The eigenvector corresponding to cn be determined from This homogeneous system hs rnk = 1, so we cn choose the vlues of 3-1= unknowns rbitrrily. From the theorem tht rnk A nullity A n nullity A n rnk A 3 1 This mens tht the dimension of null spce equl to, or there re two linerly independent vectors tht stisfy the eqution. If we choose = 1 nd 3 = 0, we get 1 = -1. If we choose = 0 nd 3 = 1, we get 1 =

84 Therefore, the eigenvectors corresponding to 3 re 1 0 nd The geometric multiplicity of 3 thus equls to. IMPORTANT NOTES The entries on the min digonl of tringulr mtri re its eigenvlues. The eigenvlues of symmetric mtri re rel numbers The eigenvlues of skew-symmetric mtri re pure imginry or zero. The eigenvlues of n orthogonl mtri re rel or comple conjugtes in pirs nd hve bsolute vlue equl

85 SIMILARITY OF MATRICES An n n mtri  is clled similr to n n n mtri A if ˆ 1 A T AT for some nonsingulr n n mtri T. This trnsformtion, which gives  from A, is clled similrity trnsformtion. Theorem 1: Eigenvlues nd eigenvectors of similr mtrices If  is similr to A, then  hs the sme eigenvlues s A. Furthermore, if is n eigenvector of A, then y = T -1 is n eigenvector of  corresponding to the sme eigenvlue. Proof Let be n eigenvlue of A with s the corresponding eigenvector. A 1 1 T A T T ATT T ˆ ( 1 ) ( 1 AT T ) Hence, is n eigenvlue of corresponding eigenvector.  with T -1 s the PROPERTIES OF EIGENVECTORS Theorem : Liner independence of eigenvectors Let 1,,..., k be distinct eigenvlues of n n mtri. Then corresponding eigenvectors 1,,..., k form linerly independent set. 1-85

86 Theorem 3: Bsis of eigenvectors If n n n mtri A hs n distinct eigenvlues, then A hs n bsis of eigenvectors for. Theorem 4: Eigenvectors of symmetric mtri A symmetric mtri hs n orthonorml bsis of n eigenvectors for. Definition of An Orthonorml Set If i nd j re vectors in n orthonorml set, i j 1 if i j 0 if i j Emple 5 Given: A The eigenvectors corresponding to 1 1 nd 6 re 1/ 5 1 nd / 5 We cn check tht / 5 1/

87 DIAGONALIZATION An n n mtri A is digonlizble if nd only if there eists 1 mtri X such tht X AX is digonl mtri. Theorem 5: Digonliztion of mtri If n n n mtri A hs bsis of eigenvectors, then D 1 X AX is digonl, with the eigenvlues of A s the entries on the min digonl. Here X is the mtri with these eigenvectors s column vectors. Also m 1 m D X A X Emple 3 A hs eigenvlues =5, -3, nd -3. columns vectors re the eigenvectors of A, is 1 3 X 1 0 nd X The mtri X, whose Then, we cn obtin D X AX

88 Emple The given mtri 4 3 A hs eigenvlues = -, -, nd 1. Let s try to find the eigenvector corresponding to 1. A I The rnk of the coefficient mtri of this homogeneous system equls to. The nullity (dimension of the null spce) is then equl to 3 1. Hence, there is only one rbitrry unknown. The eigenvector for this eigenvlue is We cn see tht the geometric multiplicity m of this eigenvlue 1 equls to 1, while the lgebric multiplicity M equls to. There re not enough bses of eigenvectors to digonlize A, so the mtri A is not digonlizble. 1-88

89 NUMERICAL METHODS FOR EIGENVALUE PROBS Power method In this method, we strt from ny vector 0 ( 0) nd compute successively 1 A 0, 1 A, A 1 For simplifiction, we denote s1 by nd s by y, so tht y A. s s Ryleigh s Quotient T q y T is n pproimtion for the lrgest eigenvlue. We cn see tht s will become more nd more similr to the eigenvector corresponding to the lrgest (in bsolute vlue) eigenvlue. To prove this, suppose 0 cn be written s liner combintion of ll eigenvectors v 1, v,, v n of A. 0 1v1v... nv n 1 A0 1Av1Av... nav n 1 A0 1 1v1v... nnv n s 0 s s... s s A v v n n v n We will see tht the term with the lrgest eigenvlue (in bsolute vlue) will eventully dominte other terms, s s gets lrger nd lrge. In ddition, the size of the vector s will differ from vector s1 by tht eigenvlue. 1-89

90 Emple A Choose Then, we cn obtin successively Tke = 3 nd y = 4 m 0 = T = m 1 = T y = q = m 1 /m 0 = The correct eigenvlue is 1. To obtin more ccurte result, we need to continue the multipliction of A. We cn see tht the size of the vector is getting lrger nd lrger. In prctice, the vector is scled down by T. 1-90

91 T Note tht A T Note tht A A Note tht q T 3 4 T

92 Inverse Vector Itertion In the power method, we cn determine only the lrgest eigenvlue, but this inverse vector itertion method cn overcome tht disdvntge. We need the following three properties of eigenvlues of mtri. 1. If A hs the eigenvlues 1,,..., n, then eigenvlues ,,..., n. 1 A hs the. If A hs the eigenvlues 1,,..., n, then A ki hs the eigenvlues 1 k, k,..., n k. (Spectrl Shift Theorem) 3. Eigenvectors of A re lso eigenvectors of A -1. The concept in this method is tht we use the power method to determine the lrgest eigenvlue of A -1, which will give us the smllest eigenvlue of A. Then we pply some spectrl shift (A-kI) nd gin use the power method to determine the lrgest eigenvlue of (A-kI) -1, which will give us the smllest eigenvlue of shifted (A-kI). Then, eigenvlue of A = (eigenvlue of A-kI)+ k which is eigenvlue of A nerest to the shift k. We cn keep chnging the shift nd repet the process until ll eigenvlues nd eigenvectors re obtined. 1-9

93 Emple Given: A 5 First compute 1 1 A 6 5 nd choose Then, we cn obtin successively 1 =A Tke = 3 nd y = 4 m 0 = T = 1.80 m 1 = T y = q = m 1 /m 0 = -1 Eigenvlue of A -1 An eigenvlue of A = -1 which comes from inverse of eigenvlue of A -1 To find other eigenvlues of A, we need to pply spectrl 1 A IA I shift (just tril k = -4) k

94 k Compute A I A 4I nd choose Then, we cn obtin successively 1 =A T 7 8 Then compute Ryleigh s quotient q T which is the lrgest eigenvlue of A ki 1. Smllest eigenvlue of Note tht eigenvlue of 1 AkIA( 4) I A ki = (eigenvlue of A) - k eigenvlue of A = (eigenvlue of A ki) + k Another eigenvlue of A = (-4) = which is the eigenvlue of A nerest to the tril shift k. 1-94