Lectures - Week 4 Matrx norms, Condtonng, Vector Spaces, Lnear Independence, Spannng sets and Bass, Null space and Range of a Matrx Matrx Norms Now we turn to assocatng a number to each matrx. We could choose our norms analogous to the way we dd for vector norms; e.g., we could assocate the number max j a j. However, ths s actually not very useful because remember our goal s to study lnear systems A x = b. The general defnton of a matrx norm s a map from all m n matrces to IR 1 whch satsfes certan propertes. However, the most useful matrx norms are those that are generated by a vector norm; agan the reason for ths s that we want to solve A x = b so f we take the norm of both sdes of the equaton t s a vector norm and on the left hand sde we have the norm of a matrx tmes a vector. We wll defne an nduced matrx norm as the largest amount any vector s magnfed when multpled by that matrx,.e., A = max A x Note that all norms on the rght hand sde are vector norms. We wll denote a vector and matrx norm usng the same notaton; the dfference should be clear from the argument. We say that the vector norm on the rght hand sde nduces the matrx norm on the left. Note that sometmes the defnton s wrtten n an equvalent way as A = sup A x What s I? Clearly t s just one. The problem wth the defnton s that t doesn t tell us how to compute a matrx norm for a general matrx A. The followng theorem gves us a way to calculate matrx norms nduced by the l and l 1 norms; the matrx norm nduced by l 2 norm wll be addressed later after we have ntroduced egenvalues. Theorem Let A be an m n matrx. Then A = max 1 m A 1 = max 1 j n [ m [ n =1 ] a j ] a j (max absolute row sum) (max absolute column sum) 1
Determne A and A 1 where A = 4 3 0 12 20 We have A 1 = max{(1 + 3 + 20), (2 + 1), (4 + 12 + 2)} = max{24, 3, 18} = 24 A = max{(1 + 2 + 4), (3 + 12), (20 + 1 + 2)} = max{7, 15, 23} = 23 Proof We wll prove that A s the maxmum row sum (n absolute value). We wll do ths by provng that A max 1 m [ n Frst recall that f A x = b then b = ] a j and then showng a j x j b = max For the frst nequalty we know that by defnton Now lets smplfy the numerator to get A x = max A = max a j x j max b = max A x A max 1 m a j x j a j x j max [ n a j ] a j Thus the rato reduces to A x n max a j = max a j and hence A = max A x Now for the second nequalty we know that max a j. A A y y 2
for any y IR n because equalty n the defnton holds here for the maxmum of ths rato. So now we wll choose a partcular y and we wll construct t so that t has y = 1. Frst let p be the row where A has ts maxmum row sum (or there are two rows, take the frst),.e., max a j = a pj Now we wll take the entres of y to be ±1 so ts nfnty norm s one. Specfcally we choose y = { 1 f apj 0 1 f a pj < 0 Defnng y n ths way means that a j y j = a j. Usng ths and the fact that y = 1 we have A A y y = max a j y j a pj y j = n a pj = a pj but the last quantty on the rght s must the maxmum row sum and the proof s complete. What are the propertes that any matrx norm (and thus an nduced norm) satsfy? You wll recognze most of them as beng analogous to the propertes of a vector norm. () A 0 and = 0 only f a j = 0 for all, j. () ka = k A for scalars k () AB A B (v) A + B A + B A very useful nequalty s A x A for any nduced norm Why s ths true? A = max A x A x A x A The Condton Number of a Matrx We sad one of our goals was to determne f small changes n our data of a lnear system produces small changes n the soluton. Now lets assume we want to solve A x = b, b 0 but nstead we solve A y = b + δb that s, we have perturbed the rght hand sde by a small about δb. We assume that A s nvertble,.e., A 1 exsts. For smplcty, we have not perturbed the coeffcent matrx 3
A. What we want to see s how much y dffers from x. Lets wrte y as x + δx and so our change n the soluton wll be δx. The two systems are A x = b A( x + δx) = b + δb What we would lke to get s an estmate for the relatve change n the soluton,.e., δx n terms of the relatve change n b where denotes any nduced vector norm. Subtractng these two equatons gves A δx = δb whch mples δx = A 1 δb Now we take the (vector) norm of both sdes of the equaton and then use our favorte nequalty above δx = A 1 δb A 1 δb Remember our goal s to get an estmate for the relatve change n the soluton so we have a bound for the change. What we need s a bound for the relatve change. Because A x = b we have A x = b b A 1 b 1 A Now we see that f we dvde our prevous result for δx by A > 0 we can use ths result to ntroduce b n the denomnator. We have δx A A 1 δb A Multplyng by A gves the desred result A 1 δb b δx A A 1 δb b If the quantty A A 1 s small, then ths means small relatve changes n b result n small relatve changes n the soluton but f t s large, we could have a large relatve change n the soluton. Defnton The condton number of a square matrx A s defned as K(A) A A 1 Note that the condton number depends on what norm you are usng. We say that a matrx s well-condtoned f K(A) s small and ll-condtoned otherwse. 4
Fnd the condton number of the dentty matrx usng the nfnty norm. Clearly t s one because the nverse of the dentty matrx s tself. Fnd the condton number for each of the followng matrces usng the nfnty norm.. Explan geometrcally why you thnk ths s happenng. A 1 = A 4 3 2 = 0.998 2 Frst we need to fnd the nverse of each matrx and then take the norms. Note the followng trck for takng the nverse of a 2 2 matrx A 1 1 = 1 3 2 A 1 500 500 5 4 1 2 = 249.5 250 Now K (A 1 ) = (7)(1) = 7 K (A 2 ) = (3)(1000) = 3000 Calculate the K (A) where A s a b A = c d and comment on when the condton number wll be large. A 1 1 d b = ad bc c a So 1 1 K (A) = max{ a + b, c + d } max{ d + b, c + a } ad bc ad bc Consequently f the determnant ad bc 0 then the condton number can be qute large;.e., when the matrx s almost not nvertble,.e., almost sngular. A classc example of an ll condtoned matrx s the Hlbert matrx whch we have already encountered Here are some of the condton numbers (usng the matrx norm nduced by the l 2 vector norm). n approxmate K 2 (A) 2 19 3 524 4 15,514 5 476,607 6 1.495 10 7 5
Vector (or Lnear) Spaces What we want to do now s take the concept of vectors and the propertes of Eucldean space IR n and generalze them to a collecton of objects wth two operatons defned, addton and scalar multplcaton. We want to defne these operatons so that the usual propertes hold,.e., x + y = y + x, k(x + y) = kx + ky for objects other than vectors, such as matrces, contnuous functons, etc. We want to be able to add two elements of our set and get another element of the set. Defnton A vector or lnear space V s a set of objects, whch we wll call vectors, for whch addton and scalar multplcaton are defned and satsfy the followng propertes. () x + y = y + x () x + (y + z) = (x + y) + z () there exsts a zero element 0 V such that x + 0 = 0 + x = x (v) for each x V there exsts x V such that x + ( x) = 0 (v) 1x = x (v) (α + β)x = αx + βx, for scalars α, β (v) α(x + y) = αx + αy (v) (αβ)x = α(βx) IR n s a vector space wth the usual operaton of addton and scalar multpl- caton. The set of all m n matrces wth the usual defnton of addton and scalar multplcaton forms a vector space. All polynomals of degree less than or equal to two on [ 1, 1] form a vector space wth usual defnton of addton and scalar multplcaton. Here p(x) = a 0 + a 1 x + a 2 x 2 All contnuous functons defned on [0, 1]. Note that f v, w V then v + w V ; we say that V s closed under addton. Also f v V and k s a scalar, kv V and t s closed under scalar multplcaton. Defnton A subspace S of a vector space V s a nonempty subset of V such that f we take any lnear combnaton of vectors n S t s also n S. A lne s a subspace of IR 2 because f we take add two ponts on the lne, the result s stll a pont on the lne and f we multply any pont on the lne by a constant then t s stll on the lne. matrces. All lower trangular matrces form a subspace of the vector space of all square 6
Is the set of all ponts on the gven lnes a subspace of IR 2? y = x y = 2x + 1 The frst forms a subspace but the second does not. It s not closed under addton or scalar multplcaton, e.g., 2(x, 2x + 1) = (2x, 4x + 2) whch s not on the lne y = 2x + 1 Lnear ndependence, spannng sets and bass We now want to nvestgate how to characterze a vector space, f possble. To do ths, we need the concepts of () lnear ndependence/dependence of vectors () a spannng set () the dmenson of a vector space (v) the bass of a fnte dmensonal vector space In IR 2 the vectors v 1 = (1, 0) T and v 2 = (0, 1) T are specal because () every other vector n IR 2 can be wrtten as a lnear combnaton of these two vectors, e.g., f x = (x 1, x 2 ) T then x = x 1 v 1 + x 2 v 2. () the vectors v 1, v 2 are dfferent n the sense that the only way we can combne them and get the zero vector s wth zero coeffcents,.e., C 1 v 1 + C 2 v 2 = 0 C 1 = C 2 = 0 Of course there are many other sets of two vectors n IR 2 that have the same propertes. Defnton Let V be a vector space. Then the set of vectors { v }, = 1,..., n () span V f any w V can be wrtten as a lnear combnaton of the v,.e., () are lnearly ndependent f w = C j v j C j v j = 0 C = 0, otherwse they are lnearly dependent. Note that ths says the only way we can combne the vectors and get the zero vector s f all coeffcents are zero. () form a bass for V f they are lnearly ndependent and span V. 7
Let V be the vector space of all polynomals of degree less than or equal to two on [0, 1]. Then the set of vectors (.e., polynomals) n V {1, x, x 2, 2 + 6x} span V but they are NOT lnearly ndependent because 2(1) + 6(x) + 0(x 2 ) (2 + 6x) = 0 and thus they do not form a bass for V. The set of vectors {1, x} are lnearly ndependent but they do not span V because, e.g., x 2 V and t can t be wrtten as a lnear combnaton of {1, x}. Defnton The number of elements n the bass for V s called the dmenson of V. It s the smallest number of vectors needed to completely characterze V,.e., that span V. Fnd a bass for () the vector space of all 2 2 matrces and () the vector space of all 2 2 symmetrc matrces. Gve the dmenson. A bass for all 2 2 matrces conssts of the four matrces 1 0, 0 1,, 1 0 0 1 and so ts dmenson s 4. A bass for all 2 2 symmetrc matrces conssts of the three matrces 1 1,, 0 1 1 0 and so ts dmenson s 2. Four Important Spaces 1. The column space (or equvalently the range) of A, where A s m n matrx s all lnear combnatons of the columns of A. We denote ths by R(A). By defnton t s a subspace of IR m. Consder the overdetermned system 3 4 x = b 1 b y 2 5 6 b 3 We know that ths s solvable f there exsts x, y such that x 1 3 5 + y 2 4 = 6 b 1 b 2 b 3 whch says that b has to be n the column space (or range) of A. 8
Ths says that an equvalent statement to A x = b beng solvable s that b s n the range or column space of A. 2. The null space of A, denoted N (A), where A s m n matrx s the set of all vectors z IR n such that A z = 0. Fnd the null space of each matrx A 1 = 3 4 A 2 = 2 4 A 3 = For A 1 we have that N (A 1 ) = 0 because the matrx s nvertble. To see ths we could take the determnant or perform GE and get the result. For A 2 we see that N (A 2 ) s α( 2x 2, x 2 ) T,.e., the all ponts on the lne through the orgn y =.5x. To see ths consder GE for the system ( 2 4 ) ( Ths says that x 2 s arbtrary and x 1 = 2x 2. For A 3 we see that N (A 2 ) s all of IR 2. ) 0 x 2 = 0, x 1 + 2x 2 = 0 What are the possble null spaces of a 3 3 matrx? For an nvertble matrx t s the () zero vector n IR 3, we could have () a lne through the orgn, () a plane through the orgn or all of (v) IR 3. 9