APPENDIX A Some Linear Algebra

APPENDIX A Some Lnear Algebra The collecton of m, n matrces A.1 Matrces a 1,1,..., a 1,n A = a m,1,..., a m,n wth real elements a,j s denoted by R m,n. If n = 1 then A s called a column vector. Smlarly, f m = 1 then A s a row vector. We let R m denote the collecton of all column or row vectors wth m real components. A.1.1 Nonsngular matrces, and nverses. Defnton A.1. A collecton of vectors a 1,..., a n R m s lnearly ndependent f x 1 a 1 + + x n a n = 0 for some real numbers x 1,..., x n, mples that x 1 = = x n = 0. Suppose a 1,..., a n are the columns of a matrx A R m,n. For a vector x = (x 1,..., x n ) T R n we have Ax = n j=1 x ja j. It follows that the collecton a 1,..., a n s lnearly ndependent f and only f Ax = 0 mples x = 0. Defnton A.2. A square matrx A such that Ax = 0 mples x = 0 s sad to be nonsngular. Defnton A.3. A square matrx A R n,n s sad to be nvertble f for some B R n,n where I R n,n s the dentty matrx. BA = AB = I, An nvertble matrx A has a unque nverse B = A 1. If A, B, and C are square matrces, and A = BC, then A s nvertble f and only f both B and C are also nvertble. Moreover, the nverse of A s the product of the nverses of B and C n reverse order, A 1 = C 1 B 1. 213

214 A.1.2 Determnants. The determnant of a square matrx A wll be denoted det(a) or a 1,1,..., a 1,n... a n,1,..., a n,n Recall that the determnant of a 2 2 matrx s a 1,1 a 1,2 a 2,1 a 2,2 = a 1,1a 2,2 a 1,2 a 2,1. A.1.3 Crtera for nonsngularty and sngularty. We state wthout proof the followng crtera for nonsngularty. Theorem A.4. The followng s equvalent for a square matrx A R n,n. 1. A s nonsngular. 2. A s nvertble. 3. Ax = b has a unque soluton x = A 1 b for any b R n. 4. A has lnearly ndependent columns. 5. A T s nonsngular. 6. A has lnearly ndependent rows. 7. det(a) 0. We also have a number of crtera for a matrx to be sngular. Theorem A.5. The followng s equvalent for a square matrx A R n,n. 1. There s a nonzero x R n so that Ax = 0. 2. A has no nverse. 3. Ax = b has ether no soluton or an nfnte number of solutons. 4. A has lnearly dependent columns. 5. There s a nonzero x so that x T A = 0. 6. A has lnearly dependent rows. 7. det(a) = 0. Corollary A.6. A matrx wth more columns than rows has lnearly dependent columns. Proof. Suppose A R m,n wth n > m. By addng n m rows of zeros to A we obtan a square matrx B R n,n. Ths matrx has lnearly dependent rows. By Theorem A.4 the matrx B has lnearly dependent columns. But then the columns of A are also lnearly dependent.

A.2. VECTOR NORMS 215 A.2 Vector Norms Formally, a vector norm = x, s a functon : R n [0, ) that satsfes for x, y, R n, and α R the followng propertes 1. x = 0 mples x = 0. 2. αx = α x. 3. x + y x + y. (A.1) Property 3 s known as the Trangle Inequalty. For us the most useful class of norms are the p or l p norms. They are defned for p 1 and x = (x 1, x 2,..., x n ) T R n by x p = ( x 1 p + x 2 p + + x n p ) 1/p. x = max x. (A.2) Snce x x p n 1/p x, p 1 (A.3) and lm p n 1/p = 1 for any n N we see that lm p x p = x. The 1,2, and norms are the most mportant. We have x 2 2 = x 2 1 + + x 2 n = x T x. (A.4) Lemma A.7 (The Hölder nequalty). We have for 1 p and x, y R n x y x p y q, where =1 1 p + 1 q = 1. (A.5) Proof. We base the proof on propertes of the exponental functon. Recall that the exponental functon s convex,.e. wth f(x) = e x we have the nequalty f(λx + (1 λ)y) λf(x) + (1 λ)f(y) (A.6) for every λ [0, 1] and x, y R. If x = 0 or y = 0, there s nothng to prove. Suppose x, y 0. Defne u = x/ x p and v = y/ y q. Then u p = v q = 1. If we can prove that u v 1, we are done because then x y = x p y q u v x p y q. Snce u v = u v, we can assume that u 0 and v 0. Moreover, we can assume that u > 0 and v > 0 because a zero term contrbutes no more to the left hand sde than to the rght hand sde of (A.5). Let s, t be such that u = e s /p, v = e t /q. Takng f(x) = e x, λ = 1/p, 1 λ = 1/q, x = s and y = t n (A.6) we fnd But then u v = Ths completes the proof of (A.5). e s /p+t /q 1 p es + 1 q et. e s /p+t /q 1 e s + 1 e t = 1 u p p q p + 1 v q q = 1 p + 1 q = 1.

216 When p = 2 then q = 2 and the Hölder nequalty s assocated wth the names Bunakowsk-Cauchy-Schwarz. Lemma A.8 (The Mnkowsk nequalty). We have for 1 p and x, y R x + y p x p + y p. (A.7) Proof. Let u = (u 1,..., u n ) wth u = x + y p 1. Snce q(p 1) = p and p/q = p 1, we fnd u q = ( x + y q(p 1) ) 1/q = ( x + y p ) 1/q = x + y p/q p = x + y p 1 p. Usng ths and the Hölder nequalty we obtan x + y p p = x + y p u x + u y ( x p + y p ) u q ( x p + y p ) x + y p 1 p. Dvdng by x + y p 1 p proves Mnkowsk. Usng the Mnkowsk nequalty t follows that the p norms satsfes the axoms for a vector norm. In (A.3) we establshed the nequalty x x p n 1/p x, p 1. More generally, we say that two vector norms and are equvalent f there exsts postve constants µ and M such that µ x x M x (A.8) for all x R n. Theorem A.9. All vector norms on R n are equvalent. Proof. It s enough to show that a vector norm s equvalent to the l norm,. Let x R n and let e, = 1,..., n be the unt vectors n R n. Wrtng x = x 1 e 1 + +x n e n we have x x e x M, M = e. To fnd µ > 0 such that x µ x for all x R n s less elementary. Consder the functon f gven by f(x) = x defned on the l unt ball S = {x R n : x = 1}. S s a closed and bounded set. From the nverse trangle nequalty x y x y, x, y R n.

A.3. VECTOR SPACES OF FUNCTIONS 217 t follows that f s contnuous on S. But then f attans ts maxmum and mnmum on S,.e. there s a pont x S such that x = mn x S x. Moreover, snce x s nonzero we have µ := x > 0. x = x/ x S. Thus If x R n s nonzero then x µ x = = 1 x, x x and ths establshes the lower nequalty. It can be shown that for the p norms we have for any q wth 1 q p x p x q n 1/q 1/p x p, x R n. (A.9) < A.3 Vector spaces of functons In R m we have the operatons x + y and ax of vector addton and multplcaton by a scalar a R. Such operatons can also be defned for functons. As an example, f f(x) = x, g(x) = 1, and a, b are real numbers then af(x) + bg(x) = ax + b. In general, f f and g are two functons defned on the same set I and a R, then the sum f + g and the product af are functons defned on I by (f + g)(x) = f(x) + g(x), (af(x) = af(x). Two functons f and g defned on I are equal f f(x) = g(x) for all x I. We say that f s the zero functon,.e. f = 0, f f(x) = 0 for all x I. Defnton A.10. Suppose S s a collecton of real valued or vector valued functons, all defned on the same set I. The collecton S s called a vector space f af + bg S for all f, g S and all a, b R. A subset T of S s called a subspace of S f T tself s a vector space. Example A.11. Vector spaces All polynomals π d of degree at most d. All polynomals of all degrees. All trgonometrc polynomals a 0 + d k=1 (a k cos kx + b k sn kx of degree at most d. The set C(I) of all contnuous real valued functons defned on I. The set C r (I) of all real valued functons defned on I wth contnuous j th dervatve for j = 0, 1,..., r.

218 Defnton A.12. A vector space S s sad to be fnte dmesonal f S = span(φ 1,..., φ n ) = { n c j φ j : c j R}, for a fnte number of functons φ 1,..., φ n n S. The functons φ 1,..., φ n are sad to span or generate S. Of the examples above the space π d = span(1, x, x 2,... x d ) generated by the monomals 1, x, x 2,... x d s fnte dmensonal. Also the trgonometrc polynomals are fnte dmensonal. The space of all polynomals of all degrees s not fnte dmensonal. To see ths we observe that any fnte set cannot generate the monomal x d+1 where d s the maxmal degree of the elements n the spannng set. Fnally we observe that C(I) and C r (I) contan the space of polynomals of all degrees as a subspace. Hence they are not fnte dmensonal, If f S = span(φ 1,..., φ n ) then f = n j=1 c jφ j for some c = (c 1,..., c n ). Wth φ = (φ 1,..., φ n ) T we wll often use the vector notaton j=1 f(x) = φ(x) T c (A.10) for f. A.3.1 Lnear ndependence and bases All vector spaces n ths secton wll be fnte dmensonal. Defnton A.13. A set of functons φ = (φ 1,..., φ n ) T n a vector space S s sad to be lnearly ndependent on a subset J of I f φ(x) T c = c 1 φ 1 (x) + + c n φ n (x) = 0 for all x J mples that c = 0. If J = I then we smply say that φ s lnearly ndependent. If φ s lnearly ndependent then the representaton n (A.10) s unque. For f f = φ T c = φ T b for some c, b R n then f = φ T (c b) = 0. Snce φ s lnearly ndependent we have c b = 0, or c = b. Defnton A.14. A set of functons φ T = (φ 1,..., φ n ) n a vector space S s a bass for S f the followng two condtons hold 1. φ s lnearly ndependent. 2. S = span(φ). Theorem A.15. The monomals 1, x, x 2,... x d are lnearly ndependent on any set J R contanng at least d + 1 dstnct ponts. In partcular these functons form as bass for π d. Proof. Let x 0,..., x d be d+1 dstnct ponts n J, and let p(x) = c 0 +c 1 x+ +c d x d = 0 for all x J. Then p(x ) = 0, for = 0, 1,..., d. Snce a nonzero polynomal of degree d can have at most d zeros we conclude that p must be the zero polynomal. But then c k = p (k) (0)/k! = 0 for k = 0, 1,..., d. It follows that the monomal s a bass for π d snce they span π d by defnton. To prove some basc results about bases n a vector space of functons t s convenent to ntroduce a matrx transformng one bass nto another.

A.3. VECTOR SPACES OF FUNCTIONS 219 Lemma A.16. Suppose S and T are fnte dmensonal vector spaces wth S T, and let φ = (φ 1,..., φ n ) T be a bass for S and ψ = (ψ 1,..., ψ m ) T a bass for T. Then φ = A T ψ, (A.11) for some matrx A R m,n. If f = φ T c S s gven then f = ψ T b wth b = Ac. (A.12) Moreover A has lnearly ndependent columns. Proof. Snce φ j T there are real numbers a,j such that φ j = m a,j ψ, for j = 1,..., n, =1 Ths equaton s smply the component verson of (A.11). If f S then f T and f = ψ T b for some b. By (A.11) we have φ T = ψ T A and f = φ T c = ψ T Ac or ψ T b = ψ T Ac. Snce ψ s lnearly ndependent we get (A.12). Fnally, to show that A has lnearly ndependent columns suppose Ac = 0. Defne f S by f = φ T c. By (A.11) we have f = ψ T Ac = 0. But then f = φ T c = 0. Snce φ s lnearly ndependent we conclude that c = 0. The matrx A n Lemma A.16 s called a change of bass matrx. A bass for a vector space generated by n functons can have at most n elements. Lemma A.17. If ψ = (ψ 1..., ψ k ) T s a lnearly ndependent set n a vector space S = span(φ 1,..., φ n ), then k n. Proof. Wth φ = (φ 1,..., φ n ) T we have ψ = A T φ, for some A R n,k. If k > n then A s a rectangular matrx wth more columns than rows. From Corollary A.6 we know that the columns of such a matrx must be lnearly dependent; I.e. there s some nonzero c R k such that Ac = 0. But then ψ T c = φ T Ac = 0, for some nonzero c. Ths mples that ψ s lnearly dependent, a contradcton. We conclude that k n. Lemma A.18. Every bass for a vector space must have the same number of elements. Proof. Suppose φ = (φ 1,..., φ n ) T and ψ = (ψ 1,..., ψ m ) T are two bases for the vector space. We need to show that m = n. Now φ = A T ψ, for some A R m,n, ψ = B T φ, for some B R n,m. By Lemma A.16 we know that both A and B have lnearly ndependent columns. But then by Corollary A.6 we see that m = n. Defnton A.19. The number of elements n a bass n a vector space S s called the dmenson of S, and s denoted by dm(s).

220 The followng lemma shows that every set of lnearly ndependent functons n a vector space S can be extended to a bass for S. In partcular every fnte dmensonal vector space has a bass. Lemma A.20. A set φ T = (φ 1,..., φ k ) of lnearly ndependent elements n a fnte dmensonal vector space S, can be extended to a bass ψ T = (ψ 1,..., ψ m ) for S. Proof. Let S k = span(ψ 1,..., ψ k ) where ψ j = φ j for j = 1,..., k. If S k = S then we set m = k and stop. Otherwse there must be an element ψ k+1 S such that ψ 1,..., ψ k+1 are lnearly ndependent. We defne a new vector space S k+1 by S k+1 = span(ψ 1,..., ψ k+1 ). If S k+1 = S then we set m = k + 1 and stop the process. Otherwse we contnue to generate vector spaces S k+2, S k+3,. Snce S s fntely generated we must by Lemma A.17 eventually fnd some m such that S m = S. The followng smple, but useful lemma, shows that a spannng set must be a bass f t contans the correct number of elements. Lemma A.21. Suppose S = span(φ). If φ contans dm(s) elements then φ s a bass for S. Proof. Let n = dm(s) and suppose φ = (φ 1,..., φ n ) s a lnearly dependent set. Then there s one element, say φ n whch can be wrtten as a lnear combnaton of φ 1,..., φ n 1. But then S = span(φ 1,..., φ n 1 ) and dm(s) < n by Lemma A.17, a contradcton to the assumpton that φ s lnearly dependent. A.4 Normed Vector Spaces Suppose S s a vector space of functons. A norm = f, s a functon : S [0, ) that satsfes for f, g, S, and α R the followng propertes 1. f = 0 mples f = 0. 2. αf = α f. 3. f + g f + g. (A.13) Property 3 s known as the Trangle Inequalty. The par (S, ) s called a normed vector space (of functons). In the rest of ths secton we assume that the functons n S are contnuous, or at least pecewse contnuous on some nterval [a, b]. Analogous to the p or l p norms for vectors n R n we have the p or L p norms for functons. They are defned for 1 p and f S by f p = f L p [a,b] ( b 1/p = a dx) f(x) p, p 1, f = f L [a,b] = max a x b f(x). (A.14) The 1,2, and norms are the most mportant. We have for 1 p and f, g S the Hölder nequalty b a f(x)g(x) dx f p g q, where 1 p + 1 q = 1, (A.15)

A.4. NORMED VECTOR SPACES 221 and the Mnkowsk nequalty f + g p f p + g p. (A.16) For p = 2 (A.15) s known as the Schwarz nequalty, the Cauchy-Schwarz nequalty, or the Bunakowsk-Cauchy- Schwarz nequalty.