McFadden Chapter 2. Analysis and Linear Algebra in a Nutshell 18

Transcription

1 McFadde Chapter. alyss ad Lear lgebra a Nutshell 8 Chapter. NLYSIS ND LINER LGEBR IN NUTSHELL. Sme Elemets f Mathematcal alyss Real umbers are deted by lwer case Greek r Rma umbers; the space f real umbers s the real le, deted by R. The abslute value f a real umber a s deted by a. Cmplex umbers are rarely requred ecmetrcs befre the study f tme seres ad dyamc systems. Fr future referece, a cmplex umber s q====6 wrtte a + bι, where a ad b are real umbers ad ι = e, wth a termed the real part ad ιb termed the magary part. The cmplex umber ca als be wrtte as q====================6 r(cs θ + ι s θ), where r = ea +b s the mdulus f the umber ad θ = cs (a/r). The prpertes f cmplex umbers we wll eed basc ecmetrcs are the rules fr sums, (a+ιb) + (c+ιd) = (a+c)+ι(b+d), ad prducts, (a+ιb) (c+ιd) = (abcd)+ι(ad+bc). Fr sets f bjects ad B, the u B s the set f bjects ether; the tersect B s the set f bjects bth; ad \B s the set f bjects that are t B. The empty set s deted. Set clus s deted B; we say s ctaed B. c ctas t) s deted. The cmplemet f a set (relatve t a set B that famly f sets s dsjt f the tersect f each par s empty. The symbl a meas that a s a member f ; ad a meas that a s t a member f. The symbl meas "there exsts", the symbl meas "fr all", ad the symbl meas "such that". The terms fuct, mappg, ad trasfrmat are used syymusly, ad the tat f: L B wll mea that each bject a the dma s mapped t a bject b =f(a) the rage B. The symbl f(c), termed the mage f C, s used fr the set f all bjects f(a) fr a C. Fr D B, the symbl f (D) detes the

2 McFadde Chapter. alyss ad Lear lgebra a Nutshell 9 verse mage f D: the set f all a such that f(a) D. The fuct f s t f B = f(); t s ete f t s t ad f a,c ad a c mples f(a) f(c). Whe f s ete, the mappg f s a fuct frm B t. If C, defe the fuct : L R by (a) = fr a C, ad (a) = 0 therwse; ths C C C s called the dcatr fuct fr the set C. fuct s termed realvalued f ts rage s R. If s a set f real umbers, the the fmum f, deted f, s the greatest real umber that s less tha r equal t every umber. The supremum f, deted sup, s the least real umber that s greater tha r equal t every umber. typcal applcat has a fuct f:c L R ad = f(c); the sup f(c) s used t dete sup. If the supremum s acheved by a bject d C, s c C f(d) =supf(c), the we wrte f(d) = max f(c) ad d = ar gmax f(c). Ths tat s c C c C c C ambgus whe there s a uque maxmzg argumet; we wll assume that ar gmax f(c) s a select f ay e f the maxmzg argumets. c C defts hld fr f ad m. algus If a s a sequece f real umbers dexed by =,,..., the the sequece s sad t have a lmt (equal t a ) f fr each ε > 0, there exsts such that a a < ε fr all ; the tat fr a lmt s l m a = a r a L a. L The Cauchy crter says that a sequece a has a lmt f ad ly f, fr each ε > 0, there exsts such that a a j < ε fr,j. The tat lmsup a L meas the lmt f the supremum f the sets {a,a +,...}; because t s creasg, t always exsts (but may equal + r ). aalgus deft hlds fr lmf. realvalued fuct ρ(a,b) defed fr pars f bjects a set s a dstace fuct f t s egatve, gves a pstve dstace betwee all

3 McFadde Chapter. alyss ad Lear lgebra a Nutshell 0 dstct pts f, has ρ(a,b) = ρ(b,a), ad satsfes the tragle equalty ρ(a,b) ρ(a,c) + ρ(c,b). set wth a dstace fuct ρ s termed a metrc space. typcal example s the real le R, wth the abslute value f the dfferece f tw umbers take as the dstace betwee them. (ε)eghbrhd f a pt a a metrc space (fr ε > 0) s a set f the frm {b ρ(a,b) <ε}. set C s pe f fr each pt C, sme eghbrhd f ths pt s als ctaed C. set C s clsed f ts cmplemet s pe. The clsure f a set C s the tersect f all clsed sets that cta C. The terr f C s the u f all pe sets ctaed C. cverg f a set C s a famly f pe sets whse u ctas C. The set C s sad t be cmpact f every cverg ctas a fte subfamly whch s als a cverg. famly f sets s sad t have the ftetersect prperty f every fte subfamly has a empty tersect. ther characterzat f a cmpact set s that every famly f clsed subsets wth the fte tersect prperty has a empty tersect. metrc space s separable f there exsts a cutable subset B such that every eghbrhd ctas a member f B. ll f the metrc spaces ecutered ecmetrcs wll be separable. sequece a a separable metrc space s cverget (t a pt a ) f the sequece s evetually ctaed each eghbrhd f a; we wrte a L a r lm a = a t dete a cverget sequece. L set C s cmpact f ad ly f every sequece C has a cverget subsequece (whch cverges t a cluster pt f the rgal sequece). Csder separable metrc spaces ad B, ad a fuct f: L B. The fuct f s ctuus f the verse mage f every pe set s pe. ther characterzat f ctuty s that fr ay sequece satsfyg a L a, e has f(a ) L f(a ); the fuct s sad t be ctuus C f ths prperty

4 McFadde Chapter. alyss ad Lear lgebra a Nutshell hlds fr each a C. Stated ather way, f s ctuus C f fr each ε >0 ad a C, there exsts δ > 0 such that fr each b a δeghbrhd f a, f(b) s a εeghbrhd f f(a). Ctuty f realvalued fucts f ad g s preserved by the perats f abslute value f(a), multplcat f(a) g(a), addt f(a)+g(a), ad maxma r mma max{f(a),g(a)} ad m{f(a),g(a)}. The fuct f s ufrmly ctuus C f fr each ε > 0, there exsts δ > 0 such that fr all a C ad b wth b a δeghbrhd f a, e has f(b) a εeghbrhd f f(a). The dstct betwee ctuty ad ufrm ctuty s that fr the latter a sgle δ > 0 wrks fr all a C. fuct that s ctuus a cmpact set s ufrmly ctuus. Csder a realvalued fuct f R. The dervatve f f at a, deted f (a ), f(a ), r df(a )/da, has the prperty f t exsts that f(b) f(a ) f (a )(ba ) ε(ba ) (ba ), where l m ε(c) = 0. The fuct s cl 0 ctuusly dfferetable at a f f s a ctuus fuct at a. If a fuct s ktmes ctuusly dfferetable a eghbrhd f a pt a, the t has a Taylr s expas k (ba ) ( )(ba ) k s () (k) (k) f(b) = f(a ) + {f (λb+(λ)a )f (a )}, t! k! =0 9 0 () where f detes the th dervatve, λ s a scalar betwee zer ad e, ad b s the eghbrhd. The expetal fuct e a, als wrtte exp(a), ad atural lgarthm lg(a) appear frequetly ecmetrcs. The expetal fuct s defed fr bth real ad cmplex argumets, ad has the prpertes that e a+b = e a e b, e 0 =, ad the Taylr s expas e a = s a t! that s vald fr all a. The trgmetrc fucts =0

5 McFadde Chapter. alyss ad Lear lgebra a Nutshell cs(a) ad s(a) are als defed fr bth real ad cmplex argumets, ad have + s () a s Taylr s expass cs(a) =, ad s(a) = (). a These t ()! t (+)! =0 =0 expass cmbe t shw that e a+ιb = e a (cs(b) + ιs(b)). The lgarthm s defed fr pstve argumets, ad has the prpertes that lg() = 0, lg(a b) = a s lg(a) + lg(b), ad lg(e ) = a. It has a Taylr s expas lg(+a) = a, t = vald fr a <. useful bud lgarthms s that fr a < /3 ad b < /3, Lg(+a+b) a <4 b +3 a.. Vectrs ad Lear Spaces.. ftedmesal lear space s a set wth the prpertes (a) that lear cmbats f pts the set are defed ad are aga the set, ad (b) there s a fte umber f pts the set (a bass) such that every pt the set s a lear cmbat f ths fte umber f pts. The dmes f the space s the mmum umber f pts eeded t frm a bass. pt x a lear space f dmes has a rdate represetat x =(x,x,...,x ), gve a bass fr the space {b,...,b }, where x,...,x are real umbers such that x = x b x b. The pt x s called a vectr, ad x,...,x are called ts cmpets. The tat (x) wll smetmes als be used fr cmpet f a vectr x. I ecmetrcs, we wrk mstly wth ftedmesal real space. Whe ths space s f dmes, t s deted R. Pts ths space are vectrs f real umbers (x,...,x ); ths crrespds t the prevus termlgy wth the bass fr R beg the ut vectrs (,0,..,0), (0,,0,..,0),..., (0,..,0,). Usually, we assume ths represetat wthut beg explct abut the bass fr the space.

6 McFadde Chapter. alyss ad Lear lgebra a Nutshell 3 Hwever, t s wrth tg that the crdate represetat f a vectr depeds the partcular bass chse fr a space. Smetmes ths fact ca be used t chse bases whch vectrs ad trasfrmats have partcularly smple crdate represetats. q=====================================6 The Eucldea rm f a vectr x s NxN = x +...+x. Ths rm ca be used t e defe the dstace betwee vectrs, r eghbrhds f a vectr. Other pssble rms are NxN = x x, NxN = max { x,..., x }, r fr p < +, q e p p p NxN = x x Each rm defes a tplgy the lear space, based p z c eghbrhds f a vectr that are less tha each pstve dstace away. The space R wth the rm NxN ad asscated tplgy s called Eucldea space. The vectr prduct f x ad y R s defed as x y =x y +...+x y. Other tats fr vectr prducts are <x,y> r (whe x ad y are terpreted as rw vectrs) xy r (whe x ad y are terpreted as clum vectrs) x y... lear subspace f a lear space such as R s a subset that has the prperty that all lear cmbats f ts members rema the subset. Examples f lear subspaces are the plae {(a,b,c) b = 0} ad the le {(a,b,c) a = b = c} 3 R. The lear subspace spaed by a set f vectrs {x,...,x J } s the set f all lear cmbats f these vectrs, J L = {xα +...+x α (α,...,α ) R J J J }. The vectrs {x,...,x J } are learly depedet f ad ly f e cat be wrtte as a lear cmbat f the remader. The lear subspace that s spaed by a set f J learly depedet vectrs s sad t be f dmes J. Cversely, each lear space f dmes J ca be represeted as the set f lear cmbats f J

7 McFadde Chapter. alyss ad Lear lgebra a Nutshell 4 learly depedet vectrs, whch are fact a bass fr the subspace. lear subspace f dmes e s a le (thrugh the rg), ad a lear subspace f dmes () s a hyperplae (thrugh the rg). If L s a subspace, the L = {x R x y = 0 fr all y L} s termed the cmplemetary subspace. Subspaces L ad M wth the prperty that x y = 0 fr all y L ad x M are termed rthgal, ad deted L M. The agle θ betwee subspaces L ad M s defed by cs θ = M {x y y L, NyN =,x M, NxN = }. The, the agle betwee rthgal subspaces s π/, ad the agle betwee subspaces that have a zer pt cmm s zer. subspace that s traslated by addg a zer vectr c t all pts the subspace s termed a affe subspace..3. The ccept f a ftedmesal space ca be geeralzed. Fr example, csder, fr p < +, the famly L (R ) f realvalued fucts f R such p p that the tegral NfN = f(x) dx s welldefed ad fte. Ths s a lear p j R space wth rm NfN sce lear cmbats f fucts that satsfy ths p prperty als satsfy (usg cvexty f the rm fuct) ths prperty. Oe ca thk f the fuct f as a vectr L (R ), ad f(x) fr a partcular value f x p as a cmpet f ths vectr. May, but t all, f the prpertes f ftedmesal space exted t fte dmess. I basc ecmetrcs, we wll t eed the ftedmesal geeralzat. It appears mre advaced ecmetrcs, stchastc prcesses tme seres, ad lear ad parametrc prblems.

8 McFadde Chapter. alyss ad Lear lgebra a Nutshell 5 3. Lear Trasfrmats ad Matrces 3.. mappg frm e lear space (ts dma) t ather (ts rage) s a lear trasfrmat f t satsfes (x+z) = (x) + (z) fr ay x ad z the dma. Whe the dma ad rage are ftedmesal lear spaces, a lear trasfrmat ca be represeted as a matrx. Specfcally, a lear trasfrmat frm m R t R ca be represeted by a m array wth elemets a fr m ad, wth y = (x) havg cmpets y = s a x fr m. t j I j= matrx tat, ths s wrtte y = x. matrx s real f all ts elemets are real umbers, cmplex f sme f ts elemets are cmplex umbers. Thrughut these tes, matrces are assumed t be real uless explctly assumed therwse. The set N ={x R x = 0} s termed the ull space f the trasfrmat. The set N f all lear cmbats f the clum vectrs f s termed the clum space f ; t s the cmplemetary subspace t N. If detes a m matrx, the detes ts m traspse (rws becme clums ad vce versa). The detty matrx f dmes s wth e s dw the dagal, zer s elsewhere, ad s deted I, r I f the dmes s clear frm the ctext. permutat matrx s btaed by permutg the clums f a detty matrx. If s a m matrx ad B s a p matrx, the the matrx prduct C = B s f dmes m p wth elemets c k s a b fr m ad k p. t jk j= Fr the matrx prduct t be defed, the umber f clums must equal the umber f rws B (.e., the matrces must be cmmesurate). matrx s square f t has the same umber f rws ad clums. square matrx s symmetrc f =, dagal f all ffdagal elemets are zer, upper (lwer) tragular f

9 McFadde Chapter. alyss ad Lear lgebra a Nutshell 6 all ts elemets belw (abve) the dagal are zer, ad demptet f t s symmetrc ad =. matrx s clum rthrmal f = I; smply rthrmal f t s bth square ad clum rthrmal. Each clum f a m matrx s a vectr m R. The rak f, deted ρ(), s the largest umber f clums that are learly depedet. Frm the deft, s f rak f ad ly f x = 0 s the ly slut t x = 0. matrx f rak s termed sgular; ths matrx has a verse matrx such that = = I f ad ly f s sgular. m 3.. The fllwg tables defe useful matrx ad vectr perats. Table. Basc Operats q===================================================================================================================================================================================================================================================================================================================================================e p p p Name Ntat Deft [] s.matrx Prduct C = B Fr m ad p B,c = ab k t jk j=.scalar Multplcat C = b Fr a scalar b, c = ba 3. Matrx Sum C = +B Fr ad B m, c = a + b 4.Matrx Iverse C = Fr sgular, = I m s 5.Trace c = tr() Fr,c = a t = z ===================================================================================================================================================================================================================================================================================================================================================c $ $ $

10 McFadde Chapter. alyss ad Lear lgebra a Nutshell 7 Table. Operats Elemets q===================================================================================================================================================================================================================================================================================================================================================================e p p p Name Ntat Deft [].Elemet Prduct C =.*B Fr, B m, c = a b.elemet Dvs C=. B Fr, B m, c = a /b 3.Lgcal Cdt C=. B Fr, B m, c = (a b ) [Nte ] 4. Rw Mmum C = vm() Fr m, c = m a [Nte ] j 5. Clum Mmum C = m() Fr m, c = m a [Nte 3] kj k s 6.Cumulatve Sum C = cumsum()fr m,c = a t kj k= z ===================================================================================================================================================================================================================================================================================================================================================================c $ $ $ Table 3. Shapg Operats q===========================================================================================================================================================================================================================================================================================================================================================e p p p Name Ntat Deft [].Krecker Prduct C = B Nte 4.Drect Sum C = B Nte 5 3.dag C = dag(x) C a dagal matrx wth vectr x dw the dagal 4.vec c = vec() vectr c ctas rws f, by rw 5. vech c = vech() vectr c ctas upper tragle f, rw by rw, stacked 6.vecd c = vecd() vectr c ctas dagal f 7.hrztal C = {,B} Partted matrx C = [ B ] ctatat 8.vertcal C = {;B} Partted matrx C = [ B ] ccatat 9.reshape C = rsh(,k) Nte 6 z ===========================================================================================================================================================================================================================================================================================================================================================c $ $ $ NOTES TO TBLES ND 3:. (P) s e f P s true, zer therwse. The cdt s als defed fr the lgcal perats "<", ">", " ", "=", ad " "..C s a m matrx. The perat s als defed fr "max". 3. C s a m matrx, wth all rws the same. The perat s als defed fr "max". 4. ls termed the drect prduct, the Krecker prduct s defed fr a m matrx ad a p q matrx B as the (mp) (q) array

11 McFadde Chapter. alyss ad Lear lgebra a Nutshell 8 q e a B a B... a B a B a B... a B B =. a B a B a B m m m z c 5. The drect sum s defed fr a m matrx ad a p q matrx B by the q e 0 (m+p) (+q) partted array B =. 0 B z c 6. If s m, the k must be a dvsr f m. The perat takes the elemets f rw by rw, ad rewrtes the successve elemets as rws f a matrx C that has k rws ad m /k clums. I addt t the perats the tables abve, there are statstcal perats that ca be perfrmed a matrx whe ts clums are vectrs f bservats varus varables. Dscuss f these perats s pstped utl later. Mst f the perats Tables 3 are avalable as part f the matrx prgrammg laguages ecmetrcs cmputer packages such as SST, TSP, GUSS, r MTLB. The tat these tables s clse t the tat fr the crrespdg matrx cmmads SST ad GUSS The determat f a matrx s deted r det(), ad has the gemetrc terpretat as the vlume f the parallelpped frmed by the clum vectrs f. The matrx s sgular f ad ly f det() 0. mr f a matrx (f rder r) s the determat f a submatrx frmed by strkg ut r rws ad clums. prcpal mr s frmed by strkg ut symmetrc rws ad clums f. leadg prcpal mr (f rder r) s frmed by strkg ut the last r rws ad clums. The mr f a elemet a f s the determat f the submatrx frmed by strkg ut rw ad clum j f. Determats satsfy the recurs relat

12 McFadde Chapter. alyss ad Lear lgebra a Nutshell 9 s +j s det() = () a det( ) = () +j a det( ), t t = j= wth the frst equalty hldg fr ay j ad the secd hldg fr ay. Ths frmula ca be used as a recursve deft f determats, startg frm the result that the determat f a scalar s the scalar. s () +j a det( )/det() = δ, t k kj = where δ kj s e f k = j ad zer therwse. useful related frmula s 3.4. We lst wthut prf a umber f useful elemetary prpertes f matrces: () ( ) =. () If exsts, the ( ) =. (3) If exsts, the ( ) = ( ). (4) (B) =B. (5) If,B are square, sgular, ad cmmesurate, the (B) = B. (6) If s m, the M {m,} ρ() = ρ( ) =ρ( ) = ρ( ). (7) If ad B are cmmesurate, the ρ(b) m(ρ(),ρ(b)). (8) ρ(+b) ρ() + ρ(b). (9) If s, the det() 0 f ad ly f ρ() =. (0) If B ad C are sgular ad cmmesurate wth, the ρ(bc) = ρ(). () If, B are, the ρ(b) ρ() + ρ(b). () det(b) = det() det(b). (3) If c s a scalar ad s, the det(c) = c det() (4) The determat f a matrx s uchaged f a scalar tmes e clum (rw) s added t ather clum (rw).

13 McFadde Chapter. alyss ad Lear lgebra a Nutshell 30 (5) The determat f a dagal r upper tragular matrx s the prduct f the dagal elemets. (6) det( ) = /det(). (7) If s ad B =, the b = () +j det( )/det(). (8) The determat f a rthrmal matrx s + r. (9) If s m ad B s m, the tr(b) = tr(b). (0) tr(i ) =. () tr(+b) = tr() + tr(b). () permutat matrx P s rthrmal; hece, P =P. (3) The verse f a (upper) tragular matrx s (upper) tragular, ad the verse f a dagal matrx D s dagal, wth the recprcals f the dagal elemets f D dw the dagal f D. (4) The prduct f rthrmal matrces s rthrmal, ad the prduct f permutat matrces s a permutat matrx. 4. Egevalues ad Egevectrs egevalue f a matrx s a scalar λ such that x = λx fr sme vectr x 0. The vectr x s called a (rght) egevectr. The cdt (λi)x = 0 asscated wth a egevalue mples λi smgular, ad hece det(λi) = 0. The determetal equat defes a plymal λ f rder ; the rts f ths plymal are the egevalues. Fr each egevalue λ, the cdt that λi s less tha rak mples the exstece f e r mre learly depedet egevectrs; the umber equals the multplcty f the rt λ. The fllwg basc prpertes f egevalues ad egevectrs are stated wthut prf:

14 McFadde Chapter. alyss ad Lear lgebra a Nutshell 3 () If s real ad symmetrc, the ts egevalues ad egevectrs geeral are cmplex. If s real ad symmetrc, the ts egevalues ad egevectrs are real. () The umber f zer egevalues f equals ts rak ρ(). (3) If λ s a egevalue f, the λ k s a egevalue f k, ad /λ s a egevalue f (f the verse exsts). (4) If s real ad symmetrc, ad Λ s a dagal matrx wth the rts f the plymal det(λi) alg the dagal, the there exsts a rthrmal matrx C (whse clums are egevectrs f ) such that C C = I ad C = CΛ, ad hece C C = Λ ad CΛC =. The trasfrmat C s sad t dagalze. (5) If s real ad symmetrc, there exsts a sgular cmplex matrx Q ad a upper tragular cmplex matrx T wth the egevalues f ts dagal such that Q Q = T. (6) real ad symmetrc mples tr() equals the sum f the egevalues f. [Sce = CΛC, tr() = tr(cλc ) = tr(c CΛ) = tr(λ) by.3.9.] (7) If,..., p are real ad symmetrc, the there exsts C rthrmal such that C C, C C,...,C p C are all dagal f ad ly f j = j fr all ad j. Results (4) ad (5) cmbed wth the result.3. that the determat f a matrx prduct s the prduct f the determats f the matrces, mples that the determat f a matrx s the prduct f ts egevalues. The trasfrmats (4) ad (5) are called smlarty trasfrmats, ad ca be terpreted as represetats f the trasfrmat whe the bass f the dma s trasfrmed by C (r Q) ad the bass f the rage s trasfrmed by C (r Q ). These trasfrmats are used extesvely ecmetrc thery.

15 McFadde Chapter. alyss ad Lear lgebra a Nutshell 3 5. Partted Matrces It s smetmes useful t partt a matrx t submatrces, q e =, z c where s m, s m, sm, sm, sm, ad m +m = m ad + =. Matrx prducts ca be wrtte fr partted matrces, applyg the usual algrthm t the partt blcks, prvded the blcks are cmmesurate. Fr q e B example, f B s p ad s partted B = B where B s p ad B s p, z c e has q eq e q e B B + B B = =. B B + B z cz c z c Partted matrces have the fllwg elemetary prpertes: () square ad square ad sgular mples det() = det( ) det( ). () ad square ad sgular mples q e + C C = C C z c wth C =.

16 McFadde Chapter. alyss ad Lear lgebra a Nutshell Quadratc Frms The scalar fuct Q(x,) = x x, where s a matrx ad x s a vectr, s termed a quadratc frm; we call x the wgs ad the ceter f the quadratc frm. The value f a quadratc frm s uchaged f s replaced by ts symmetrzed vers (+ )/. Therefre, wll be assumed symmetrc fr the dscuss f quadratc frms. quadratc frm Q(x,) may fall t e f the classes the table belw: q============================================================================================================================================================================================e p Class Defg Cdt [] Pstve Defte x 0 Q(x,) > 0 Pstve Semdefte x 0 Q(x,) 0 Negatve Defte x 0 Q(x,) < 0 Negatve Semdeftex 0 Q(x,) 0 z ============================================================================================================================================================================================c $ The basc prpertes f quadratc frms are lsted belw: () If B s m ad s f rak ρ(b) = r, the B B ad BB are bth pstve semdefte; ad f r = m, the B B s pstve defte. () If s symmetrc ad pstve semdefte (pstve defte), the the egevalues f are egatve (pstve). Smlarly, f s symmetrc ad egatve semdefte (egatve defte), the the egevalues f are pstve (egatve). (3) Every symmetrc pstve semdefte matrx has a symmetrc pstve / semdefte square rt [By.4.4, C C = D fr sme C rthrmal ad D a dagal matrx wth the egatve egevalues dw the dagal. The, / / / = CDC ad = CD C wth D a dagal matrx f the pstve square rts f the dagal f D.] (4) If s pstve defte, the s pstve defte.

17 McFadde Chapter. alyss ad Lear lgebra a Nutshell 34 (5) B pstve defte ad B pstve semdefte mply B pstve semdefte. (6) The fllwg cdts are equvalet: () s pstve defte () The prcpal mrs f are pstve () The leadg prcpal mrs f are pstve. 7. The LDU ad Chlesky Factrzats f a Matrx matrx has a LDU factrzat f t ca be wrtte = LDU, where D s a dagal matrx, L s a lwer tragular matrx, ad U s a upper tragular matrx. Ths factrzat s useful fr cmputat f verses, as tragular matrces are easly verted by recurs. Therem. Fr each matrx, = PLDUQ, where P ad Q are permutat matrces, L s a lwer tragular matrx ad U s a upper tragular matrx, each wth es the dagal, ad D s a dagal matrx. If the leadg prcpal mrs f are all zer, the P ad Q ca be take t be detty matrces. Prf: Frst assume that the leadg prcpal mrs f are all zer. We gve a recursve cstruct f the requred L ad U. Suppse the result has bee establshed fr matrces up t rder. The, wrte the requred decmpst = LDU fr a matrx partted frm q e q eq eq e L 0 D 0 U U =, L 0 D 0 z c z cz cz c where, L, D, ad U, are () (), L s (), U s (), ad ad D are. ssume that L, D, ad U have bee defed s that =

18 McFadde Chapter. alyss ad Lear lgebra a Nutshell 35 L D U, ad that L ad U als exst ad have bee cmputed. Let S = L ad T = U, ad partt S ad T cmmesurately wth L ad U. The, the remag elemets satsfy the equats = L D U L = U D T D = L D U U = D L D S =L D U +D D = T D S = S = L S T = T U where det() = det( ) det( ) 0 mples D 0. Sce the decmpst s trval fr =, ths recurs establshes the result, ad furthermre gves the tragular matrces S ad T frm the same recurs that ca be multpled t gve = TD S. Nw assume that s f rak r <, ad that the frst r clums f are learly depedet, wth zer leadg prcpal mrs up t rder r. Partt q e q eq e q e L 0 D 0 U U =, L I I z c z cz c z c where s r r ad the remag blcks are cmmesurate. The, U = D S ad L = T D, ad e must satsfy = L D U = T D S =. But the rak cdt requres that the last

19 McFadde Chapter. alyss ad Lear lgebra a Nutshell 36 q e r clums ca be wrtte as lear cmbats f the frst r clums, r z c q e = C fr sme r (r) matrx C. But = C mples C = ad hece z c = C = as requred. Fally, csder ay real matrx f rak r. By clum permutats, the frst r clums ca be made learly depedet. The, by rw permutats, the frst r rws f these r clums ca be made learly depedet. Repeat ths prcess recursvely the remag rthwest prcpal submatrces t bta prducts f permutat matrces that gve zer leadg prcpal mrs up t rder r. Ths defes P ad Q, ad cmpletes the prf f the therem. p Crllary. If s symmetrc, the L = U. Crllary. (LU Factrzat) If has zer leadg prcpal mrs, the ca be wrtte = LU, ~ where U ~ = DU s upper tragular wth a dagal ccdg wth that f D. Crllary 3. (Chlesky Factrzat) If s symmetrc ad pstve defte, the ca be wrtte = U U, ~~ where U ~ s upper tragular wth a pstve dagal. Crllary 4. symmetrc pstve semdefte mples = PU UP, ~~ wth U ~ upper tragular wth a egatve dagal, P a permutat matrx. Crllary 5. If s m wth m, the there exsts a factrzat = PLDUQ, wth D dagal, P a m m permutat matrx, Q a permutat matrx, U a upper tragular matrx wth es the dagal, ad L a m lwer q e L tragular matrx wth es the dagal (.e., L has the frm L = L wth L z c

20 McFadde Chapter. alyss ad Lear lgebra a Nutshell 37 ad lwer tragular wth es the dagal, ad L (m). Further, f ρ() =, the ( ) = QUD(L L) L P. T shw Crllary 3, te that a pstve defte matrx has pstve leadg prcpal mrs, ad te frm the prf f the therem that ths mples that the ~ / / dagal f D s pstve. Take U = D U, where D s the pstve square rt. The same cstruct appled t the LDU factrzat f after permutat gves Crllary 4. T shw Crllary 5, te frst that the rws f ca be permuted s that the q e frst rws are f maxmum rak ρ(). Suppse = s f ths frm, ad apply z c the therem t bta = P L DUQ. The rak cdt mples that = F fr sme (m) array F. The, = L DUQ, wth L = FP L, s that q eq e P 0 L = 0 I L DUQ. m z cz c T cmplete the prf, apply a left permutat f ecessary t ud the tal rw permutat f. mplcat f the last result Crllary 5 s that f the system f equats x = y wth m f rak has a slut, the the slut s gve by x = ( ) y = QU D (L L) L P y. The recurs the prf f the therem s called Crut s algrthm, ad s the methd fr matrx vers used may cmputer prgrams. It s uecessary t d the permutats advace f the factrzats; they ca als be carred ut recursvely, brgg rws ( what s termed a pvt) t make the successve elemets f D as large magtude as pssble. Ths pvt step s mprtat fr umercal accuracy.

21 McFadde Chapter. alyss ad Lear lgebra a Nutshell The Sgular Value Decmpst f a Matrx factrzat that s useful as a tl fr fdg the egevalues ad egevectrs f a symmetrc matrx, ad fr calculat f verses f mmet matrces f data wth hgh multcllearty, s the sgular value decmpst (SVD): Every real m matrx f rak r ca be decmpsed t a prduct = UDV, where D s a r r dagal matrx wth pstve creasg elemets dw the dagal, U s m r, V s r, ad U ad V are clumrthrmal;.e., U U =I r = V V. T prve that the SVD s pssble, te frst that the m m matrx s symmetrc ad pstve semdefte. The, there exsts a m m rthrmal matrx W, partted W = [W W ] wth W f dmes m r, such that W ( )W = Λ s dagal wth pstve, creasg dagal elemets, ad W ( )W = 0, mplyg W = 0. Defe D frm Λ by replacg the dagal elemets f Λ by ther pstve square rts. Nte that W W=I=WW W W +W W. Defe U = W ad V =D U. The, U U = I r ad V V =D U UD = D ΛD = I r. Further, = (I m W W ) = UU = UDV. Ths establshes the decmpst. If s symmetrc, the U s the array f egevectrs f crrespdg t the zer rts, s that U = UD, wth D the r r dagal matrx wth the zer egevalues decedg magtude dw the dagal. I ths case, V = UD = UD D. Sce the elemets f D ad D are detcal except pssbly fr sg, the clums f U ad V are ether equal (fr pstve rts) r reversed sg (fr egatve rts). The, the SVD f a square symmetrc sgular matrx prvdes the peces ecessary t wrte dw ts egevalues ad egevectrs. Fr a pstve defte matrx, the cect s drect.

22 McFadde Chapter. alyss ad Lear lgebra a Nutshell 39 Whe the m matrx s f rak, s that s symmetrc ad pstve defte, the SVD prvdes a methd f calculatg ( ) that s partcularly umercally accurate: Substtutg the frm = UDV, e btas ( ) = VD V. / Oe als btas cveet frms fr a square rt f ad ts verse, ( ) / = VDV ad ( ) = VD V. The umercal accuracy f the SVD s mst advatageus whe m s large ad sme f the rws f are early learly depedet. The, rudff errrs the matrx prduct ca lead t qute accurate results whe a matrx verse f s cmputed drectly. The SVD extracts the requred frmat frm befre the rudff errrs are trduced. Cmputer prgrams fr the Sgular Value Decmpst ca be fud Press et al, Numercal Recpes, Cambrdge Uversty Press, Prjects ad Idemptet Matrces Let x be a vectr R, expressed as a array, ad let L be a subspace f ~ R. The prject f x the subspace L s the pt y = ar gm NzxN that z L mmzes the Eucldea dstace betwee x ad L. Suppse L s f dmes k ad {b,...,b } s a bass fr L, ad let B be the k matrx whse clums are these k k bass vectrs. The L = {Bw w R }. The prject f x L ca the be calculated by slvg m w (xbw) (xbw). Oe ca shw by wrtg ths ut ad usg calculus that the mmum ccurs at w ~ = (B B) B x ad that the pt L clsest t xs ~ y = Bw ~ B(B B) B x. T verfy that ~ y achves the mmum, te that (xy) B ~ = x [IB(B B) B ]B = 0;.e., xy ~ s rthgal t L. Let c = w w, ~ ad rewrte the crter ~ ~ (xbw) (xbw) = (xbwbc) (xbwbc)

23 McFadde Chapter. alyss ad Lear lgebra a Nutshell 40 = (xbw) (xbw) ~ ~ + c B Bc (xbw) Bc ~ = (xbw) (xbw) ~ ~ + c B Bc (xbw) (xbw). ~ ~ The lear trasfrmat = B(B B) B s termed a prject matrx, ad s smetmes wrtte P B r P t emphasze that t s a trasfrmat that prjects L vectrs R t the subspace L r L B (.e., the subspace spaed by B). It has the prpertes that t s symmetrc wth =, s that t s demptet. Gemetrcally, ths says that ce a vectr s prjected t a subspace, tha repeated applcats f the same prject leave t uchaged. Cversely, every demptet matrx ca be terpreted as a prject matrx t sme subspace. mprtat ad useful prperty f prjects s that they deped ly the subspace, ad t the partcular chce f bass fr ths subspace. Thus, f B ad C are matrces that spa the cmm kdmesal subspace L, the P B = P C = P. L Further, t s rrelevat whether the clums f B r C are all learly depedet, as e ca smply dscard learly depedet clums befre frmg the prject matrx = B(B B) B. Sme f the prpertes f a demptet matrx are lsted belw: () The egevalues f are ether zer r e. () The rak f s the dmes f the lear space t whch the prject s made, ad ρ() = tr(). (3) The matrces I, 0, ad I are demptet. (4) If B s a rthrmal matrx, the B B s demptet. (5) If ρ() = r, the there exsts a r matrx B f rak r such that = B(B B) B, ad thus = P B. (6), B demptet mples B = 0 f ad ly f +B demptet. (7), B demptet ad B = B mples B demptet.

24 McFadde Chapter. alyss ad Lear lgebra a Nutshell 4 (8), B demptet mples B demptet f ad ly f B = B. Suppse X s r f rak r ad Z s s f rak s, ad csder the subspaces L X ad L Z ad the prjects = P X = X(X X) X ad B = P Z = Z(Z Z) Z. If X ad Z are clumrthgal (.e., X Z = 0), the B = 0 ad (6) mples that +B s the prject t L. If X ctas all the clums f Z, s that L L [X Z] Z X, the a prject e stage t L s equvalet t a prject t L Z X fllwed a secd stage by a further prject t L Z, s that B = B, ad a prject t L s left varat by a further prject t L Z X, s that B = B. Frm (8), ths mples that B s demptet; ths s the prject t the subspace f L X that x s rthgal t L ;.e., the subspace L L. Every vectr w R Z X Z has a uque x decmpst w = x + z + y, where x = Bw L, z = (B)w = (IB)w L L X X Z, ad y x (I)w L X. The prjects B ad I are rthgal, mplyg that I+B s als a x prject, t L L X Z. drect matrx mapulat gves a alteratve demstrat f (8). Suppse X = [Z W] s a partted matrx where Z s s, W s t, ad X s f rak r = +t <. Defe = X(X X) X ad B = Z(Z Z) Z. The q eq e q e Z Z Z W Z I B = [Z W] Z(Z Z) Z =[ZW] W Z (Z Z) Z = Z(Z Z) Z =B, W W W 0 z c z c z c s that (8) mples B s demptet, thus als I+B. 0. Geeralzed Iverses k m matrx s a MrePerse geeralzed verse f a m k matrx f t has three prpertes: () =, () =, () ad are symmetrc.

25 McFadde Chapter. alyss ad Lear lgebra a Nutshell 4 There are ther geeralzed verse defts that have sme, but t all, f these prpertes. Results geeralzed verses, partcularly fr partted r brdered matrces, ca be fud R.M. Prgle ad. Rayer Geeralzed Iverse Matrces, Grff, 97. Let r dete the rak f. Recall that has a sgular value decmpst = UDV, where D s a r r dagal matrx wth a pstve creasg umbers dw the dagal, U s k r ad clumrthrmal, ad V s m r ad clumrthrmal. The MrePerse geeralzed verse f s the the matrx = VD U. It s easy t check that ths deft satsfes prpertes ()() abve, as well as the fllwg prpertes: () = f s square ad sgular. () ad are demptet, ad are prject matrces, respectvely, t the subspace f m k R spaed by the clums f ad t the subspace f R spaed by the rws f. (3) The system f equats x = y, wth m k, has a slut f ad ly f y = y (.e., y s the lear subspace spaed by the clums f ). If t has a slut, the the affe lear subspace f all sluts s k k {x R x = y + [I ]z fr all z R }. slut s uque f ad ly f = I (.e., s f rak k.) (4) If s square, symmetrc, ad pstve semdefte, the there exsts Q pstve defte, R demptet, such that = QRQ ad = Q RQ. (5) If s demptet, the =. (6) If = BCB wth B sgular, the = (B ) C B.

26 McFadde Chapter. alyss ad Lear lgebra a Nutshell 43 (7) If s k k ad pstve semdefte wth rak r, the there exsts a k r clumrthrmal matrx U such that U U = I r, UU = D s a dagal matrx wth a pstve dagal, = UDU, ad = UD U. (8) If s square, symmetrc, ad pstve semdefte, the t has a / symmetrc square rt B =, ad = B B. (9) ( ) = = ( ) =(). (0) ( ) = ( ) () ( ) = ( ) s s () If = wth = 0 ad = 0 fr j, the =. t j j t Mst f these results are easy csequeces f the sgular value decmpst, r ca be checked by verfyg that cdts ()() hld. T prve (4), let W = [U V] be a rthrmal matrx dagalzg. The, U U = D, a dagal matrx f q e q e / I 0 pstve egevalues, ad V = 0. Defe Q = W D 0 W ad R = Wr W z c z c. Krecker Prducts If s a m matrx ad B s a p q matrx, the the Krecker (drect) prduct f ad B s the (mp) (q) partted array q e a B a B... a B a B a B... a B B =. a B a B a B m m m z c I geeral, B B. The Krecker prduct has the fllwg prpertes: () Fr a scalar c, (c) B = (cb) = c( B). () ( B) C = (B C).

27 McFadde Chapter. alyss ad Lear lgebra a Nutshell 44 (3) ( B) =( ) (B ). (4) tr( B) = (tr()) (tr(b)). (5) If the matrx prducts C ad BF are defed, the ( B)(C F) = (C) (BF). (6) If ad B are square ad sgular, the ( B) = B. (7) If ad B are rthrmal, the B s rthrmal. (8) If ad B are pstve semdefte, the B s pstve semdefte. (9) If s k k ad B s, the det( B) = det() det(b) k. (0) ρ( B) = ρ() ρ(b). () (+B) C = C +B C.. Shapg Operats The mst cmm perats used t reshape vectrs ad matrces are () C = dag(x) whch creates a dagal matrx wth the elemets f the vectr x dw the dagal; () c = vec() whch creates a vectr by stackg the clums f ; (3) c = vech() whch creates a vectr by stackg the prprts f the clums f that are the upper tragle f the matrx; ad (4) c = vecd() whch creates a vectr ctag the dagal f. There are a few rules that ca be used t mapulate these perats: () If x ad y are cmmesurate vectrs, dag(x+y) = dag(x) + dag(y). () vec(+b) = vec() + vec(b). (3) If s m k ad B s k, the vec(b) = (I )vec(b) = (B I m )vec(). (4) If s m k, B s k, C s p, the vec(bc) = (I p (B))vec(C) = (C )vec(b) = ((C B ) I m )vec().

28 McFadde Chapter. alyss ad Lear lgebra a Nutshell 45 (5) If s, the vech() s f legth (+)/. (6) vecd(dag(x)) = x ad dag(vecd()) =. 3. Vectr ad Matrx Dervatves The dervatves f fucts wth respect t the elemets f vectrs r matrces ca smetmes be expressed a cveet matrx frm. Frst, a scalar fuct f a vectr f varables, f(x), has partal dervatves that are usually wrtte as the arrays q e q e df/dx df /dx df /dx dx... df /dx dx df/dx df/dx = df /dx dx df /dx... df /dx dx, df /dxdx =. df/dx df /dx dx df /dx dx... df /dx z c z c Other cmm tat s f (x) r f(x) fr the vectr f frst dervatves, ad x x f (x) r f(x) fr the matrx f secd dervatves. Smetmes, the vectr f xx x frst dervatves wll be terpreted as a rw vectr rather tha a clum vectr. Sme examples f scalar fucts f a vectr are the lear fuct f(x) = a x, whch has f = a, ad the quadratc fuct f(x) = x x, whch has f = x. x x k Whe f s a clum vectr f scalar fucts, f(x) = [f (x) f (x)... f (x)], the the array f frst partal dervatves s called the Jacbea matrx ad s wrtte q e df /dx df /dx... df /dx J(x) = df /dx df /dx... df /dx. k k k df /dx df /dx... df /dx z c

29 McFadde Chapter. alyss ad Lear lgebra a Nutshell 46 Whe calculatg multvarate tegrals f the frm g(y)dy, where y R, R, j ad g s a scalar r vectr fuct f y, e may wat t make a lear ete trasfrmat f varables y = f(x). I terms f the trasfrmed varables, the tegral becmes g(y)dy = g(f(x)) det(j(x)) dx, j j f () where f () s the set f x vectrs that map t, ad the Jacbea matrx s square ad sgular fr wellbehaved ete trasfrmats. The tut fr the presece f the determat f the Jacbea the trasfrmed tegral s that "dy" s the vlum f a small rectagle yspace, ad because determats gve the vlum f the parallelpped frmed by the clums f a lear trasfrmat, det(j(x))dx gves the vlum (wth a plus r mus sg) f the mage xspace f the "dy" rectagle yspace. It s useful t defe the dervatve f a scalar fuct wth respect t a matrx as a array f cmmesurate dmess. Csder the blear frm f() = x y, where x s, y s m, ad s m. By cllectg the dvdual terms df/d = xy, e btas the result df/d = xy. ther example fr a matrx s f() = tr(), whch has df/d = I. There are a few ther dervatves that are partcularly useful fr statstcal applcats. I these frmulas, s a square sgular matrx. We d t requre that be symmetrc, ad the dervatves d t mpse symmetry. Oe wll stll get vald calculats vlvg dervatves whe these expresss are evaluated at matrces that hape t be symmetrc. There

30 McFadde Chapter. alyss ad Lear lgebra a Nutshell 47 are alteratve, ad smewhat mre cmplcated, dervatve frmulas that hld whe symmetry s mpsed. Fr aalyss, t s uecessary t trduce ths cmplcat. () If det() > 0, the dlg(det())/d =. () If s sgular, the d(x y)/d = xy. (3) If = TT, wth T square ad sgular, the d(x y)/dt =xy T. s We prve the frmulas rder. Recall that det() = () +k a det( k ), where k t k k s the mr f a. The, ddet()/d = () +j det( k ). Frm.3.7, the elemet f s () +j det( )/det(). Ths establshes (). T shw (), frst apply the cha rule t the detty I t get + d /d 0, where detes a matrx wth a e rw ad clum j, zers elsewhere. The, dx y/d = x y = ( x) ( y) j. Ths establshes (). T shw (3), frst te that d /dt = δ T +δ T. Cmbe ths wth () t get rs r js jr s ss dx y/dt = ( x) ( y) (δ T +δ T ) rs tt j r js jr s j s s = ( x) ( y)t + ( x)( y) T t r j js t r s j = ( xy T). rs 4. Updatg ad Backdatg Matrx Operats Ofte statstcal applcats, e eeds t mdfy the calculat f a matrx verse r ther matrx perat t accmdate the addt f data, r

31 McFadde Chapter. alyss ad Lear lgebra a Nutshell 48 delet f data btstrap methds. It s cveet t have quck methds fr these calculats. Sme f the useful frmulas are gve belw: () If s ad sgular, ad has bee calculated, ad f B ad C are arrays that are k f rak k, the (+BC ) = B(I +C B) C, k prvded I k +C B s sgular. requred the updatg. If k =, the matrx vers s () If s m wth m ad ρ() =, s that t has a LDU factrzat = PLDUQ wth D dagal, P ad Q permutat matrces, L lwer q e tragular, U upper tragular, the the array, wth B k, has the LDU B z c q e q eq e P 0L factrzat = 0 I DUQ, where C = BQU D. B kc z c z cz c q e ~ (3) Suppse s m f rak, ad B = ( ) y. Suppse = wth C k, C z c q e ~ y y= ~ ~ ~ ~ ~ wth w k, ad B = ( ) y. The, w z c BB ~ = ( ) C [I +C( ) C ] k (wcb) ~~ ~~ =( ) C [I C( ) C ] (wcb). ~ k Oe ca verfy () by multplcat. T shw (), use Crllary 5 f Therem 5.. T shw (3), apply () t ~~ = +C C, r t = ~~ C C, ad use ~~ y = y + Cw.

32 McFadde Chapter. alyss ad Lear lgebra a Nutshell 49 NOTES ND COMMENTS The basc results f lear algebra, cludg the results stated wthut prf ths summary, ca be fud stadard lear algebra texts, such as G. Hadley (96) Lear lgebra, ddswesley r F. Graybll (983) Matrces wth pplcats Statstcs, Wadswrth. The rgazat f ths summary s based the admrable sypss f matrx thery the frst chapter f F. Graybll (96) Itrduct t Lear Statstcal Mdels, McGrawHll. Fr cmputats vlvg matrces, W. Press et al (986) Numercal Recpes, Cambrdge Uv. Press, prvdes a gd dscuss f algrthms ad accmpayg cmputer cde. Fr umercal ssues statstcal cmputat, see R.Thsted (988) Elemets f Statstcal Cmputg, Chapma ad Hall.