Chapters 4-5 Linear Models & Matrix Algebra

Transcription

1 Chpters 4-5 Lier Models & Mtrix Algebr A. L. Cuchy ( , Frce) Giuseppe Peo (858 93, Itly) 5. - Lier Models & Mtrix Algebr: Summry Mtrix lgebr c be used:. to express the system of equtios i compct ottio; b. to fid out whether solutio to system of equtios exist; d c. to obti the solutio if it exists. d. If is smll, we c fid A -, but i geerl, we will void this step. We will resort to more efficiet methods to solve for x*, likely usig Cholesky decompositio with Gussi elimitio. Ax x A x * * d A d dja det A dja d A

2 5. - Nottio d Defiitios: Summry A (Upper cse letters) = mtrix b (Lower cse letters) = vector xm = rows, m colums rk(a) = umber of lierly idepedet vectors of A trce(a) = tr(a) = sum of digol elemets of A Null mtrix = ll elemets equl to ero. Digol mtrix = ll off-digol elemets re ero. I = idetity mtrix (digol elemets:, off-digol: 0) A = det(a) = determit of A A - = iverse of A A =A T = Trspose of A M ij = Mior of A A=A T => Symmetric mtrix A T A =AA T => Norml mtrix A T =A - => Orthogol mtrix A =A => Idempotet mtrix 3 5. Eigevlues d Digol Systems A set of lier simulteous equtios: Ab = d Ais o-sigulr d b d d re comformble vectors. Uder certi circumstces we c digolie this system: Λν=υ where Λ is digol mtrix. Eigevlues (Chrcteristic Roots) Ax = x This is the eigevlue problem λ is the eigevlue (chrcteristic root) x is the eigevector (chrcteristic vector) Cuchy discovered them studyig how to fid ew coordite xes for the grph of the qudrtic equtio x +bxy+cy =d so tht the equtio with the ew xes would be of the form Ax +Cy =D.

3 5. Eigevlues d Digol Systems Ax = x (Bsic equtio of eigevlue problem) For the squre mtrix A, there is vector x such tht the product of Ax such tht the result is sclr,, tht, whe multiplied by x, results i the sme product. The multiplictio of vector x by sclr is the sme s stretchig or shrikig the coordites by costt vlue. (The mtrix A just scles the vector x!) Ax = Ix => [ A - I ] x = 0 K = [ A - I] Chrcteristic mtrix of mtrix A Kx= 0 Homogeeous equtios. 5. Eigevlues d Digol Systems Homogeeous equtios: Kx= 0 - Trivil solutio x = 0 (If K 0, from Crmer s rule) - Notrivil solutio (x 0) c occur if K = 0. Tht is, do ll mtrices hve eigevlues? No. They must be squre d K = A- I = 0. Eigevectors re ot uique. If x is eigevector, the x is lso eigevector: A(x)= (x) To clculte eigevectors d eigevlues, expd the equtio A- I =0 The resultig equtio is clled chrcteristic equtio. 3

4 5. Eigevlues d Digol Systems Chrcteristic equtio: A- I =0 Exmple: For x mtrix: [A I] 0 0 A I 0 For -dimesiol problem, we hve simple qudrtic equtio with two solutios for Eigevlues d Digol Systems For =, we hve simple qudrtic equtio with two solutios for. I fct, there is geerlly oe eigevlue for ech dimesio, but some my be ero, d some complex. 0 4 Note: The solutio for λ c be writte s: λ = ½trce(A)± ½[trce(A) - 4 A ] / Three cses: ) Rel differet roots: trce(a) > 4 A ) Oe rel root: trce(a) = 4 A 3) Complex roots: trce(a) < 4 A 4

5 5. Eigevlues d Digol Systems Note: If A is symmetric, the eigevlues re rel. Tht is, we eed to hve trce(a) > 4 A. For =, we check this coditio: Eigevlues d Digol Systems Some properties: - The product of the eigevlues = A - The sum of the eigevlues = trce(a) - The eigevlues of A k re λ k, λ k,..., λ k. -If A is idempotet mtrix, its λ i s re ll 0 or. -If A is orthogol mtrix (A T =A - ), its λ i s re ll. -If A is symmetric (Hermiti) mtrix: -its λ i s re ll rel. - its eigevectors re orthogol. - All eigevectors derived from uequl eigevlues re lierly idepedet. ( eigevectors c form orthoroml bsis!). 5

6 5. Eigevlues d Digol Systems Geometric Iterprettio: The x,y vlues of A c be see s represetig poits o ellipse cetered t (0,0). The eigevectors re i the directios of the mjor d mior xes of the ellipse, d the eigevlues re the legths of these xes to the ellipse from (0,0). 5. Eigevlues d Digol Systems Exmple: A correltio mtrix A λ = ½trce(A) ± ½[trce(A) - 4 A ] / = ½ ± ½[ - 4x0.4735] / = ± ½[.5] / = ± ½[.5] = 0.5;.75 x = [ ]; [ ] Note: x is ot uique. It is usully imposed tht x =. 6

7 5. Eigevlues d Digol Systems Grphicl iterprettio: Correltio s ellipse, whose mjor xis is oe eigevlue d the mior xis legth is the other: No correltio yields circle, d perfect correltio yields lie. 5. Eigevlues: Exmple d order multivrible equtios: x + kxy + by = c Represeted i qudrtic form with symmetric mtrix A: x T A x = c, where Eigevector decompositio: λ =.764, x =[.557,-.8507] λ =6.36, x =[-.8507,-.557] Symmetric A => orthogol e-vectors! Geometricl iterprettio: Pricipl Axes of Ellipse Positive Defiite A => positive rel eigevlues! 7

8 5. Eigevlues d Digol Systems Geerl x cse: The chrcteristic determit D(λ) = det (A - λ I) is clerly polyomil i λ: D() = Chrcteristic equtio: D() = = 0 There re solutios to this polyomil. The set of eigevlues is clled the spectrum of A. The lrgest of the bsolute vlues of the eigevlues of A is clled the spectrl rdius of A. Eigevlues re computed usig the QR lgorithm (950s) or the divide-d-coquer eigevlue lgorithm (990s). They re computtiolly itesive. They tke 4 3 /3 flops. 5. Digol (Eige) decompositio Let A be squre x mtrix with lierly idepedet eigevectors, x i (i=,,,). The A c be fctoried s A=X X - where X is the squre (x) mtrix whose i th colum is the eigevector x i of A d Λ is the digol mtrix whose digol elemets re the correspodig eigevlues, i.e., Λ ii = λ i. The eigevectors re usully ormlied, but they eed ot be. A o-ormlied set of eigevectors c lso be used s the colums of X. Proof: Ax = λx => AX=X => A=XX - (X - exists) Coversely: X - A X = If X - X = I, A is orthogolly digolible. 8

9 5. Digol decompositio: Exmple Let A ;, 3. The eigevectors: x =[,-], x =[,]. Let X be the mtrix of eigevectors: X Ivertig, we hve X / / / / The, A=XX = 0 / / 0 3 / / 5. Digol decompositio: Exmple Digoliig system of equtios: A x = y Pre-multiply both sides by X - : X - A x = X - y = ν X - A (X X - )x = ν (Let υ = X - x) => υ = ν Usig the (x) previous exmple: 0 0 / 3 / / x / x / / / y / y υ = ν 3υ = ν υ = ½ x -½ x υ = ½ x + ½ x ν = ½ y -½ y ν = ½ y + ½ y 9

10 5. Digol decompositio: Applictio Let M be the squre x mtrix defied by: M = I - Z(Z Z - )Z, where Z is xk mtrix, with rk(z)=k. Let s clculte the trce(m): trce(m) = trce(i - Z(Z Z - )Z ) = trce(i ) - trce( Z(Z Z - )Z ) = = - trce( (Z Z - )Z Z) = - trce(i k ) = k. It is esy to check tht M is idempotet (λ i s re ll 0 or ) d symmetric (λ i s re ll rel d x re orthogol). Write orthogol digolitio: M = X X - (X X - = I). Agi, let s clculte the trce(m = X X - ): trce(m) = trce(xx - ) = trce(x - X) = trce() = Σ i λ i Tht is, M hs k o-ero eigevlues. 5. Why do Eigevlues/vectors mtter? 0

11 5. Why do Eigevlues/vectors mtter? Eigevectors re ivrits of A Do t chge directio whe operted A Use to determie the defiiteess of mtrix. A sigulr mtrix hs t lest oe ero eigevlue. Solutios of multivrible differetil equtios (the bred-dbutter i lier systems) correspod to solutios of lier lgebric eigevlue equtios. Eigevlues re used to study the stbility of utoregressive time series models. The orthogol bsis of eigevectos forms the core of pricipl compoets lysis (PCA). 5. Sig of qudrtic form: Eigevlue tests Suppose we re iterestig i optimitio problem for =f(x,y). We set the first order coditios (f.o.c.), solve for x* d y*, d, the, check the secod order coditios (s.o.c.). Let s re-write the s.o.c. of =f(x,y): d q q f xx dx f f f f f dxdy dy xx xy dx dy u ' Hu xy yy xy dx dy The s.o.c. of =f(x,y) is qudrtic form, with symmetric mtrix, H. f yy To determie wht type of extreme poits we hve, we eed to check the sig of the qudrtic form.

12 5. Sig of qudrtic form: Eigevlue tests.qudrtic form: q = u H u (ote: the Hessi (H) is symmetric mtrix) Let u=ty, where T is the mtrix of eigevectors of H, such tht T T = I. The, q = y T HTy = y Λ y (T HT= Λ) q = λ y + λ y λ i y i λ y => The sig(q) depeds o the λ i s oly. We sy: q is positive defiite iff λ i >0 for ll i. q is positive semi-defiite iff λ i 0 for ll i. q is egtive semi-defiite iff λ i 0 for ll i. q is egtive defiite iff λ i <0 for ll i. q is idefiite if some λ i >0 d some λ i < Sig of qudrtic form: Eigevlue tests Exmple: Fid extreme vlues for =f(x,y), d determie if they re mx or mi. x F.o.c. xy y f x x y 0, f y x 4y 0 y* 0, x* 0, * 3 Clculte mtrix of secod derivtives H d re * f f 3 f f is miimum, 4 positive, q is positive.5858; defiite

13 5.3 Vector multiplictio: Geometric iterprettio Thik of vector ( Euclidi vector) s directed lie segmet i N- dimesios! (hs legth d directio ) Sclr multiplictio ( scles the vector i.e., chges legth) Source of lier depedece U x 3 U 6 4 U x Vector Additio: Geometric iterprettio x v' = [ 3] 5 u' = [3 ] w = v'+u' = [5 5] Note: Two vectors plus the cocepts of dditio d multiplictio c crete twodimesiol spce. 4 3 u v w u x A vector spce is mthemticl structure formed by collectio of vectors, which my be dded together d multiplied by sclrs. (It s closed uder multiplictio d dditio.) Giuseppe Peo i 888 gve precise defiitio to this cocept. 6 3

14 5.3 Vector (Lier) Spce We itroduce lgebric structure clled vector spce over field. We use it to provide bstrct otio of vector: elemet of such lgbebric structure. Give field R d set V of objects, o which vector dditio (VxV V), deoted by +, d sclr multiplictio (RxV V), deoted by., re defied. If the followig xioms re true for ll objects u, v, d w V d ll sclrs c d k i R, the V is clled vector spce d the objects i V re clled vectors.. u+v V (closed uder dditio).. u + v = v + u (vector dditio is commuttive). 3. Ø V, such tht u+ Ø = u (Ø = ull elemet). 4. u + (v+w) = (v + u) +w (distributive lw of vector dditio) Vector Spce 5. For ech v, there is v, such tht v+(-v) = Ø 6. c.u V (closed uder sclr multiplictio). 7. c. (k. u) = (c.k) u (sclr multiplictio is ssocitive). 8. c. (v + u) = (c. v) + (c. u) 9. (c + k). u = (c. u) + (k. u) 0..u=u (uit elemet).. 0.u= Ø (ero elemet). We c write S = {V,R,+,.}to deote bstrct vector spce. This is geerl defiitio. If the field R represets the rel umbers, the we defie rel vector spce. Giuseppe Peo (858 93, Itly) 8 4

15 5.3 Vector Spce Defiitio: Lier Combitio Give vectors u,...,u k,, the vector w = c u c k u k is clled lier combitio of the vectors u,...,u k,. Nottio: < u,...,u k > is the set of ll lier combitios of u j s. Defiitio: Subspce Give the vector spce V d W set of vectors, such tht W V. The, W is subspce iff: - u, v W => u+v V, d -c.u V for every c R. Thus, oempty subset W of vector spce V tht is closed uder dditio d sclr multiplictio (d cotis the 0- vector of V) is subspce of V. Tht is, subspce is subset of V tht c be cosidered vector spce! Vector Spce Defiitio: Spig set Give the set Z i V d U={u,...,u k } i Z, we sy U sps Z, or U is spig set for Z, if Z <u,...,u k > or Z <U>. Defiitio: Bsis set ( bsis ) Give U={u,...,u k } d subspce W V. We sy tht if U is LI d it is spig set for W, the U is clled bsis set for W. Exmple: The N-dimesiol subspce W N of the V spce (N=). 30 5

16 5.3 Vector Spce Theorem: If vector spce hs bsis with fiite umber N of elemets, the every other bsis lso hs N elemets. Defiitio: Spce dimesio If vector spce V hs bsis with N< elemets, we sy tht V is fiite dimesiol vector spce d tht V hs dimesio N, or N = dim(v). Theorem: The vector spce cosistig of -colum vectors, with vector dditio d multiplictio correspodig to mtrix opertios is -dimesiol vector spce (Euclide -spce), which we will deote R Vector Spce Defiitio: Mximlly lier idepedet (mx-li) subset Give set U={u,...,u k } of vectors i vector spce V. If T ={u i,,...,u i,q } U cotiig q vectors is LI d every subset with more th q vectors is LD, the T is clled mximlly lier idepedet subset of U. Moreover, we will cll q the rk of the set U, writte q = rk(u). It is commo to sy the rk of mtrix A equls the umber of LI colums (rows) i A. This is OK, but keep i mid tht i geerl there is ot uique subset of q LI colums or rows tht re mx-li. Defiitio: Full rk Give A (mx). We sy A hs full rk if rk(a)=mi(m,). 3 6

17 5.3 Vector Spce As we defied them, vector spces do ot provide eough structure to study issues i rel lysis, for exmple covergece of sequeces. More structure is eeded. For exmple, we c itroduce s dditiol structure the cocept of order ( ), to compre vectors. This dditiol structure cretes ordered vector spces. We c itroduce orm, which we will use to mesure the legth or mgitude of vectors. This cretes ormed vector spce, deoted s pir (V,. ) where V is vector spce d. is orm o V Notes o Vector Opertios A (mx) colum vector u d (x) row vector v, yield product mtrix uv of dimesio (mx). u 3 x u v 3 x v x A mtrix u ' v x A sclr 34 7

18 8 The dot product produces sclr! y = c =x(=xx)= c. Note tht from the defiitio, the dot product is commuttive. Whe c is vector of s, usully oted s ι, the: 35 i i i c c c c c c c c ' c c c y The dot product,, is fuctio tht tkes pirs of vectors d produces umber. For two vectors, c d, it is defied s: 5.3 Vector Multiplictio: Dot (ier) Product i i 3... It is possible to defie ier product for fuctios. Isted of sum over the correspodig elemets of vector, the ier product o fuctios is defied s itegrl over some itervl. Some ituitio. - The dot product is used s tool to defie sie for vectors: - Now, we c compre vectors d mesure distces betwee vectors d, evetully, covergece! 5.3 Vectors: Dot Product Dot products i ecoometrics re commo. For exmple, the Residul Sum of Squres (RSS), where e is vector of residuls: i i e e e e e e e...ee'ee 3 / ] [ T

19 5.3 Vectors: Dot Product - Properties The dot product fulfils the followig properties if α,β, d γ re rel vectors d k is sclr.. Commuttive: α β = <α,β> = β α Note: We sy α d β (o-ero vectors) re orthogol iff α β = 0. Distributive over vector dditio: α (β + γ) = α β + α γ 3. Bilier: α (kβ + γ) = k (α β+)+ α γ 4. Sclr multiplictio: (k α) (k β) = k k (α β) Note: Nice, ituitive properties. Nottio: α β=<α,β> 5.3 Vectors: Dot Product & Sie The mgitude (legth or sie) is the squre root of the dot product of vector with itself (just like the Pythgore theorem): T / [ ]... Now, we c tlk bout the sie of vectors. We c pply this defiitio to defie other cocepts, for exmple, covergece, i similr fshio s i clculus: A sequece of vectors x coverge to poit c if x c decreses to 0 s icreses. Useful property: If k is sclr, the the sie of vector times k is k times the sie of the vector. k [ kk ] k Note: If we set k = / α => (/ α ) α =. => Nice result used to ormlie vectors. T / 9

20 5.3 Vectors: Dot Product Geometry There is geometric iterprettio to the dot product. Ay two vectors, sy α d β, determie ple. We c mesure the gle betwee the two vectors. The ier product coects the legth d the gle betwee the vectors:. cos( ) The dot product is relted to the gle betwee the two vectors but it does t tell us the gle. Notes: As the cos(θ=90)=0, the dot product of two perpediculr (orthogol) vectors is ero. I the CLM, we hve y = Xb + e = Projectio + residul The: X e = X (y-xb) = X y - X X(X X) - X y = 0 => (X e). 5.3 Vectors: Mgitude d Phse (directio) v ( x, x,, x )' v x i (Mgitude or -orm ) i If v, v is uit vector (uit vector => pure directio) y v x Alterte represettios: Polr coords: ( v, ) Complex umbers: v e j phse cos x, y x y 0

21 5.3 Vectors: Norm Give vector spce V, the fuctio g:v R is clled orm iff: ) g(x) 0, for ll x V ) g(x)=0 iff x=θ (empty set) 3) g(αx) = α g(x) for ll α R, x V 4) g(x+y) g(x)+g(y) ( trigle iequlity ) for ll x,y V The orm is geerlitio of the otio of sie or legth of vector. Exmple: O R, the Euclidi orm of x=(x, x,..., x ) is give by x / [ xx'] x x... x while the Mhtt (Txicb) orm is defied s: Note: Euclidi orm = L orm (-orm). Mhtt orm = L orm (-orm). x i x i 5.3 Vectors: Norm We c geerlie the cocept of orm. Defiitio: L p orm For rel umber p, the L p -orm (or p-orm) of x is defied by x p p p p / p x... x ) ( x A ifiite umber of fuctios c be show to qulify s orms. For vectors i R, we hve the followig exmples: g(x)=mx i (x i ), g(x)= i x i, g(x)=[ i (x i ) 4 ] ¼ Give orm o vector spce, we c defie mesure of how fr prt two vectors re, usig the cocept of metric.

22 5.3 Vectors: Metric Give vector spce V, the fuctio d: VxV R is clled metric or distce fuctio if d oly if: ) d(x,y) 0, ( positive ) for ll x,y V ) d(x,y) = 0 ( o-degeerte ) iff x=y 3) d(x,y) = d(y,x) ( symmetric) for ll x,y V 4) d(x+y) d(x,) + d(,y) ( trigle iequlity ) for ll x,y, V Give orm g(.), we c defie metric by the equtio: d(x,y) = g(x-y). Check: ) d ) follow immeditely from properties of g(.) 3) d(x,y) = g(x-y) = g((-)(y-x)) = - g(y-x) = g(y-x) = d(y,x). 4) (x-y) = (x-) + (-y) => g(x-y) g(x-) + g(-y) => d(x,y) d(x,) + d(,y) 5.3 Vectors: Metric Exmple: O R, the distce betwee two poits is usully give by the -orm distce. But, other distces re possible. -orm distce: -orm distce: p-orm distce: d( x, y) x i y i i d( x, y) ( x i y i ) i d( x, y) ( x i y i ) i / p / p The red, yellow d blue lies hve the sme legth () i.e., sme L distce. The gree lies is the L distce, the shortest distce, which is uique.

23 5.3 Vectors: Metric Theorem: Cuchy-Schwr Iequlity If u d v re vectors i rel ier product spce, the u, v u v Note: This result c be writte s u, v u, u v, v u, v u v Geerl Proof: Trivil proof whe v = 0, so we ssume tht <v, v> Vectors: Metric Theorem: u, v u v Proof: Let δ be y umber i the field F. The, 0 u v uv, uv u, u v, u u, v v, v Choose the vlue of δ tht miimies this qudrtic form: u, v v, v (A quick wy to remember this vlue of δ is to imgie F to be the rel umbers, d set the derivtive equl to ero to pick δ. ) We get 0 u, u u, v v, v which is true if d oly if u, v u, u v, v or equivletly: u, v u v 3

24 5.3 Vectors: Metric Spce Defiitio: Metric Spce A metric o spce M is mppig d(.,.): M M [0, ) stisfyig the metric properties () through (4) for ll x, y d i M. A spce edowed with metric is clled metric spce. M is ot ecessrily vector spce. By defiitio, y spce edowed with metric is metric spce. For exmple, the spce of desity fuctios o [0,] edowed with the metric: This spce is ot vector spce (& ot possible to defie <.>!) 5.3 Vectors: Hilbert Spce Defiitio: pre-hilbert spce A vector spce V edowed with ier product <x,y> d ssocited orm x =sqrt(<y,x>) d metric x-y is clled pre-hilbert spce. We sy pre becuse fudmetl property is still missig, mely tht every Cuchy sequece hs limit i V. Defiitio: Cuchy sequece A sequece of elemets x of metric spce with metric d(.,.) is clled Cuchy sequece if for every ε>0 there exists 0 (ε) such tht for ll k,m 0 (ε), d(x k, x m ) < ε. Exmple: I R p with fiite dimesio p every Cuchy sequece coverges to limit i R p. 4

25 5.3 Vectors: Hilbert Spce We wt the Hilbert spce to be complete i.e., every Cuchy sequece hs limit i the spce. A very useful property. If we hve coverget sequece of poits, the it ctully coverges to poit i the spce. Cotrst tht with, sy, the rtiol umbers, Q. We c hve coverget sequece sy, pproximtios to π tht do ot coverge to poit i Q, becuse, obviouslyπ is irrtiol. This situtio does ot occur i Hilbert spce: If we hve coverget sequece of vectors to poit p, the, p is i the spce. Now, the techiques of clculus c be used. Completeess lso mkes Hilbert spce closed uder covergece, which geertes useful properties. 5.3 Vectors: Hilbert Spce Defiitio: Hilbert spce A Hilbert spce H is vector spce edowed with ier product d ssocited orm d metric such tht every Cuchy sequece i H hs limit i H. A Hilbert spce is specil cse of Bch spce. A Bch spce is complete ormed vector spce. I Hilbert spce we specified orm, the ier product. If the spce is ot complete, H is kow s ier product spce. Usully, i lier lgebr, we re fmilir with some vector spces. They re R or C. These re lso Hilbert spces. 5

26 5.3 Vectors: Hilbert Spce Hilbert spces pper frequetly i mthemtics, sttistics, d physics, typiclly s ifiite-dimesiol fuctio spces. Hilbert spce methods helped i the developmet of fuctiol lysis. Exmple I: The spce V of rdom vribles defied o commo probbility spce {Ω,F, P} with fiite secod momets, edowed with the ier product <X,Y>=E[XY] d ssocited orm X =sqrt(<x,x>) d metric X-Y. The spce V is Hilbert spce. Exmple II: L, the set of ll fuctios f:r R such tht the itegrl of f over the whole rel lie is fiite. I this cse, the ier product is <f,g> = f(x) g(x) dx 5.3 Vectors: Hilbert Spce Typicl exmples of Hilbert spces: Euclide spces, spces of squre-itegrble fuctios, spces of sequeces. Hilbert Spce - Summry: - Geerlitio of Euclidi spce (R, R 3 ). - Abstrct vector (lier) spce with ier product, complete. - Nice properties: lier spce, ier product, sums tht should coverge do coverge, clculus c be used. Dvid Hilbert (86 943, Germy) 6

27 5.3 Vectors: Orthogolity Theorem: (Geerlied Lw of Pythgors) If u d v re orthogol vectors i ier product spce, the u + v = u + v Proof: It follows from <u,v>=0. Defiitio: Orthogol Complemet Let W be subspce of ier product spce V. A vector u i V is sid to be orthogol to W if it is orthogol to every vector i W. The set of ll vectors i V tht re orthogol to W is clled the orthogol complemet of W. Nottio: We deote the orthogol complemet of subspce W by W. [red W perp ] 5.3 Vectors: Orthogolity Theorem: Properties of Orthogol Complemets If W is subspce of fiite-dimesiol ier product spce V, the - W is subspce of V. - The oly vector commo to both W d W is 0. - The orthogol complemet of W is W; tht is (W ) = W. Theorem: If W is subspce of R, the dim(w) + dim(w )=. Furthermore, if {u,...,u k } is bsis for W d {u k+,...,u }is bsis for W, the {u,...,u k, u k+,...,u } is bsis for R. 7

28 5.3 Vectors: Projectios Defiitio: Projectio Let u d v be two o-ero vectors i ier product spce V. The, the sclr projectio of u oto v is defied s v u, v k u, v v The vector projectio of u oto v is v u, v p k v v v, v Derivtio: Give two vector v i S d u i R. We wt to fid p, the vector i S closest to u. Let p=kv. To miimie u-p with respect to k. u-p = u-kv = (u-kv) (u-kv) = u u - ku v + k v v d( u-p )/dk = -u v + kv v = 0 k = u v/v v p = (u v/v v) v 5.3 Vectors: Projectios Lemm: Let v be o-ero vector d p be the projectio of u oto v. The, u (i) (u-p) p u p (ii) u =p u=k v for some k Proof: Recll p = kv = (u v/v v) v (i) <p,u-p> =<p,u> - <p,p> = v p = <u,v> / v - <u,v> / v = 0 => (u-p) p (ii) strightforwrd. 8

29 5.3 Vectors: Projectios - CLM I the CLM, we hve Projectio mtrix, P: P = X(X X) - X (X is Nxk P is NxN) Fetures Py = X(X X) - X y = Xb = ŷ (fitted vlues) Py is the projectio of y ito the colum spce of X. PM = P[I T - X(X X) - X ] = MP = 0 (M: residul mker) PX = X Properties - P is symmetric P = P - P is idempotet P*P = P - P is sigulr P - does ot exist. rk(p)=k 5.3 Vectors: Projectios - CLM We hve two wys to look t y: y = X + = Coditiol me + disturbce y = Xb + e = Projectio + residul Note: X e = X (y-xb) = X y - X X(X X) - X y = 0. 9

30 30 Bsis: spce is totlly defied by set of vectors y poit is lier combitio of the bsis Ortho-Norml: orthogol + orml Orthogol: dot product is ero i.e., vectors re perpediculr Norml: mgitude is oe Exmple: X, Y, Z (but do t hve to be!) T T T y x y x y x X, Y, Z is orthoorml bsis. We c describe y 3D poit s lier combitio of these vectors. 5.3 Vectors: Orthoorml Bsis c b v c v b v u c u b u v u v u v u c b (ot ctul formul just wy of thikig bout it) To chge poit from oe coordite system to other, compute the dot product of ech coordite row with ech of the bsis vectors. How do we express y poit s combitio of ew bsis U, V, N, give X, Y, Z? 5.3 Vectors: Orthoorml Bsis

31 You kow too much lier lgebr whe... You look t the log row of milk crtos t Whole Foods --soy, skim,.5% low-ft, % low-ft, % low-ft, d whole-- d thik: "Why so my? Are't soy, skim, d whole bsis?" 3