Linear Algebra Concepts

Transcription

1 Ler Algebr Cocepts Nuo Vscocelos (Ke Kreutz-Delgdo) UCSD

2 Vector spces Defto: vector spce s set H where ddto d sclr multplcto re defed d stsf: ) +( + ) = (+ )+ 5) H 2) + = + H 6) = 3) H, + = 7) ( ) = ( ) 4) H, - + = 8) (+ ) = + ( sclr;,, H ) 9) ( + ) = + the cocl emple s R d wth stdrd vector ddto d sclr multplcto e d + e d e e e 2 e 2

3 Vector spces But there re much more terestg emples E.g., the spce of fuctos f:x R wth (f + g)() = f() + g() ( f)() = f() R d s vector spce of fte dmeso, e.g. f = ( f,..., f d ) T Whe d goes to ft we hve fucto f = f (t ) The spce of ll fuctos s fte dmesol vector spce

4 Dt Vector Spces I ths course we wll tlk lot bout dt Dt wll lws be represeted vector spce: emple s rell just pot ( dtpot ) o such spce from bove we kow how to perform bsc opertos o dtpots ths s ce, becuse dtpots c be qute bstrct e.g. mges: mge s fucto o the mge ple t ssgs color f(,) to ech mge locto (,) the spce of mges s vector spce (ote: ssumes tht mges c be egtve) ths mge s pot

5 Imges Becuse of ths we c mpulte mges b mpultg ther vector represettos E.g., Suppose oe wts to morph (,) to b(,): Oe w to do ths s v the pth log the le from to b. c( ) = + (b-) = (- ) + b for = we hve for = we hve b for (,) we hve pot o the le betwee d b To morph mges we c smpl ppl ths rule to ther vector represettos! b b- (b-)

6 Imges Whe we mke c(,) = (- ) (,) + we get mge morphg : b(,) = =.2 =.4 b b- (b-) =.6 =.8 = The pot s tht ths s possble becuse the mges re pots vector spce.

7 Imges Imges re usull represeted s pots R d Smple (dscretze) mge o fte grd to get rr of pels (,) (,j) Imges re lws stored lke ths o dgtl computers stck ll the rows to vector. E.g. 3 3 mge s coverted to 9 vector s follows: I geerl m mge vector s trsformed to m vector Note tht ths s et other vector spce The pot s tht there re geerll multple dfferet, but somorphc, vector spces whch the dt c be represeted

8 Tet Aother commo tpe of dt s tet Documets re represeted b word couts: ssocte couter wth ech word slde wdow through the tet wheever the word occurs cremet ts couter Ths s the w serch eges represet web pges

9 Tet E.g. word couts for three documets cert corpus (ol 2 words show for clrt) Note tht: Ech documet s d = 2 dmesol vector If I dd two word cout vectors (documets), I get ew word cout vector (documet) If I multpl word cout vector (documet) b sclr, I get word cout vector Note: oce g we ssume word couts could be egtve (to mke ths hppe we c smpl subtrct the verge vlue) Ths mes: We re oce g vector spce (postve subset of R d ) A documet s pot ths spce

10 Bler forms Ier product vector spces re populr becuse the llow us to mesure dstces betwee dt pots We wll see tht ths s crucl for clssfcto The m tool for ths s the er product ( dot-product ). We c defe the dot-product usg the oto of bler form. Defto: bler form o rel vector spce H s bler mppg Q: H H R (, ) Q(, ) B-ler mes tht,, ) Q[( + ), ] = Q(, ) + Q(, ) ) Q[,( + )] = Q(,) + Q(, ) H

11 Ier Products Defto: er product o rel vector spce H s bler form <.,. >: H H R (, ) <, > such tht ) <,>, H ) <,> = f d ol f = ) <,> = <,> for ll d The postve-defteess codtos ) d ) mke the er product turl mesure of smlrt othg c be more smlr to th tself Ths becomes more precse wth troducto of orm

12 Ier Products d Norms A er product duces orm v 2 = <,> B defto, orm must obe the followg propertes Postve-defteess:, & ff Homogeet: = Trgle Ieqult: + + A orm defes correspodg metrc d(,) = - whch s mesure of the dstce betwee d Alws remember tht the duced orm chges wth dfferet choce of er product!

13 Ier Product Bck to our emples: I R d the stdrd er product s Whch leds to the stdrd Euclde orm R d The dstce betwee two vectors s the stdrd Euclde dstce R d d T, d T 2 d T d 2 ) ( ) ( ) ( ), (

14 Ier Products d Norms Note, e.g., tht ths mmedtel gves mesure of smlrt betwee web pges compute word cout vector from pge, for ll dstce betwee pge d pge j c be smpl defed s: T d(, j ) j ( j ) ( j ) Ths llows us to fd, the web, the most smlr pge to gve pge j. I fct, ths s ver close to the mesure of smlrt used b most serch eges! Wht bout mges d other cotuous vlued sgls?

15 Ier Products o Fucto Spces Recll tht the spce of fuctos s fte dmesol vector spce The stdrd er product s the turl eteso of tht R d (just replce summtos b tegrls) f ( ), g( ) f ( ) g( ) d The orm becomes the eerg of the fucto 2 2 f ( ) f ( ) d The dstce betwee fuctos the eerg of the dfferece betwee them 2 2 d( f ( ), g( )) f ( ) g( ) [ f ( ) g( )] d

16 Bss Vectors We kow how to mesure dstces vector spce Aother terestg propert s tht we c full chrcterze the vector spce b oe of ts bses A set of vectors,, k s bss of vector spce H f d ol f (ff) the re lerl depedet c c, d the sp H : for v H, v c be wrtte s v c These two codtos me tht uquel represeted ths form. v H c be

17 Bss Note tht B mkg the vectors the colums of mtr X, these two codtos c be compctl wrtte s Codto. The vectors re ler depedet: Xc c Codto 2. The vectors sp H v, c such tht v Xc Also, ll bses of H hve the sme umber of vectors, whch s clled the dmeso of H Ths s vld for vector spce!

18 Bss emple A bss of the vector spce of mges of fces The fgure ol shows the frst 6 bss vectors but there ctull more These vectors re orthoorml

19 Orthogolt Two vectors re orthogol ff ther er product s zero e.g. 2 2 s s( )cos( ) d 2 the spce of fuctos defed o [,2 ], cos() d s() re orthogol Two subspces V d W re orthogol, V W, f ever vector V s orthogol to ever vector W set of vectors,, k s clled orthogol f ll prs of vectors re orthogol. orthoorml f ll vectors lso hve ut orm., j,, f f j j 2

20 Mtr m mtr represets ler opertor tht mps vector from the dom X = R to vector the codom Y = R m E.g. the equto = A seds R to R m ccordg to X e m m e m m Y A e e e 2 ote tht there s othg mgcl bout ths, t follows rther mechcll from the defto of mtr-vector multplcto

21 Mtr-Vector Multplcto I Cosder = A,.e. = j= j j We c thk of ths s, =,,m j j j (m rows) where ( ) mes the th row of A. Hece the th compoet of s the er product of ( ) d. s the projecto of o the subspce (of the dom spce) sped b the rows of A e X A s cto X m - m - X e 2 e

22 Mtr-Vector Multplcto II But there s more. Let = A,.e. = j= j j, ow be wrtte s where wth bove d below mes the th colum of A. hece m j j j s the th compoet of the subspce (of the co-dom) sped b the colums of A s ler combto of the colums of A m m e X A mps from X to Y Y e 2 e

23 Mtr-Vector Multplcto two ltertve (dul) pctures of = A: = coordtes of row spce of A (The X = R vewpot) Dom X = R m - m - Dom X = R vewpot e ( m rows) A e 2 e Codom Y = R m vewpot = coordtes of colum spce of A (Y = R m vewpot)

24 A cool trck the mtr multplcto formul C AB c j k k b kj lso pples to block mtrces whe these re defed properl for emple, f A,B,C,D,E,F,G,H re mtrces, A B E F AE BG AF BH C D G H CE DG CF DH ol but mportt cvet: the szes of A,B,C,D,E,F,G,H hve to be such tht the termedte opertos mke sese! (the hve to be coforml )

25 Mtr-Vector Multplcto Ths mkes t es to derve the two ltertve pctures The row spce pcture (or vewpot): s just lke sclr multplcto, wth blocks ( -) d The colum spce pcture (or vewpot): s just er product, wth (sclr) blocks d the colum blocks of A. m m

26 Mtr-Vector Multplcto two ltertve (dul) pctures of = A: = coordtes of row spce of A (The X = R vewpot) Dom X = R m - m - Dom X = R vewpot e ( m rows) A e 2 e Codom Y = R m vewpot = coordtes of colum spce of A (Y = R m vewpot)

27 Squre mtrces ths cse m = d the row d colum subspces re both equl to (copes of) R - - e A e e 2 2 2

28 Orthogol mtrces A mtr s clled orthogol f t s squre d hs orthoorml colums. Importt propertes: ) The verse of orthogol mtr s ts trspose ths c be esl show wth the block mtr trck. (Also see lter.) T T A A j 2) A proper (det(a) = ) orthogol mtr s rotto mtr ths follows from the fct tht t does ot chge the orms ( szes ) of the vectors o whch t opertes, d does ot duce reflecto. 2 T T T T 2 A ( A) ( A) A A,

29 Rotto mtrces The combto of. opertor terpretto 2. block mtr trck s useful m stutos Poll: Wht s the mtr R tht rottes the ple R 2 b degrees? e 2 e

30 Rotto mtrces The ke s to cosder how the mtr opertes o the vectors e of the cocl bss ote tht R seds e to e e 2 e' r r 2 r r 2 22 s usg the colum spce pcture e' r 2 r2 r22 r from whch we hve the frst colum of the mtr R e' r r 2 22 cos s r r 2 22 r r 2 cos e

31 Rotto Mtrces d we do the sme for e 2 R seds e 2 to e 2 e' r r r2 r22 r2 r22 r r r r 2 22 from whch e 2 cos R e' e' 2 cos s s cos s check cos e R T cos s cos s R I s cos s cos -s

32 Alss/sthess oe terestg cse s tht of mtrces wth orthogol colums ote tht, ths cse, the colums of A re bss of the colum spce of A bss of the row spce of A T ths leds to terestg terpretto of the two pctures cosder the projecto of to the row spce of A T = A T due to orthoormlt, c the be stheszed b usg the colum spce pcture = A

33 Alss/sthess ote tht ths s our most commo use of bss let the colums of A be the bss vectors the operto = A T projects the vector to the bss, e.g. The vector c the be recostructed b computg = A, e.g. Q: s the stheszed lws equl to? ' ' ' ths s clled the cocl bss of R

34 Projectos A: ot ecessrl! Recll = A T d = A = f d ol f AA T = I! ths mes tht A hs to be orthoorml. wht hppes whe ths s ot the cse? we get the projecto of o the colum spce of A e.g. let the A d ' e 3 e e 2 colum spce of A = row spce of A T

35 Null Spce of Mtr Wht hppes to the prt tht s lost? Ths s the ull spce of A T N A T T A e 3 e e 2 colum spce of A = row spce of A T ull spce of A T I the emple, ths s comprsed of ll vectors of the tpe sce A T FACT: N(A) s lws orthogol to the row spce of A: s the ull spce ff t s orthogol to ll rows of A For the prevous emple ths mes tht N(A T ) s orthogol to the colum spce of A

36 Orthoorml mtrces Q: wh s the orthoorml cse specl? becuse here there s o ull spce of A T recll tht for ll N(A T ) A T A the ol vector the ull spce s ths mkes sese: A hs orthoorml colums, e.g. these sp ll of R there s o etr room for orthogol spce the ull spce of A T hs to be empt the projecto to row spce of A T (=colum spce of A) s the vector tself ths cse, we s tht the mtr hs full rk A

37 The Four Fudmetl Subspces These est for mtr: Colum Spce: spce sped b the colums Row Spce: spce sped b the rows Nullspce: spce of vectors orthogol to ll rows (lso kow s the orthogol complemet of the row spce) Left Nullspce: spce of vectors orthogol to ll colums (lso kow s the orthogol complemet of the colum spce) You c thk of these the followg w Row d Nullspce chrcterze the dom spce (puts) Colum d Left Nullspce chrcterze the codom spce (outputs)

38 Dom vewpot Dom X = R = coordtes of row spce of A Row spce: spce of useful puts, whch A mps to o-zero output Null spce: spce of useless puts, mpped to zero Operto of mtr o ts dom X = R ( m rows) N( A) A e Null spce m - m - A e - - e 2 Q: wht s the ull spce of low-pss flter?

39 Codom vewpot Codom Y = R m = coordtes of colum spce of A Colum spce: spce of possble outputs, whch A c rech Left Null spce: spce of mpossble outputs, cot be reched Operto of mtr o ts codom Y = R m T L( A) A e A Left Null spce e e 2 Q: wht s the colum spce of low-pss flter?

40 The Four Fudmetl Subspces Assume Dom of A = Codom of A. The: Specl Cse I: Squre Smmetrc Mtrces (A = A T ): Colum Spce s equl to the Row Spce Nullspce s equl to the Left Nullspce, d s therefore orthogol to the Colum Spce Specl Cse II: Orthogol Mtrces (A T A = AA T = I) Colum Spce = Row Spce = R Nullspce = Left Nullspce = {} = the Trvl Subspce

41 Ler sstems s mtrces A ler d tme vrt sstem of mpulse respose h[] respods to sgl [] wth output ths s the covoluto of [] wth h[] The sstem s chrcterzed b mtr ote tht [ ] [ k] g [ k], wth g [ k] h[ k] k [ ] [ k] h[ k] the output s the projecto of the put o the spce sped b the fuctos g [k] k [] [2] [ ] g g g 2 h[] h[] h[ ] h[ ] h[] h[ 2] h[ h[ ( )] ( 2)] h[] [] [2] [ ]

42 Ler sstems s mtrces the mtr h[] h[ ] h[ ( )] A h[] h[] h[ ( 2)] h[ ] h[ 2] h[] chrcterzes the respose of the sstem to put the sstem projects the put to shfted d flpped copes of ts mpulse respose h[] ote tht the colum spce s the spce sped b the vectors h[], h[-], ths s the reso wh the mpulse respose determes the output of the sstem e.g. low-pss flter s flter such tht the colum spce of A ol cots low-pss low pss sgls e.g. f h[] s the delt fucto, A s the dett

43