Fundamentals of Linear Algebra

Transcription

1 -7/8-797 Mchine Lerning for Signl Processing Fundmentls of Liner Alger Administrivi Registrtion: Anone on witlist still? Homewor : Will pper over weeend Liner lger Clss Aug Instructor: Bhish Rj -7/8-797 Aug -7/8-797 Aug Overview Vectors nd mtrices Bsic vector/mtri opertions Vector products Mtri products Vrious mtri tpes Projections Boo Fundmentls of Liner Alger, Gilert Strng Importnt to e ver comfortle with liner lger Appers repetedl in the form of Eigen nlsis, SVD, Fctor nlsis Appers through vrious properties of mtrices tht re used in mchine lerning, prticulrl when pplied to imges nd sound Tod s lecture: Definitions Ver smll suset of ll tht s used Importnt suset, intended to help ou recollect -7/8-797 Aug -7/8-797 Aug Incentive to use liner lger Prett nottion! T A j j i i ij And other things ou cn do Frequenc From Bch s Fugue in Gm Esier intuition Rell convenient geometric interprettions Opertions es to descrie verll Es code trnsltion! for i:n for j:m c(i)c(i)(j)*(i)*(i,j) end end C*A* Rottion Projection Scling Mnipulte Imges Mnipulte Sounds Time Decomposition (MF) -7/8-797 Aug -7/8-797 Aug 6

2 Sclrs, vectors, mtrices, Vectors A sclr is numer,, -, etc A vectoris liner rrngement of collection of sclrs [ ], A mtriais rectngulr rrngement of collection of sclrs A MATLAB snt: [ ], A[ ; ] Vectors usull hold sets of numericl ttriutes,, Z coordintes [,, ] Ernings, losses, suicides [$ $,, ] A loction in Mnhttn [v st] Vectors re either column or row vectors c, r c c [ ], s [ ] A sound cn e vector, series of dil tempertures cn e vector, etc [-v 6st] [v st] [v 8st] -7/8-797 Aug 7-7/8-797 Aug 8 Vectors in the strct Ordered collection of numers Emples: [],[ c d], []![] Order is importnt Tpicll viewed s identifing (the pth from origin to) loction in n -dimensionl spce (,,) (,,) Mtrices Mtrices cn e squre or rectngulr S c d, R c d e f, M Imges cn e mtri, collections of sounds cn e mtri, etc A mtri cn e verticl stcing of row vectors R c d e f Or horiontl rrngement of column vectors R c d e f -7/8-797 Aug 9-7/8-797 Aug Dimensions of mtri The mtri sie is specified the numer of rows nd columns c mtri: rows nd column r mtri: row nd columns S c S mtri R mtri c, r c, R d d e Pcmn 99 mtri [ c] c f v Representing n imge s mtri 6 [ ] Vlues onl; nd re implicit pcmen A 99 mtri Row nd Column position A 879 mtri Triples of, nd vlue A 879 vector Unrveling the mtri ote: All of these cn e recst s the mtri tht forms the imge Representtions nd re equivlent The position is not represented -7/8-797 Aug -7/8-797 Aug

3 Vectors vs Mtrices (,,) A vector is geometric nottion for how to get from (,) to some loction in the spce A mtri is simpl collection of destintions! Properties of mtrices re vergeproperties of the trveller s pth to these destintions Bsic rithmetic opertions Addition nd sutrction Element-wise opertions MATLAB snt: nd - A B -7/8-797 Aug -7/8-797 Aug Vector Opertions Opertions emple (,,) Opertions tell us how to get from ({}) to the result of the vector opertions (,,) (,-,-) (6,,) (,-,-) - (6,,) - [ ] 6 Adding rndom vlues to different representtions of the imge 6 Rndom(,columns(M)) -7/8-797 Aug -7/8-797 Aug 6 Vector norm Mesure of how ig vector is: Represented s [ ] Geometricll the shortest distnce to trvel from the origin to the destintion As the crow flies Assuming Eucliden Geometr MATLAB snt: norm() Length sqrt( ) (,,) [-v 7st] [-6v st] Trnsposition A trnsposed row vector ecomes column (nd vice vers), T c c A trnsposed mtri gets ll its row (or column) vectors trnsposed in order c d e f, T c MATLAB snt: [ ] d e f [ c], T c M, M T Aug -7/ Aug -7/

4 Vector multipliction Multipliction is not element-wise! Dot product, or inner product Vectors must hve the sme numer of elements Row vector times column vector sclr d [ c ] e d e c f f Outer product or vector direct product Column vector times row vector mtri d e f d e f [ ] d e f c c d c e c f MATLAB snt: * Vector dot product in Mnhttn Emple: Coordintes re rds, not ve/st [ 6], [77 ] The dot product of the two vectors reltes to the length of projection How much of the first vector hve we covered following the second one? Must normlie the length of the trget vector T 77 [ 6] 9d 6 [ ] [d 6d] norm 6 norm 9d [77d d] norm 86-7/8-797 Aug 9-7/8-797 Aug Vector dot product Sqrt(energ) frequenc frequenc frequenc [ 9 ] [ 6 ] [ ] Vectors re spectr C E C Energ t discrete set of frequencies Actull 96 is is the inde of the numer in the vector Represents frequenc is is the vlue of the numer in the vector Represents mgnitude Vector dot product Sqrt(energ) frequenc frequenc frequenc [ 9 ] [ 6 ] [ ] How much of C is lso in E C E C How much cn ou fe C pling n E CE / C E ot ver much How much of C is in C? CC / C / C ot d, ou cn fe it To do this, C, E, nd C must e the sme sie -7/8-797 Aug -7/8-797 Aug Vector outer product Multipling vector mtri Generlition of vector multipliction Left multipliction: Dot product of ech vector pir The column vector is the spectrum The row vector is n mplitude modultion The crossproduct is spectrogrm Shows how the energ in ech frequenc vries with time The pttern in ech column is scled version of the spectrum Ech row is scled version of the modultion A B Dimensions must mtch!! o of columns of mtri sie of vector Result inherits the numer of rows from the mtri MATLAB snt: * -7/8-797 Aug -7/8-797 Aug

5 Multipling vector mtri Generlition of vector multipliction Right multipliction: Dot product of ech vector pir A B [ ] [ ] Dimensions must mtch!! o of rows of mtri sie of vector Result inherits the numer of columns from the mtri MATLAB snt: * Multipliction of vector spce mtri 7 6 The mtri rottes nd scles the spce Including its own vectors -7/8-797 Aug -7/8-797 Aug 6 Multipliction of vector spce mtri Mtri Multipliction 7 6 The normls to the row vectors in the mtri ecome the new es is norml to the second row vector Scled the inverse of the length of the first row vector Aug -7/ The -th is corresponds to the norml to the hperplne represented the -,-th row vectors in the mtri An set of K- vectors represent hperplne of dimension K- or less The distnce long the new is equls the length of the projection on the -th row vector Epressed in inverse-lengths of the vector Aug -7/ Mtri Multipliction: Column spce d e c c f d e f So much for spces wht does multipling mtri vector rell do? It mies the column vectors of the mtri using the numers in the vector The column spce of the Mtri is the complete set of ll vectors tht cn e formed miing its columns Mtri Multipliction: Row spce d c f [ ] [ c] [ d e f ] e Left multipliction mies the row vectors of the mtri The row spce of the Mtri is the complete set of ll vectors tht cn e formed miing its rows -7/8-797 Aug 9-7/8-797 Aug

6 6-7/8-797 Mtri multipliction: Miing vectors A phsicl emple The three column vectors of the mtri re the spectr of three notes The multipling column vector is just miing vector The result is sound tht is the miture of the three notes 9 7 Aug -7/8-797 Mtri multipliction: Miing vectors Miing two imges The imges re rrnged s columns position vlue not included The result of the multipliction is rerrnged s n imge 7 Aug -7/8-797 Multipling mtrices Generlition of vector multipliction Outer product of dot products!! Dimensions must mtch!! Columns of first mtri rows of second Result inherits the numer of rows from the first mtri nd the numer of columns from the second mtri MATLAB snt: * A B Aug -7/8-797 Mtri multipliction: nother view Wht does this men? K M M K K K M M B A [ ] [ ] [ ] K M K M K M K K M M The outer product of the first column of A nd the first row of B outer product of the second column of A nd the second row of B Aug -7/8-797 Wh is tht useful? Sounds: Three notes modulted independentl Aug -7/8-797 Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl Aug 6

7 Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl -7/8-797 Aug 7-7/8-797 Aug 8 Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl -7/8-797 Aug 9-7/8-797 Aug Mtri multipliction: Imge trnsition Mtri multipliction: Imge trnsition i i j j Imge fdes out linerl Imge fdes in linerl i i j j Ech column is one imge i 9 i 8 i i 9 i 8 i i 9 i 8 i The columns represent sequence of imges of decresing intensit Imge fdes out linerl -7/8-797 Aug -7/8-797 Aug 7

8 Mtri multipliction: Imge trnsition Mtri multipliction: Imge trnsition i i j j i i j j Imge fdes in linerl Imge fdes out linerl Imge fdes in linerl -7/8-797 Aug -7/8-797 Aug The Identit Mtri Digonl Mtri An identit mtri is squre mtri where All digonl elements re All off-digonl elements re Multipliction n identit mtri does not chnge vectors All off-digonl elements re ero Digonl elements re non-ero Scles the es M flip es -7/8-797 Aug -7/8-797 Aug 6 Digonl mtri to trnsform imges Stretching 6 How? Loction-sed representtion Scling mtri onl scles the is The is nd piel vlue re scled identit ot good w of scling -7/8-797 Aug 7-7/8-797 Aug 8 8

9 9-7/8-797 Stretching Better w D ) ( ewpic EA A Aug 9-7/8-797 Modifing color Scle onl Green P ewpic B G R P Aug -7/8-797 Permuttion Mtri A permuttion mtri simpl rerrnges the es The row entries re is vectors in different order The result is comintion of rottions nd reflections The permuttion mtri effectivel permutes the rrngement of the elements in vector (,,) Z (old ) (old Z) Z (old ) Aug -7/8-797 Permuttion Mtri Reflections nd 9 degree rottions of imges nd ojects P P 6 Aug -7/8-797 Permuttion Mtri Reflections nd 9 degree rottions of imges nd ojects Oject represented s mtri of -Dimensionl position vectors Positions identif ech point on the surfce P P Aug -7/8-797 Rottion Mtri A rottion mtri rottes the vector some ngle q Alterntel viewed, it rottes the es The new es re t n ngle to the old one ' ' cos sin sin cos new R (,) new R (,) (, ) cos sin ' sin cos ' Aug

10 Rotting picture -D Rottion 6 R cos sin ote the representtion: -row mtri Rottion onl pplies on the coordinte rows The vlue does not chnge Wh is pcmn grin? sin cos new Znew Z new degrees of freedom seprte ngles α Wht will the rottion mtri e? -7/8-797 Aug -7/8-797 Aug 6 Mtri Opertions: Properties Projections AB BA AB! BA Wht would we see if the cone to the left were trnsprent if we looed t it from ove the plne shown the grid? orml to the plne Answer: the figure to the right How do we get this? Projection Aug -7/ /8-797 Aug 8 Projection Mtri 9degrees Projection Mtri 9degrees W W W projection W projection Consider n plne specified set of vectors W, W Or mtri [W W ] An vector cn e projected onto this plne The mtri A tht rottes nd scles the vector so tht it ecomes its projection is projection mtri Given set of vectors W, W, which form mtri W [W W ] The projection mtri tht trnsforms n vector to its projection on the plne is P W (W T W) - W T We will visit mtri inversion shortl Mgic n set of vectors from the sme plne tht re epressed s mtri will give ou the sme projection mtri P V (V T V) - V T -7/8-797 Aug 9-7/8-797 Aug 6

11 Projections Projections HOW? Drw n two vectors W nd W tht lie on the plne A two so long s the hve different ngles Compose mtri W [W W] Compose the projection mtri P W (W T W) - W T Multipl ever point on the cone P to get its projection View it I m missing step here wht is it? -7/8-797 Aug 6-7/8-797 Aug 6 Projections Projection mtri properties The projection ctull projects it onto the plne, ut ou re still seeing the plne in D The result of the projection is -D vector P W (W T W) - W T, P*Vector The imge must e rotted till the plne is in the plne of the pper The Z is in this cse will lws e ero nd cn e ignored How will ou rotte it? (rememer ou now W nd W) Aug -7/ The projection of n vector tht is lred on the plne is the vector itself P if is on the plne If the oject is lred on the plne, there is no further projection to e performed The projection of projection is the projection P (P) P Tht is ecuse Pis lred on the plne Projection mtrices re idempotent P P Follows from the ove 6 Perspective An side on Perspective The picture is the equivlent of pinting the viewed scener on glss window Feture: The lines connecting n point in the scener nd its projection on the window merge t common point The ee Aug -7/ Perspective is the result of convergence of the imge to point Convergence cn e to multiple points Top Left: One-point perspective Top Right: Two-point perspective Right: Three-point perspective Aug -7/

12 Centrl Projection,, Representing Perspective, α ' α' Propert of line through origin α' ' ' ' The positions on the window re scled long the line To compute (,) position on the window, we need (distnce of window from ee), nd (,, ) (loction eing projected) Aug -7/ Perspective ws not lws understood Crefull represented perspective cn crete illusions Aug -7/ Projections: A more phsicl mening Projection : n emple with sounds Let W, W W e ses We wnt to eplin our dt in terms of these ses We often cnnot do so But we cn eplin significnt portion of it The portion of the dt tht cn e epressed in terms of our vectors W, W, W, is the projection of the dt on the W W (hper) plne In our previous emple, the dt were ll the points on cone, nd the ses were vectors on the plne The spectrogrm (mtri) of piece of music How much of the ove music ws composed of the ove notes Ie how much cn it e eplined the notes -7/8-797 Aug 69-7/8-797 Aug 7 Projection: one note Projection: one note clened up M M The spectrogrm (mtri) of piece of music The spectrogrm (mtri) of piece of music W W M spectrogrm; W note P W (W T W) - W T Projected Spectrogrm P * M Floored ll mtri vlues elow threshold to ero -7/8-797 Aug 7-7/8-797 Aug 7

13 Projection: multiple notes Projection: multiple notes, clened up M M The spectrogrm (mtri) of piece of music The spectrogrm (mtri) of piece of music W W P W (W T W) - W T Projected Spectrogrm P * M P W (W T W) - W T Projected Spectrogrm P * M -7/8-797 Aug 7-7/8-797 Aug 7 Projection nd Lest Squres Projection ctull computes lest squred error estimte For ech vector V in the music spectrogrm mtri Approimtion: V ppro *note *note c*note Error vector E V V ppro Squred error energ for V e(v) norm(e) V ppro note note note c Totl error sum over ll V { e(v) } Σ V e(v) Projection computes V ppro for ll vectors such tht Totl error is minimied It does not give ou,, c Though Tht needs different opertion the inverse / pseudo inverse Aug -7/