-7/8-797 Mchine Lerning for Signl rocessing Fundmentls of Liner Alger Administrivi Registrtion: Anone on witlist still? Homework : Will e hnded out with clss Liner lger Clss - Sep Instructor: Bhiksh Rj Overview Vectors nd mtrices Bsic vector/mtri opertions Vector products Mtri products Vrious mtri tpes Mtri inversion Mtri interprettion Eigennlsis Singulr vlue decomposition Book Fundmentls of Liner Alger, Gilert Strng Importnt to e ver comfortle with liner lger Appers repetedl in the form of Eigen nlsis, SVD, Fctor nlsis Appers through h vrious properties of mtrices tht t re used in mchine lerning, prticulrl when pplied to imges nd sound Tod s lecture: Definitions Ver smll suset of ll tht s used Importnt suset, intended to help ou recollect Incentive to use liner lger rett nottion! T A j j i i ij And other things ou cn do Frequenc From Bch s Fugue in Gm Esier intuition Rell convenient geometric interprettions Opertions es to descrie verll Es code trnsltion! for i=:n for j=:m c(i)=c(i)+(j)*(i)*(i,j) end end C=*A* Rottion + rojection + Scling Mnipulte Imges Mnipulte Sounds Time Decomposition (MF)
Sclrs, vectors, mtrices, A sclr is numer =, =, = -, etc A vector is liner rrngement of collection of sclrs, A mtri A is rectngulr rrngement of collection of vectors A MATLAB snt: =[ ], A=[ ; ] 7 Vector/Mtri tpes nd shpes Vectors re either column or row vectors c, r c c, s A sound cn e vector, series of dil tempertures cn e vector, etc Mtrices cn e squre or rectngulr S c d, R c d e f, M Imges cn e mtri, collections of sounds cn e mtri, etc 8 Dimensions of mtri The mtri sie is specified the numer of rows nd columns c, r c c c = mtri: rows nd column r = mtri: ti row nd d columns S c, R d d c e f S = mtri R = mtri cmn = 99 mtri v Representing n imge s mtri Vlues onl; nd re implicit pcmen A 99 mtri Row nd Column = position A 879 mtri Triples of, nd vlue A 879 vector Unrveling the mtri ote: All of these cn e recst s the mtri tht forms the imge Representtions nd re equivlent The position is not represented 9 Emple of vector Vectors usull hold sets of numericl ttriutes,, vlue [,, ] Ernings, losses, suicides [$ $ ] Etc Consider reltive Mnhttn vector rovides reltive position giving numer of venues nd streets to cross, eg [v st] [-v st] [v st] [v 8st] Vectors Ordered collection of numers Emples: [ ], [ c d], [ ]!= [ ] Order is importnt Tpicll viewed s identifing (the pth from origin to) loction in n -dimensionl spce (,,) (,,)
Vectors vs Mtrices (,,) A vector is geometric nottion for how to get from (,) to some loction in the spce A mtri is simpl collection of destintions! roperties of mtrices re verge properties of the trveller s pth to these destintions Bsic rithmetic opertions Addition nd sutrction Element-wise opertions A B MATLAB snt: + nd - Vector Opertions Opertions emple (,,) (,-,-) - - (,,) Opertions tell us how to get from ({}) to the result of the vector opertions (,,) + (,-,-) = (,,) Adding rndom vlues to different representtions of the imge + + Rndom(,columns(M)) Vector norm Mesure of how ig vector is: otted s In Mnhttn vectors mesure of distnce 7 7 MATLAB snt: norm() [-v 7st] [-v st] 7 Vector orm Length = sqrt( + + ) (,,) Geometricll the shortest distnce to trvel from the origin to the destintion As the crow flies Assuming Eucliden Geometr 8
Trnsposition A trnsposed row vector ecomes column (nd vice vers), c c c, T c A trnsposed mtri gets ll its row (or column) vectors trnsposed in order c d e f, T c MATLAB snt: d e f M, M T 9 Vector multipliction Multipliction is not element-wise! Dot product, or inner product Vectors must hve the sme numer of elements Row vector times column vector = sclr d c e d e c f f f Cross product, outer product or vector direct product Column vector times row vector = mtri d e f d e f d e f c c d c e c f MATLAB snt: * Vector dot product in Mnhttn Multipling the rd vectors Insted of venue/street we ll use rds = [ ], = [77 ] The dot product of the two vectors reltes to the length of projection How much of the first vector hve we covered following the second one? The nswer comes ck s unit of the first vector so we divide its length 77 T 9d [d d] norm norm 9d [77d d] norm 8 Vector dot product Sqrt(energ) D S D frequenc frequenc frequenc 9 Vectors re spectr Energ t discrete set of frequencies Actull 9 is is the inde of the numer in the vector Represents frequenc is is the vlue of the numer in the vector Represents mgnitude Vector dot product Sqrt(energ) D S D Vector cross product frequenc frequenc frequenc 9 How much of D is lso in S How much cn ou fke D pling n S DS / D S = ot ver much How much of D is in D? DD / D / D = ot d, ou cn fke it To do this, D, S, nd D must e the sme sie The column vector is the spectrum The row vector is n mplitude modultion The crossproduct is spectrogrm Shows how the energ in ech frequenc vries with time The pttern in ech column is scled version of the spectrum Ech row is scled version of the modultion
Mtri multipliction Generlition of vector multipliction Dot product of ech vector pir A B Dimensions must mtch!! Columns of first mtri = rows of second Result inherits the numer of rows from the first mtri nd the numer of columns from the second mtri MATLAB snt: * Multipling Vector Mtri (,:) 9 (,:) 8 9 8 9 9 Multipliction of vector mtri epresses the vector in terms of projections of on the row vectors of the mtri It scles nd rottes the vector Alterntel viewed, it scles nd rottes the spce the underling plne Mtri Multipliction Mtri Multipliction 7 The mtri rottes nd scles the spce Including its own vectors 7 The normls to the row vectors in the mtri ecome the new es is = norml to the second row vector Scled the inverse of the length of the first row vector 8 Mtri Multipliction is projection The k-th is scorresponds dstot the norml tot the hperplne pep e represented the k-,k+-th row vectors in the mtri An set of K- vectors represent hperplne of dimension K- or less The distnce long the new is equls the length of the projection on the k-th row vector Epressed in inverse-lengths of the vector Mtri Multipliction: Column spce d e c c f d e f So much for spces wht does multipling mtri vector rell do? It mies the column vectors of the mtri using the numers in the vector The column spce of the Mtri is the complete set of ll vectors tht cn e formed miing its columns 9
Mtri Multipliction: Row spce d e c f c d e f Left multipliction mies the row vectors of the mtri The row spce of the Mtri is the complete set of ll vectors tht cn e formed miing its rows Mtri multipliction: Miing vectors 9 7 = A phsicl emple The three column vectors of the mtri re the spectr of three notes The multipling column vector is just miing vector The result is sound tht is the miture of the three notes Mtri multipliction: Miing vectors 7 Miing two imges The imges re rrnged s columns position vlue not included The result of the multipliction is rerrnged s n imge M Mtri multipliction: nother view A B M M Wht does this men? M K K k k k Mk k k k k kkk kk Mk K K K K M M M K The outer product of the first column of A nd the first row of B + outer product of the second column of A nd the second row of B + Wh is tht useful? Mtri multipliction: Miing modulted spectr 9 Sounds: Three notes modulted independentl 7 7 9 7 7 7 8 8 9 9 9 9 9 Sounds: Three notes modulted independentl 7 7 9 7 7 7 8 8 9 9 9 9
Mtri multipliction: Miing modulted spectr 7 7 9 7 7 8 9 9 9 Sounds: Three notes modulted independentl Mtri multipliction: Miing modulted spectr 9 7 7 9 7 7 8 9 9 Sounds: Three notes modulted independentl 7 8 Mtri multipliction: Miing modulted spectr 9 7 7 9 7 7 8 9 9 Sounds: Three notes modulted independentl Mtri multipliction: Miing modulted spectr Sounds: Three notes modulted independentl 9 Mtri multipliction: Imge trnsition Mtri multipliction: Imge trnsition i j i j Imge fdes out linerl Imge fdes in linerl 9 8 7 7 8 9 i j i j 9 8 7 7 8 9 i 9i 8i i 9 i 8 i i 9 i 8 i Ech column is one imge The columns represent sequence of imges of decresing intensit Imge fdes out linerl 7
Mtri multipliction: Imge trnsition Mtri multipliction: Imge trnsition i j i j Imge fdes in linerl 9 8 7 7 8 9 i j i j Imge fdes out linerl Imge fdes in linerl 9 8 7 7 8 9 The Identit Mtri Digonl Mtri An identit mtri is squre mtri where All digonl elements re All off-digonl elements re Multipliction n identit mtri does not chnge vectors All off-digonl elements re ero Digonl elements re non-ero Scles the es M flip es Digonl mtri to trnsform imges Stretching How? Loction-sed representtion Scling mtri onl scles the is The is nd piel vlue re scled identit ot good w of scling 7 8 8
9 Stretching D = -7/8-797 Better w ) ( ewpic EA A Sep 9 Modifing color B G R -7/8-797 Scle onl Green ewpic Sep ermuttion Mtri (,,) Z (old ) (old Z) Z (old ) -7/8-797 A permuttion mtri simpl rerrnges the es The row entries re is vectors in different order The result is comintion of rottions nd reflections The permuttion mtri effectivel permutes the rrngement of the elements in vector ( ) Sep ermuttion Mtri -7/8-797 Reflections nd 9 degree rottions of imges nd ojects Sep ermuttion Mtri -7/8-797 Reflections nd 9 degree rottions of imges nd ojects Oject represented s mtri of -Dimensionl position vectors ositions identif ech point on the surfce Sep Rottion Mtri ' ' cos sin sin cos new R (,) new R (,) (, ) cos sin ' sin cos ' -7/8-797 A rottion mtri rottes the vector some ngle Alterntel viewed, it rottes the es The new es re t n ngle to the old one Sep
Rotting picture cos R sin sin cos 8 7 8 ote the representtion: -row mtri Rottion onl pplies on the coordinte rows The vlue does not chnge Wh is pcmn grin? -D Rottion new new Znew Z degrees of freedom seprte ngles Wht will the rottion mtri e? rojections rojection Mtri 9degrees W W projection Wht would we see if the cone to the left were trnsprent if we looked t it long the norml to the plne The plne goes through the origin Answer: the figure to the right How do we get this? rojection Consider n plne specified set of vectors W, W Or mtri [W W ] An vector cn e projected onto this plne The mtri A tht rottes nd scles the vector so tht it ecomes its projection is projection mtri 7 8 rojection Mtri 9degrees rojections W W projection Given set of vectors W, W, which form mtri W = [W W ] The projection mtri tht trnsforms n vector to its projection on the plne is = W (W T W) - W T We will visit mtri inversion shortl Mgic n set of vectors from the sme plne tht re epressed s mtri will give ou the sme projection mtri = V (V T V) - V T 9 HOW?
rojections rojections Drw n two vectors W nd W tht lie on the plne A two so long s the hve different ngles Compose mtri W = [W W] Compose the projection mtri = W (W T W) - W T Multipl ever point on the cone to get its projection View it I m missing step here wht is it? The projection ctull projects it onto the plne, ut ou re still seeing the plne in D The result of the projection is -D vector = W (W T W) - W T =, *Vector = The imge must e rotted till the plne is in the plne of the pper The Z is in this cse will lws e ero nd cn e ignored How will ou rotte it? (rememer ou know W nd W) rojection mtri properties The projection of n vector tht is lred on the plne is the vector itself = if is on the plne If the oject is lred on the plne, there is no further projection to e performed The projection of projection is the projection () = Tht is ecuse is lred on the plne rojection mtrices re idempotent = Sep Follows from the ove -7/8-797