Mchine Lerning for Signl Processing Fundmentls of Liner Alger Clss 6 Sep 6 Instructor: Bhiksh Rj -755/8-797
Overview Vectors nd mtrices Bsic vector/mtrix opertions Vrious mtrix types Projections -755/8-797
Book Fundmentls of Liner Alger, Gilert Strng Importnt to e very comfortle with liner lger Appers repetedly in the form of Eigen nlysis, SVD, Fctor nlysis Appers through vrious properties of mtrices tht re used in mchine lerning Often used in the processing of dt of vrious kinds Will use sound nd imges s exmples Tody s lecture: Definitions Very smll suset of ll tht s used Importnt suset, intended to help you recollect -755/8-797 3
Incentive to use liner lger Simplified nottion! x T Ay Esier intuition Relly convenient geometric interprettions Esy code trnsltion! j y j i x i ij for i=:n for j=:m c(i)=c(i)+y(j)*x(i)*(i,j) end end C=x*A*y -755/8-797 4
And other things you cn do From Bch s Fugue in Gm Frequency Rottion + Projection + Scling + Perspective Time Decomposition (NMF) Mnipulte Dt Extrct informtion from dt Represent dt Etc -755/8-797 5
Sclrs, vectors, mtrices, A sclr is numer =, = 34, = -, etc A vector is liner rrngement of collection of sclrs 3, 34 3 A mtrix A is rectngulr rrngement of collection of sclrs A 3 6 5-755/8-797 6
Vectors in the strct Ordered collection of numers Exmples: [3 4 5], [ c d], [3 4 5]!= [4 3 5] Order is importnt Typiclly viewed s identifying (the pth from origin to) loction in n N-dimensionl spce (3,4,5) 5 (4,3,5) z 3 y 4 x -755/8-797 7
Vectors usully hold sets of numericl ttriutes Vectors in relity X, Y, Z coordintes [,, ] [height(cm) weight(kg)] [75 7] A loction in Mnhttn [3v 33st] A series of dily tempertures Smples in n udio signl Etc [-5v 6st] [v 4st] [v 8st] -755/8-797 8
Vector norm Mesure of how long vector is: Represented s x Length = sqrt(3 + 4 + 5 ) (3,4,5) 5 Geometriclly the shortest distnce to trvel from the origin to the destintion As the crow flies Assuming Eucliden Geometry 3 [-v 7st] [-6v st] -755/8-797 9 4
Vector Opertions: Multipliction y sclr (75,, 5) (3,4,5) Multipliction y sclr stretches the vector Vector multipliction y sclr: ech component multiplied y sclr 5 x (3,4,5) = (75,, 5) Note: s result, vector norm is lso multiplied y the sclr 5 x (3,4,5) = 5x (3, 4, 5) 3-755/8-797
Vector Opertions: Addition (3,4,5) 3 5 (3,-,-3) - -3 4 (6,,) Vector ddition: individul components dd (3,4,5) + (3,-,-3) = (6,,) 3 Only pplicle if oth vectors re the sme size -755/8-797
An introduction to spces Conventionl notion of spce : geometric construct of certin numer of dimensions Eg the 3-D spce tht this room nd every oject in it lives in -755/8-797
A vector spce A vector spce is n infinitely lrge set of vectors the following properties The set includes the zero vector (of ll zeros) The set is closed under ddition If X nd Y re in the set, X + Y is lso in the set for ny two sclrs nd For every X in the set, the set lso includes the dditive inverse Y = -X, such tht X + Y = -755/8-797 3
Additionl Properties Additionl requirements: Sclr multiplictive identity element exists: X = X Addition is ssocitive: X + Y = Y + X Addition is commuttive: (X+Y)+Z = X+(Y+Z) Sclr multipliction is commuttive: (X) = () X Sclr multipliction is distriutive: (+)X = X + X (X+Y) = X + Y -755/8-797 4
Exmple of vector spce Set of ll three-component column vectors Note we used the term three-component, rther thn threedimensionl The set includes the zero vector For every X in the set α R, every X is in the set For every X, Y in the set, X + Y is in the set -X is in the set Etc -755/8-797 5
Exmple: function spce -755/8-797 6
Dimension of spce Every element in the spce cn e composed of liner comintions of some other elements in the spce For ny X in S we cn write X = Y + Y + cy 3 for some other Y, Y, Y 3 in S Trivil to prove -755/8-797 7
Dimension of spce Wht is the smllest suset of elements tht cn compose the entire set? There my e multiple such sets The elements in this set re clled ses The set is sis set The numer of elements in the set is the dimensionlity of the spce -755/8-797 8
Dimensions: Exmple Wht is the dimensionlity of this vector spce -755/8-797 9
Dimensions: Exmple Wht is the dimensionlity of this vector spce? First confirm this is proper vector spce Note: ll elements in Z re lso in S (slide 9) Z is suspce of S -755/8-797
Dimensions: Exmple Wht is the dimensionlity of this spce? -755/8-797
Moving on -755/8-797
Interpreting Mtrices Two interprettions of mtrix As trnsform tht modifies vectors nd vector spces As continer for dt (vectors) In the next two clsses we ll focus on the first view But we will mostly consider the second view for ML lgorithms in the our discussions of signl representtions -755/8-797 3 d c o n m l k j i h g f e d c
Interpreting Mtrices s collections of vectors A mtrix cn e verticl stcking of row vectors R d e c f The spce of ll vectors tht cn e composed from the rows of the mtrix is the row spce of the mtrix Or horizontl rrngement of column vectors R d e c f The spce of ll vectors tht cn e composed from the columns of the mtrix is the column spce of the mtrix -755/8-797 4
Dimensions of mtrix The mtrix size is specified y the numer of rows nd columns c = 3x mtrix: 3 rows nd column r = x3 mtrix: row nd 3 columns S = x mtrix R = x 3 mtrix Pcmn = 3 x 399 mtrix -755/8-797 5 c c r c, f e d c d c R S,
Representing n imge s mtrix Y X v 5 6 Vlues only; X nd Y re implicit 3 pcmen A 3 x 399 mtrix Row nd Column = position A 3 x 879 mtrix Triples of x,y nd vlue A x 879 vector Unrveling the mtrix Note: All of these cn e recst s the mtrix tht forms the imge Representtions nd 4 re equivlent The position is not represented -755/8-797 6
Bsic rithmetic opertions Addition nd sutrction (vectors nd mtrices) Element-wise opertions 3 3 3 3 3 3 3 3 A B -755/8-797 7
Vector/Mtrix Trnsposition A trnsposed row vector ecomes column (nd vice vers) A trnsposed mtrix gets ll its row (or column) vectors trnsposed in order x X c d e f, XT, x T c c c d e f y c, y T M, M T c -755/8-797 8
Vector multipliction Multipliction y sclr d d c d d dc d d c cd Dot product, or inner product Vectors must hve the sme numer of elements Row vector times column vector = sclr d c e d e c f f Outer product or vector direct product Column vector times row vector = mtrix d e f c d e f d e f c d c e c f -755/8-797 9
Vector dot product Exmple: Coordintes re yrds, not ve/st = [ 6], = [77 3] [yd 6yd] norm 6 The dot product of the two vectors reltes to the length of projection How much of the first vector hve we covered y following the second one? Must normlize y the length of the trget vector T 77 6 3 393yd 6 norm 393yd [77yd 3yd] norm 86-755/8-797 3
Sqrt(energy) Vector dot product C E C frequency frequency Vectors re spectr Energy t discrete set of frequencies Actully x 496 X xis is the index of the numer in the vector Represents frequency Y xis is the vlue of the numer in the vector Represents mgnitude frequency 9 54 3 4 6 4 3 3-755/8-797 3
Sqrt(energy) Vector dot product C E C frequency frequency How much of C is lso in E How much cn you fke C y plying n E CE / C E = Not very much How much of C is in C? CC / C / C = 5 Not d, you cn fke it To do this, C, E, nd C must e the sme size frequency 9 54 3 4 6 4 3 3-755/8-797 3
Vector outer product The column vector is the spectrum The row vector is n mplitude modultion The outer product is spectrogrm Shows how the energy in ech frequency vries with time The pttern in ech column is scled version of the spectrum Ech row is scled version of the modultion -755/8-797 33
Multiplying mtrix y sclr Multiplying mtrix y sclr multiplies every element of the mtrix A 3 3 3 3 33 4 4 34 3 3 3 3 33 4 4 34 Note: A = A -755/8-797 34
Multiplying vector y mtrix Multiplying vector y mtrix trnsforms the vector A B 3 3 3 3 33 4 4 34 3 4 3 3 3 3 33 3 3 3 Dimensions must mtch!! No of columns of mtrix = size of vector Result inherits the numer of rows from the mtrix -755/8-797 35
Multiplying vector y mtrix Generliztion of vector scling Left multipliction: Dot product of ech vector pir Dimensions must mtch!! No of columns of mtrix = size of vector Result inherits the numer of rows from the mtrix -755/8-797 36 B A cd d d d c
Multiplying vector y mtrix Generliztion of vector multipliction Right multipliction: Dot product of ech vector pir A B d c d d dc Dimensions must mtch!! No of columns of mtrix = size of vector Result inherits the numer of rows from the mtrix -755/8-797 37
Mtrix Multipliction: Column spce Right multipliction y mtrix trnsform row spce vector to column spce vector It mixes the column vectors of the mtrix using the numers in the vector The column spce of the Mtrix is the complete set of ll vectors tht cn e formed y mixing its columns -755/8-797 38 f c z e y d x z y x f e d c
Mtrix Multipliction: Row spce d x y x c yd e f e c f Left multipliction mixes the row vectors of the mtrix Converts vector in the column spce to one in the row spce The row spce of the Mtrix is the complete set of ll vectors tht cn e formed y mixing its rows -755/8-797 39
Multipliction of vector spce y mtrix Y 3 3 7 6 Row spce Column spce The mtrix rottes nd scles the spce Including its own vectors -755/8-797 4
Multipliction of vector spce y mtrix 3 Y 3 7 6 The normls to the row vectors in the mtrix ecome the new xes X xis = norml to the second row vector Scled y the inverse of the length of the first row vector -755/8-797 4
Mtrix Multipliction The k-th xis corresponds to the norml to the hyperplne represented y the k-,k+n-th row vectors in the mtrix Any set of K- vectors represent hyperplne of dimension K- or less The distnce long the new xis equls the length of the projection on the k-th row vector Expressed in inverse-lengths of the vector c d e f g h i -755/8-797 4
Mtrix multipliction: Mixing vectors A physicl exmple The three column vectors of the mtrix X re the spectr of three notes The multiplying column vector Y is just mixing vector The result is sound tht is the mixture of the three notes -755/8-797 43 4 9 3 X Y 7 =
Mtrix multipliction: Mixing vectors x x x 4 x 5 75 x 4 x Mixing two imges The imges re rrnged s columns position vlue not included The result of the multipliction is rerrnged s n imge -755/8-797 44
Multiplying mtrices Simple vector multipliction: Vector outer product -755/8-797 45
Multiplying mtrices Generliztion of vector multipliction Outer product of dot products!! Dimensions must mtch!! A B Columns of first mtrix = rows of second Result inherits the numer of rows from the first mtrix nd the numer of columns from the second mtrix -755/8-797 46
Multiplying mtrices: Another view Simple vector multipliction: Vector inner product -755/8-797 47
Mtrix multipliction: nother view -755/8-797 48 NK N MN N K M K M NK N NK MN M N N The outer product of the first column of A nd the first row of B + outer product of the second column of A nd the second row of B + Sum of outer products B A
Why is tht useful? 9 3 4 5 5 9 6 75 7 5 7 8 Y 75 9 5 5 95 X Sounds: Three notes modulted independently -755/8-797 49
Mtrix multipliction: Mixing modulted spectr 9 3 4 5 5 9 6 75 7 5 7 8 Y 75 9 5 5 95 X Sounds: Three notes modulted independently -755/8-797 5
Mtrix multipliction: Mixing modulted spectr 9 3 4 5 5 9 6 75 7 7 5 8 75 Y 9 5 5 95 X Sounds: Three notes modulted independently -755/8-797 5
Mtrix multipliction: Mixing modulted spectr 9 3 4 5 5 9 6 75 7 7 5 8 75 9 5 5 95 X Sounds: Three notes modulted independently -755/8-797 5
Mtrix multipliction: Mixing modulted spectr 9 3 4 X 5 Sounds: Three notes modulted independently 5 9 6 75 7 7 5 8 75 9 5 5 95-755/8-797 53
Mtrix multipliction: Mixing modulted spectr Sounds: Three notes modulted independently -755/8-797 54
Mtrix multipliction: Imge trnsition i i j j Imge fdes out linerly Imge fdes in linerly 9 8 7 3 6 4 5 5 4 6 3 7 8 9-755/8-797 55
Mtrix multipliction: Imge trnsition i i j j i i i N 9 9i 3 4 8i Ech column is one imge The columns represent sequence of imges of decresing intensity Imge fdes out linerly 8 9i 9i N 7 6 8i 8i N 5 5 4 6 3 7 8-755/8-797 56 9
Mtrix multipliction: Imge trnsition i i j j 9 8 7 3 6 4 5 5 4 6 3 7 8 9 Imge fdes in linerly -755/8-797 57
Mtrix multipliction: Imge trnsition i i j j 9 8 7 3 6 4 5 5 4 6 3 7 8 9 Imge fdes out linerly Imge fdes in linerly -755/8-797 58
Mtrix Opertions: Properties A+B = B+A Actul interprettion: for ny vector x (A+B)x = (B+A)x (column vector x of the right size) x(a+b) = x(b+a) (row vector x of the pproprite size) A + (B + C) = (A + B) + C AB!= BA -755/8-797 59
The Spce of Mtrices The set of ll mtrices of given size (eg ll 3x4 mtrices) is spce! Addition is closed Sclr multipliction is closed Zero mtrix exists Mtrices hve dditive inverses Associtivity nd commuttivity rules pply! -755/8-797 6
The Identity Mtrix Y An identity mtrix is squre mtrix where All digonl elements re All off-digonl elements re Multipliction y n identity mtrix does not chnge vectors -755/8-797 6
Digonl Mtrix Y All off-digonl elements re zero Digonl elements re non-zero Scles the xes My flip xes -755/8-797 6
Digonl mtrix to trnsform imges How? -755/8-797 63
Stretching 6 5 Loction-sed representtion Scling mtrix only scles the X xis The Y xis nd pixel vlue re scled y identity Not good wy of scling -755/8-797 64
Stretching D = 5 5 5 A 5 Newpic DA Better wy Interpolte ( NxN) N is the width of the originl imge -755/8-797 65
Modifying color Scle only Green P Newpic B G R P -755/8-797 66
Permuttion Mtrix A permuttion mtrix simply rerrnges the xes The row entries re xis vectors in different order The result is comintion of rottions nd reflections The permuttion mtrix effectively permutes the rrngement of the elements in vector -755/8-797 67 x z y z y x 3 4 5 (3,4,5) X Y Z 4 5 3 X (old Y) Y (old Z) Z (old X)
Permuttion Mtrix Reflections nd 9 degree rottions of imges nd ojects -755/8-797 68 P P 6 5
Permuttion Mtrix Reflections nd 9 degree rottions of imges nd ojects Oject represented s mtrix of 3-Dimensionl position vectors Positions identify ech point on the surfce -755/8-797 69 P P N N N z z z y y y x x x
Rottion Mtrix Y x' y' x cosq xsinq y sinq y cosq (x,y) R q cosq sinq sinq cosq x X y x' X new y' y y Y R q X q X new (x,y ) (x,y) X X x x A rottion mtrix rottes the vector y some ngle q Alterntely viewed, it rottes the xes The new xes re t n ngle q to the old one -755/8-797 7
Rotting picture Note the representtion: 3-row mtrix Rottion only pplies on the coordinte rows The vlue does not chnge Why is pcmn griny? -755/8-797 7 6 5 cos45 sin 45 sin 45 cos45 R 8 7 3 3 8 4 3
3-D Rottion Xnew Ynew Z Y Znew q X degrees of freedom seprte ngles Wht will the rottion mtrix e? -755/8-797 7
Projections Wht would we see if the cone to the left were trnsprent if we looked t it from ove the plne shown y the grid? Norml to the plne Answer: the figure to the right How do we get this? Projection -755/8-797 73
Projections 9degrees projection Actul prolem: for ech vector Wht is the corresponding vector on the plne tht is closest pproximtion to it? Wht is the trnsform tht converts the vector to its pproximtion on the plne? -755/8-797 74
Projections error 9degrees Arithmeticlly: Find the mtrix P such tht For every vector X, PX lies on the plne The plne is the column spce of P X PX is the smllest possile projection -755/8-797 75
Projection Mtrix 9degrees W W projection Consider ny set of independent vectors W, W on the plne Arrnged s mtrix [W W ] The plne is the column spce of the mtrix Any vector cn e projected onto this plne The mtrix P tht rottes nd scles the vector so tht it ecomes its projection is projection mtrix -755/8-797 76
Projection Mtrix 9degrees W W projection Given set of vectors W, W which form mtrix W = [W W ] The projection mtrix to trnsform vector X to its projection on the plne is P = W (W T W) - W T We will visit mtrix inversion shortly Mgic ny set of independent vectors from the sme plne tht re expressed s mtrix will give you the sme projection mtrix P = V (V T V) - V T -755/8-797 77
Projections HOW? -755/8-797 78
Projections Drw ny two vectors W nd W W tht lie on the plne ANY two so long s they hve different ngles Compose mtrix W = [W W ] Compose the projection mtrix P = W (W T W) - W T Multiply every point on the cone y P to get its projection -755/8-797 79
Projections The projection ctully projects it onto the plne, ut you re still seeing the plne in 3D The result of the projection is 3-D vector P = W (W T W) - W T = 3x3, PX = 3x The imge must e rotted till the plne is in the plne of the pper The Z xis in this cse will lwys e zero nd cn e ignored How will you rotte it? (rememer you know W nd W ) -755/8-797 8
Projection mtrix properties The projection of ny vector tht is lredy on the plne is the vector itself PX = X if X is on the plne If the oject is lredy on the plne, there is no further projection to e performed The projection of projection is the projection P(PX) = PX Projection mtrices re idempotent P = P -755/8-797 8
Projections: A more physicl mening Let W, W W k e ses We wnt to explin our dt in terms of these ses We often cnnot do so But we cn explin significnt portion of it The portion of the dt tht cn e expressed in terms of our vectors W, W, W k, is the projection of the dt on the W W k (hyper) plne In our previous exmple, the dt were ll the points on cone, nd the ses were vectors on the plne -755/8-797 8
Projection : n exmple with sounds The spectrogrm (mtrix) of piece of music How much of the ove music ws composed of the ove notes Ie how much cn it e explined y the notes -755/8-797 83
Projection: one note M = The spectrogrm (mtrix) of piece of music W = M = spectrogrm; W = note P = W (W T W) - W T Projected Spectrogrm = P * M -755/8-797 84
Projection: one note clened up M = The spectrogrm (mtrix) of piece of music W = Floored ll mtrix vlues elow threshold to zero -755/8-797 85
Projection: multiple notes M = The spectrogrm (mtrix) of piece of music W = P = W (W T W) - W T Projected Spectrogrm = P * M -755/8-797 86
Projection: multiple notes, clened up M = The spectrogrm (mtrix) of piece of music W = P = W (W T W) - W T Projected Spectrogrm = P * M -755/8-797 87
note note note3 Projection nd Lest Squres Projection ctully computes lest squred error estimte For ech vector V in the music spectrogrm mtrix Approximtion: V pprox = *note + *note + c*note3 V pprox c Error vector E = V V pprox Squred error energy for V e(v) = norm(e) Totl error = sum over ll V { e(v) } = S V e(v) Projection computes V pprox for ll vectors such tht Totl error is minimized It does not give you,, c Though Tht needs different opertion the inverse / pseudo inverse -755/8-797 88