Linear Algebra Primer Juan Carlos Niebles and Ranja Krishna Sanford Vision and Learning Lab Anoher, ver in-deph linear algebra review from CS229 is available here: hp://cs229.sanford.edu/secion/cs229-linalg.pdf And a video discussion of linear algebra from EE263 is here (lecures 3 and 4): hps://see.sanford.edu/course/ee263 Sanford Universi /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculus Sanford Universi 2 /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculus Vecors and marices are jus collecions of ordered numbers ha represen somehing: movemens in space, scaling facors, piel brighness, ec. We ll define some common uses and sandard operaions on hem. Sanford Universi 3 /2/7
Vecor A column vecor where A row vecor where denoes he ranspose operaion Sanford Universi 4 /2/7
Vecor We ll defaul o column vecors in his class You ll wan o keep rack of he orienaion of our vecors when programming in phon You can ranspose a vecor V in phon b wriing V.. (Bu in class maerials, we will alwas use V T o indicae ranspose, and we will use V o mean V prime ) Sanford Universi 5 /2/7
Vecors have wo main uses Vecors can represen an offse in 2D or 3D space Poins are jus vecors from he origin Daa (piels, gradiens a an image kepoin, ec) can also be reaed as a vecor Such vecors don have a geomeric inerpreaion, bu calculaions like disance can sill have value Sanford Universi 6 /2/7
Mari A mari is an arra of numbers wih size b, i.e. m rows and n columns. If, we sa ha is square. Sanford Universi 7 /2/7
Images Phon represens an image as a mari of piel brighnesses Noe ha he upper lef corner is [,] (,) Sanford Universi 8 /2/7
Color Images Grascale images have one number per piel, and are sored as an m n mari. Color images have 3 numbers per piel red, green, and blue brighnesses (RGB) Sored as an m n 3 mari Sanford Universi 9 /2/7
Basic Mari Operaions We will discuss: Addiion Scaling Do produc Muliplicaion Transpose Inverse / pseudoinverse Deerminan / race Sanford Universi /2/7
Mari Operaions Addiion Can onl add a mari wih maching dimensions, or a scalar. Scaling Sanford Universi /2/7
Vecors Norm More formall, a norm is an funcion ha saisfies 4 properies: Non-negaivi: For all Definieness: f() if and onl if. Homogenei: For all Triangle inequali: For all Sanford Universi 2 /2/7
Mari Operaions Eample Norms General norms: Sanford Universi 3 /2/7
Mari Operaions Inner produc (do produc) of vecors Mulipl corresponding enries of wo vecors and add up he resul is also Cos( he angle beween and ) Sanford Universi 4 /2/7
Mari Operaions Inner produc (do produc) of vecors If B is a uni vecor, hen A B gives he lengh of A which lies in he direcion of B Sanford Universi 5 /2/7
Mari Operaions The produc of wo marices Sanford Universi 6 /2/7
Muliplicaion Mari Operaions The produc AB is: Each enr in he resul is (ha row of A) do produc wih (ha column of B) Man uses, which will be covered laer Sanford Universi 7 /2/7
Mari Operaions Muliplicaion eample: Each enr of he mari produc is made b aking he do produc of he corresponding row in he lef mari, wih he corresponding column in he righ one. Sanford Universi 8 /2/7
Mari Operaions The produc of wo marices Sanford Universi 9 /2/7
Mari Operaions Powers B convenion, we can refer o he mari produc AA as A 2, and AAA as A 3, ec. Obviousl onl square marices can be muliplied ha wa Sanford Universi 2 /2/7
Mari Operaions Transpose flip mari, so row becomes column A useful ideni: Sanford Universi 2 /2/7
Deerminan reurns a scalar Represens area (or volume) of he parallelogram described b he vecors in he rows of he mari For, Properies: Mari Operaions Sanford Universi 22 /2/7
Mari Operaions Trace Invarian o a lo of ransformaions, so i s used someimes in proofs. (Rarel in his class hough.) Properies: Sanford Universi 23 /2/7
Mari Operaions Vecor Norms Mari norms: Norms can also be defined for marices, such as he Frobenius norm: Sanford Universi 24 /2/7
Special Marices Ideni mari I Square mari, s along diagonal, s elsewhere I [anoher mari] [ha mari] Diagonal mari Square mari wih numbers along diagonal, s elsewhere A diagonal [anoher mari] scales he rows of ha mari Sanford Universi 25 /2/7
Special Marices Smmeric mari Skew-smmeric mari 2 2 3 5 42 75 5 7 Sanford Universi 26 /2/7
Linear Algebra Primer Juan Carlos Niebles and Ranja Krishna Sanford Vision and Learning Lab Anoher, ver in-deph linear algebra review from CS229 is available here: hp://cs229.sanford.edu/secion/cs229-linalg.pdf And a video discussion of linear algebra from EE263 is here (lecures 3 and 4): hps://see.sanford.edu/course/ee263 Sanford Universi 27 /2/7
Announcemens par HW submied las nigh HW is due ne Monda HW2 will be released onigh Class noes from las Thursda due before class in eacl 48 hours Sanford Universi 28 /2/7
Announcemens par 2 Fuure homework assignmens will be released via gihub Will allow ou o keep rack of changes IF he happen. Submissions for HW onwards will be done all hrough gradescope. NO MORE CORN SUBMISSIONS You will have separae submissions for he iphon pdf and he phon code. Sanford Universi 29 /2/7
Recap - Vecor A column vecor where A row vecor where denoes he ranspose operaion Sanford Universi 3 /2/7
Recap - Mari A mari is an arra of numbers wih size b, i.e. m rows and n columns. If, we sa ha is square. Sanford Universi 3 /2/7
Recap - Color Images Grascale images have one number per piel, and are sored as an m n mari. Color images have 3 numbers per piel red, green, and blue brighnesses (RGB) Sored as an m n 3 mari Sanford Universi 32 /2/7
Recap - Vecors Norm More formall, a norm is an funcion ha saisfies 4 properies: Non-negaivi: For all Definieness: f() if and onl if. Homogenei: For all Triangle inequali: For all Sanford Universi 33 /2/7
Recap projecion Inner produc (do produc) of vecors If B is a uni vecor, hen A B gives he lengh of A which lies in he direcion of B Sanford Universi 34 /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculus Mari muliplicaion can be used o ransform vecors. A mari used in his wa is called a ransformaion mari. Sanford Universi 35 /2/7
Transformaion Marices can be used o ransform vecors in useful was, hrough muliplicaion: A Simples is scaling: (Verif o ourself ha he mari muliplicaion works ou his wa) Sanford Universi 36 /2/7
Roaion How can ou conver a vecor represened in frame o a new, roaed coordinae frame? Sanford Universi 37 /2/7
Roaion How can ou conver a vecor represened in frame o a new, roaed coordinae frame? Remember wha a vecor is: [componen in direcion of he frame s ais, componen in direcion of ais] Sanford Universi 38 /2/7
Roaion So o roae i we mus produce his vecor: [componen in direcion of new ais, componen in direcion of new ais] We can do his easil wih do producs! New coordinae is [original vecor] do [he new ais] New coordinae is [original vecor] do [he new ais] Sanford Universi 39 /2/7
Roaion Insigh: his is wha happens in a mari*vecor muliplicaion Resul coordinae is: [original vecor] do [mari row ] So mari muliplicaion can roae a vecor p: Sanford Universi 4 /2/7
Roaion Suppose we epress a poin in he new coordinae ssem which is roaed lef If we plo he resul in he original coordinae ssem, we have roaed he poin righ Thus, roaion marices can be used o roae vecors. We ll usuall hink of hem in ha sense-- as operaors o roae vecors Sanford Universi 4 /2/7
2D Roaion Mari Formula Couner-clockwise roaion b an angle q P q cosθ -sin θ cosθ + sin θ P é écosq - sin q é sin q cosq P R P Sanford Universi 42 /2/7
Transformaion Marices Muliple ransformaion marices can be used o ransform a poin: p R 2 R S p Sanford Universi 43 /2/7
Transformaion Marices Muliple ransformaion marices can be used o ransform a poin: p R 2 R S p The effec of his is o appl heir ransformaions one afer he oher, from righ o lef. In he eample above, he resul is (R 2 (R (S p))) Sanford Universi 44 /2/7
Transformaion Marices Muliple ransformaion marices can be used o ransform a poin: p R 2 R S p The effec of his is o appl heir ransformaions one afer he oher, from righ o lef. In he eample above, he resul is (R 2 (R (S p))) The resul is eacl he same if we mulipl he marices firs, o form a single ransformaion mari: p (R 2 R S) p Sanford Universi 45 /2/7
Homogeneous ssem In general, a mari muliplicaion les us linearl combine componens of a vecor This is sufficien for scale, roae, skew ransformaions. Bu noice, we can add a consan! L Sanford Universi 46 /2/7
Homogeneous ssem The (somewha hack) soluion? Sick a a he end of ever vecor: Now we can roae, scale, and skew like before, AND ranslae (noe how he muliplicaion works ou, above) This is called homogeneous coordinaes Sanford Universi 47 /2/7
Homogeneous ssem In homogeneous coordinaes, he muliplicaion works ou so he righmos column of he mari is a vecor ha ges added. Generall, a homogeneous ransformaion mari will have a boom row of [ ], so ha he resul has a a he boom oo. Sanford Universi 48 /2/7
Homogeneous ssem One more hing we migh wan: o divide he resul b somehing For eample, we ma wan o divide b a coordinae, o make hings scale down as he ge farher awa in a camera image Mari muliplicaion can acuall divide So, b convenion, in homogeneous coordinaes, we ll divide he resul b is las coordinae afer doing a mari muliplicaion Sanford Universi 49 /2/7
2D Translaion P P Sanford Universi 5 /2/7
Sanford Universi /2/7 5 2D Translaion using Homogeneous Coordinaes P P é é é + + P,), ( ), (,), ( ), ( P P
Sanford Universi /2/7 52 2D Translaion using Homogeneous Coordinaes P P é é é + + P,), ( ), (,), ( ), ( P P
Sanford Universi /2/7 53 2D Translaion using Homogeneous Coordinaes P P é é é + + P,), ( ), (,), ( ), ( P P
Sanford Universi /2/7 54 2D Translaion using Homogeneous Coordinaes P P é é é + + P,), ( ), (,), ( ), ( P P
Sanford Universi /2/7 55 2D Translaion using Homogeneous Coordinaes P P é é é + + P,), ( ), (,), ( ), ( P P P T P I é
Scaling P P Sanford Universi 56 /2/7
Scaling Equaion s P P (, ) P (s,s ) P P (, ) (,,) s P ( s, s ) ( s, s,) Sanford Universi 57 /2/7
Sanford Universi Scaling Equaion P s P s é é é s s s s P,), ( ), (,), ( ), ( s s s s P P ),s (s ), ( P P /2/7 58
Sanford Universi Scaling Equaion P s P s é é é s s s s P,), ( ), (,), ( ), ( s s s s P P S P S P S é ),s (s ), ( P P /2/7 59
Scaling & Translaing P P P S P P T P P T P T (S P) T S P Sanford Universi 6 /2/7
Sanford Universi Scaling & Translaing P T S P " # $ $ $ $ % & s s " # $ $ $ $ % & " # $ $ $ % & s s " # $ $ $ $ % & " # $ $ $ % & s + s + " # $ $ $ $ % & S " # $ % & " # $ $ $ % & A /2/7 6
Sanford Universi Scaling & Translaing P T S P " # $ $ $ $ % & s s " # $ $ $ $ % & " # $ $ $ % & s s " # $ $ $ $ % & " # $ $ $ % & s + s + " # $ $ $ $ % & S " # $ % & " # $ $ $ % & /2/7 62
Sanford Universi Translaing & Scaling versus Scaling & Translaing é + + é é é é é s s s s s s P S T P /2/7 63
Sanford Universi Translaing & Scaling! Scaling & Translaing é + + é é é é é s s s s s s s s s s P T S P é + + é é é é é s s s s s s P S T P /2/7 64
Sanford Universi Translaing & Scaling! Scaling & Translaing é + + é é é é é s s s s s s s s s s P T S P é + + é é é é é s s s s s s P S T P /2/7 65
Roaion P P Sanford Universi 66 /2/7
Roaion Equaions Couner-clockwise roaion b an angle q P q P cosθ -sin θ cosθ + sin θ é écosq - sin q é sin q cosq P R P Sanford Universi 67 /2/7
Roaion Mari Properies é écosq - sin q é sin q cosq A 2D roaion mari is 22 Noe: R belongs o he caegor of normal marices and saisfies man ineresing properies: R R T R T R I de( R) Sanford Universi 68 /2/7
Roaion Mari Properies Transpose of a roaion mari produces a roaion in he opposie direcion R R T de( R) R The rows of a roaion mari are alwas muuall perpendicular (a.k.a. orhogonal) uni vecors (and so are is columns) R T I Sanford Universi 69 /2/7
Sanford Universi Scaling + Roaion + Translaion P (T R S) P é é é - é s s cosθ sinθ sin θ cosθ R P S T P é é é - s s cosθ sinθ sin θ cosθ é é é é é R S S R /2/7 7 This is he form of he general-purpose ransformaion mari
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculae The inverse of a ransformaion mari reverses is effec Sanford Universi 7 /2/7
Given a mari A, is inverse A - is a mari such ha AA - A - A I E.g. Inverse Inverse does no alwas eis. If A - eiss, A is inverible or non-singular. Oherwise, i s singular. Useful ideniies, for marices ha are inverible: Sanford Universi 72 /2/7
Mari Operaions Pseudoinverse Sa ou have he mari equaion AXB, where A and B are known, and ou wan o solve for X Sanford Universi 73 /2/7
Mari Operaions Pseudoinverse Sa ou have he mari equaion AXB, where A and B are known, and ou wan o solve for X You could calculae he inverse and pre-mulipl b i: A - AXA - B XA - B Sanford Universi 74 /2/7
Mari Operaions Pseudoinverse Sa ou have he mari equaion AXB, where A and B are known, and ou wan o solve for X You could calculae he inverse and pre-mulipl b i: A - AXA - B XA - B Phon command would be np.linalg.inv(a)*b Bu calculaing he inverse for large marices ofen brings problems wih compuer floaing-poin resoluion (because i involves working wih ver small and ver large numbers ogeher). Or, our mari migh no even have an inverse. Sanford Universi 75 /2/7
Mari Operaions Pseudoinverse Forunael, here are workarounds o solve AXB in hese siuaions. And phon can do hem! Insead of aking an inverse, direcl ask phon o solve for X in AXB, b ping np.linalg.solve(a, B) Phon will r several appropriae numerical mehods (including he pseudoinverse if he inverse doesn eis) Phon will reurn he value of X which solves he equaion If here is no eac soluion, i will reurn he closes one If here are man soluions, i will reurn he smalles one Sanford Universi 76 /2/7
Phon eample: Mari Operaions >> impor nump as np >> np.linalg.solve(a,b). -.5 Sanford Universi 77 /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculae The rank of a ransformaion mari ells ou how man dimensions i ransforms a vecor o. Sanford Universi 78 /2/7
Linear independence Suppose we have a se of vecors v,, v n If we can epress v as a linear combinaion of he oher vecors v 2 v n, hen v is linearl dependen on he oher vecors. The direcion v can be epressed as a combinaion of he direcions v 2 v n. (E.g. v.7 v 2 -.7 v 4 ) Sanford Universi 79 /2/7
Linear independence Suppose we have a se of vecors v,, v n If we can epress v as a linear combinaion of he oher vecors v 2 v n, hen v is linearl dependen on he oher vecors. The direcion v can be epressed as a combinaion of he direcions v 2 v n. (E.g. v.7 v 2 -.7 v 4 ) If no vecor is linearl dependen on he res of he se, he se is linearl independen. Common case: a se of vecors v,, v n is alwas linearl independen if each vecor is perpendicular o ever oher vecor (and non-zero) Sanford Universi 8 /2/7
Linear independence Linearl independen se No linearl independen Sanford Universi 8 /2/7
Mari rank Column/row rank Column rank alwas equals row rank Mari rank Sanford Universi 82 /2/7
Mari rank For ransformaion marices, he rank ells ou he dimensions of he oupu E.g. if rank of A is, hen he ransformaion p Ap maps poins ono a line. Here s a mari wih rank : All poins ge mapped o he line 2 Sanford Universi 83 /2/7
Mari rank If an m m mari is rank m, we sa i s full rank Maps an m vecor uniquel o anoher m vecor An inverse mari can be found If rank < m, we sa i s singular A leas one dimension is geing collapsed. No wa o look a he resul and ell wha he inpu was Inverse does no eis Inverse also doesn eis for non-square marices Sanford Universi 84 /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors(SVD) Mari Calculus Sanford Universi 85 /2/7
Eigenvecor and Eigenvalue An eigenvecor of a linear ransformaion A is a non-zero vecor ha, when A is applied o i, does no change direcion. Sanford Universi 86 /2/7
Eigenvecor and Eigenvalue An eigenvecor of a linear ransformaion A is a non-zero vecor ha, when A is applied o i, does no change direcion. Appling A o he eigenvecor onl scales he eigenvecor b he scalar value λ, called an eigenvalue. Sanford Universi 87 /2/7
Eigenvecor and Eigenvalue We wan o find all he eigenvalues of A: Which can we wrien as: Therefore: Sanford Universi 88 /2/7
Eigenvecor and Eigenvalue We can solve for eigenvalues b solving: Since we are looking for non-zero, we can insead solve he above equaion as: Sanford Universi 89 /2/7
Properies The race of a A is equal o he sum of is eigenvalues: Sanford Universi 9 /2/7
Properies The race of a A is equal o he sum of is eigenvalues: The deerminan of A is equal o he produc of is eigenvalues Sanford Universi 9 /2/7
Properies The race of a A is equal o he sum of is eigenvalues: The deerminan of A is equal o he produc of is eigenvalues The rank of A is equal o he number of non-zero eigenvalues of A. Sanford Universi 92 /2/7
Properies The race of a A is equal o he sum of is eigenvalues: The deerminan of A is equal o he produc of is eigenvalues The rank of A is equal o he number of non-zero eigenvalues of A. The eigenvalues of a diagonal mari D diag(d,... dn) are jus he diagonal enries d,... dn Sanford Universi 93 /2/7
Specral heor We call an eigenvalue λ and an associaed eigenvecor an eigenpair. The space of vecors where (A λi) is ofen called he eigenspace of A associaed wih he eigenvalue λ. The se of all eigenvalues of A is called is specrum: Sanford Universi 94 /2/7
Specral heor The magniude of he larges eigenvalue (in magniude) is called he specral radius Where C is he space of all eigenvalues of A Sanford Universi 95 /2/7
Specral heor The specral radius is bounded b infini norm of a mari: Proof: Turn o a parner and prove his! Sanford Universi 96 /2/7
Specral heor The specral radius is bounded b infini norm of a mari: Proof: Le λ and v be an eigenpair of A: Sanford Universi 97 /2/7
Diagonalizaion An n n mari A is diagonalizable if i has n linearl independen eigenvecors. Mos square marices (in a sense ha can be made mahemaicall rigorous) are diagonalizable: Normal marices are diagonalizable Marices wih n disinc eigenvalues are diagonalizable Lemma: Eigenvecors associaed wih disinc eigenvalues are linearl independen. Sanford Universi 98 /2/7
Diagonalizaion An n n mari A is diagonalizable if i has n linearl independen eigenvecors. Mos square marices are diagonalizable: Normal marices are diagonalizable Marices wih n disinc eigenvalues are diagonalizable Lemma: Eigenvecors associaed wih disinc eigenvalues are linearl independen. Sanford Universi 99 /2/7
Diagonalizaion Eigenvalue equaion: Where D is a diagonal mari of he eigenvalues Sanford Universi /2/7
Diagonalizaion Eigenvalue equaion: Assuming all λ i s are unique: Remember ha he inverse of an orhogonal mari is jus is ranspose and he eigenvecors are orhogonal Sanford Universi /2/7
Smmeric marices Properies: For a smmeric mari A, all he eigenvalues are real. The eigenvecors of A are orhonormal. Sanford Universi 2 /2/7
Smmeric marices Therefore: where So, if we waned o find he vecor ha: Sanford Universi 3 /2/7
Smmeric marices Therefore: where So, if we waned o find he vecor ha: Is he same as finding he eigenvecor ha corresponds o he larges eigenvalue. Sanford Universi 4 /2/7
Some applicaions of Eigenvalues PageRank Schrodinger s equaion PCA Sanford Universi 5 /2/7
Ouline Vecors and marices Basic Mari Operaions Deerminans, norms, race Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors(SVD) Mari Calculus Sanford Universi 6 /2/7
Mari Calculus The Gradien Le a funcion ake as inpu a mari A of size m n and reurns a real value. Then he gradien of f: Sanford Universi 7 /2/7
Mari Calculus The Gradien Ever enr in he mari is: he size of A f(a) is alwas he same as he size of A. So if A is jus a vecor : Sanford Universi 8 /2/7
Eercise Eample: Find: Sanford Universi 9 /2/7
Eercise Eample: From his we can conclude ha: Sanford Universi /2/7
Mari Calculus The Gradien Properies Sanford Universi /2/7
Mari Calculus The Hessian The Hessian mari wih respec o, wrien or simpl as H is he n n mari of parial derivaives Sanford Universi 2 /2/7
Mari Calculus The Hessian Each enr can be wrien as: Eercise: Wh is he Hessian alwas smmeric? Sanford Universi 3 /2/7
Mari Calculus The Hessian Each enr can be wrien as: The Hessian is alwas smmeric, because This is known as Schwarzs heorem: The order of parial derivaives don maer as long as he second derivaive eiss and is coninuous. Sanford Universi 4 /2/7
Mari Calculus The Hessian Noe ha he hessian is no he gradien of whole gradien of a vecor (his is no defined). I is acuall he gradien of ever enr of he gradien of he vecor. Sanford Universi 5 /2/7
Mari Calculus The Hessian Eg, he firs column is he gradien of Sanford Universi 6 /2/7
Eercise Eample: Sanford Universi 7 /2/7
Eercise Sanford Universi 8 /2/7
Eercise Divide he summaion ino 3 pars depending on wheher: i k or j k Sanford Universi 9 /2/7
Eercise Sanford Universi 2 /2/7
Eercise Sanford Universi 2 /2/7
Eercise Sanford Universi 22 /2/7
Eercise Sanford Universi 23 /2/7
Eercise Sanford Universi 24 /2/7
Eercise Sanford Universi 25 /2/7
Eercise Sanford Universi 26 /2/7
Eercise Sanford Universi 27 /2/7
Eercise Sanford Universi 28 /2/7
Wha we have learned Vecors and marices Basic Mari Operaions Special Marices Transformaion Marices Homogeneous coordinaes Translaion Mari inverse Mari rank Eigenvalues and Eigenvecors Mari Calculae Sanford Universi 29 /2/7