Linear Algebra Primer

Transcription

1 Linear Algebra Primer Dr. Juan Carlos Niebles Sanford AI Lab Prof. Fei- Fei Li Sanford Vision Lab Anoher, ver in- deph linear algebra review from CS229 is available here: hip://cs229.sanford.edu/seclon/cs229- linalg.pdf And a video discussion of linear algebra from EE263 is here (lecures 3 and 4): hips://see.sanford.edu/course/ee263

2 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm 2

3 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm Vecors and marices are jus colleclons of ordered numbers ha represen somehing: movemens in space, scaling facors, piel brighness, ec. We ll define some common uses and sandard operalons on hem. 3

4 Vecor A column vecor where A row vecor where denoes he ranspose operalon 4

5 Vecor We ll defaul o column vecors in his class You ll wan o keep rack of he orienalon of our vecors when programming in MATLAB You can ranspose a vecor V in MATLAB b wrilng V. (Bu in class maerials, we will alwas use V T o indicae ranspose, and we will use V o mean V prime ) 5

6 Vecors have wo main uses Vecors can represen an offse in 2D or 3D space Poins are jus vecors from he origin Daa (piels, gradiens a an image kepoin, ec) can also be reaed as a vecor Such vecors don have a geomeric inerprealon, bu calculalons like disance can slll have value 6

7 Mari A mari is an arra of numbers wih size b, i.e. m rows and n columns. If, we sa ha is square. 7

8 Images MATLAB represens an image as a mari of piel brighnesses Noe ha mari coordinaes are NOT Caresian coordinaes. The upper leg corner is [,] (,) 8

9 Color Images Grascale images have one number per piel, and are sored as an m n mari. Color images have 3 numbers per piel red, green, and blue brighnesses (RGB) Sored as an m n 3 mari 9

10 Basic Mari OperaLons We will discuss: AddiLon Scaling Do produc MulLplicaLon Transpose Inverse / pseudoinverse Deerminan / race

11 Mari OperaLons AddiLon Can onl add a mari wih maching dimensions, or a scalar. Scaling

12 Mari OperaLons Inner produc (do produc) of vecors MulLpl corresponding enries of wo vecors and add up he resul is also Cos( he angle beween and ) 2

13 Mari OperaLons Inner produc (do produc) of vecors If B is a uni vecor, hen A B gives he lengh of A which lies in he direclon of B 3

14 MulLplicaLon Mari OperaLons The produc AB is: Each enr in he resul is (ha row of A) do produc wih (ha column of B) Man uses, which will be covered laer 4

15 Mari OperaLons MulLplicaLon eample: Each enr of he mari produc is made b aking he do produc of he corresponding row in he leg mari, wih he corresponding column in he righ one. 5

16 Mari OperaLons Powers B convenlon, we can refer o he mari produc AA as A 2, and AAA as A 3, ec. Obviousl onl square marices can be mullplied ha wa 6

17 Mari OperaLons Transpose flip mari, so row becomes column A useful idenl: 7

18 Deerminan reurns a scalar Represens area (or volume) of he parallelogram described b he vecors in he rows of he mari For, ProperLes: Mari OperaLons 8

19 Mari OperaLons Trace Invarian o a lo of ransformalons, so i s used somelmes in proofs. (Rarel in his class hough.) ProperLes: 9

20 Special Marices IdenL mari I Square mari, s along diagonal, s elsewhere I [anoher mari] [ha mari] Diagonal mari Square mari wih numbers along diagonal, s elsewhere A diagonal [anoher mari] scales he rows of ha mari 2

21 Special Marices Smmeric mari Skew- smmeric mari

22 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm Mari mullplicalon can be used o ransform vecors. A mari used in his wa is called a ransformalon mari. 22

23 TransformaLon Marices can be used o ransform vecors in useful was, hrough mullplicalon: A Simples is scaling: (Verif o ourself ha he mari mullplicalon works ou his wa) 23

24 RoaLon How can ou conver a vecor represened in frame o a new, roaed coordinae frame? Remember wha a vecor is: [componen in direclon of he frame s ais, componen in direclon of ais] 24

25 RoaLon So o roae i we mus produce his vecor: [componen in direclon of new ais, componen in direclon of new ais] We can do his easil wih do producs! New coordinae is [original vecor] do [he new ais] New coordinae is [original vecor] do [he new ais] 25

26 RoaLon Insigh: his is wha happens in a mari*vecor mullplicalon Resul coordinae is: [original vecor] do [mari row ] So mari mullplicalon can roae a vecor p: 26

27 RoaLon Suppose we epress a poin in he new coordinae ssem which is roaed leg If we plo he resul in he original coordinae ssem, we have roaed he poin righ Thus, roalon marices can be used o roae vecors. We ll usuall hink of hem in ha sense- - as operaors o roae vecors 27

28 2D RoaLon Mari Formula Couner-clockwise roaion b an angle θ P θ ' cosθ sin θ ' cosθ + sin θ P ' ' cosθ sin θ sin θ cosθ P' R P 28

29 TransformaLon Marices MulLple ransformalon marices can be used o ransform a poin: p R 2 R S p The effec of his is o appl heir ransformalons one ager he oher, from righ o le3. In he eample above, he resul is (R 2 (R (S p))) The resul is eacl he same if we mullpl he marices firs, o form a single ransformalon mari: p (R 2 R S) p 29

30 Homogeneous ssem In general, a mari mullplicalon les us linearl combine componens of a vecor This is sufficien for scale, roae, skew ransformalons. Bu nolce, we can add a consan! L 3

31 Homogeneous ssem The (somewha hack) solulon? SLck a a he end of ever vecor: Now we can roae, scale, and skew like before, AND ranslae (noe how he mullplicalon works ou, above) This is called homogeneous coordinaes 3

32 Homogeneous ssem In homogeneous coordinaes, he mullplicalon works ou so he righmos column of he mari is a vecor ha ges added. Generall, a homogeneous ransformalon mari will have a boiom row of [ ], so ha he resul has a a he boiom oo. 32

33 Homogeneous ssem One more hing we migh wan: o divide he resul b somehing For eample, we ma wan o divide b a coordinae, o make hings scale down as he ge farher awa in a camera image Mari mullplicalon can acuall divide So, b conven7on, in homogeneous coordinaes, we ll divide he resul b is las coordinae ager doing a mari mullplicalon 33

34 2D TranslaLon P P 34

35 35 2D TranslaLon using Homogeneous Coordinaes P P + + ' P,), ( ), (,), ( ), ( P P P T P I

36 Scaling P P 36

37 Scaling EquaLon P s P s ' s s s s P,), ( ), ( ',), ( ), ( s s s s P P S P S P S ' ),s (s ' ), ( P P 37

38 Scaling & TranslaLng P P P S P P T P P T P T (S P) T S P A P 38

39 Scaling & TranslaLng + + '' S s s s s s s P S T P A 39

40 TranslaLng & Scaling! Scaling & TranslaLng + + s s s s s s s s s s ''' P T S P + + s s s s s s '' ' P S T P 4

41 RoaLon P P 4

42 RoaLon EquaLons Couner-clockwise roaion b an angle θ P θ P ' cosθ sin θ ' cosθ + sin θ ' ' cosθ sin θ sin θ cosθ P' R P 42

43 RoaLon Mari ProperLes Transpose of a roalon mari produces a roalon in he opposie direclon R R T de( R) R The rows of a roalon mari are alwas muuall perpendicular (a.k.a. orhogonal) uni vecors (and so are is columns) R T I 43

44 ProperLes ' ' cosθ sin θ sin θ cosθ A 2D roalon mari is 22 Noe: R belongs o he caegor of normal marices and saisfies man ineresing properies: R R T R T R I de( R) 44

45 Scaling + RoaLon + TranslaLon P (T R S) P s s cosθ sinθ sin θ cosθ R ' P S T P s s cosθ sinθ sin θ cosθ R S S R 45 This is he form of he general- purpose ransformalon mari

46 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm The inverse of a ransformalon mari reverses is effec 46

47 Given a mari A, is inverse A - is a mari such ha AA - A - A I E.g. Inverse Inverse does no alwas eis. If A - eiss, A is inver2ble or non- singular. Oherwise, i s singular. Useful idenlles, for marices ha are inverlble: 47

48 Mari OperaLons Pseudoinverse Sa ou have he mari equalon AXB, where A and B are known, and ou wan o solve for X You could use MATLAB o calculae he inverse and premullpl b i: A - AXA - B XA - B MATLAB command would be inv(a)*b Bu calculalng he inverse for large marices ogen brings problems wih compuer floalng- poin resolulon (because i involves working wih ver small and ver large numbers ogeher). Or, our mari migh no even have an inverse. 48

49 Mari OperaLons Pseudoinverse Forunael, here are workarounds o solve AXB in hese siualons. And MATLAB can do hem! Insead of aking an inverse, direcl ask MATLAB o solve for X in AXB, b ping A\B MATLAB will r several appropriae numerical mehods (including he pseudoinverse if he inverse doesn eis) MATLAB will reurn he value of X which solves he equalon If here is no eac solulon, i will reurn he closes one If here are man solulons, i will reurn he smalles one 49

50 MATLAB eample: Mari OperaLons >> A\B

51 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm The rank of a ransformalon mari ells ou how man dimensions i ransforms a vecor o. 5

52 Linear independence Suppose we have a se of vecors v,, v n If we can epress v as a linear combinalon of he oher vecors v 2 v n, hen v is linearl dependen on he oher vecors. The direclon v can be epressed as a combinalon of he direclons v 2 v n. (E.g. v.7 v v 4 ) If no vecor is linearl dependen on he res of he se, he se is linearl independen. Common case: a se of vecors v,, v n is alwas linearl independen if each vecor is perpendicular o ever oher vecor (and non- zero) 52

53 Linear independence Linearl independen se No linearl independen 53

54 Mari rank Column/row rank Column rank alwas equals row rank Mari rank 54

55 Mari rank For ransformalon marices, he rank ells ou he dimensions of he oupu E.g. if rank of A is, hen he ransformalon p Ap maps poins ono a line. Here s a mari wih rank : All poins ge mapped o he line 2 55

56 Mari rank If an m m mari is rank m, we sa i s full rank Maps an m vecor uniquel o anoher m vecor An inverse mari can be found If rank < m, we sa i s singular A leas one dimension is ge}ng collapsed. No wa o look a he resul and ell wha he inpu was Inverse does no eis Inverse also doesn eis for non- square marices 56

57 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm SVD is an algorihm ha represens an mari as he produc of 3 marices. I is used o discover inereslng srucure in a mari. 57

58 Singular Value DecomposiLon (SVD) There are several compuer algorihms ha can facorize a mari, represenlng i as he produc of some oher marices The mos useful of hese is he Singular Value DecomposiLon. Represens an mari A as a produc of hree marices: UΣV T MATLAB command: [U,S,V]svd(A) 58

59 Singular Value DecomposiLon (SVD) UΣV T A Where U and V are roalon marices, and Σ is a scaling mari. For eample: 59

60 Singular Value DecomposiLon (SVD) Beond 2D: In general, if A is m n, hen U will be m m, Σ will be m n, and V T will be n n. (Noe he dimensions work ou o produce m n ager mullplicalon) 6

61 Singular Value DecomposiLon (SVD) U and V are alwas roalon marices. Geomeric roalon ma no be an applicable concep, depending on he mari. So we call hem uniar marices each column is a uni vecor. Σ is a diagonal mari The number of nonzero enries rank of A The algorihm alwas sors he enries high o low 6

62 SVD ApplicaLons We ve discussed SVD in erms of geomeric ransformalon marices Bu SVD of an image mari can also be ver useful To undersand his, we ll look a a less geomeric inerprealon of wha SVD is doing 62

63 SVD ApplicaLons Look a how he mullplicalon works ou, leg o righ: Column of U ges scaled b he firs value from Σ. The resullng vecor ges scaled b row of V T o produce a conribulon o he columns of A 63

64 SVD ApplicaLons + Each produc of (column i of U) (value i from Σ) (row i of V T ) produces a componen of he final A. 64

65 SVD ApplicaLons We re building A as a linear combinalon of he columns of U Using all columns of U, we ll rebuild he original mari perfecl Bu, in real- world daa, ogen we can jus use he firs few columns of U and we ll ge somehing close (e.g. he firs A par)al, above) 65

66 SVD ApplicaLons We can call hose firs few columns of U he Principal Componens of he daa The show he major paierns ha can be added o produce he columns of he original mari The rows of V T show how he principal componens are mied o produce he columns of he mari 66

67 SVD ApplicaLons We can look a Σ o see ha he firs column has a large effec while he second column has a much smaller effec in his eample 67

68 SVD ApplicaLons For his image, using onl he firs of 3 principal componens produces a recognizable reconsruclon So, SVD can be used for image compression 68

69 Principal Componen Analsis Remember, columns of U are he Principal Componens of he daa: he major paierns ha can be added o produce he columns of he original mari One use of his is o consruc a mari where each column is a separae daa sample Run SVD on ha mari, and look a he firs few columns of U o see paierns ha are common among he columns This is called Principal Componen Analsis (or PCA) of he daa samples 69

70 Principal Componen Analsis Ogen, raw daa samples have a lo of redundanc and paierns PCA can allow ou o represen daa samples as weighs on he principal componens, raher han using he original raw form of he daa B represenlng each sample as jus hose weighs, ou can represen jus he mea of wha s differen beween samples. This minimal represenalon makes machine learning and oher algorihms much more efficien 7

71 Ouline Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm Compuers can compue SVD ver quickl. We ll briefl discuss he algorihm, for hose who are ineresed. 7

72 Addendum: How is SVD compued? For his class: ell MATLAB o do i. Use he resul. Bu, if ou re ineresed, one compuer algorihm o do i makes use of Eigenvecors The following maerial is presened o make SVD less of a magical black bo. Bu ou will do fine in his class if ou rea SVD as a magical black bo, as long as ou remember is properles from he previous slides. 72

73 Eigenvecor definilon Suppose we have a square mari A. We can solve for vecor and scalar λ such ha A λ In oher words, find vecors where, if we ransform hem wih A, he onl effec is o scale hem wih no change in direclon. These vecors are called eigenvecors (German for self vecor of he mari), and he scaling facors λ are called eigenvalues An m m mari will have m eigenvecors where λ is nonzero 73

74 Finding eigenvecors Compuers can find an such ha A λ using his ieralve algorihm: random uni vecor while( hasn converged) A normalize will quickl converge o an eigenvecor Some simple modificalons will le his algorihm find all eigenvecors 74

75 Finding SVD Eigenvecors are for square marices, bu SVD is for all marices To do svd(a), compuers can do his: Take eigenvecors of AA T (mari is alwas square). These eigenvecors are he columns of U. Square roo of eigenvalues are he singular values (he enries of Σ). Take eigenvecors of A T A (mari is alwas square). These eigenvecors are columns of V (or rows of V T ) 75

76 Finding SVD Moral of he sor: SVD is fas, even for large marices I s useful for a lo of suff There are also oher algorihms o compue SVD or par of he SVD MATLAB s svd() command has oplons o efficienl compue onl wha ou need, if performance becomes an issue A deailed geomeric eplanalon of SVD is here: hip:// column/fcarc- svd 76

77 Wha we have learned Vecors and marices Basic Mari OperaLons Special Marices TransformaLon Marices Homogeneous coordinaes TranslaLon Mari inverse Mari rank Singular Value DecomposiLon (SVD) Use for image compression Use for Principal Componen Analsis (PCA) Compuer algorihm 77