3D Geometry for Computer Graphcs Lesso 2: PCA & SVD
Last week - egedecomposto We wat to lear how the matrx A works: A 2
Last week - egedecomposto If we look at arbtrary vectors, t does t tell us much. A 3
Spectra ad dagoalzato If A s symmetrc, the egevectors are orthogoal (ad there s always a egebass). A A = UΛU λ1 λ2 Λ= Au = λ u λ 4
Why SVD Dagoalzable matrx s essetally a scalg. Most matrces are ot dagoalzable they do other thgs alog wth scalg (such as rotato) So, to uderstad how geeral matrces behave, oly egevalues are ot eough SVD tells us how geeral lear trasformatos behave, ad other thgs 5
he pla for today Frst we ll see some applcatos of PCA Prcpal Compoet Aalyss that uses spectral decomposto. he look at the theory. SVD Basc tuto Formal defto Applcatos 6
PCA the geeral dea PCA fds a orthogoal bass that best represets gve data set. y y x x he sum of dstaces 2 from the x axs s mmzed. 7
PCA the geeral dea PCA fds a orthogoal bass that best represets gve data set. z 3D pot set stadard bass x y PCA fds a best approxmatg plae (aga, terms of Σdstaces 2 ) 8
Applcato: fdg tght boudg box A axs-alged boudg box: agrees wth the axes y maxy mx maxx x my 9
Applcato: fdg tght boudg box Oreted boudg box: we fd better axes! y x 10
Applcato: fdg tght boudg box hs s ot the optmal boudg box z x y 11
Applcato: fdg tght boudg box Oreted boudg box: we fd better axes! 12
Usage of boudg boxes (boudg volumes) Serve as very smple approxmato of the object Fast collso detecto, vsblty queres Wheever we eed to kow the dmesos (sze) of the object he models cosst of thousads of polygos o quckly test that they do t tersect, the boudg boxes are tested Sometmes a herarchy of BB s s used he tghter the BB the less false alarms we have Sample 13
Scaed meshes 14
Pot clouds Scaers gve us raw pot cloud data How to compute ormals to shade the surface? ormal 15
Pot clouds Local PCA, take the thrd vector 16
Notatos Deote our data pots by x 1, x 2,, x R d 1 1 1 x 1 x 2 x 2 2 2 x1 x2 x x,,, 1 = x = = 2 x d d d x 1 x 2 x 17
he org of the ew axes he org s zero-order approxmato of our data set (a pot) It wll be the ceter of mass: m It ca be show that: = 1 = 1 x m = argm x -x x = 1 2 18
Scatter matrx Deote y = x m, = 1, 2,, S = YY where Y s d matrx wth y k as colums (k = 1, 2,, ) S 1 1 1 1 2 d y1 y2 y y1 y1 y 1 2 2 2 1 2 d y1 y2 y y2 y2 y2 = d d d 1 2 d y1 y2 y y y y Y Y 19
Varace of projected pots I a way, S measures varace (= scatteress) of the data dfferet drectos. Let s look at a le L through the ceter of mass m, ad project our pots x oto t. he varace of the projected pots x s: 1 2 var( L) = x m = 1 L L L L Orgal set Small varace Large varace 20
Varace of projected pots Gve a drecto v, v = 1, the projecto of x oto L = m + vt s: x m = < v, x m> / v = < v, y > = v y x L m v x 21
Varace of projected pots So, 1 2 1 2 1 2 x-m vy v Y = 1 = 1 var( L) = = ( ) = = = Y v = < Y v, Y v>= v YY v= v Sv= < Sv, v> 1 2 1 1 1 1 2 2 1 1 1 1 y y1 y2 y 2 2 2 2 2 1 2 d y 1 2 d y1 y2 y 2 ( ) ( v v v ) ( v v v ) vy = = = vy = 1 = 1 d d d d y y1 y2 y 22
Drectos of maxmal varace So, we have: var(l) = <Sv, v> heorem: Let f : {v R d v = 1} R, f (v) = <Sv, v> (ad S s a symmetrc matrx). he, the extrema of f are attaed at the egevectors of S. So, egevectors of S are drectos of maxmal/mmal varace! 23
Summary so far We take the cetered data vectors y 1, y 2,, y R d Costruct the scatter matrx S = YY S measures the varace of the data pots Egevectors of S are drectos of maxmal varace. 24
Scatter matrx - egedecomposto S s symmetrc S has egedecomposto: S = VΛV λ 1λ2 v 1 S = v 1 v 2 v d v 2 λ d v d he egevectors form orthogoal bass 25
Prcpal compoets Egevectors that correspod to bg egevalues are the drectos whch the data has strog compoets (= large varace). If the egevalues are more or less the same there s o preferable drecto. Note: the egevalues are always o-egatve. 26
Prcpal compoets here s o preferable drecto S looks lke ths: V λ V λ Ay vector s a egevector here s a clear preferable drecto S looks lke ths: λ V V μ μ s close to zero, much smaller tha λ. 27
How to use what we got For fdg oreted boudg box we smply compute the boudg box wth respect to the axes defed by the egevectors. he org s at the mea pot m. v 3 v 1 v 2 28
For approxmato y v 2 v 1 x y y x x hs le segmet approxmates the orgal data set he projected data set approxmates the orgal data set 29
For approxmato I geeral dmeso d, the egevalues are sorted descedg order: λ 1 λ 2 λ d he egevectors are sorted accordgly. o get a approxmato of dmeso d < d, we take the d frst egevectors ad look at the subspace they spa (d = 1 s a le, d = 2 s a plae ) 30
For approxmato o get a approxmatg set, we project the orgal data pots oto the chose subspace: x = m + α 1 v 1 + α 2 v 2 + + α d v d + +α d v d Projecto: x = m + α 1 v 1 + α 2 v 2 + + α d v d +0 v d +1 + + 0 v d 31
SVD
Geometrc aalyss of lear trasformatos We wat to kow what a lear trasformato A does Need some smple ad comprehedble represetato of the matrx of A. Let s look what A does to some vectors Sce A(αv) = αa(v), t s eough to look at vectors v of ut legth A 33
he geometry of lear trasformatos A lear (o-sgular) trasform A always takes hyper-spheres to hyper-ellpses. A A 34
he geometry of lear trasformatos hus, oe good way to uderstad what A does s to fd whch vectors are mapped to the ma axes of the ellpsod. A A 35
Geometrc aalyss of lear trasformatos If we are lucky: A = V Λ V, V orthogoal (true f A s symmetrc) he egevectors of A are the axes of the ellpse A 36
37 Symmetrc matrx: ege decomposto I ths case A s just a scalg matrx. he ege decomposto of A tells us whch orthogoal axes t scales, ad by how much: [ ] [ ] 1 2 1 2 1 2 A λ λ λ = v v v v v v A 1 1 λ 2 λ 1 A v v λ =
Geeral lear trasformatos: SVD I geeral A wll also cota rotatos, ot just scales: 1 1 A σ 2 σ 1 A = U V A = σ 1 σ 2 [ u u u ] [ v v v ] σ 1 2 1 2 38
Geeral lear trasformatos: SVD 1 1 A σ 2 σ 1 AV = U A orthoormal [ v v v ] = [ u u u ] 1 2 1 2 orthoormal σ 1 σ 2 σ A v = σ u, σ 0 39
SVD more formally SVD exsts for ay matrx Formal defto: For square matrces A R, there exst orthogoal matrces U, V R ad a dagoal matrx Σ, such that all the dagoal values σ of Σ are o-egatve ad A = UΣV = A U Σ V 40
SVD more formally he dagoal values of Σ (σ 1,, σ ) are called the sgular values. It s accustomed to sort them: σ 1 σ 2 σ he colums of U (u 1,, u ) are called the left sgular vectors. hey are the axes of the ellpsod. he colums of V (v 1,, v ) are called the rght sgular vectors. hey are the premages of the axes of the ellpsod. A = UΣV = A U Σ V 41
Reduced SVD For rectagular matrces, we have two forms of SVD. he reduced SVD looks lke ths: he colums of U are orthoormal Cheaper form for computato ad storage = A U Σ V 42
Full SVD We ca complete U to a full orthogoal matrx ad pad Σ by zeros accordgly = A U Σ V 43
Some hstory SVD was dscovered by the followg people: E. Beltram (1835 1900) M. Jorda (1838 1922) J. Sylvester (1814 1897) E. Schmdt (1876-1959) H. Weyl (1885-1955) 44
SVD s the workg horse of lear algebra here are umercal algorthms to compute SVD. Oce you have t, you have may thgs: Matrx verse ca solve square lear systems Numercal rak of a matrx Ca solve least-squares systems PCA May more 45
Matrx verse ad solvg lear systems Matrx verse: So, to solve Ax= x A = U V ( ) ( ) 1 1 1 1 1 A = U V = V U = = V 1 σ 1 = V U 1 σ b U 1 b 46
Matrx rak he rak of A s the umber of o-zero sgular values σ 1σ2 m = σ A U Σ V 47
Numercal rak If there are very small sgular values, the A s close to beg sgular. We ca set a threshold t, so that umerc_rak(a) = #{σ σ > t} If rak(a) < the A s sgular. It maps the etre space R oto some subspace, lke a plae (so A s some sort of projecto). 48
Back to PCA We wated to fd prcpal compoets y y x x 49
PCA Move the ceter of mass to the org p = p m y y x x 50
PCA Costructed the matrx X of the data pots. X p p p = 1 2 he prcpal axes are egevectors of S = XX λ 1 XX = S = U U λ d 51
PCA We ca compute the prcpal compoets by SVD of X: X = UΣV XX = UΣV ( UΣ V ) = = Σ Σ = Σ 2 U V V U U U hus, the left sgular vectors of X are the prcpal compoets! We sort them by the sze of the sgular values of X. 52
Shape matchg We have two objects correspodece Wat to fd the rgd trasformato that algs them 53
Shape matchg Whe the objects are alged, the legths of the coectg les are small. 54
Shape matchg formalzato Alg two pot sets P = p,, p ad Q= q,, q. { } { } 1 1 Fd a traslato vector t ad rotato matrx R so that: p = 1 2 ( Rq + t) s mmzed 55
Shape matchg soluto urs out we ca solve the traslato ad rotato separately. heorem: f (R, t) s the optmal trasformato, the the pots {p } ad {Rq + t} have the same ceters of mass. p 1 p = 1 = q 1 q = 1 = p 1 = + = 1 ( Rq t ) 1 p = R q + = R + t q t = 1 t = p Rq 56
Fdg the rotato R o fd the optmal R, we brg the cetrods of both pot sets to the org: p = p p q = q q We wat to fd R that mmzes p Rq = 1 2 57
Fdg the rotato R 2 ( ) ( ) = 1 = 1 = 1 p Rq = p Rq p Rq = ( p ) p p Rq q R p q R Rq = + I hese terms do ot deped o R, so we ca gore them the mmzato 58
Fdg the rotato R ( ) ( p ) Rq q R p = p Rq + q R p m max. = 1 = 1 R = R = R ( ) q p q p p q ( ) ( p ) Rq p Rq max 2 = max = 1 = 1 ths s a scalar 59
Fdg the rotato R p q qp qp = 1 = 1 = 1 ( R ) = race R = race R H = = 1 qp = 3 1 1 3 = 3 3 race( A) = = 1 A 60
Fdg the rotato R So, we wat to fd R that maxmzes race( RH ) heorem: f M s symmetrc postve defte (all egevalues of M are postve) ad B s ay orthogoal matrx the race M race BM ( ) ( ) So, let s fd R so that RH s symmetrc postve defte. he we kow for sure that race(rh) s maxmal. 61
Fdg the rotato R hs s easy! Compute SVD of H: Defe R: Check RH: H = UΣV R = VU ( )( ) RH = VU UΣ V = VΣV hs s a symmetrc matrx, Its egevalues are σ 0 So RH s postve! 62
Summary of rgd algmet: raslate the put pots to the cetrods: Compute the covarace matrx Compute the SVD of H: he optmal rotato s he traslato vector s p = p p = q q H = qp = 1 H = UΣV R = VU t = p Rq q 63
Complexty Numercal SVD s a expesve operato We always eed to pay atteto to the dmesos of the matrx we re applyg SVD to. 64
Small somewhat related example 65
he Ed