Why Reduce Dimensionality? Feature Selection vs Extraction. Subset Selection

Transcription

1 Dimenionality Reduction Why Reduce Dimenionality? Olive lide: Alpaydin Numbeed blue lide: Haykin, Neual Netwok: A Compehenive Foundation, Second edition, Pentice-Hall, Uppe Saddle Rive:NJ,. Black lide: eta content. Reduce time compleity: Le computation Reduce pace compleity: Fewe paamete Save the cot of obeving the featue Simple model ae moe obut on mall dataet Moe intepetable; imple eplanation Data viualization (tuctue, goup, outlie, etc) if plotted in o dimenion Featue Selection v Etaction Subet Selection Featue election: Chooing k<d impotant featue, ignoing the emaining d k Subet election algoithm Featue etaction: Poject the oiginal i, i =,...,d dimenion to new k<d dimenion, z j, j =,...,k hee ae d ubet of d featue Fowad each: Add the bet featue at each tep Set of featue F initially Ø. At each iteation, find the bet new featue j = agmin i E ( F i ) Add j to F if E ( F j ) < E ( F ) Hill-climbing O(d ) algoithm Backwad each: Stat with all featue and emove one at a time, if poible. Floating each (Add k, emove l)

2 Pincipal Component Analyi (PCA) Note: Q mean eigenvecto mati of the covaiance mati, in Haykin lide..... Motivation cloud.dat How can we poject the given data o that the vaiance in the pojected point i maimized? Eigenvalue/Eigenvecto Fo a quae mati A, if a vecto and a cala value λ eit o that (A λi) = Eigenvecto/Eigenvalue Eample then i called an eigenvecto of A and λ an eigenvalue. Note, the above i imply A = λ An intuitive meaning i: i the diection in which applying the linea tanfomation A only change the magnitude of (by λ) but not the angle. hee can be a many a n eigenvecto/eigenvalue fo an n n mati. Red: oiginal data Geen: pojected data uing A = Blue: Eigenvecto v =(.,.), v =(-.,.), λ =., λ =.. Octave/Matlab code: [V,Lamba]=eig(A) Magenta: A time eigenvecto..

3 Red: oiginal data Eigenvecto/Eigenvalue Eample Geen: pojected data uing A = Blue: Eigenvecto; Magenta: A time eigenvecto. A i a ymmetic mati, o eigenvecto ae othogonal.. Pincipal Component Analyi Find a low-dimenional pace uch that when i pojected thee, infomation lo i minimized. he pojection of on the diection of w i: z = w Find w uch that Va(z) i maimized Va(z) = Va(w ) = E[(w w μ) ] = E[(w w μ)(w w μ)] = E[w ( μ)( μ) w] = w E[( μ)( μ) ]w = w w whee Va()= E[( μ)( μ) ] = Maimize Va(z) ubject to w = maw w w w = αw that i, w i an eigenvecto of Chooe the one with the laget eigenvalue fo Va(z) to be ma Second pincipal component: Ma Va(z ),.t., w = and othogonal to w maw w w w = α w that i, w i anothe eigenvecto of and o on. w w w w w w What PCA doe z = W ( m) whee the column of W ae the eigenvecto of and m i ample mean Cente the data at the oigin and otate the ae

4 How to chooe k? Popotion of Vaiance (PoV) eplained k k d when λ i ae oted in decending ode ypically, top at PoV>. Scee gaph plot of PoV v k, top at elbow PCA: Uage Poject input to the pincipal diection: a = Q. We can alo ecove the input fom the pojected point a: = (Q ) a = Qa. Note that we don t need all m pincipal diection, depending on how much vaiance i captued in the fit few eigenvalue: We can do dimenionality eduction.

5 PCA: Dimenionality Reduction Encoding: We can ue the fit l eigenvecto to encode. [a, a,..., a l ] = [q, q,..., q l ]. Note that we only need to calculate l pojection a, a,..., a l, whee l m. Decoding: Once [a, a,..., a l ] i obtained, we want to econtuct the full [,,..., l,..., m ]. = Qa [q, q,..., q l ][a, a,..., a l ] = ˆ. O, altenatively ˆ = Q[a, a,..., a l,,,..., ] }{{}. m l zeo PCA: otal Vaiance he total vaiance of th em component of the data vecto i m σ j = m j= j= λ j. he tuncated veion with the fit l component have vaiance l σ j = j= j= he lage the vaiance in the tuncated veion, i.e., the malle the vaiance in the emaining component, the moe accuate the dimenionality eduction. l λ j PCA Eample line line line line inp=[andn(,)/+.;andn(,)/+one(,)]; Q = λ = [ [ ] ] Facto Analyi Find a mall numbe of facto z, which when combined geneate : i µ i = v i z + v i z v ik z k + ε i whee z j, j =,...,k ae the latent facto with E[ z j ]=, Va(z j )=, Cov(z i,, z j )=, i j, ε i ae the noie ouce E[ ε i ]= ψ i, Cov(ε i, ε j ) =, i j, Cov(ε i, z j ) =, and v ij ae the facto loading

6 PCA v FA Facto Analyi PCA Fom to z z = W ( µ) FA Fom z to µ = Vz + ε In FA, facto z j ae tetched, otated and tanlated to geneate z z Singula Value Decompoition and Mati Factoization Singula value decompoition: X=VAW V i NN and contain the eigenvecto of XX W i dd and contain the eigenvecto of X X and A i Nd and contain ingula value on it fit k diagonal X=u a v +...+u k a k v k whee k i the ank of X Multidimenional Scaling Given paiwie ditance between N point, d ij, i,j =,...,N place on a low-dim map.t. ditance ae peeved (by featue embedding) z = g ( θ ) E X,, Find θ that min Sammon te z g z g

7 Map of Euope by MDS Manifold La H. Rohwedde, Wikimedia Common A topological pace that i locally Euclidean (flat, not cuved). Dimenionality of the manifold = dimenionality of the Euclidean pace it eemble, locally. Staight line, wiggly cuve, etc. ae D manifold. Map fom CIA he Wold Factbook: Flat plane, uface of phee, etc. ae D manifold. Detecting cuvatue of pace: um of intenal angle of tiangle = o? Manifold Leaning Iomap Geodeic ditance i the ditance along the manifold that the data lie in, a oppoed to the Euclidean ditance in the input pace A: D manifold embedded in D embedding pace. B: Data point etaced fom A. C: Recoveed D tuctue. ak: ecove C fom B, without knowledge of A.

8 Geodeic Ditance Geodeic ditance = Shotet path. A: Manifold with two point. B: Euclidean ditance between the two point. C: Geodeic ditance between the two point. Iomap Intance and ae connected in the gaph if - <e o if i one of the k neighbo of he edge length i - Fo two node and not connected, the ditance i equal to the hotet path between them Once the NN ditance mati i thu fomed, ue MDS to find a lowe-dimenional mapping Optdigit afte Iomap (with neighbohood gaph). Locally Linea Embedding Matlab ouce fom Given find it neighbo (). Find W that minimize E ( W X) W( ). Find the new coodinate z that minimize E ( z W) z Wz( )

9 LLE on Optdigit Matlab ouce fom Refeence