Unsupervised Learning: Dimension Reduction

Transcription

1 Unsupervised Learning: Diension Reduction by Prof. Seungchul Lee isystes Design Lab UNIST Table of Contents I.. Principal Coponent Analysis (PCA) II. 2. PCA Algorith I. 2.. Pre-processing II Maxiize variance III Miniize the su-of-squared error IV Diension Reduction ethod (n k) III. 3. Python Codes IV. 4. PCA Exaple V. 5. PCA Visualization I. 5.. Eigenvalues II Eigenvectors. Principal Coponent Analysis (PCA) Motivation: Can we describe high-diensional data in a "sipler" way? Diension reduction without losing too uch inforation Find a low-diensional, yet useful representation of the data Why diensionality reduction? insights into the low-diensinal structures in the data (visualization) Fewer diensions Less chances of overfitting Better generalization Speeding up learning algoriths Most algoriths scale badly with increasing data diensionality Less storage requireents (data copression) Note: Diensionality Reduction is different fro Feature Selection.. although the goals are kind of the sae Diensionality reduction is ore like Feature Extraction Constructing a sall set of new features fro the original features

2 How? idea: highly correlated data contains redundant features x {, } - Each exaple has 2 features x x 2 - Consider ignoring the feature x 2 for each exaple - Each 2-diensional exaple x now becoes -diensional x = { x } - Are we losing uch inforation by throwing away x 2? - No. Most of the data spread is along x (very little variance along x 2 ) x {, } - Each exaple has 2 features x x 2 - Consider ignoring the feature x 2 for each exaple - Each 2-diensional exaple x now becoes -diensional x = { x } - Are we losing uch inforation by throwing away x 2? - Yes, the data has substantial variance along both features (_i.e._ both axes) - Now consider a change of axes - Each exaple x has 2 features { u, u 2 } - Consider ignoring the feature u 2 for each exaple - Each 2-diensional exaple x now becoes -diensional x = { u } - No. Most of the data spread is along u (very little variance along u 2 )

3 Data projection onto unit vector u^ PCA is used when we want projections capturing axiu variance directions Principal Coponents (PC): directions of axiu variability in the data Roughly speaking, PCA does a change of axes that can represent the data in a succinct anner HOW?. Maxiize variance (ost separable) 2. Miniize the su-of-squares (iniu squared error) 2. PCA Algorith

4 2.. Pre-processing Given data x (i) =, X = x (i) x (i) n (x () ) T (x (2) ) T (x () ) T Shifting (zero ean) and rescaling (unit variance). Shift to zero ean μ = i= x (i) x (i) x (i) μ (zero ean) 2. [optional] Rescaling (unit variance) σ 2 j = i= ( x ) (i) j 2 x (i) j x(i) j σ j 2.2. Maxiize variance Find unit vector u such that axiizes variance of projections for big data variance of projected data = ( u T x (i) ) 2 = ( x (i)t u) 2 i= i= = ( x (i)t u ) T ( u) = x (i)t u T x (i) x (i)t u i= i= Note: = u T ( x (i) x (i)t ) u i= = u T Su (S = X : saple covariance atrix) XT In an optiization for axiize subject to u T Su u T u = u T Su = u T λu = λu T u = λ (Eigen analysis : Su = λu) pick the largest eigenvalue λ of covariance atrix S u = u is the λ s corresponding eigenvector u is the first principal coponent (direction of highest variance in the data)

5 2.3. Miniize the su-of-squared error e (i) 2 = x (i) 2 ( x (i)t u) 2 = x (i) 2 ( x (i)t u ) T ( x (i)t u) = x (i) 2 u T x (i) x (i)t u i= e (i) 2 = x (i) 2 u T x (i) x (i)t u i= i= = x (i) 2 u T ( x (i) x (i)t ) u i= i= In an optiization for iniize x (i) 2 i= constant given x i u T i= u T ( x (i) x (i)t ) u i= axiize axiize ( x (i) x (i)t ) u = ax u T Su iniize erro r 2 = axiize variance

6 2.4. Diension Reduction ethod ( n k). Choose top (orthonoral) eigenvectors, u u 2 2. Project x i onto span { u, u 2,, u k } u T x(i) k U = [,,, ] z (i) Pictorial suary of PCA u T 2 x(i) u k = or z = U T x u T k x(i) x (i) projection onto unit vector u u T x (i) = distance fro the origin along u 3. Python Codes In [3]: iport nupy as np iport atplotlib.pyplot as plt fro pl_toolkits.plot3d iport axes3d %atplotlib inline

7 In [2]: # data generation = 5000 u = np.array([0, 0]) siga = np.array([[3,.5], [.5, ]]) X = np.rando.ultivariate_noral(u, siga, ) X = np.asatrix(x) fig = plt.figure(figsize=(0, 6)) plt.plot(x[:,0], X[:,], 'k.') plt.axis('equal') plt.show() In [3]: S = /(-)*X.T*X S = np.asatrix(s) D, V = np.linalg.eig(s) idx = np.argsort(-d) D = D[idx] V = V[:,idx] print(d) print(v) [ ] [[ ] [ ]]

8 In [4]: h = V[,0]/V[0,0] xp = np.arange(-6, 6, 0.) yp = h*xp fig = plt.figure(figsize=(0, 6)) plt.plot(x[:,0], X[:,], 'k.') plt.plot(xp, yp, 'r', linewidth=4.0) plt.axis('equal') plt.show()

9 In [5]: Z = X*V[:,] plt.figure(figsize=(0, 6)) plt.hist(z, 5) plt.show() 4. PCA Exaple ultiple video caera records of spring and ass syste source: (

10 In [6]: %%htl <center><ifrae width="560" height="35" src=" fraeborder ="0" allowfullscreen> </ifrae></center> 3 cut In [7]: %%htl <center><ifrae width="560" height="35" src=" fraeborder ="0" allowfullscreen> </ifrae></center> 4 cut

11 In [8]: %%htl <center><ifrae width="560" height="35" src=" fraeborder ="0" allowfullscreen> </ifrae></center> 5 cut x (i) x in caera y in caera x in caera 2 =, X = y in caera 2 x in caera 3 y in caera 3 ( x () ) ( x (2) ) ( x () ) In [4]: fro six.oves iport cpickle X = cpickle.load(open('./data_files/pca_spring.pkl','rb')) X = np.asatrix(x.t) print(x.shape) = X.shape[0] (273, 6)

12 In [5]: plt.figure(figsize=(5, 7)) plt.subplot(3) plt.plot(x[:,0], -X[:,],'r') plt.axis('equal') plt.title('caera ') plt.subplot(32) plt.plot(x[:,2], -X[:,3],'b') plt.axis('equal') plt.title('caera 2') plt.subplot(33) plt.plot(x[:,4], -X[:,5],'k') plt.axis('equal') plt.title('caera 3') plt.show()

13 In [6]: X = X - np.ean(x,axis=0) S = /(-)*X.T*X S = np.asatrix(s) D, V = np.linalg.eig(s) idx = np.argsort(-d) D = D[idx] V = V[:,idx] print(d) print(v) [ e e e e e e+00] [[ ] [ ] [ ] [ ] [ ] [ ]] In [7]: plt.figure(figsize=(0,6)) plt.ste(np.sqrt(d)) plt.show()

14 In [9]: # relative agnitutes of the principal coponents Z = X*V xp = np.arange(0,)/24 plt.figure(figsize=(0, 6)) plt.plot(xp, Z) plt.yticks([]) plt.show()

15 In [0]: ## projected onto the first principal coponent # 6 di -> di (di reduction) # relative agnitute of the first principal coponent Z = X*V[:,0] plt.figure(figsize=(0, 6)) plt.plot(z) plt.yticks([]) plt.show() Reference: John P Cunningha & Byron M Yu, Diensionality reduction for large-scale neural recordings, Nature Neuroscience 7, (204)

16 5. PCA Visualization 5.. Eigenvalues, λ λ 2 indicates variance along the eigenvectors, respectively. The larger eigenvalue is, the ore doinante feature (eigenvector) is 5.2. Eigenvectors given basis {, } x^ x^2 to transfored basis { u^, u^2 } [ ] = [ ] [ ] u^ u^2 x^ x^2 c c 3 c 2 c 4 In [5]: %%javascript $.getscript(' c.js')