Kernel-Based Principal Component Analysis (KPCA) and Its Applications. Nonlinear PCA

Transcription

1 Kernel-Based Principal Component Analysis (KPCA) and Its Applications 4//009 Based on slides originaly from Dr. John Tan 1 Nonlinear PCA Natural phenomena are usually nonlinear and standard PCA is intrinsically a linear technique. Nonlinear PCA Principal Curves Nonlinear PCA by Neural Network Kernel PCA 4//009 Based on slides originaly from Dr. John Tan 1

2 Principal Curves (PCS) Trevor Hastie; Werner Stuetzle, Principal Curves, Journal of the American Statistical Association, Vol. 84, No (Jun. 1989), pp //009 Based on slides originaly from Dr. John Tan 3 William W. Hsieh and Benyang Tang. 1998: Applying Neural Network Models to Prediction and Data Analysis in Meteorology and Oceanography. Bulletin of the American Meteorological Society: Vol. 79, No. 9, pp NLPCA-NN 4//009 Based on slides originaly from Dr. John Tan 4

3 Limitation on PCS and NNPCA Small input dimensions NNPCA used PCA to reduce the dimensions of spatio-temporal data 4//009 Based on slides originaly from Dr. John Tan 5 Kernel PCA B. Scholkopf, A. Smola, and K.-R. Muller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10: , input space y X Φ F feature space v y Nonlinear map from input space to a richer feature space PCA in the feature space Preimage: map back to the input space 4//009 Based on slides originaly from Dr. John Tan 6 3

4 Major Steps Original spatio-temporal data ψ(,t): M points (dimensions); N observations; N M-dimensional vectors Nonlinear Mapping: An M-dimensional vector is transformed into a M F -dimensional vector 4//009 Based on slides originaly from Dr. John Tan 7 Feature Space Data Matri Matri for the eigenvalue problem: 4//009 Based on slides originaly from Dr. John Tan 8 4

5 Mapping Compleity Dot products for covariance matri element: K + = ( 1, ) (, y) = ( y) = ( 1 y1 y Dimension numbers may be high: For eample, the polynomial kernel with d = More General case: K (, y) + ) Φ( ) = ( 1,, 1 ) d = ( y c d = positive integer, c is constant Φ( ) = (1, 1,, 1,, 1 ) Compleity for the polynomial kernel is ~M d. Where N is the dimensionality of the input space. For eample, the polynomial kernel with d = 5 and a 1616 piel image (M = 56) would yield a dimensionality of 10 10! 4//009 Based on slides originaly from Dr. John Tan 9 ) Kernel Trick Elements of matri K is evaluated by a kernel function. Since Kernel is defined to a vector in the input space, we can compute elements of D Φ D Φ but not D Φ D Φ. Therefore, the size of K is always N, the total number of observations. 4//009 Based on slides originaly from Dr. John Tan 10 5

6 What is a valid kernel? Mercer s Theorem Any (semi) positive definite, symmetric function is a Kernel. K corresponds to the dot product of two mapped data points in the feature space Matri generated from K is symmetric and positive (semi) definite. K (, y) = ( Φ( ) Φ( y)) = f (, y) Linear Kernel K(, y) = y Gaussian RBF Kernel K y σ (, y) = e 4//009 Based on slides originaly from Dr. John Tan 11 Kernels New kernels can be constructed from eisting kernels. The set of Kernels is closed under some operations. Given the following properties, eisting Kernels can be used to generate new ones. Assuming that K1, K are valid Kernels: K1 + K is a Kernel C*K1 is a Kernel for C > 0 A*K1 + B*K is a Kernel for A, B > 0 There are more properties that can be eploited, see Schölkopf and Smola (00) Comple Kernels can be generated from known ones and these can be used to generate additional Kernels. 4//009 Based on slides originaly from Dr. John Tan 1 6

7 Physical Components Eigenvalue problem: and α i (i=1,, N) representing the time series (temporal components) Spatial pattern: (Spatial Component) 4//009 Based on slides originaly from Dr. John Tan 13 New Issues in Earth Science Applications Preimage: transformation back to the input space Mapping may be epensive, it can be highdimensional Mapping to this space is performed using a Mercer s Kernel Higher dimensionalities in feature space More comparable eigenvalues Variance is not conserved A new algorithm for pattern selection 4//009 Based on slides originaly from Dr. John Tan 14 7

8 Preimage Problem The mapping from X to F. The preimage problem is illustrated by the point Ψ which has no corollary in X. Required for earth science applications. Spatial patterns. 4//009 Based on slides originaly from Dr. John Tan 15 ( Ψ Φ( ) ρ1( = Φ( Φ( Preimages The mapping from input space to feature space is performed implicitly through the use of a kernel. This makes the calculation of the inverse transform, the preimage, interesting. The preimage in many cases may not eist however a good approimate preimage can be calculated [SMB99]. * ρ ( = Ψ Φ( ρ 1 ( Ψ ρ ( Φ(X ) ( Ψ Φ( ) ( Ψ Φ( ) * Φ( Ψ = Ψ Φ( Φ( Φ( Φ( F 4//009 Based on slides originaly from Dr. John Tan 16 8

9 K (, y) = ( y) PCA Eamples Data: Lorenz Attractor K y σ (, y) = e KPCA: Gaussian kernel P α e i i= 1 z n+ 1 = P i= 1 α i zn σ i zn σ ie i 4//009 Based on slides originaly from Dr. John Tan 17 KPCA for Spatial-Temporal Data New methodology for KPCA Determine (spatial) patterns correlated with a (temporal) signal Recall α represents the temporal principal components and is related to the implicitly defined spatial components v by: v I p = I Φ α ν ν p Mλ p 4//009 Based on slides originaly from Dr. John Tan 18 9

10 Finding the Temporal Signal The temporal principal components α can be combined to determine a direction set that will correlate better with the signal of interest. Instead of searching for the underlying source of a principal component direction, the source is specified and a preferred direction is determined based on the principal components. -This allows us to specify the corresponding spatial components v. 4//009 Based on slides originaly from Dr. John Tan 19 Procedures in PCA & KPCA Standard PCA Analysis Eamine principal components that contain the highest variance for inherent low dimensional structure Correlate temporal components (loadings) with signal of interest Determine component with highest correlation Eamine associated spatial principal component for new patterns KPCA Analysis for Spatial-Temporal Data Use α vectors as temporal components of mapped data Determine temporal components and correlate with signal of interest Determine set of α vectors that have highest correlation with signal of interest Calculate preimage of the associated spatial components and eamine for new patterns Linear combination of α vectors Sort vectors from highest to lowest wrt correlation score Combine vectors in descending order, keeping vectors that increase the score 4//009 Based on slides originaly from Dr. John Tan 0 10

11 Selection Algorithm 4//009 Based on slides originaly from Dr. John Tan 1 Signal Detection piel_value = 80*noise + 0*signal and at specific regions piel_value = 100*noise otherwise 4//009 Based on slides originaly from Dr. John Tan 11

12 Data Sets Normalized Difference Vegetation Inde (NDVI) NDVI = (near IR band - red band) / (near IR band + red band) NASA GES Distributed Active Archive Center 1 o 1 o Level 3 data set covering the globe January 198 to December 001 Sea Surface Temperature (SST) International Comprehensive Ocean-Atmosphere Data Set (ICOADS). Latitude 18 o S to 18 o N and Longitude 10 o E to 75 o W, at o o resolution. January 1951 to December 004 Southern Oscillation Inde (SOI) NOAA National Weather Service, Climate Prediction Center. January 1951-December 004 4//009 Based on slides originaly from Dr. John Tan 3 Masking NDVI NDVI Red - valid, Blue - invalid SST 1951 SST: all available Red - invalid, Blue-valid 4//009 Based on slides originaly from Dr. John Tan 4 1

13 Sea Surface Temperature Anomaly I Positive anomaly around , , , , and These correspond to El Nino years. The two regions of highest positive anomaly and are also strong El Nino years. raw deseasoned 4//009 Based on slides originaly from Dr. John Tan 5 piels month Global NDVI Anomaly I raw Like SSTA the deseasoned NDVI data has anomaly around the El Nino years. However the anomaly patterns are less distinct, SST has a direct link with El Nino. Vegetation has an indirect connection. deseasoned -Jan 88(73) -Jan 86(49) -Jan 84(5) -Jan 8(1) -Sep 01(33) -Jan 00(13) -Jan 98(189) -Jan 96(165) -Jan 94(145) -Jan 9(11) -Jan 90(97) 4//009 Based on slides originaly from * , Dr. John , Tan , , and

14 A Specific Eample Data Sets Normalized Difference Vegetation Inde (NDVI) NDVI = (near IR band - red band) / (near IR band + red band) NASA GES Distributed Active Archive Center 1 o 1 o Level 3 data set covering the globe January 198 to December 001 Southern Oscillation Inde (SOI) NOAA National Weather Service, Climate Prediction Center. January 1951-December 004 Gaussian Kernel: y σ = 6% of SD σ (Empirical optimal value) K(, y) = e 4//009 Based on slides originaly from Dr. John Tan 7 Results (PCA, One Component) Spatial pattern from PC 4 of standard PCA. (r = Percent of variance eplained by PC 4 = 3.8%) 4//009 Based on slides originaly from Dr. John Tan 8 14

15 Results (Combined KPCA) Combined spatial pattern from KPCA (r = 0.68 with SOI). 4//009 Based on slides originaly from Dr. John Tan 9 Results (Combined PCA) Combined spatial pattern from Standard PCA (r = 0.56 with SOI). 4//009 Based on slides originaly from Dr. John Tan 30 15

16 Comparisons PCA The drought patterns from the El Nino taken from the National Drought Mitigation Center ( show that KPCA pattern matches the drought patterns to a higher degree than standard PCA KPCA //009 Based on slides originaly from Dr. John Tan 31 Conclusions First known application of KPCA to Earth Science data KPCA yields correlations that are significantly higher than PCA results. The results show new spatial patterns and has a more refined regional structure The compleity is on the same order of operations and memory requirements as standard PCA The results depend on the choice of Kernel Eigenvalue problem of size N (# of observations) 4//009 Based on slides originaly from Dr. John Tan 3 16

17 Future Works Improve efficiency of preimage algorithms Test different kernels for earth science applications Study the atmospheric circulation regimes 4//009 Based on slides originaly from Dr. John Tan 33 PCA Applications Atmospheric Regimes Corti, S., F. Molteni, and T. Palmer, 1999: Signature of recent climate change in frequencies of natural atmospheric circulation regimes. Nature, 398, //009 Based on slides originaly from Dr. John Tan 34 17

18 KPCA vs. PCS/NNPCA/PCA KPCA NNPCA/PCS PCA Fleibility Good. factor analysis, dim reduction, nonlinear analysis Primarily dimension reduction, nonlinear analysis OK. linear characterization only Large training set: M Bad. But greedy methods available OK OK if input dim is small Large input dimension: d OK Bad OK Algorithm Symmetric Eigenvalue Problem Nonlinear variational optimization Problem Symmetric Eigenvalue Problem Variance Kernel dependent Well understood Well understood Inverse Trans. Preimage Given Matri Multiplication 4//009 Based on slides originaly from Dr. John Tan 35 SVM (Support Vector Machines) SVM is the tool of choice for the data mining classification problem. SVM is a statistical learning system for predictive data mining -- for estimating regression functions. Loads of information available here: 4//009 Based on slides originaly from Dr. John Tan 36 18

19 SVM Classification SVM attempts to find an optimal separating hyperplane between members of the two initial classifications. Class A Class B separating hyperplane 4//009 Based on slides originaly from Dr. John Tan 37 SVM Kernel Construction The data attributes can be transformed to a higher dimensional space (feature space) by applying a kernel function. This transformation can have the effect of allowing a separating hyperplane to be found. 4//009 Based on slides originaly from Dr. John Tan 38 19