Kernel Sliced Inverse Regression With Applications to Classification

Transcription

1 May 21-24, 2008 in Durham, NC Kernel Sliced Inverse Regression With Applications to Classification Han-Ming Wu (Hank) Department of Mathematics, Tamkang University Taipei, Taiwan 2008/05/22

2 Outline 2/27 Kernel Methods, Kernel Trick Kernel Data and Its Properties SIR in the Euclidean Space Kernel SIR in a Non-linear Feature Space KSIR for Nonlinear Dimension Reduction and Data Visualization Experiments on Classification Conclusion and Future Direction

3 Kernel Methods (1) 3/27 Aronszajn (1950) and Parzen (1962) first to employ kernel methods in statistics. Aizerman et al. (1964) used positive definite kernels which was closer to kernel trick, argued that a positive definite kernel is identical to a dot product in the feature space.

4 Kernel Methods (2) 4/27 Boser et al (1992) construct SVMs, a generalization of the so-called optimal hyperplane algorithm. Scholkopf et al (1998) point out that kernels can be used to construct generalization of any algorithm that can be carried out in terms of dot products. For last 10 years there have seen a large number of kernelization of various algorithms. (e.g., PCA, LDA, CCA, PLS, )

5 Prepare Kernel Data 5/27 theoretically In fact x i x j Φ(x i )? Φ(x j )

6 Data Representation 6/27 Data are not represented individually anymore, but only through a set of pairwise comparisons. The size of the matrix used to represent a dataset of n objects is always n by n.

7 Kernel as Inner Product 7/27 (Aronszajn, 1950)

8 Kernel Trick 8/27 The kernel trick transforms any algorithm that solely dependents on the dot product between two vectors. Wherever a dot product is used, it is replaced with the kernel function. The non-linear algorithm is the linear algorithm operating in the feature space. Kernelization: the operation that transforms a linear algorithm into a more general kernel method.

9 SIR in the Euclidean Space 9/27 Sufficient Dimension Reduction NOTE: For more details, please see Dr. Dennis Cook, School of Statistics, University of Minnesota. ( > 50 related articles published!)

10 Classical SIR: Algorithm 10/27 Weighted PCA

11 Kernel SIR in a Non-linear Feature Space 11/27 Kernel SIR: Kernelize the SIR algorithm

12 KSIR: Algorithm (1) 12/27

13 KSIR: Algorithm (2) 13/27

14 KSIR: Algorithm (3) 14/27

15 Normalization and Projection 15/27

16 Reduced Features 16/27 For Theoretical details: Lee, Y.J. and Huang, S.Y. (2006), Reduced support vector machines: a statistical theory, IEEE Transactions on Neural Networks, accepted.

17 KSIR for Nonlinear Dimensional Reduction and Data Visualization 17/27 Simulation Data Square Data (150x2, na) Three Clusters Data (220x2, no.class=3) Li Data Model (6.3) (400x10, y=conti) Real Data Wine Data (178x18, no.class=3) Pendigit Data (7494x16, no.class=10)

18 Visualization (1): Square Data H=8 18/27 d = 1 d = 2 d = 3 d = 4 d = 1 d = 2 d = 3 d = 4 V 1 V 1 V 2 V 2 V 3 V 3 KPCA KSIR s = 0.01 s = 0.1 s = 1 s = 10 s = 0.01 s = 0.1 s = 1 s = 10 V 1 V 1 V 2 V 2 V 3 V 3

19 Visualization (2): Three Clusters Data 19/27 d = 1 d = 2 d = 3 d = 4 d = 1 d = 2 d = 3 d = 4 V 1 V 1 V 2 V 2 V 3 V 3 KPCA KSIR s = 0.01 s = 0.1 s = 1 s = 10 s = 0.01 s = 0.1 s = 1 s = 10 V 1 V 1 V 2 V 2 V 3 V 3

20 20/27 Visualization (3): Li Data Model (6.3) PCA SIR H=13 Orig KPCA Gaussian s=0.05 KSIR

21 Visualization (4): Wine Data Wine data (n=178) are the results of a chemical analysis of wines grown in the same region in Italy but derived from three different cultivars. PCA SIR 21/27 The analysis determined the quantities of 13 constituents found in each of the three types of wines. KPCA Gaussian s=0.05 KSIR

22 Pen-based recognition of handwritten Digits Visualization (5): Pendigit Data PCA SIR 22/ instances, 16 attributes 10 classes KPCA Gaussian 0.05 Random sampling 200 KSIR

23 Classification (1): UCI Data Sets 23/27 Gaussian 0.05 Random sampling 200 Linear Support Vector Machine 10-fold classification error rates

24 10-fold Classification ERs: UCI Data Sets 24/27

25 Classification: Microarray Data Sets 25/27 Linear Support Vector Machine Leave-one-out classification error rates

26 Conclusion and Future Direction 26/27 Use Kernel Trick to study the linear algorithm of SIR in the Feature Space. Nonlinear dimension reduction subspace from X viewpoint Linear dimension reduction subspace in H k Nonlinear Dimension Reduction and Visualization For Classification. Apply to Clustering Problem. SIR/KSIR: A tool for feature extraction and data exploratory analysis. Theoretical Prosperities of Kernel SIR. Selection of Kernel Parameters (model selection).

27 jdrcluster: Dimension Reduction and Cluster Analysis 27/27 Thank You!